123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263 |
- Introduction
- ============
- This document describes how to use the dynamic debug (ddebug) feature.
- Dynamic debug is designed to allow you to dynamically enable/disable kernel
- code to obtain additional kernel information. Currently, if
- CONFIG_DYNAMIC_DEBUG is set, then all pr_debug()/dev_dbg() calls can be
- dynamically enabled per-callsite.
- Dynamic debug has even more useful features:
- * Simple query language allows turning on and off debugging statements by
- matching any combination of 0 or 1 of:
- - source filename
- - function name
- - line number (including ranges of line numbers)
- - module name
- - format string
- * Provides a debugfs control file: <debugfs>/dynamic_debug/control which can be
- read to display the complete list of known debug statements, to help guide you
- Controlling dynamic debug Behaviour
- ===================================
- The behaviour of pr_debug()/dev_dbg()s are controlled via writing to a
- control file in the 'debugfs' filesystem. Thus, you must first mount the debugfs
- filesystem, in order to make use of this feature. Subsequently, we refer to the
- control file as: <debugfs>/dynamic_debug/control. For example, if you want to
- enable printing from source file 'svcsock.c', line 1603 you simply do:
- nullarbor:~ # echo 'file svcsock.c line 1603 +p' >
- <debugfs>/dynamic_debug/control
- If you make a mistake with the syntax, the write will fail thus:
- nullarbor:~ # echo 'file svcsock.c wtf 1 +p' >
- <debugfs>/dynamic_debug/control
- -bash: echo: write error: Invalid argument
- Viewing Dynamic Debug Behaviour
- ===========================
- You can view the currently configured behaviour of all the debug statements
- via:
- nullarbor:~ # cat <debugfs>/dynamic_debug/control
- # filename:lineno [module]function flags format
- /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:323 [svcxprt_rdma]svc_rdma_cleanup - "SVCRDMA Module Removed, deregister RPC RDMA transport\012"
- /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:341 [svcxprt_rdma]svc_rdma_init - "\011max_inline : %d\012"
- /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:340 [svcxprt_rdma]svc_rdma_init - "\011sq_depth : %d\012"
- /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svc_rdma.c:338 [svcxprt_rdma]svc_rdma_init - "\011max_requests : %d\012"
- ...
- You can also apply standard Unix text manipulation filters to this
- data, e.g.
- nullarbor:~ # grep -i rdma <debugfs>/dynamic_debug/control | wc -l
- 62
- nullarbor:~ # grep -i tcp <debugfs>/dynamic_debug/control | wc -l
- 42
- Note in particular that the third column shows the enabled behaviour
- flags for each debug statement callsite (see below for definitions of the
- flags). The default value, no extra behaviour enabled, is "-". So
- you can view all the debug statement callsites with any non-default flags:
- nullarbor:~ # awk '$3 != "-"' <debugfs>/dynamic_debug/control
- # filename:lineno [module]function flags format
- /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svcsock.c:1603 [sunrpc]svc_send p "svc_process: st_sendto returned %d\012"
- Command Language Reference
- ==========================
- At the lexical level, a command comprises a sequence of words separated
- by spaces or tabs. So these are all equivalent:
- nullarbor:~ # echo -c 'file svcsock.c line 1603 +p' >
- <debugfs>/dynamic_debug/control
- nullarbor:~ # echo -c ' file svcsock.c line 1603 +p ' >
- <debugfs>/dynamic_debug/control
- nullarbor:~ # echo -n 'file svcsock.c line 1603 +p' >
- <debugfs>/dynamic_debug/control
- Command submissions are bounded by a write() system call.
- Multiple commands can be written together, separated by ';' or '\n'.
- ~# echo "func pnpacpi_get_resources +p; func pnp_assign_mem +p" \
- > <debugfs>/dynamic_debug/control
- If your query set is big, you can batch them too:
- ~# cat query-batch-file > <debugfs>/dynamic_debug/control
- At the syntactical level, a command comprises a sequence of match
- specifications, followed by a flags change specification.
- command ::= match-spec* flags-spec
- The match-spec's are used to choose a subset of the known dprintk()
- callsites to which to apply the flags-spec. Think of them as a query
- with implicit ANDs between each pair. Note that an empty list of
- match-specs is possible, but is not very useful because it will not
- match any debug statement callsites.
- A match specification comprises a keyword, which controls the attribute
- of the callsite to be compared, and a value to compare against. Possible
- keywords are:
- match-spec ::= 'func' string |
- 'file' string |
- 'module' string |
- 'format' string |
- 'line' line-range
- line-range ::= lineno |
- '-'lineno |
- lineno'-' |
- lineno'-'lineno
- // Note: line-range cannot contain space, e.g.
- // "1-30" is valid range but "1 - 30" is not.
- lineno ::= unsigned-int
- The meanings of each keyword are:
- func
- The given string is compared against the function name
- of each callsite. Example:
- func svc_tcp_accept
- file
- The given string is compared against either the full pathname, the
- src-root relative pathname, or the basename of the source file of
- each callsite. Examples:
- file svcsock.c
- file kernel/freezer.c
- file /usr/src/packages/BUILD/sgi-enhancednfs-1.4/default/net/sunrpc/svcsock.c
- module
- The given string is compared against the module name
- of each callsite. The module name is the string as
- seen in "lsmod", i.e. without the directory or the .ko
- suffix and with '-' changed to '_'. Examples:
- module sunrpc
- module nfsd
- format
- The given string is searched for in the dynamic debug format
- string. Note that the string does not need to match the
- entire format, only some part. Whitespace and other
- special characters can be escaped using C octal character
- escape \ooo notation, e.g. the space character is \040.
- Alternatively, the string can be enclosed in double quote
- characters (") or single quote characters (').
- Examples:
- format svcrdma: // many of the NFS/RDMA server dprintks
- format readahead // some dprintks in the readahead cache
- format nfsd:\040SETATTR // one way to match a format with whitespace
- format "nfsd: SETATTR" // a neater way to match a format with whitespace
- format 'nfsd: SETATTR' // yet another way to match a format with whitespace
- line
- The given line number or range of line numbers is compared
- against the line number of each dprintk() callsite. A single
- line number matches the callsite line number exactly. A
- range of line numbers matches any callsite between the first
- and last line number inclusive. An empty first number means
- the first line in the file, an empty line number means the
- last number in the file. Examples:
- line 1603 // exactly line 1603
- line 1600-1605 // the six lines from line 1600 to line 1605
- line -1605 // the 1605 lines from line 1 to line 1605
- line 1600- // all lines from line 1600 to the end of the file
- The flags specification comprises a change operation followed
- by one or more flag characters. The change operation is one
- of the characters:
- -
- remove the given flags
- +
- add the given flags
- =
- set the flags to the given flags
- The flags are:
- f
- Include the function name in the printed message
- l
- Include line number in the printed message
- m
- Include module name in the printed message
- p
- Causes a printk() message to be emitted to dmesg
- t
- Include thread ID in messages not generated from interrupt context
- Note the regexp ^[-+=][flmpt]+$ matches a flags specification.
- Note also that there is no convenient syntax to remove all
- the flags at once, you need to use "-flmpt".
- Debug messages during boot process
- ==================================
- To be able to activate debug messages during the boot process,
- even before userspace and debugfs exists, use the boot parameter:
- ddebug_query="QUERY"
- QUERY follows the syntax described above, but must not exceed 1023
- characters. The enablement of debug messages is done as an arch_initcall.
- Thus you can enable debug messages in all code processed after this
- arch_initcall via this boot parameter.
- On an x86 system for example ACPI enablement is a subsys_initcall and
- ddebug_query="file ec.c +p"
- will show early Embedded Controller transactions during ACPI setup if
- your machine (typically a laptop) has an Embedded Controller.
- PCI (or other devices) initialization also is a hot candidate for using
- this boot parameter for debugging purposes.
- Examples
- ========
- // enable the message at line 1603 of file svcsock.c
- nullarbor:~ # echo -n 'file svcsock.c line 1603 +p' >
- <debugfs>/dynamic_debug/control
- // enable all the messages in file svcsock.c
- nullarbor:~ # echo -n 'file svcsock.c +p' >
- <debugfs>/dynamic_debug/control
- // enable all the messages in the NFS server module
- nullarbor:~ # echo -n 'module nfsd +p' >
- <debugfs>/dynamic_debug/control
- // enable all 12 messages in the function svc_process()
- nullarbor:~ # echo -n 'func svc_process +p' >
- <debugfs>/dynamic_debug/control
- // disable all 12 messages in the function svc_process()
- nullarbor:~ # echo -n 'func svc_process -p' >
- <debugfs>/dynamic_debug/control
- // enable messages for NFS calls READ, READLINK, READDIR and READDIR+.
- nullarbor:~ # echo -n 'format "nfsd: READ" +p' >
- <debugfs>/dynamic_debug/control
|