diff options
| author | Ross Zwisler <zwisler@chromium.org> | 2023-01-25 14:32:51 -0700 |
|---|---|---|
| committer | Jonathan Corbet <corbet@lwn.net> | 2023-01-31 14:02:30 -0700 |
| commit | 2abfcd293b79baf62895894fa3ea2386ffbe6338 (patch) | |
| tree | b74867f24b7c6d6b6909856063d6c184a29195e0 /Documentation/trace | |
| parent | b05ada5615f844c3fdcf5e2a0fa35a4bc1a394f2 (diff) | |
| download | linux-2abfcd293b79baf62895894fa3ea2386ffbe6338.tar.gz linux-2abfcd293b79baf62895894fa3ea2386ffbe6338.tar.bz2 linux-2abfcd293b79baf62895894fa3ea2386ffbe6338.zip | |
docs: ftrace: always use canonical ftrace path
The canonical location for the tracefs filesystem is at /sys/kernel/tracing.
But, from Documentation/trace/ftrace.rst:
Before 4.1, all ftrace tracing control files were within the debugfs
file system, which is typically located at /sys/kernel/debug/tracing.
For backward compatibility, when mounting the debugfs file system,
the tracefs file system will be automatically mounted at:
/sys/kernel/debug/tracing
Many parts of Documentation still reference this older debugfs path, so
let's update them to avoid confusion.
Signed-off-by: Ross Zwisler <zwisler@google.com>
Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Link: https://lore.kernel.org/r/20230125213251.2013791-1-zwisler@google.com
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
Diffstat (limited to 'Documentation/trace')
| -rw-r--r-- | Documentation/trace/events-msr.rst | 4 | ||||
| -rw-r--r-- | Documentation/trace/events-nmi.rst | 6 | ||||
| -rw-r--r-- | Documentation/trace/events.rst | 74 | ||||
| -rw-r--r-- | Documentation/trace/ftrace.rst | 6 | ||||
| -rw-r--r-- | Documentation/trace/histogram-design.rst | 12 | ||||
| -rw-r--r-- | Documentation/trace/histogram.rst | 190 | ||||
| -rw-r--r-- | Documentation/trace/kprobetrace.rst | 26 | ||||
| -rw-r--r-- | Documentation/trace/mmiotrace.rst | 20 | ||||
| -rw-r--r-- | Documentation/trace/postprocess/trace-pagealloc-postprocess.pl | 4 | ||||
| -rw-r--r-- | Documentation/trace/postprocess/trace-vmscan-postprocess.pl | 4 | ||||
| -rw-r--r-- | Documentation/trace/tracepoint-analysis.rst | 8 | ||||
| -rw-r--r-- | Documentation/trace/uprobetracer.rst | 22 | ||||
| -rw-r--r-- | Documentation/trace/user_events.rst | 18 |
13 files changed, 197 insertions, 197 deletions
diff --git a/Documentation/trace/events-msr.rst b/Documentation/trace/events-msr.rst index 810481e530b6..35d06dc66bc2 100644 --- a/Documentation/trace/events-msr.rst +++ b/Documentation/trace/events-msr.rst @@ -8,7 +8,7 @@ at https://www.intel.com/sdm (Volume 3) Available trace points: -/sys/kernel/debug/tracing/events/msr/ +/sys/kernel/tracing/events/msr/ Trace MSR reads: @@ -34,7 +34,7 @@ rdpmc The trace data can be post processed with the postprocess/decode_msr.py script:: - cat /sys/kernel/debug/tracing/trace | decode_msr.py /usr/src/linux/include/asm/msr-index.h + cat /sys/kernel/tracing/trace | decode_msr.py /usr/src/linux/include/asm/msr-index.h to add symbolic MSR names. diff --git a/Documentation/trace/events-nmi.rst b/Documentation/trace/events-nmi.rst index 9e0a7289d80a..22ac1be0ea6f 100644 --- a/Documentation/trace/events-nmi.rst +++ b/Documentation/trace/events-nmi.rst @@ -4,7 +4,7 @@ NMI Trace Events These events normally show up here: - /sys/kernel/debug/tracing/events/nmi + /sys/kernel/tracing/events/nmi nmi_handler @@ -31,13 +31,13 @@ really hogging a lot of CPU time, like a millisecond at a time. Note that the kernel's output is in milliseconds, but the input to the filter is in nanoseconds! You can filter on 'delta_ns':: - cd /sys/kernel/debug/tracing/events/nmi/nmi_handler + cd /sys/kernel/tracing/events/nmi/nmi_handler echo 'handler==0xffffffff81625600 && delta_ns>1000000' > filter echo 1 > enable Your output would then look like:: - $ cat /sys/kernel/debug/tracing/trace_pipe + $ cat /sys/kernel/tracing/trace_pipe <idle>-0 [000] d.h3 505.397558: nmi_handler: perf_event_nmi_handler() delta_ns: 3236765 handled: 1 <idle>-0 [000] d.h3 505.805893: nmi_handler: perf_event_nmi_handler() delta_ns: 3174234 handled: 1 <idle>-0 [000] d.h3 506.158206: nmi_handler: perf_event_nmi_handler() delta_ns: 3084642 handled: 1 diff --git a/Documentation/trace/events.rst b/Documentation/trace/events.rst index 8f86eef1bada..8b33850b8bf9 100644 --- a/Documentation/trace/events.rst +++ b/Documentation/trace/events.rst @@ -24,27 +24,27 @@ tracing information should be printed. --------------------------------- The events which are available for tracing can be found in the file -/sys/kernel/debug/tracing/available_events. +/sys/kernel/tracing/available_events. To enable a particular event, such as 'sched_wakeup', simply echo it -to /sys/kernel/debug/tracing/set_event. For example:: +to /sys/kernel/tracing/set_event. For example:: - # echo sched_wakeup >> /sys/kernel/debug/tracing/set_event + # echo sched_wakeup >> /sys/kernel/tracing/set_event .. Note:: '>>' is necessary, otherwise it will firstly disable all the events. To disable an event, echo the event name to the set_event file prefixed with an exclamation point:: - # echo '!sched_wakeup' >> /sys/kernel/debug/tracing/set_event + # echo '!sched_wakeup' >> /sys/kernel/tracing/set_event To disable all events, echo an empty line to the set_event file:: - # echo > /sys/kernel/debug/tracing/set_event + # echo > /sys/kernel/tracing/set_event To enable all events, echo ``*:*`` or ``*:`` to the set_event file:: - # echo *:* > /sys/kernel/debug/tracing/set_event + # echo *:* > /sys/kernel/tracing/set_event The events are organized into subsystems, such as ext4, irq, sched, etc., and a full event name looks like this: <subsystem>:<event>. The @@ -53,29 +53,29 @@ file. All of the events in a subsystem can be specified via the syntax ``<subsystem>:*``; for example, to enable all irq events, you can use the command:: - # echo 'irq:*' > /sys/kernel/debug/tracing/set_event + # echo 'irq:*' > /sys/kernel/tracing/set_event 2.2 Via the 'enable' toggle --------------------------- -The events available are also listed in /sys/kernel/debug/tracing/events/ hierarchy +The events available are also listed in /sys/kernel/tracing/events/ hierarchy of directories. To enable event 'sched_wakeup':: - # echo 1 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable + # echo 1 > /sys/kernel/tracing/events/sched/sched_wakeup/enable To disable it:: - # echo 0 > /sys/kernel/debug/tracing/events/sched/sched_wakeup/enable + # echo 0 > /sys/kernel/tracing/events/sched/sched_wakeup/enable To enable all events in sched subsystem:: - # echo 1 > /sys/kernel/debug/tracing/events/sched/enable + # echo 1 > /sys/kernel/tracing/events/sched/enable To enable all events:: - # echo 1 > /sys/kernel/debug/tracing/events/enable + # echo 1 > /sys/kernel/tracing/events/enable When reading one of these enable files, there are four results: @@ -126,7 +126,7 @@ is the size of the data item, in bytes. For example, here's the information displayed for the 'sched_wakeup' event:: - # cat /sys/kernel/debug/tracing/events/sched/sched_wakeup/format + # cat /sys/kernel/tracing/events/sched/sched_wakeup/format name: sched_wakeup ID: 60 @@ -215,19 +215,19 @@ to the 'filter' file for the given event. For example:: - # cd /sys/kernel/debug/tracing/events/sched/sched_wakeup + # cd /sys/kernel/tracing/events/sched/sched_wakeup # echo "common_preempt_count > 4" > filter A slightly more involved example:: - # cd /sys/kernel/debug/tracing/events/signal/signal_generate + # cd /sys/kernel/tracing/events/signal/signal_generate # echo "((sig >= 10 && sig < 15) || sig == 17) && comm != bash" > filter If there is an error in the expression, you'll get an 'Invalid argument' error when setting it, and the erroneous string along with an error message can be seen by looking at the filter e.g.:: - # cd /sys/kernel/debug/tracing/events/signal/signal_generate + # cd /sys/kernel/tracing/events/signal/signal_generate # echo "((sig >= 10 && sig < 15) || dsig == 17) && comm != bash" > filter -bash: echo: write error: Invalid argument # cat filter @@ -277,7 +277,7 @@ above points: Clear the filters on all events in the sched subsystem:: - # cd /sys/kernel/debug/tracing/events/sched + # cd /sys/kernel/tracing/events/sched # echo 0 > filter # cat sched_switch/filter none @@ -287,7 +287,7 @@ Clear the filters on all events in the sched subsystem:: Set a filter using only common fields for all events in the sched subsystem (all events end up with the same filter):: - # cd /sys/kernel/debug/tracing/events/sched + # cd /sys/kernel/tracing/events/sched # echo common_pid == 0 > filter # cat sched_switch/filter common_pid == 0 @@ -298,7 +298,7 @@ Attempt to set a filter using a non-common field for all events in the sched subsystem (all events but those that have a prev_pid field retain their old filters):: - # cd /sys/kernel/debug/tracing/events/sched + # cd /sys/kernel/tracing/events/sched # echo prev_pid == 0 > filter # cat sched_switch/filter prev_pid == 0 @@ -313,7 +313,7 @@ exists, will filter all events from tracing any task that does not have the PID listed in the set_event_pid file. :: - # cd /sys/kernel/debug/tracing + # cd /sys/kernel/tracing # echo $$ > set_event_pid # echo 1 > events/enable @@ -409,14 +409,14 @@ The following commands are supported: specifies that this enablement happens only once:: # echo 'enable_event:kmem:kmalloc:1' > \ - /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger + /sys/kernel/tracing/events/syscalls/sys_enter_read/trigger The following trigger causes kmalloc events to stop being traced when a read system call exits. This disablement happens on every read system call exit:: # echo 'disable_event:kmem:kmalloc' > \ - /sys/kernel/debug/tracing/events/syscalls/sys_exit_read/trigger + /sys/kernel/tracing/events/syscalls/sys_exit_read/trigger The format is:: @@ -426,10 +426,10 @@ The following commands are supported: To remove the above commands:: # echo '!enable_event:kmem:kmalloc:1' > \ - /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger + /sys/kernel/tracing/events/syscalls/sys_enter_read/trigger # echo '!disable_event:kmem:kmalloc' > \ - /sys/kernel/debug/tracing/events/syscalls/sys_exit_read/trigger + /sys/kernel/tracing/events/syscalls/sys_exit_read/trigger Note that there can be any number of enable/disable_event triggers per triggering event, but there can only be one trigger per @@ -448,13 +448,13 @@ The following commands are supported: kmalloc tracepoint is hit:: # echo 'stacktrace' > \ - /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger + /sys/kernel/tracing/events/kmem/kmalloc/trigger The following trigger dumps a stacktrace the first 5 times a kmalloc request happens with a size >= 64K:: # echo 'stacktrace:5 if bytes_req >= 65536' > \ - /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger + /sys/kernel/tracing/events/kmem/kmalloc/trigger The format is:: @@ -463,16 +463,16 @@ The following commands are supported: To remove the above commands:: # echo '!stacktrace' > \ - /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger + /sys/kernel/tracing/events/kmem/kmalloc/trigger # echo '!stacktrace:5 if bytes_req >= 65536' > \ - /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger + /sys/kernel/tracing/events/kmem/kmalloc/trigger The latter can also be removed more simply by the following (without the filter):: # echo '!stacktrace:5' > \ - /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger + /sys/kernel/tracing/events/kmem/kmalloc/trigger Note that there can be only one stacktrace trigger per triggering event. @@ -488,20 +488,20 @@ The following commands are supported: capture those events when the trigger event occurred:: # echo 'snapshot if nr_rq > 1' > \ - /sys/kernel/debug/tracing/events/block/block_unplug/trigger + /sys/kernel/tracing/events/block/block_unplug/trigger To only snapshot once:: # echo 'snapshot:1 if nr_rq > 1' > \ - /sys/kernel/debug/tracing/events/block/block_unplug/trigger + /sys/kernel/tracing/events/block/block_unplug/trigger To remove the above commands:: # echo '!snapshot if nr_rq > 1' > \ - /sys/kernel/debug/tracing/events/block/block_unplug/trigger + /sys/kernel/tracing/events/block/block_unplug/trigger # echo '!snapshot:1 if nr_rq > 1' > \ - /sys/kernel/debug/tracing/events/block/block_unplug/trigger + /sys/kernel/tracing/events/block/block_unplug/trigger Note that there can be only one snapshot trigger per triggering event. @@ -519,20 +519,20 @@ The following commands are supported: trigger event:: # echo 'traceoff:1 if nr_rq > 1' > \ - /sys/kernel/debug/tracing/events/block/block_unplug/trigger + /sys/kernel/tracing/events/block/block_unplug/trigger To always disable tracing when nr_rq > 1:: # echo 'traceoff if nr_rq > 1' > \ - /sys/kernel/debug/tracing/events/block/block_unplug/trigger + /sys/kernel/tracing/events/block/block_unplug/trigger To remove the above commands:: # echo '!traceoff:1 if nr_rq > 1' > \ - /sys/kernel/debug/tracing/events/block/block_unplug/trigger + /sys/kernel/tracing/events/block/block_unplug/trigger # echo '!traceoff if nr_rq > 1' > \ - /sys/kernel/debug/tracing/events/block/block_unplug/trigger + /sys/kernel/tracing/events/block/block_unplug/trigger Note that there can be only one traceon or traceoff trigger per triggering event. diff --git a/Documentation/trace/ftrace.rst b/Documentation/trace/ftrace.rst index 21f01d32c959..b927fb2b94dc 100644 --- a/Documentation/trace/ftrace.rst +++ b/Documentation/trace/ftrace.rst @@ -830,10 +830,10 @@ Error conditions The extended error information and usage takes the form shown in this example:: - # echo xxx > /sys/kernel/debug/tracing/events/sched/sched_wakeup/trigger + # echo xxx > /sys/kernel/tracing/events/sched/sched_wakeup/trigger echo: write error: Invalid argument - # cat /sys/kernel/debug/tracing/error_log + # cat /sys/kernel/tracing/error_log [ 5348.887237] location: error: Couldn't yyy: zzz Command: xxx ^ @@ -843,7 +843,7 @@ Error conditions To clear the error log, echo the empty string into it:: - # echo > /sys/kernel/debug/tracing/error_log + # echo > /sys/kernel/tracing/error_log Examples of using the tracer ---------------------------- diff --git a/Documentation/trace/histogram-design.rst b/Documentation/trace/histogram-design.rst index 088c8cce738b..5765eb3e9efa 100644 --- a/Documentation/trace/histogram-design.rst +++ b/Documentation/trace/histogram-design.rst @@ -14,7 +14,7 @@ tracing_map.c. Note: All the ftrace histogram command examples assume the working directory is the ftrace /tracing directory. For example:: - # cd /sys/kernel/debug/tracing + # cd /sys/kernel/tracing Also, the histogram output displayed for those commands will be generally be truncated - only enough to make the point is displayed. @@ -905,7 +905,7 @@ means it will be automatically converted into a field variable:: # echo 'hist:keys=next_pid:wakeup_lat=common_timestamp.usecs-$ts0: \ onmatch(sched.sched_waking).wakeup_latency($wakeup_lat,next_pid)' >> - /sys/kernel/debug/tracing/events/sched/sched_switch/trigger + /sys/kernel/tracing/events/sched/sched_switch/trigger The diagram for the sched_switch event is similar to previous examples but shows the additional field_vars[] array for hist_data and shows @@ -1112,7 +1112,7 @@ sched_switch event fields, next_pid and next_comm, to generate a wakeup_latency trace event. The next_pid and next_comm event fields are automatically converted into field variables for this purpose:: - # echo 'hist:keys=next_pid:wakeup_lat=common_timestamp.usecs-$ts0:onmatch(sched.sched_waking).wakeup_latency($wakeup_lat,next_pid,next_comm)' >> /sys/kernel/debug/tracing/events/sched/sched_switch/trigger + # echo 'hist:keys=next_pid:wakeup_lat=common_timestamp.usecs-$ts0:onmatch(sched.sched_waking).wakeup_latency($wakeup_lat,next_pid,next_comm)' >> /sys/kernel/tracing/events/sched/sched_switch/trigger The sched_waking hist_debug output shows the same data as in the previous test example:: @@ -1305,7 +1305,7 @@ and event name for the onmatch() handler:: The commands below can be used to clean things up for the next test:: - # echo '!hist:keys=next_pid:wakeup_lat=common_timestamp.usecs-$ts0:onmatch(sched.sched_waking).wakeup_latency($wakeup_lat,next_pid,next_comm)' >> /sys/kernel/debug/tracing/events/sched/sched_switch/trigger + # echo '!hist:keys=next_pid:wakeup_lat=common_timestamp.usecs-$ts0:onmatch(sched.sched_waking).wakeup_latency($wakeup_lat,next_pid,next_comm)' >> /sys/kernel/tracing/events/sched/sched_switch/trigger # echo '!hist:keys=pid:ts0=common_timestamp.usecs' >> events/sched/sched_waking/trigger @@ -1363,13 +1363,13 @@ with the save() and snapshot() actions. For example:: # echo 'hist:keys=next_pid:wakeup_lat=common_timestamp.usecs-$ts0: \ onmax($wakeup_lat).save(next_comm,prev_pid,prev_prio,prev_comm)' >> - /sys/kernel/debug/tracing/events/sched/sched_switch/trigger + /sys/kernel/tracing/events/sched/sched_switch/trigger or:: # echo 'hist:keys=next_pid:wakeup_lat=common_timestamp.usecs-$ts0: \ onmax($wakeup_lat).snapshot()' >> - /sys/kernel/debug/tracing/events/sched/sched_switch/trigger + /sys/kernel/tracing/events/sched/sched_switch/trigger save() action field variable test --------------------------------- diff --git a/Documentation/trace/histogram.rst b/Documentation/trace/histogram.rst index f95459aa984f..7b7e4893b8f6 100644 --- a/Documentation/trace/histogram.rst +++ b/Documentation/trace/histogram.rst @@ -102,12 +102,12 @@ Documentation written by Tom Zanussi trigger, read its current contents, and then turn it off:: # echo 'hist:keys=skbaddr.hex:vals=len' > \ - /sys/kernel/debug/tracing/events/net/netif_rx/trigger + /sys/kernel/tracing/events/net/netif_rx/trigger - # cat /sys/kernel/debug/tracing/events/net/netif_rx/hist + # cat /sys/kernel/tracing/events/net/netif_rx/hist # echo '!hist:keys=skbaddr.hex:vals=len' > \ - /sys/kernel/debug/tracing/events/net/netif_rx/trigger + /sys/kernel/tracing/events/net/netif_rx/trigger The trigger file itself can be read to show the details of the currently attached hist trigger. This information is also displayed @@ -169,13 +169,13 @@ Documentation written by Tom Zanussi aggregation on and off when conditions of interest are hit:: # echo 'hist:keys=skbaddr.hex:vals=len:pause' > \ - /sys/kernel/debug/tracing/events/net/netif_receive_skb/trigger + /sys/kernel/tracing/events/net/netif_receive_skb/trigger # echo 'enable_hist:net:netif_receive_skb if filename==/usr/bin/wget' > \ - /sys/kernel/debug/tracing/events/sched/sched_process_exec/trigger + /sys/kernel/tracing/events/sched/sched_process_exec/trigger # echo 'disable_hist:net:netif_receive_skb if comm==wget' > \ - /sys/kernel/debug/tracing/events/sched/sched_process_exit/trigger + /sys/kernel/tracing/events/sched/sched_process_exit/trigger The above sets up an initially paused hist trigger which is unpaused and starts aggregating events when a given program is executed, and @@ -218,7 +218,7 @@ Extended error information event. The fields that can be used for the hist trigger are listed in the kmalloc event's format file:: - # cat /sys/kernel/debug/tracing/events/kmem/kmalloc/format + # cat /sys/kernel/tracing/events/kmem/kmalloc/format name: kmalloc ID: 374 format: @@ -238,7 +238,7 @@ Extended error information the kernel that made one or more calls to kmalloc:: # echo 'hist:key=call_site:val=bytes_req.buckets=32' > \ - /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger + /sys/kernel/tracing/events/kmem/kmalloc/trigger This tells the tracing system to create a 'hist' trigger using the call_site field of the kmalloc event as the key for the table, which @@ -252,7 +252,7 @@ Extended error information file in the kmalloc event's subdirectory (for readability, a number of entries have been omitted):: - # cat /sys/kernel/debug/tracing/events/kmem/kmalloc/hist + # cat /sys/kernel/tracing/events/kmem/kmalloc/hist # trigger info: hist:keys=call_site:vals=bytes_req:sort=hitcount:size=2048 [active] { call_site: 18446744072106379007 } hitcount: 1 bytes_req: 176 @@ -292,7 +292,7 @@ Extended error information the trigger info, which can also be displayed by reading the 'trigger' file:: - # cat /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger + # cat /sys/kernel/tracing/events/kmem/kmalloc/trigger hist:keys=call_site:vals=bytes_req:sort=hitcount:size=2048 [active] At the end of the output are a few lines that display the overall @@ -323,7 +323,7 @@ Extended error information command history and re-execute it with a '!' prepended:: # echo '!hist:key=call_site:val=bytes_req' > \ - /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger + /sys/kernel/tracing/events/kmem/kmalloc/trigger Finally, notice that the call_site as displayed in the output above isn't really very useful. It's an address, but normally addresses @@ -331,9 +331,9 @@ Extended error information value, simply append '.hex' to the field name in the trigger:: # echo 'hist:key=call_site.hex:val=bytes_req' > \ - /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger + /sys/kernel/tracing/events/kmem/kmalloc/trigger - # cat /sys/kernel/debug/tracing/events/kmem/kmalloc/hist + # cat /sys/kernel/tracing/events/kmem/kmalloc/hist # trigger info: hist:keys=call_site.hex:vals=bytes_req:sort=hitcount:size=2048 [active] { call_site: ffffffffa026b291 } hitcount: 1 bytes_req: 433 @@ -376,9 +376,9 @@ Extended error information trigger:: # echo 'hist:key=call_site.sym:val=bytes_req' > \ - /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger + /sys/kernel/tracing/events/kmem/kmalloc/trigger - # cat /sys/kernel/debug/tracing/events/kmem/kmalloc/hist + # cat /sys/kernel/tracing/events/kmem/kmalloc/hist # trigger info: hist:keys=call_site.sym:vals=bytes_req:sort=hitcount:size=2048 [active] { call_site: [ffffffff810adcb9] syslog_print_all } hitcount: 1 bytes_req: 1024 @@ -426,9 +426,9 @@ Extended error information the 'sort' parameter, along with the 'descending' modifier:: # echo 'hist:key=call_site.sym:val=bytes_req:sort=bytes_req.descending' > \ - /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger + /sys/kernel/tracing/events/kmem/kmalloc/trigger - # cat /sys/kernel/debug/tracing/events/kmem/kmalloc/hist + # cat /sys/kernel/tracing/events/kmem/kmalloc/hist # trigger info: hist:keys=call_site.sym:vals=bytes_req:sort=bytes_req.descending:size=2048 [active] { call_site: [ffffffffa046041c] i915_gem_execbuffer2 [i915] } hitcount: 2186 bytes_req: 3397464 @@ -467,9 +467,9 @@ Extended error information name, just use 'sym-offset' instead:: # echo 'hist:key=call_site.sym-offset:val=bytes_req:sort=bytes_req.descending' > \ - /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger + /sys/kernel/tracing/events/kmem/kmalloc/trigger - # cat /sys/kernel/debug/tracing/events/kmem/kmalloc/hist + # cat /sys/kernel/tracing/events/kmem/kmalloc/hist # trigger info: hist:keys=call_site.sym-offset:vals=bytes_req:sort=bytes_req.descending:size=2048 [active] { call_site: [ffffffffa046041c] i915_gem_execbuffer2+0x6c/0x2c0 [i915] } hitcount: 4569 bytes_req: 3163720 @@ -506,9 +506,9 @@ Extended error information allocated in a descending order:: # echo 'hist:keys=call_site.sym:values=bytes_req,bytes_alloc:sort=bytes_alloc.descending' > \ - /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger + /sys/kernel/tracing/events/kmem/kmalloc/trigger - # cat /sys/kernel/debug/tracing/events/kmem/kmalloc/hist + # cat /sys/kernel/tracing/events/kmem/kmalloc/hist # trigger info: hist:keys=call_site.sym:vals=bytes_req,bytes_alloc:sort=bytes_alloc.descending:size=2048 [active] { call_site: [ffffffffa046041c] i915_gem_execbuffer2 [i915] } hitcount: 7403 bytes_req: 4084360 bytes_alloc: 5958016 @@ -549,7 +549,7 @@ Extended error information value 'stacktrace' for the key parameter:: # echo 'hist:keys=stacktrace:values=bytes_req,bytes_alloc:sort=bytes_alloc' > \ - /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger + /sys/kernel/tracing/events/kmem/kmalloc/trigger The above trigger will use the kernel stack trace in effect when an event is triggered as the key for the hash table. This allows the @@ -559,7 +559,7 @@ Extended error information every callpath in the system that led up to a kmalloc (in this case every callpath to a kmalloc for a kernel compile):: - # cat /sys/kernel/debug/tracing/events/kmem/kmalloc/hist + # cat /sys/kernel/tracing/events/kmem/kmalloc/hist # trigger info: hist:keys=stacktrace:vals=bytes_req,bytes_alloc:sort=bytes_alloc:size=2048 [active] { stacktrace: @@ -658,9 +658,9 @@ Extended error information keeps a per-process sum of total bytes read:: # echo 'hist:key=common_pid.execname:val=count:sort=count.descending' > \ - /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/trigger + /sys/kernel/tracing/events/syscalls/sys_enter_read/trigger - # cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_read/hist + # cat /sys/kernel/tracing/events/syscalls/sys_enter_read/hist # trigger info: hist:keys=common_pid.execname:vals=count:sort=count.descending:size=2048 [active] { common_pid: gnome-terminal [ 3196] } hitcount: 280 count: 1093512 @@ -699,9 +699,9 @@ Extended error information counts for the system during the run:: # echo 'hist:key=id.syscall:val=hitcount' > \ - /sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/trigger + /sys/kernel/tracing/events/raw_syscalls/sys_enter/trigger - # cat /sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/hist + # cat /sys/kernel/tracing/events/raw_syscalls/sys_enter/hist # trigger info: hist:keys=id.syscall:vals=hitcount:sort=hitcount:size=2048 [active] { id: sys_fsync [ 74] } hitcount: 1 @@ -753,9 +753,9 @@ Extended error information hitcount sum as the secondary key:: # echo 'hist:key=id.syscall,common_pid.execname:val=hitcount:sort=id,hitcount' > \ - /sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/trigger + /sys/kernel/tracing/events/raw_syscalls/sys_enter/trigger - # cat /sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/hist + # cat /sys/kernel/tracing/events/raw_syscalls/sys_enter/hist # trigger info: hist:keys=id.syscall,common_pid.execname:vals=hitcount:sort=id.syscall,hitcount:size=2048 [active] { id: sys_read [ 0], common_pid: rtkit-daemon [ 1877] } hitcount: 1 @@ -803,9 +803,9 @@ Extended error information can use that to filter out all the other syscalls:: # echo 'hist:key=id.syscall,common_pid.execname:val=hitcount:sort=id,hitcount if id == 16' > \ - /sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/trigger + /sys/kernel/tracing/events/raw_syscalls/sys_enter/trigger - # cat /sys/kernel/debug/tracing/events/raw_syscalls/sys_enter/hist + # cat /sys/kernel/tracing/events/raw_syscalls/sys_enter/hist # trigger info: hist:keys=id.syscall,common_pid.execname:vals=hitcount:sort=id.syscall,hitcount:size=2048 if id == 16 [active] { id: sys_ioctl [ 16], common_pid: gmain [ 2769] } hitcount: 1 @@ -846,9 +846,9 @@ Extended error information each process:: # echo 'hist:key=common_pid.execname,size:val=hitcount:sort=common_pid,size' > \ - /sys/kernel/debug/tracing/events/syscalls/sys_enter_recvfrom/trigger + /sys/kernel/tracing/events/syscalls/sys_enter_recvfrom/trigger - # cat /sys/kernel/debug/tracing/events/syscalls/sys_enter_recvfrom/hist + # cat /sys/kernel/tracing/events/syscalls/sys_enter_recvfrom/hist # trigger info: hist:keys=common_pid.execname,size:vals=hitcount:sort=common_pid.execname,size:size=2048 [active] { common_pid: smbd [ 784], size: 4 } hitcount: 1 @@ -899,9 +899,9 @@ Extended error information much smaller number, say 256:: # echo 'hist:key=child_comm:val=hitcount:size=256' > \ - /sys/kernel/debug/tracing/events/sched/sched_process_fork/trigger + /sys/kernel/tracing/events/sched/sched_process_fork/trigger - # cat /sys/kernel/debug/tracing/events/sched/sched_process_fork/hist + # cat /sys/kernel/tracing/events/sched/sched_process_fork/hist # trigger info: hist:keys=child_comm:vals=hitcount:sort=hitcount:size=256 [active] { child_comm: dconf worker } hitcount: 1 @@ -935,9 +935,9 @@ Extended error information displays as [paused]:: # echo 'hist:key=child_comm:val=hitcount:size=256:pause' >> \ - /sys/kernel/debug/tracing/events/sched/sched_process_fork/trigger + /sys/kernel/tracing/events/sched/sched_process_fork/trigger - # cat /sys/kernel/debug/tracing/events/sched/sched_process_fork/hist + # cat /sys/kernel/tracing/events/sched/sched_process_fork/hist # trigger info: hist:keys=child_comm:vals=hitcount:sort=hitcount:size=256 [paused] { child_comm: dconf worker } hitcount: 1 @@ -972,9 +972,9 @@ Extended error information again, and the data has changed:: # echo 'hist:key=child_comm:val=hitcount:size=256:cont' >> \ - /sys/kernel/debug/tracing/events/sched/sched_process_fork/trigger + /sys/kernel/tracing/events/sched/sched_process_fork/trigger - # cat /sys/kernel/debug/tracing/events/sched/sched_process_fork/hist + # cat /sys/kernel/tracing/events/sched/sched_process_fork/hist # trigger info: hist:keys=child_comm:vals=hitcount:sort=hitcount:size=256 [active] { child_comm: dconf worker } hitcount: 1 @@ -1026,7 +1026,7 @@ Extended error information netif_receive_skb event:: # echo 'hist:key=stacktrace:vals=len:pause' > \ - /sys/kernel/debug/tracing/events/net/netif_receive_skb/trigger + /sys/kernel/tracing/events/net/netif_receive_skb/trigger Next, we set up an 'enable_hist' trigger on the sched_process_exec event, with an 'if filename==/usr/bin/wget' filter. The effect of @@ -1037,7 +1037,7 @@ Extended error information hash table keyed on stacktrace:: # echo 'enable_hist:net:netif_receive_skb if filename==/usr/bin/wget' > \ - /sys/kernel/debug/tracing/events/sched/sched_process_exec/trigger + /sys/kernel/tracing/events/sched/sched_process_exec/trigger The aggregation continues until the netif_receive_skb is paused again, which is what the following disable_hist event does by @@ -1045,7 +1045,7 @@ Extended error information filter 'comm==wget':: # echo 'disable_hist:net:netif_receive_skb if comm==wget' > \ - /sys/kernel/debug/tracing/events/sched/sched_process_exit/trigger + /sys/kernel/tracing/events/sched/sched_process_exit/trigger Whenever a process exits and the comm field of the disable_hist trigger filter matches 'comm==wget', the netif_receive_skb hist @@ -1058,7 +1058,7 @@ Extended error information $ wget https://www.kernel.org/pub/linux/kernel/v3.x/patch-3.19.xz - # cat /sys/kernel/debug/tracing/events/net/netif_receive_skb/hist + # cat /sys/kernel/tracing/events/net/netif_receive_skb/hist # trigger info: hist:keys=stacktrace:vals=len:sort=hitcount:size=2048 [paused] { stacktrace: @@ -1142,12 +1142,12 @@ Extended error information again, we can just clear the histogram first:: # echo 'hist:key=stacktrace:vals=len:clear' >> \ - /sys/kernel/debug/tracing/events/net/netif_receive_skb/trigger + /sys/kernel/tracing/events/net/netif_receive_skb/trigger Just to verify that it is in fact cleared, here's what we now see in the hist file:: - # cat /sys/kernel/debug/tracing/events/net/netif_receive_skb/hist + # cat /sys/kernel/tracing/events/net/netif_receive_skb/hist # trigger info: hist:keys=stacktrace:vals=len:sort=hitcount:size=2048 [paused] Totals: @@ -1162,21 +1162,21 @@ Extended error information sched_process_exit events as such:: # echo 'enable_event:net:netif_receive_skb if filename==/usr/bin/wget' > \ - /sys/kernel/debug/tracing/events/sched/sched_process_exec/trigger + /sys/kernel/tracing/events/sched/sched_process_exec/trigger # echo 'disable_event:net:netif_receive_skb if comm==wget' > \ - /sys/kernel/debug/tracing/events/sched/sched_process_exit/trigger + /sys/kernel/tracing/events/sched/sched_process_exit/trigger If you read the trigger files for the sched_process_exec and sched_process_exit triggers, you should see two triggers for each: one enabling/disabling the hist aggregation and the other enabling/disabling the logging of events:: - # cat /sys/kernel/debug/tracing/events/sched/sched_process_exec/trigger + # cat /sys/kernel/tracing/events/sched/sched_process_exec/trigger enable_event:net:netif_receive_skb:unlimited if filename==/usr/bin/wget enable_hist:net:netif_receive_skb:unlimited if filename==/usr/bin/wget - # cat /sys/kernel/debug/tracing/events/sched/sched_process_exit/trigger + # cat /sys/kernel/tracing/events/sched/sched_process_exit/trigger enable_event:net:netif_receive_skb:unlimited if comm==wget disable_hist:net:netif_receive_skb:unlimited if comm==wget @@ -1192,7 +1192,7 @@ Extended error information saw in the last run, but this time you should also see the individual events in the trace file:: - # cat /sys/kernel/debug/tracing/trace + # cat /sys/kernel/tracing/trace # tracer: nop # @@ -1226,15 +1226,15 @@ Extended error information other things:: # echo 'hist:keys=skbaddr.hex:vals=len if len < 0' >> \ - /sys/kernel/debug/tracing/events/net/netif_receive_skb/trigger + /sys/kernel/tracing/events/net/netif_receive_skb/trigger # echo 'hist:keys=skbaddr.hex:vals=len if len > 4096' >> \ - /sys/kernel/debug/tracing/events/net/netif_receive_skb/trigger + /sys/kernel/tracing/events/net/netif_receive_skb/trigger # echo 'hist:keys=skbaddr.hex:vals=len if len == 256' >> \ - /sys/kernel/debug/tracing/events/net/netif_receive_skb/trigger + /sys/kernel/tracing/events/net/netif_receive_skb/trigger # echo 'hist:keys=skbaddr.hex:vals=len' >> \ - |
