histogram.txt

			     Event Histograms

		    Documentation written by Tom Zanussi

1. Introduction
===============

  Histogram triggers are special event triggers that can be used to
  aggregate trace event data into histograms.  For information on
  trace events and event triggers, see Documentation/trace/events.txt.


2. Histogram Trigger Command
============================

  A histogram trigger command is an event trigger command that
  aggregates event hits into a hash table keyed on one or more trace
  event format fields (or stacktrace) and a set of running totals
  derived from one or more trace event format fields and/or event
  counts (hitcount).

  The format of a hist trigger is as follows:

        hist:keys=<field1[,field2,...]>[:values=<field1[,field2,...]>]
          [:sort=<field1[,field2,...]>][:size=#entries][:pause][:continue]
          [:clear][:name=histname1] [if <filter>]

  When a matching event is hit, an entry is added to a hash table
  using the key(s) and value(s) named.  Keys and values correspond to
  fields in the event's format description.  Values must correspond to
  numeric fields - on an event hit, the value(s) will be added to a
  sum kept for that field.  The special string 'hitcount' can be used
  in place of an explicit value field - this is simply a count of
  event hits.  If 'values' isn't specified, an implicit 'hitcount'
  value will be automatically created and used as the only value.
  Keys can be any field, or the special string 'stacktrace', which
  will use the event's kernel stacktrace as the key.  The keywords
  'keys' or 'key' can be used to specify keys, and the keywords
  'values', 'vals', or 'val' can be used to specify values.  Compound
  keys consisting of up to two fields can be specified by the 'keys'
  keyword.  Hashing a compound key produces a unique entry in the
  table for each unique combination of component keys, and can be
  useful for providing more fine-grained summaries of event data.
  Additionally, sort keys consisting of up to two fields can be
  specified by the 'sort' keyword.  If more than one field is
  specified, the result will be a 'sort within a sort': the first key
  is taken to be the primary sort key and the second the secondary
  key.  If a hist trigger is given a name using the 'name' parameter,
  its histogram data will be shared with other triggers of the same
  name, and trigger hits will update this common data.  Only triggers
  with 'compatible' fields can be combined in this way; triggers are
  'compatible' if the fields named in the trigger share the same
  number and type of fields and those fields also have the same names.
  Note that any two events always share the compatible 'hitcount' and
  'stacktrace' fields and can therefore be combined using those
  fields, however pointless that may be.

  'hist' triggers add a 'hist' file to each event's subdirectory.
  Reading the 'hist' file for the event will dump the hash table in
  its entirety to stdout.  If there are multiple hist triggers
  attached to an event, there will be a table for each trigger in the
  output.  The table displayed for a named trigger will be the same as
  any other instance having the same name. Each printed hash table
  entry is a simple list of the keys and values comprising the entry;
  keys are printed first and are delineated by curly braces, and are
  followed by the set of value fields for the entry.  By default,
  numeric fields are displayed as base-10 integers.  This can be
  modified by appending any of the following modifiers to the field
  name:

        .hex        display a number as a hex value
	.sym        display an address as a symbol
	.sym-offset display an address as a symbol and offset
	.syscall    display a syscall id as a system call name
	.execname   display a common_pid as a program name
	.log2       display log2 value rather than raw number
	.usecs      display a common_timestamp in microseconds

  Note that in general the semantics of a given field aren't
  interpreted when applying a modifier to it, but there are some
  restrictions to be aware of in this regard:

    - only the 'hex' modifier can be used for values (because values
      are essentially sums, and the other modifiers don't make sense
      in that context).
    - the 'execname' modifier can only be used on a 'common_pid'.  The
      reason for this is that the execname is simply the 'comm' value
      saved for the 'current' process when an event was triggered,
      which is the same as the common_pid value saved by the event
      tracing code.  Trying to apply that comm value to other pid
      values wouldn't be correct, and typically events that care save
      pid-specific comm fields in the event itself.

  A typical usage scenario would be the following to enable a hist
  trigger, read its current contents, and then turn it off:

  # echo 'hist:keys=skbaddr.hex:vals=len' > \
    /sys/kernel/debug/tracing/events/net/netif_rx/trigger

  # cat /sys/kernel/debug/tracing/events/net/netif_rx/hist

  # echo '!hist:keys=skbaddr.hex:vals=len' > \
    /sys/kernel/debug/tracing/events/net/netif_rx/trigger

  The trigger file itself can be read to show the details of the
  currently attached hist trigger.  This information is also displayed
  at the top of the 'hist' file when read.

  By default, the size of the hash table is 2048 entries.  The 'size'
  parameter can be used to specify more or fewer than that.  The units
  are in terms of hashtable entries - if a run uses more entries than
  specified, the results will show the number of 'drops', the number
  of hits that were ignored.  The size should be a power of 2 between
  128 and 131072 (any non- power-of-2 number specified will be rounded
  up).

  The 'sort' parameter can be used to specify a value field to sort
  on.  The default if unspecified is 'hitcount' and the default sort
  order is 'ascending'.  To sort in the opposite direction, append
  .descending' to the sort key.

  The 'pause' parameter can be used to pause an existing hist trigger
  or to start a hist trigger but not log any events until told to do
  so.  'continue' or 'cont' can be used to start or restart a paused
  hist trigger.

  The 'clear' parameter will clear the contents of a running hist
  trigger and leave its current paused/active state.

  Note that the 'pause', 'cont', and 'clear' parameters should be
  applied using 'append' shell operator ('>>') if applied to an
  existing trigger, rather than via the '>' operator, which will cause
  the trigger to be removed through truncation.

- enable_hist/disable_hist

  The enable_hist and disable_hist triggers can be used to have one
  event conditionally start and stop another event's already-attached
  hist trigger.  Any number of enable_hist and disable_hist triggers
  can be attached to a given event, allowing that event to kick off
  and stop aggregations on a host of other events.

  The format is very similar to the enable/disable_event triggers:

      enable_hist:<system>:<event>[:count]
      disable_hist:<system>:<event>[:count]

  Instead of enabling or disabling the tracing of the target event
  into the trace buffer as the enable/disable_event triggers do, the
  enable/disable_hist triggers enable or disable the aggregation of
  the target event into a hash table.

  A typical usage scenario for the enable_hist/disable_hist triggers
  would be to first set up a paused hist trigger on some event,
  followed by an enable_hist/disable_hist pair that turns the hist
  aggregation on and off when conditions of interest are hit:

  # echo 'hist:keys=skbaddr.hex:vals=len:pause' > \
    /sys/kernel/debug/tracing/events/net/netif_receive_skb/trigger

  # echo 'enable_hist:net:netif_receive_skb if filename==/usr/bin/wget' > \
    /sys/kernel/debug/tracing/events/sched/sched_process_exec/trigger

  # echo 'disable_hist:net:netif_receive_skb if comm==wget' > \
    /sys/kernel/debug/tracing/events/sched/sched_process_exit/trigger

  The above sets up an initially paused hist trigger which is unpaused
  and starts aggregating events when a given program is executed, and
  which stops aggregating when the process exits and the hist trigger
  is paused again.

  The examples below provide a more concrete illustration of the
  concepts and typical usage patterns discussed above.


6.2 'hist' trigger examples
---------------------------

  The first set of examples creates aggregations using the kmalloc
  event.  The fields that can be used for the hist trigger are listed
  in the kmalloc event's format file:

    # cat /sys/kernel/debug/tracing/events/kmem/kmalloc/format
    name: kmalloc
    ID: 374
    format:
	field:unsigned short common_type;	offset:0;	size:2;	signed:0;
	field:unsigned char common_flags;	offset:2;	size:1;	signed:0;
	field:unsigned char common_preempt_count;		offset:3;	size:1;	signed:0;
	field:int common_pid;					offset:4;	size:4;	signed:1;

	field:unsigned long call_site;				offset:8;	size:8;	signed:0;
	field:const void * ptr;					offset:16;	size:8;	signed:0;
	field:size_t bytes_req;					offset:24;	size:8;	signed:0;
	field:size_t bytes_alloc;				offset:32;	size:8;	signed:0;
	field:gfp_t gfp_flags;					offset:40;	size:4;	signed:0;

  We'll start by creating a hist trigger that generates a simple table
  that lists the total number of bytes requested for each function in
  the kernel that made one or more calls to kmalloc: