Category: These are items grouped according to subcomponent or related areas B - buffering R - Recording Interface C - Trace Control I - Instrumentation F - Formatting S - Spooling Targets: intended audience/audience critically affected SS - system serviceability, NW - network diag, PF - performace, US - user, DP - dynamic probes Provided by/ present in: LP - IBM LinProf, K42 - IBM K42 OS Reseach Project, LTT - Linux Trace Toolkit, LKST - Hitachi Linux Kernel State Trace |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | Item | Cat | LI# | Prty| Target| Provided by| Description | | # | | | | | | | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 1 |B, R | |H |SS,PF | |Init-time tracing | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 2 |B,I | |M |NW | |Able to handle throughput of network drivers such as Gb Etherner | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 3 |B | |H |ALL | |Multiuple buffering schemes - switchable globally or selctable by trace | | | | | | | |client | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 3a |B | |H |ALL |LP, LKST, |Per-CPU Buffering scheme | | | | | | |K42 | | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 3b |B | |H |ALL |LTT |Global buffer scheme | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 3c |B | |H |ALL | |Per-process buffer scheme | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 3d |B | |H |ALL | |Per-NGPT thread buffer scheme | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 3e |B | |H |ALL | |Per-component buffer scheme | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 4 |B | |H |PF |K42, LP |Minimal latency - minimal or no serialisation | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 5 |R | |H |PF |LP |Fine granularity time stamping - min=o(CPU cycle time), max=.05 Gb Ethernet | | | | | | | |interrupt rate - see note 1 | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 6 |R | |H |PF |LP |System information logged at trace start - see note 2 | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 7 |R | |H |PF |LP |Collection of process memory map information at trace start and restart. | | | | | | | |Along with what would be collected by the next item in this list, this | | | | | | | |allows address-to-name resolution for user space. | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 8 |R | |L |PF |LP |Collection of process memory map information at fork/exec/exit/... | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 9 |R | |M | |LP |Includes the facility to write system snapshots (total memory layout for | | | | | | | |kernel, drivers, and all processes) to a file. This is required for trace | | | | | | | |post-processing on a system other than the one producing the trace. | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 10 |R | |L |US | |Posix compliance | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 11 |R | |H |ALL |K42, LTT |Tracing from user and system space | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 12 |R | |H |ALL |LTT |Ability to define static trace hooks | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 13 |I | |H |ALL |LP, K42 |Simple wrapper macros for trace instrumentation | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 14 |C | |H |ALL |LP, LTT, |Ability to activate a selection of events dynamically in a uniform manner | | | | | | |LKST, K42 ? | | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 15 |C | |H |ALL |K42 |Global control menchainsm for global trace options, but local controls for | | | | | | | |component traces | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 16 |I | |H |SS, PF | |Ability to do function entry/exit tracing facility | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 17 |I | |M |PF |K42, LP |Processor performance counter sampling and recording - See note 3. | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 18 |F | |M |PF |LP, LTT |A set of performance analysis post-processing programs. | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 19 |I | |M |SS |LP |Trace mechanism and instrumentation to be separate capabilities | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 20 |R,F | |H |SS |LP, K42 |Easily expandable with new trace types | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 21 |I,R | |M |SS |K42 |Major/minor code means of identification/registration/assignment | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 22 |F | |M |SS,US | |A flexible formatting mechanism that will cater for structures and arrays of| | | | | | | |structures with recursion. | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 23 |R | |M |DP |LTT |Interface for Dynamic Probes | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 24 |F | |M |DP,US |LTT |Formatting exit capability | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 25 |F | |H |SS | |Data presentation utilities to be able to present data from multiple trace | | | | | | | |instances in a logically combined form | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 26 |S | |H |ALL |LTT |Background spooling mechanism that operates with minimal distortion to | | | | | | | |captured events. | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 27 |S,F | |M |SS, PF | |Data reduction faciltiies able to logically combine traces from more than | | | | | | | |one system | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 28 |R, B | |H |SS |LTT |Filtering and selection mechanisms within formatting utility | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 29 |R | |H |SS |LTT |Suspend & Resume capability | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 30 |B | |H |NW | |Per-packet (N/W) trace buffer instance | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 31 |B | |H |PF |K42 |Logging without locking | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 32 |B | |H |SS |K42, LTT |Variable size event records | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 33 |R, B | |L |PF |K42, LTT |Random access to trace event stream | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 34 |B,R | |H |SS, PF |K42, LTT, |Abailty to make tracing infrastructure redisent. | | | | | | |LKST, LP | | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 35 |B | |H |SS | |Support component "flight recorder" minimal level serviceability tracing | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| | 36 |I | |M |SS | |Fine grained dynamic trace instrumentation for kernel space and user | | | | | | | |subsystems | |------+-----+-----+-----+-------+------------+----------------------------------------------------------------------------| Notes These are clarifications of requirements from Niels Christiansen: 1) Fine granularity time-stamp - how fine? Fine granularity means fine enough to merge processor-specific trace files. For all practical purposes, that means cycle counter. 2) System info logged at trace start - why?, what should it include? Because we need to know the address space of all processes in order to resolve user space addresses correctly. What we need is the information in /proc/*/maps for every running process. We also need to record that information when new tasks are created, fork, exec, and die. LTT does some of this by recording to a separate file,which would do (though unhandy) for the initial state. For all the others, the stuff needs to be timestamped or we lose chronology. Ideally, all are in the tracefile, in my opinion. 3) Processor performance counter - how does this manifest itself on non-Intel processors? All modern processors have performance counters. I believe they all have counters for things like cache misses, retired instructions, and most others that may be important. The requirement is that the selected counters be recorded with every trace record so it is possible to calculate the CPI or whatever else you need between two trace points. This is especially important for entry/exit instrumentation because it gives you a way to determine path length and cache misses or whatever per function. It can also be used to do the same for other "units of work", such as system calls and used to create time-profiles showing how counted events vary over time. Richard J Moore CEng, MIEE, Consulting IT Specialist, TSM RAS Project Lead - Linux Technology Centre (ATS-PIC). http://oss.software.ibm.com/developerworks/opensource/linux Office: (+44) (0)1962-817072, Mobile: (+44) (0)7768-298183 IBM UK Ltd, MP135 Galileo Centre, Hursley Park, Winchester, SO21 2JN, UK The IBM Academy will hold a Conference on Performance Engineering in Toronto July 8-10. A High Availability Conference follows July 10-12. Details on http://w3.ibm.com/academy/