DTrace

DTrace

Developer(s)	Sun Microsystems
Initial release	January 2005
Operating system	Unix-like
Type	tracing
License	Common Development and Distribution License
Website	http://opensolaris.org/os/community/dtrace/

DTrace is a comprehensive dynamic tracing framework created by Sun Microsystems for troubleshooting kernel and application problems on production systems in real time. It was first made available for use in November 2003, and was formally released as part of Sun's Solaris 10 in January 2005. DTrace was the first component of the OpenSolaris project to have its source code released under the Common Development and Distribution License (CDDL).

Description

DTrace is designed to give operational insights that allow users to tune and troubleshoot applications and the OS itself.

Tracing programs (also referred to as scripts) can be written using the D programming language (not to be confused with other programming languages named "D"). The language is a subset of C with added functions and variables specific to tracing. D programs resemble awk programs in structure; they consist of a set of actions rather than a top-down structured program. In a DTrace program, one or more probes (instrumentation points) are enabled, and each one is associated with an action; whenever the condition for the probe is met, the associated action is executed (the probe "fires"). A probe that fires may analyze the situation at run-time by accessing context variables and the stack and evaluating expressions; it can then print out or log some information, record it in a database, or modify context variables. The reading and writing of context variables allows probes to pass information to each other, allowing them to cooperatively analyze the correlation of different events.

Special consideration has been taken to make DTrace safe to use in a production environment. For example, there is minimal probe effect when tracing is underway, and no performance impact associated with any disabled probe; this is important since there are tens of thousands of DTrace probes that can be enabled. New probes can also be created dynamically.

DTrace was designed and implemented by Bryan Cantrill, Mike Shapiro, and Adam Leventhal. The authors received recognition in 2005 for the innovations in DTrace from InfoWorld and Technology Review. DTrace won the top prize in the Wall Street Journal's 2006 Technology Innovation Awards competition. The authors were recognized by USENIX with the Software Tools User Group (STUG) award in 2008.

Supported platforms

DTrace implementations require tight integration with the operating system kernel. Although DTrace was initially written for Solaris, its source code is freely available as part of the OpenSolaris project, and work is in progress to port it to FreeBSD as a substitute for the ktrace utility (in which there has been initial success; further work on DTrace integration into FreeBSD is being sponsored by Cisco and QNX.)

Apple added DTrace support in Mac OS X 10.5 "Leopard", including a GUI called Instruments. Unlike other platforms that DTrace is supported on, Mac OS X has a flag (P_LNOATTACH) that a program may set that disallows tracing of that process by debugging utilities such as DTrace and gdb. In the original Mac OS X DTrace implementation, this could affect tracing of other system information, as unrelated probes that should fire while a program with this flag set was running would fail to do so. This problem was addressed a few months later in the Mac OS X 10.5.3 update.

A port for Linux is currently under development. The current dtrace port compiles and runs on many linux kernels, with no kernel source code changes. This is desirable to ensure portability, but also to avoid licensing conflicts (CDDL vs GPL). User space DTrace probes (USDT) are nearly working, but there's still some work in clean up and garbage collection of the shadow process structures. Syscall tracing works fine.

Command line examples

DTrace requests can be invoked directly from the command line, providing one or more probes and actions as arguments. Some examples:

# New processes with arguments,
dtrace -n 'proc:::exec-success { trace(curpsinfo->pr_psargs); }'
 
# Files opened by process,
dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }'
 
# Syscall count by program,
dtrace -n 'syscall:::entry { @num[execname] = count(); }'
 
# Syscall count by syscall,
dtrace -n 'syscall:::entry { @num[probefunc] = count(); }'
 
# Syscall count by process,
dtrace -n 'syscall:::entry { @num[pid,execname] = count(); }'
 
# Disk size by process,
dtrace -n 'io:::start { printf("%d %s %d",pid,execname,args[0]->b_bcount); }'   
 
# Pages paged in by process,
dtrace -n 'vminfo:::pgpgin { @pg[execname] = sum(arg0); }'

See also

• SystemTap
• IBM ProbeVue

References

• Bryan Cantrill (2006). "Hidden in Plain Sight". ACM Queue. 4 (1): 26–36. doi:10.1145/1117389.1117401. ISSN 1542-7730. Retrieved 2006-09-08. {{cite journal}}: Unknown parameter |month= ignored (help)
• Bryan M. Cantrill, Michael W. Shapiro and Adam H. Leventhal (2004). "Dynamic Instrumentation of Production Systems". Proceedings of the 2004 USENIX Annual Technical Conference. Retrieved 2006-09-08. {{cite conference}}: Unknown parameter |booktitle= ignored (|book-title= suggested) (help); Unknown parameter |month= ignored (help)

External links

• DTrace BigAdmin Forum
• OpenSolaris DTrace Community
• Solaris Dynamic Tracing Guide
• DTrace FAQ at Genunix
• DTrace for FreeBSD
• DTraceToolkit
• Top Ten DTrace scripts
• Understanding vmstat and mpstat output with DTrace
• Project D-Light Tutorial - a DTrace plug-in for the Sun Studio IDE
• Exploring Leopard with DTrace How to use DTrace for debugging and exploration
• Mac OS X 10.5 Leopard Review - Good description of DTrace from a writer at Ars Technica
• Google Tech Talk on DTrace given by Bryan Cantrill