DTrace is a performance analysis and troubleshooting tool that is included by default with various operating systems, including Solaris, Mac OS X and FreeBSD. A Linux port is in development.
DTrace instruments all software.
Not just user-level software, including applications, databases and webservers, but also the operating system kernel and device drivers. The name is short for Dynamic Tracing: an instrumentation technique pioneered by DTrace which dynamically patches live running instructions with instrumentation code. The DTrace facility also supports Static Tracing: where user-friendly trace points are added to code and compiled-in before deployment.
DTrace provides a language, ‘D’, for writing DTrace scripts and one-liners. The language is like C and awk, and provides powerful ways to filter and summarize data in-kernel before passing to user-land. This is an important feature that enables DTrace to be used in performance-sensitive production environments, as it can greatly reduce the overhead of gathering and presenting data.
Example
Tracing which processes are executed when you run “man ls”, along with nanosecond timestamps:
# dtrace -n 'proc:::exec-success { printf("%d %s", timestamp, curpsinfo->pr_psargs); }'
dtrace: description 'proc:::exec-success ' matched 1 probe
CPU ID FUNCTION:NAME
1 797 exec_common:exec-success 21935388676181394 man ls
0 797 exec_common:exec-success 21935388840101743 sh -c cd /usr/share/man; tbl /usr/share/man/man1/ls.1 |neqn /usr/share/lib/pub/
1 797 exec_common:exec-success 21935388858652639 col -x
0 797 exec_common:exec-success 21935388863714971 neqn /usr/share/lib/pub/eqnchar -
0 797 exec_common:exec-success 21935388867119787 tbl /usr/share/man/man1/ls.1
1 797 exec_common:exec-success 21935388881310626 nroff -u0 -Tlp -man -
0 797 exec_common:exec-success 21935388922364648 sh -c trap '' 1 15; /usr/bin/mv -f /tmp/mpDKaiGn /usr/share/man/cat1/ls.1 2> /d
1 797 exec_common:exec-success 21935388930987671 /usr/bin/mv -f /tmp/mpDKaiGn /usr/share/man/cat1/ls.1
1 797 exec_common:exec-success 21935388948485045 sh -c less -siM /tmp/mpDKaiGn
0 797 exec_common:exec-success 21935388971039204 less -siM /tmp/mpDKaiGn
^C
That’s a one-liner. Much more complex scripts can be written as desired.
Aside from using DTrace at the command line, you may use it indirectly via another interface that uses DTrace as the back-end for gathering data. Examples of these include Apple’s Instruments, Oracle’s Sun ZFS Storage Analytics, and Joyent’s Cloud Analytics.
Books
- Dynamic Tracing Guide: the reference manual, freely available online.
- DTrace: the DTrace book of scripts and strategy, 1100 pages (2011).
- Solaris Performance and Tools: takes Solaris perf analysis further with DTrace, 440 pages (2006).
Awards
The Wall Street Journal awarded DTrace top prize in its 2006 Technology Innovation Awards competition:
A Wall Street Journal editor initially screened more than 600 applications. The judges then considered 121 of the entries, selecting 12 category winners and 37 runners-up. Among the category winners are the top three award winners. In selecting winners, judges considered whether the technology truly represents a breakthrough from conventional methods, rather than just an incremental improvement.
This was inclusive of all technology areas; third prize was for a “powdered, inhalable insulin designed to replace shots for the treatment of diabetes”.
tl;dr
DTrace is a magician that conjures up rainbows, ponies and unicorns — and does it all entirely safely and in production!
dtrace.org
dtrace.org hosts blogs from software developers related to DTrace. These include Bryan Cantrill and Adam Leventhal, who with Mike Shapiro invented and developed DTrace at Sun Microsystems; and Brendan Gregg, the co-author of the DTrace book and the DTraceToolkit.
Or as someone put it neatly: “It’s just a bunch of ex-Sun guys who share the domain.”