Dynamic Tracing Guide


DTrace is a comprehensive dynamic tracing framework for the illumos™ Operating System. DTrace provides a powerful infrastructure to permit administrators, developers, and service personnel to concisely answer arbitrary questions about the behavior of the operating system and user programs. The illumos Dynamic Tracing Guide describes how to use DTrace to observe, debug, and tune system behavior. This book also includes a complete reference for bundled DTrace observability tools and the D programming language.

Note -

This illumos release supports systems that use the SPARCTM and x86 families of processor architectures: UltraSPARCTM, SPARC64, AMD64, Pentium, and Xeon EM64T. The supported systems appear in the illumos Hardware Compatibility List at http://www.illumos.org/hcl/. This document cites any implementation differences between the platform types.

In this document the term “x86” refers to 64–bit and 32–bit systems manufactured using processors compatible with the AMD64 or Intel Xeon/Pentium product families. For supported systems, see the illumos Hardware Compatibility List.

Who Should Use This Book

If you have ever wanted to understand the behavior of your system, DTrace is the tool for you. DTrace is a comprehensive dynamic tracing facility that is built into illumos. The DTrace facility can be used to examine the behavior of user programs. The DTrace facility can also be used to examine the behavior of the operating system. DTrace can be used by system administrators or application developers, and is suitable for use with live production systems. DTrace will allow you to explore your system to understand how it works, track down performance problems across many layers of software, or locate the cause of aberrant behavior. As you'll see, DTrace lets you create your own custom programs to dynamically instrument the system and provide immediate, concise answers to arbitrary questions you can formulate using the DTrace D programming language.

DTrace allows all illumos users to:

  • Dynamically enable and manage thousands of probes

  • Dynamically associate logical predicates and actions with probes

  • Dynamically manage trace buffers and buffer policies

  • Display and examine trace data from the live system or a crash dump

DTrace allows illumos developers and administrators to:

  • Implement custom scripts that use the DTrace facility

  • Implement layered tools that use DTrace to retrieve trace data

This guide will teach you everything you need to know about using DTrace. Basic familiarity with a programming language such as C or a scripting language such as awk(1) or perl(1) will help you learn DTrace and the D programming language faster, but you need not be an expert in any of these areas. If you have never written a program or script before in any language, Related Information provides references to other documents you might find useful.

How This Book Is Organized

Chapter 1, Introduction provides a whirlwind tour of the entire DTrace facility and introduces readers to the D programming language. Chapter 2, Types, Operators, and Expressions, Chapter 3, Variables, and Chapter 4, D Program Structure then discuss the fundamentals of D in greater detail, and explain how D programs are converted into dynamic instrumentation. This initial group of chapters should be read first by all readers.

Chapter 5, Pointers and Arrays, Chapter 6, Strings, Chapter 7, Structs and Unions, and Chapter 8, Type and Constant Definitions discuss the remaining D language features, most of which will be familiar already to C, C++, and JavaTM programmers. Readers who are unfamiliar with any of these languages should read these chapters; more experienced programmers may wish to proceed directly to later chapters.

Chapter 9, Aggregations and Chapter 10, Actions and Subroutines discuss DTrace's powerful primitive for aggregating data and the set of built-in actions that can be used to build tracing experiments. All readers should carefully read these chapters.

Chapter 11, Buffers and Buffering describes the DTrace policies for buffering data and how these can be configured. This chapter should be read by users once they are familiar with constructing and running D programs.

Chapter 12, Output Formatting describes the D output formatting actions as well as the default policy for formatting trace data. Readers who are familiar with the C printf function can rapidly skim this chapter. Readers who have never seen printf before should read this chapter carefully.

Chapter 13, Speculative Tracing discusses the DTrace facility for speculatively committing data to a trace buffer. This chapter should be read by users who need to use DTrace in a situation where data must be traced prior to understanding whether it is relevant to the question at hand.

Chapter 14, dtrace(1M) Utility provides a complete reference for the dtrace command-line utility, similar to the corresponding on-line manual page. Readers may wish to refer to this chapter when various command-line options are presented elsewhere in the book. Chapter 15, Scripting then discusses how the dtrace utility can be used to construct executable D scripts and process their command-line arguments, and Chapter 16, Options and Tunables describes the options that can be tuned on the command-line or from within a D program itself.

The group of chapters beginning with Chapter 17, dtrace Provider and ending with Chapter 32, fasttrap Provider discuss the various DTrace providers that can be used to instrument various aspects of the illumos system. All readers should skim these chapters to familiarize themselves with the various providers, and then return back to read particular chapters in detail as needed.

Chapter 33, User Process Tracing discusses examples of using DTrace to instrument user processes. Chapter 34, Statically Defined Tracing for User Applications describes how application programmers can add customized DTrace providers and probes to user applications. Readers who are user program developers or administrators and wish to use DTrace to investigate user process behavior should read these chapters.

Chapter 35, Security and the remaining chapters discuss advanced topics such as security, versioning, and stability attributes of DTrace, and how to perform boot-time and post-mortem tracing with DTrace. These chapters are intended for advanced DTrace users.

Related Information

These books and papers are recommended and related to the tasks that you need to perform with DTrace:

  • Kernighan, Brian W. and Ritchie, Dennis M. The C Programming Language. Prentice Hall, 1988. ISBN 0–13–110370–9

  • Vahalia, Uresh. UNIX Internals: The New Frontiers. Prentice Hall, 1996. ISBN 0-13-101908-2

  • Mauro, Jim and McDougall, Richard. Solaris Internals: Core Kernel Components. Sun Microsystems Press, 2001. ISBN 0-13-022496-0

  • Gregg, Brendan and Mauro, Jim. DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X and FreeBSD. Prentice Hall, 2011. ISBN 0-13-209151-8

You can share your DTrace experiences and scripts with the rest of the DTrace community on the DTrace discussion mailing list at http://dtrace.org/blogs/mailing-list/.

Documentation, Support, and Training

Sun Welcomes Your Comments

Sun is interested in improving its documentation and welcomes your comments and suggestions. To share your comments, go to and click Feedback.

Typographic Conventions

The following table describes the typographic conventions that are used in this book.

Typographic Conventions





The names of commands, files, and directories, and onscreen computer output

Edit your .login file.

Use ls -a to list all files.

machine_name% you have mail.


What you type, contrasted with onscreen computer output

machine_name% su



Placeholder: replace with a real name or value

The command to remove a file is rm filename.


Book titles, new terms, and terms to be emphasized

Read Chapter 6 in the User's Guide.

A cache is a copy that is stored locally.

Do not save the file.

Note: Some emphasized items appear bold online.

Shell Prompts in Command Examples

The following table shows the default UNIXTM system prompt and superuser prompt for the C shell, Bourne shell, and Korn shell.

Shell Prompts



C shell


C shell for superuser


Bourne shell and Korn shell


Bourne shell and Korn shell for superuser