We at Joyent use DTrace for understanding and debugging userland applications just as often as we do for the kernel. That is part of the reason why we’ve worked on things like flamegraphs, the Node.js ustack helper, and the integration of libusdt in node module’s like restify and bunyan.
I’ve just put back some work that makes observing userland with DTrace a lot simpler and much more powerful. Before we meet the devil in the details, let’s start with an example:
$ cat startd.d
pid$target::start_instance:entry,
pid$target::stop_instance:entry
{
printf("%s: %s\n", probefunc == "start_instance" ?
"starting" : "stopping", stringof(args[1]->ri_i.i_fmri));
}
$ dtrace -qs startd.d -p $(pgrep -o startd)
...
stopping: svc:/system/intrd:default
stopping: svc:/system/cron:default
start_instance:entry starting: svc:/system/cron:default
...
If you’re familiar with DTrace you might realize that this doesn’t really look like what you’re expecting! Hey Robert, won’t that script blow up without the copyin? Also where did the args[] come from with the pid provider?!
The answer to this minor mystery is that DTrace can now leverage type information in userland programs. Not only does the compiler know the size and layout of types, it’ll also take care of adding calls to copyin for you so you can dereference members without fear. To explain how we’ve managed all of this, we need to go into a bit of back story.
The Compiler is the Enemy
Since the beginning of programming, we’ve needed to be able to debug the programs that we’ve written. A large chunk of time has been spent on tooling to be able to understand and root cause these bugs whether working on a live system or doing a post-mortem analysis on something like a core dump.
Unfortunately, the compiler is in many ways our enemy. Its goal is to take whatever well commented and understandable code we might have and transform it not only into something that a processor can understand, but often times transforming it through optimization passes into something that no longer resembles what we originally wrote.
This problem isn’t limited to languages like C and C++. In fact, many of the same problems apply when you use any compiler, be it your current compile to JavaScript language of the day (emscripten and coffeescript) or something like lex and yacc.
Fortunately, both the compiler and the linker are just software. Shortly after we first hit this problem, they were modified to produce debugging information that could be encoded into the binaries they produced. Folks even were able to encode this kind of information in the original ‘a.out’ executable format that came around in first edition UNIX.
One of the first popular formats that was used was called stabs. It was used on many operating systems for many years and you can still convince compilers like gcc, clang, and suncc to generate it. Since then, DWARF has become the most popular and commonly used format. The initial origin of DWARF came from Bell Labs, but it was rather unpopular because the debugging data that it created was just too large. Since then DWARF has become more standardized and more compact than the first version of DWARF. However, it is a rather complicated format.
With all of these formats there is a trade-off between expressibility and size. If the debugging information takes too much space, then people stop including it. Most available OS and package distributions do not incorporate debugging information. If you’re lucky, they separate that information into different packages. This means that when you’re debugging a problem in production you very often don’t have the very information you need. Even more frustrating, when this information is in a separate file and you’re trying to do post-mortem analysis, then you need to track that down and make sure that you have the right version of the debug information that corresponds to what you were using in production.
This situation is unsatisfying, but – we have other options! Sun developed CTF in Solaris 9 with the intent of using it with mdb and eventually DTrace. In illumos, we put CTF data in every kernel module, library, and a majority of applications. We don’t store all the information that you might get in, say, DWARF, but we store what we’ve found over the years to be the most useful information.
CTF data includes the following pieces of information:
o The definitions of all types and structures
o The arguments and types of each function
o The types of function return values
o The types of global variables
All of the CTF data for a given library, binary, or kernel module is found inside of what we call a CTF Container. The CTF container is found as its own section in an ELF object. A simple way to see if something in question has CTF is to run elfdump(1). Here’s an example:
$ elfdump /lib/libc.so | grep SUNW_ctf
Section Header[37]: sh_name: .SUNW_ctf
If a library or program does not have CTF data then the section won’t show up in the list and the grep should turn up empty.
CTF and DTrace
If you’ve ever wondered how it is that DTrace knows types when you use various providers, like args[] in fbt, the answer to that is CTF. When you run DTrace, it loads relevant CTF containers for the kernel. In fact, even the basic types that D provides, such as an int or types that you define in a D script, end up in a CTF container. Consider the following dtrace invocation:
# dtrace -l -v -n 'fbt::ip_input:entry'
ID PROVIDER MODULE FUNCTION
NAME
37546 fbt ip ip_input
entry
Probe Description Attributes
Identifier Names: Private
Data Semantics: Private
Dependency Class: Unknown
Argument Attributes
Identifier Names: Private
Data Semantics: Private
Dependency Class: ISA
Argument Types
args[0]: struct ill_s *
args[1]: ill_rx_ring_t *
args[2]: mblk_t *
args[3]: struct mac_header_info_s *
The D compiler used its CTF data for the ip module to determine the arguments and their types. We can then run something like:
# dtrace -qn 'fbt::ip_input:entry{ print(*args[0]); exit(0) }'
...
struct ill_s {
pfillinput_t ill_inputfn = ip`ill_input_short_v4
ill_if_t *ill_ifptr = 0xffffff0148a27ab8
queue_t *ill_rq = 0xffffff014b65ba60
queue_t *ill_wq = 0xffffff014b65bb58
int ill_error = 0
ipif_t *ill_ipif = 0xffffff014b6fc460
uint_t ill_ipif_up_count = 0x1
uint_t ill_max_frag = 0x5dc
uint_t ill_current_frag = 0x5dc
uint_t ill_mtu = 0x5dc
uint_t ill_mc_mtu = 0x5dc
uint_t ill_metric = 0
char *ill_name = 0xffffff014bc642c8
...
}
DTrace uses the CTF data for the struct ill_s to interpret all of the data and correlate it with the appropriate members.
Bringing it to Userland
While DTrace happily consumes all of the CTF data for the various kernel modules, up to now it simply ignored all of the CTF data in userland applications. With my changes, DTrace will now consume that CTF data for referenced processes. Let’s go back to the example that we opened this blog post with. If we list those probes verbosely we see:
# dtrace -l -v -n
# 'pid$target::stop_instance:entry,pid$target::start_instance:entry' -p
# $(pgrep -o startd)
ID PROVIDER MODULE FUNCTION
NAME
62420 pid8 svc.startd stop_instance
entry
Probe Description Attributes
Identifier Names: Private
Data Semantics: Private
Dependency Class: Unknown
Argument Attributes
Identifier Names: Private
Data Semantics: Private
Dependency Class: Unknown
Argument Types
args[0]: userland scf_handle_t *
args[1]: userland restarter_inst_t *
args[2]: userland stop_cause_t
62419 pid8 svc.startd start_instance
entry
Probe Description Attributes
Identifier Names: Private
Data Semantics: Private
Dependency Class: Unknown
Argument Attributes
Identifier Names: Private
Data Semantics: Private
Dependency Class: Unknown
Argument Types
args[0]: userland scf_handle_t *
args[1]: userland restarter_inst_t *
args[2]: userland int32_t
Before this change, the Argument Types section would be empty. Because svc.startd has CTF data, DTrace was able to figure out the types of startd’s functions. Without these changes, you’d have to manually define the types of a scf_handle_t and a restarter_inst_t and cast the raw arguments to the correct type. If you ended up with a structure that has a lot of nested structures, defining all of them in D can quickly become turtles all the way down.
Look Ma, no copyin!
DTrace runs in a special context in the kernel, and often times DTrace requires you to think about what’s going on. Just as the kernel can’t access arbitrary user memory without copying it in, neither can DTrace. Consider the following classic one liner:
dtrace -n 'syscall::open:entry{ trace(copyinstr(arg0)); }'
You’ll note that we have to use copyinstr. That tells DTrace that we need to copy in the string from userland into the kernel in order to do something with it, whether that be an aggregation, printing it out, or saving it for some later action. This copyin isn’t limited to just strings. If you wanted to dereference some member of a structure, you’d have to either copy in the full structure, or manually determine the offset location of the member you care about.
At the previous illumos hackathon, Adam Leventhal had the idea of introducing a keyword into D, the DTrace language, that would tell the D compiler that a type is from userland. The D compiler would then take care of copying in the data automatically. Together we built a working prototype, with the keyword userland.
While certainly useful on its own, it really shines when combined with CTF data, as in the pid provider. The pid provider automatically applies the userland keyword to all of the types that are found in args[]. This allowed us to skip the copyin of intermediate structures and write a simple expression. eg. in our initial example we are able to do something that looks like a normal dereference in C: args[1]->ri_i.i_fmri. Before this change, you would have had to do three copyins: one for args[1], one for ri_i, and a final one for the string i_fmri.
As an example of the kinds of scripts that motivated this, here’s a portion of a D script that I used to help debug part of an issue inside of QEMU and KVM:
$ cat event.d
...
/* After its mfence */
pid$target::vring_notify:entry
/self->trace/
{
self->arg = arg1;
this->data = (char *)(copyin(self->arg + 0x28, 8));
self->sign = *(uint16_t *)(this->data+0x2);
}
...
pid$target::vring_notify:return
/self->trace && self->arg/
{
this->data = (char *)(copyin(self->arg + 0x28, 8));
printf("%d notify signal index: 0x%04x notify? %d\n", timestamp,
*(uint16_t *)(this->data + 0x2), arg1);
}
...
There are many parts of this script where I’ve had to manually encode structure offsets and structure sizes. In the larger script, I had to play games with looking at registers and the pid provider’s ability to instrument arbitrary assembly instructions. I for one am glad that I’ll have to write a lot less of these.
When you have no CTF
While no binary should be left behind, not everything has CTF data today. But the userland keyword can still be incredibly useful even without it. Whenever you’re making a cast, you can note that the type is a userland type with the userland keyword, and the D compiler will do all the heavy lifting from there.
Here’s an example from a program that has a traditional linked list, but doesn’t have any CTF data:
struct foo;
typedef struct foo {
struct foo *foo_next;
char *foo_value;
} foo_t;
pid$target::print_head:entry
{
this->p = (userland foo_t *)arg0;
trace(stringof(this->p->foo_next->foo_next->foo_value));
}
The nice thing with the userland keyword is that you don’t have to do any copyin or worry about figuring out structure sizes. The goal with all of this is to make it simpler and more intuitive to write D scripts.
Referring to types
As a part of all this, you can refer to types in arbitrary processes that are running on the system, as well as the target. The syntax is designed to be flexible enough to allow you to specify not just the pid, but the link map, library, and type name, but you can also just specify the pid and type that you want. While you can’t refer to macros inside these definitions, you can use the shorthand `pid“ to refer to the value of $TARGET.
For example, say you wanted to refer to a glob_t, which on illumos is defined via a typdef struct glob_t { ... } glob_t, there are a lot of different ways that you can do that. The following are all equivalent:
dtrace -n 'BEGIN{ trace((pid`glob_t *)0); }' -p 8
dtrace -n 'BEGIN{ trace((pid`libc.so.1`glob_t *)0); }' -p 8
dtrace -n 'BEGIN{ trace((pid`LM0`libc.so.1`glob_t *)0); }' -p 8
dtrace -n 'BEGIN{ trace((pid8`glob_t *)0); }'
dtrace -n 'BEGIN{ trace((pid8`libc.so.1`glob_t *)0); }'
dtrace -n 'BEGIN{ trace((pid8`LM0`libc.so.1`glob_t *)0); }'
dtrace -n 'BEGIN{ trace((struct pid`glob_t *)0); }' -p 8
dtrace -n 'BEGIN{ trace((struct pid`libc.so.1`glob_t *)0); }' -p 8
dtrace -n 'BEGIN{ trace((struct pid`LM0`libc.so.1`glob_t *)0); }' -p 8
dtrace -n 'BEGIN{ trace((struct pid8`glob_t *)0); }'
dtrace -n 'BEGIN{ trace((struct pid8`libc.so.1`glob_t *)0); }'
dtrace -n 'BEGIN{ trace((struct pid8`LM0`libc.so.1`glob_t *)0); }'
All of these would also work with the userland keyword. The userland keyword interacts with structs a bit differently than one might expect, so let’s show all of our above examples with the userland keyword:
dtrace -n 'BEGIN{ trace((userland pid`glob_t *)0); }' -p 8
dtrace -n 'BEGIN{ trace((userland pid`libc.so.1`glob_t *)0); }' -p 8
dtrace -n 'BEGIN{ trace((userland pid`LM0`libc.so.1`glob_t *)0); }' -p 8
dtrace -n 'BEGIN{ trace((userland pid8`glob_t *)0); }'
dtrace -n 'BEGIN{ trace((userland pid8`libc.so.1`glob_t *)0); }'
dtrace -n 'BEGIN{ trace((userland pid8`LM0`libc.so.1`glob_t *)0); }'
dtrace -n 'BEGIN{ trace((struct userland pid`glob_t *)0); }' -p 8
dtrace -n 'BEGIN{ trace((struct userland pid`libc.so.1`glob_t *)0); }' -p 8
dtrace -n 'BEGIN{ trace((struct userland pid`LM0`libc.so.1`glob_t *)0); }' -p 8
dtrace -n 'BEGIN{ trace((struct userland pid8`glob_t *)0); }'
dtrace -n 'BEGIN{ trace((struct userland pid8`libc.so.1`glob_t *)0); }'
dtrace -n 'BEGIN{ trace((struct userland pid8`LM0`libc.so.1`glob_t *)0); }'
What’s next?
From here you can get started with userland ctf and the userland keyword in DTrace in current releases of SmartOS. They’ll be making their way to an illumos distribution near you some time soon.
Now that we have this, we’re starting to make plans for the future. One idea that Adam had is to make it easy to scope a structure definition to a target process’s data model.
Another big task that we have is to make it easy to get CTF into programs and ideally, make it invisible to anyone using any compiler tool chain on illumos!
With that, happy tracing!