Back in January, I posted about a problem with Apple’s port of DTrace to Mac OS X. The heart of the issue is that their port would silently drop data such that certain experiments would be quietly invalid. Unfortunately, most reactions seized on a headline paraphrasing a line of the post — albeit with the critical negation omitted (the subject and language were, perhaps, too baroque to expect the press to read every excruciating word). The good news is that Apple has (quietly) fixed the problem in Mac OS X 10.5.3.
One issue was that timer based probes wouldn’t fire if certain applications were actively executing (e.g. iTunes). This was evident both by counting periodic probe firings, and by the absence of certain applications when profiling. Apple chose to solve this problem by allowing the probes to fire while denying any inspection of untraceable processes (and generating a verbose error in that case). This script which should count 1000 firings per virtual CPU gave sporadic results on earlier revisions of Mac OS X 10.5:
profile-1000
{
@ = count();
}
tick-1s
{
printa(@);
clear(@);
}
On 10.5.3, the output is exactly what one would expect on a 2-core CPU (1,000 executions per core):
1 22697 :tick-1s 2000 1 22697 :tick-1s 2000
On previous revisions, profiling to see what applications were spending the most time on CPU would silently omit certain applications. Now, while we can’t actually peer into those apps, we can infer the presence of stealthy apps when we encounter an error:
profile-199
{
@[execname] = count();
}
ERROR
{
@["=stealth app="] = count();
}
Running this DTrace script will generate a lot of errors as we try to evaluate the execname variable for secret applications, but at the end we’ll end up with a table like this:
Adium 1 GrowlHelperApp 1 iCal 1 kdcmond 1 loginwindow 1 Mail 2 Activity Monito 3 ntpd 3 pmTool 6 mlb-nexdef-auto 12 Terminal 14 =stealth app= 29 WindowServer 34 kernel_task 307 Safari 571
A big thank you to Apple for making progress on this issue; the situation is now much improved and considerably more palatable. That said, there are a couple of problems. The first is squarely the fault of team DTrace: we should probably have a mode where errors aren’t printed particularly if the script is already handling them explicitly using an ERROR probe as in the script above. For the Apple folks: I’d argue that revealing the name of otherwise untraceable processes is no more transparent than what Activity Monitor provides — could I have that please? Also, I’m not sure if this has always been true, but the ustack() action doesn’t seem to work from the profile action so simple profiling scripts like this one produce a bunch of errors and no output:
profile-199
/execname == "Safari"/
{
@[ustack()] = count();
}
But to reiterate: thank you thank you thank you, Steve, James, Tom, and the rest of the DTrace folks at Apple. It’s great to see these issues being addressed. The whole DTrace community appreciates it.
20 Responses
I’m constantly amazed at the worshipping of Apple. Here they are screwing up, by design, and then fixing the issue a little bit and you’re happy? I bet you’d be livid if Microsoft were to be in a similar situation. You are openly sharing DTrace with the world and here comes the most ‘closed’ company in the world, hijacks your baby and you are happy (ecstatic if I were to accurately characterize your post) that they just fixed it and that too silently without admitting any mistake! Biased comes to my mind.
What’s really annoying to me is that, if I understand their documentation correctly, I have to change my DTRACE_PROBEn macro calls for usdt probes to use their different generated macros for my provider. Do I get that right? Seems a quite gratuitous change that makes life a touch painful for cross-platform applications.
But wait a minute… If you’re root, why should there be _any_ "untraceable" processes?
poofter!
@Anantha The reality of the situation is that Apple has to apply the same restrictions in DTrace as pertain to the rest of the system (e.g. gdb). And yes, I’m happy that Apple recognized the problem and fixed it. Somewhat. Realize that sometimes the world exists in shades of gray.
@nothanks Yeah, that’s a drag. You should be using the new USDT methodology with dtrace -h.
@mind That’s a question for Apple, but it boils down preventing users from understanding the inner workings of applications that interact with copyright-protected music and movies (the euphemistically named DRM). I’m not happy about it, but then I can’t imagine music or movie companies getting onboard without it.
@anil Fair enough.
For me, if you can not attach, trace and debug any process you want, then the DTrace implementation is flawed. Some time into the future, iTunes will have a conflict with some software and DTrace will not be of any help because Apple deliberately crippled it. I’m just not interested in developing for Apple Computers anymore because P_LNOATTACH is feature compatible with the EVIL_BIT as defined in the RFC.
What would be the use of DTrace if now every software for Max OSX under the sun starts to mark their processes with P_LNOATTACH? It would be useless. Adobe will use it, Microsoft will use it, any privative software will use it, some Apple system software will use it. The megabytes DTrace takes of disk space will be wasted space and wasted expectations.
This is good news, will you all please stop your whining. If you don’t like it don’t use it, but there’s no need to make this personal.
@Anantha: I’m really tired of hearing this "if Microsoft did it" argument. In case you didn’t notice, people *were* livid when Apple did this – just look at all the anger from the Register and Slashdot articles mentioned above.
Apple receives plenty of negative feedback when they screw up, both from Mac users and everyone else.
@Eddie D I completely agree that artificially limiting the scope of DTrace is deleterious, and it’s frankly something that we would never do in Solaris (and ISVs have asked). That said, Apple’s business is different than ours and what they’ve implemented has been an attempt to maintain the same restrictions that were already in place (something that we also did with our privilege model for DTrace).
The bigger problem is that in limiting the scope, they actually damaged DTrace for uses that even Apple would deem legitimate (again, not that I’m anointing them the arbiters of legitimacy). I’m pleased to see that they’re taking steps to correct this. The lesser problem is restricting visibility in the first place, and that already applied to debuggers before DTrace was ported to Mac OS X. Fortunately Apple’s engineers apparently realize the reality that perfect enforcement would never be possible.
Isn’t it curious that this DRM limitation only exists for apps that deal with multimedia- not OS X in general? Apple doesn’t use copy protection system-wide, it only uses it when copyright owners would sue Apple if it didn’t use DRM. This anger would be better directed at Hollywood than Apple.
It’s even more fine grain that that. Application that uses the QuickTime Framework for example are untraceable only while they are reading a media that conatins DRM.
@Eddie D: I don’t see the point here. That’s not because MS, Adobe, and even Apple are using the ptrace hack that gdb is useless ? Why would it be different with dtrace (now that they fixed the major issue)
Heads up, Slashdot’s a-comin’. Be prepared for the headline: "Apple ‘Actually Damaged’ DTrace."
Not only is perfect enforcement of debugger-based inspection not possible, but as was revealed when the last round of publicity showed up about what Apple has done, their attempts to keep a debugger out can be removed with a couple of lines in .gdbinit:
future-break ptrace
command 1
return
continue
end
I don’t know what they promised ISV’s but they certainly didn’t offer what they were probably asking for.
@Paul,
That’s largely because what the ISVs really want is impossible for Apple to offer. The amusing part is that anyone with the knowledge to decode a DRM scheme using a debugger, is going to know how to recompile a working version of GDB and DTrace that don’t have Apple’s crippling in it.
At least they’ve made the tool more useful now.
@Jeff I think what they offer is pro forma protection that’s enough to satisfy executives that a mechanism is in place. If people start being noisy about methods to circumvent it, Apple’s engineers may have to put something more complex in place, but I’m sure they understand, as we do, that they cannot win that arms race: once they give you the bits, it’s game over (especially with DTrace in your toolbelt).
@Jean-Daniel I do agree that making processes unsuitable to gdb is evil. But, it is evil only if you want to trace that very process.
The Dtrace situation is much worse because it provides system-wide metrics. Apple just seems to have fixed Dtrace so it gives accurate system metrics while not allowing introspection in P_LNOATTACH processes. Think about it, is it a fix? to what extent?
At the end of the road, any well motived hacker CAN and WILL trace itunes, no matter what. Deliberately crippling DTrace is just wrong. Just imagine that the next step were to turn of ZFS autohealing features because of protecting their DRM scheme. Excuses are just shameful.
@Adam – the gdb situation is terrible for apps that dynamically load plugins. My main app does this, and when I was debugging with gdb (before finding the fix), loading a "copy protected" plugin would just break gdb instead, even though I wanted to actually debug my app. The .dylib for the plugin was marked P_LNOATTACH, and hence requires the workaround for ptrace(2).
@Anantha: Apple are hardly the most closed company in the world…
Yes they’re secretive with regards to product development, but they’re so incredibly open with their software its amazing.
This post is exactly the case in point.. Did Apple have to port DTrace? No. Did Apple have to commit their changes back to the upstream project? No. Did Apple have to fix the concerns raised earlier? No. Did they do all the above? Yes!
Just look at the amount of Open Source software they’re using and contributing to (Darwin (OSX Kernel and Userland minus Aqua), DTrace, ZFS, mDNSresponder, CUPS, Samba, etc), and you’ll find you’re so incredibly far away from the point its amazing.
@Paul I can imagine how annoying the interference with gdb would be, and I didn’t mean to defend the practice. I just mean to say that I can understand why they did it. Fortunately, it’s fairly easy to circumvent.
@Jim: In fact, since the kernel source code is freely available, P_LNOATTACH can easily be hacked out. I am sure there is a debate or war inside Apple (probably political) over this.