The interest in DTrace on Linux is heating up again — this time in an inferno on the Linux 2008 Kernel Summit discussion list. Under discussion is SystemTap, the Linux-born DTrace-knockoff, with people like Ted Ts’o explaining why they find SystemTap generally unusable (“Do you really expect system administrators to use this tool?”) and in stark contrast to DTrace (“it just works”).
While the comparison is clearly flattering, I find it a bit disappointing that no one in the discussion seems to realize that DTrace “just works” not merely my implementation, but also by design. Over and over again, we made architectural and technical design decisions that would yield an instrumentation framework that would be not just safe, powerful and flexible, but also usable. The subtle bit here is that many of those decisions were not at the surface of the system (where the discussion on the Linux list seems to be currently mired), but in its guts. To phrase it more concretely, innovations like CTF, DOF and provider-specified stability may seem like mind-numbing, arcane implementation detail (and okay, they probably are that too), but they are the foundation upon which the usability of DTrace is built. If you don’t solve the problems that they solve, you won’t have a system anywhere near as usable as DTrace.
So does SystemTap appreciate either the importance of these problems or the scope of their solutions? Almost certainly not — for if they did, they would come to the same conclusion that technologists at Apple, QNX, and the FreeBSD project have come to: the only way to have a system at parity with DTrace is to port DTrace.
Fortunately for Linux users, there are some in the community who have made this realization. In particular, Paul Fox has a nascent port of DTrace to Linux. Paul still has a long way to go (and I’m sure he could use whatever help Linux folks are willing to offer) but it’s impossible to believe that Paul isn’t on a shorter and more realistic path than SystemTap to achieving safe, powerful, flexible — and usable! — dynamic Linux instrumentation. Good luck to you Paul; we continue to be available to help where we can — and may the Linux community realize the value of your work sooner rather than later!
25 Responses
All really goes back to the kernel and how well things are done there… it’ll be interesting to see how well things work on all the flavors of linux given it’s diversity… then again System Tap isn’t everywhere yet either.
How does that work even, he’d be introducing incompatible code in to the ( GPL ) Linux kernel. seems like it’s a plan that’s liable to get you sued by GPL people, no?
Licensing is my concern only along as the CDDL is being abided by — and given its easy terms, I don’t see the CDDL being violated as a part of Paul’s work. As for the GPL, one would have to ask the FSF. But do note that the fact that both components are open source makes this different from the traditional licensing question in that Paul has the option to simply make available source code for his DTrace port, thereby skirting the distribution clauses of the GPL (which pertain explicitly to "executable or object code"). At any rate, I hope (in vain, no doubt) that the Linux community can focus on the technology and not on the inside baseball licensing naunces…
The beauty of linux is that we have one body of code that we can change however we have to to get things done.
If we have to rewrite half the code to implement zfs and d-trace, well, given enough interest, we can do it.
The /real/ bottleneck is X at this point, imo.
Reading through the comments, especially Theodore’s, it became obvious to me (I’ve taken the official Sun DTrace training course), that DTrace suffers from the problems that systemtap suffers also:
DTrace is unusable for system administrators, one of the largest consumers of DTrace information.
For example, when I attended the DTrace training course, there were so many things that were just plain "???", because, although I am very familiar with AWK, and although I am pretty familiar with C programming, I WASN’T familiar with the internal structure of the Solaris kernel.
And therein lies the predicament of just about every system administrator:
"um, wait a minute, where did that stuff come from now??? What is this structure all of a sudden, where does it come from, and how could I possibly know it exists if I’m not familiar with the kernel???"
These are the questions I found myself asking over and over and over during the DTrace course.
My point is: DTrace needs to be made easier for sysadmins to use, in order to help sysadmins obtain and consume the information they need. Brendan Gregg’s DTrace toolkit is a very good start, but DTrace itself needs to become more approachable.
Consider the alternatives: every sysadmin wanting to effectively use DTrace would basically need to eat, drink and sleep the "Solaris internals" book, from cover to cover. Perhaps you will agree, that that might not be feasible for most sysadmins out there (myself being the exception).
"the only way to have a system at parity with DTrace is to port DTrace."
Seriously? You think there was precisely one way to solve these problems, you found it, and nobody else in the world is as clever as you and nobody will ever find any ideas or solutions for these problems you didn’t anticipate?
I really ought to have done that in LOLspeak 🙂
ORLY? You think…. SRSLY?
WOW, I just went and actually read tytso’s email and realized you’ve pretty much completely mischaracterized his entire email! Hyperbole is one thing but outright deception is another antirely.
Ted’s email started with: "Actually, we don’t. The big thing that are missing are the tapsets"
There’s a world of difference between saying SystemTap needs to be replaced with a port of another technology because it’s unusable and just saying a library of macros is needed to package up the probes for administrators.
It’s not surprising that Sun is able to organize a large number of developers to build macros for every module in the kernel better better than an open source group. Open source excels when software is modularizable and modules can be delegated. Instrumenting a system with probes for Dtrace or Systemtap is inherently a cross-module project which requires a lot of developers to be on the same page at the same time.
Gerg: I don’t believe I have mischaracterized Ted’s post. Certainly, the quote is a direct quote, and he has since elaborated on his view in that thread: "In the past two years, I’ve on average tried Systemtap every 9 months or so, and each time, I’d hit a different annoying roadblock, and then I was so busy I would move on to a more productive way of solving my problems." I said that he (and others) were describing SystemTap as "generally unusable"; if hitting "different annoying roadblocks" whenever you try something doesn’t constitute a technology being "generally unusable", what does?
Gerg: No, there is not only one way to solve the problems — but I will stand by the intractability of the problems themselves: there is no way to achieve parity with DTrace without solving many of the same nasty problems that we have solved. Solving these problems takes a ton of focus, time and (yes) expertise — and experience has shown us that those who can summon these elements also come to the decision that it makes more sense to port DTrace than to reimplement it. Indeed, just porting DTrace is an extremely technical endeavor — and to know that one need look no further than the extraordinarily talented folks who ported DTrace to other platforms.
Bryan,
The "executable or object code" aspects of the GPLv2 only pertain to section 3 – section 2 must still be followed even if the distribution is in source form. 2(b) requires that all code derived from a GPLed work be released under the terms of the GPL. It’s clear that right now the dtrace code can’t be provided under the terms of the GPL by anyone other than Sun, so the question hinges on whether a Linux implementation of dtrace would be a derived work of the Linux kernel or not. You’d need a court case to make that determination, but my understanding is that most Linux vendors are working under the assumption that it would be.
That’s hugely unfortunate. It would obviously be beneficial to everyone for dtrace to be usable under Linux (Linux gains functionality, Sun gains extra testing and bugfixes), so it’s pretty clearly the case that the licensing is the only real problem. And while that licensing problem exists, focusing on the technical feasability of porting dtrace isn’t a useful way to spend time. What benefit is there in focusing resources on code that can never be shipped?
Matthew,
Yes, it’s true that all GPL-derived work must be released under the GPL — but the GPL-derived work in Paul’s work (namely, any Linux mods to support DTrace) remains GPL’d. That is, the source code of DTrace is in no way a derived work of Linux. (In my opinion, neither is the DTrace object code even when it’s running in a Linux kernel, but that is at least an arguable gray area; that the DTrace source code is not a derived work of Linux is inarguable.)
Now, before you respond to that, I would ask that you reconsider before engaging in a prolonged licensing discussion here. From previous posts both here and on Adam’s blog, I know that these never-ending discussions are a favorite hobby of yours, but can we please either talk about technology or nothing at all? To phrase it perhaps more bluntly: if you fancy yourself a technologist, then let’s talk about the technology. But if you want to be a lawyer when you grow up, then please, do us a favor and go to law school and leave us technologists to do what we do.
I’ve no interest in having a protracted legal discussion at all – I’m just interested in it being recognised that the reason the Linux community haven’t embraced dtrace is because of genuine legal concerns, not because we’re uninterested in the technology. If my employers told me that we could ship it, I’d be delighted. Until then, my life is just going to suck slightly more than it would do otherwise.
Still, maybe one day SystemTap will catch up with dtrace. And maybe one day, Solaris’s power management will catch up with Linux’s. When legal issues force us to reinvent wheels then pretty much everyone loses, but I can’t see any way around that.
Matthew,
Yes, maybe SystemTap will catch up with DTrace. Or maybe Paul will complete his port — and the Linux community will get to decide what they’re really about…
Without legal certainty, Paul’s port is a neat technology demonstration. We love technology – we just don’t love it enough to ship it to customers when we risk getting sued. In similar situations, Sun have made exactly the same choice. See the reinventing of the Openoffice optimisation solver code, for instance. Everyone just makes do with the best they can get in a clearly legally unencumbered way, even if that means more misery and pain.
Hi Bryan. I am a fan of DTrace, and I’ve enjoyed your blog in the past.
I just read this entry, and I’m leaving this comment to say that the way you spoke to Matthew Garrett on July 02, 2008 at 09:10 AM PDT was rude and disrespectful. The substance of your post — that you didn’t wish to discuss licensing issues — was of course perfectly justifiable, but the way you addressed Matthew Garrett was impolite.
Thanks for listening, and I hope to continue enjoying your blog in the future.
Regards,
Zooko O’Whielacronx
Zooko,
I appreciate the comment, and I confess that I have become a bit frustrated that whenever Adam or I blog about DTrace and Linux in the same post, the conversation quickly becomes overrun by Garrett’s comments about licensing; this is not a blog about software licensing or law, and I find it incredibly tedious when software engineers pretend that they’re lawyers. (If you know anyone who practices IP law, you know that they feel the same way.) But I appreciate what you’re saying, and it is not my intent to be rude or disrespectful. So Matthew: my apologies for any rudeness — and thanks in advance for respecting the fact that I am not interesting in getting tied up here in licensing discussions.
UX-admin, I think those are good points about DTrace and uptake by system administrators; when I created the DTraceToolkit I was also asked by Sun to teach some of their DTrace courses – so I saw these issues first hand. My students made some blunt (but realistic) comments about not having the time to read Solaris Internals + the DTrace Guide, and would rather download a handful of useful scripts. It was encouragement for me to spend more of my spare time writing DTraceToolkit scripts, and since I had already been doing performance work and writing performance analysis freeware I knew what problems needed to be solved in that space.
I don’t think it’s a fault of the DTrace implementation – it makes something very complex relatively easy to use, and includes features aimed at general usability (the concept of stable providers and stability semantics, awk-like scripting, and many examples in the DTrace Guide). But we can do more, and there are projects in the pipeline for this (including a new GUI).
It might help to also mention that the Sun DTrace course is now a bit long in the tooth – it was good for early 2005, but these days there are many newer providers and there is much less (and often no) need to dig through kernel structures – especially those that seem to come from nowhere.
I utterly respect people’s desire to avoid having to worry about licensing discussions. The frustration I have is when people appear to insist that the technological aspects of a problem should be considered without regard to the legal aspects. It’s a wonderfully utopian viewpoint and I can respect that, but I think it’s wrong to imply that people who don’t ascribe to that viewpoint aren’t interested in the technology.
Back when the last dtrace/Linux discussion came up, I was a member of the Ubuntu technical board and so was actually part of the decision making process that determined whether we could include that code in the Ubuntu Linux distribution. Decisions were made with legal advice, and I shared my understanding of that advice. The perception I had at the time was that we were unfairly criticised for not shipping code that we would have loved to ship but felt we legally couldn’t. If I gave the impression that I was attempting to engage in excessive legal wanking, then I sincerely apologise. In truth, I’m fascinated by the technology involved in dtrace and zfs and think both should be counted amongst the most important code written this decade, but when lawyers are telling me we can’t ship this code and people are criticising that then I’m going to feel forced to defend that position. We’d love to ship your code. But realistically, it’s going to take either a change in copyright law (which would allow us to change Linux’s license) or Sun’s licensing policy.
Matthew,
You misunderstand me: I don’t have a "desire to avoid having to worry about licensing discussions", rather I simply don’t want to have those discussions here. That said, I’m quite curious by what you meant when you said "back when the last DTrace/Linux discussion came up" that you were involved in the "decision making process" as to whether or not to include "that code". What code are you talking about? The DTrace port to Linux has only existed for a very short period of time; how could DTrace have been discussed in the context of Linux without a port in hand?
Sorry, I wasn’t clear. I meant to refer to this:
"At any rate, I hope (in vain, no doubt) that the Linux community can focus on the technology and not on the inside baseball licensing naunces…"
The implication (and again, sorry if I’ve misinterpreted you) seemed to be that the licensing aspect was basically unimportant and the technoloy was the relevant part of the discussion. The point I was trying to make was purely that as far as the majority of the Linux development community is concerned, the licensing nuances mean that the technological aspects are philosophically interesting but pragmatically unhelpful. As a result, talking about dtrace in Linux is kind of like talking about human society in a psot-fusion era – full of handwavy assertions about how things will be much better, but missing the point that reality doesn’t currently allow it.
My recollection about the precise discussion I had last year is slightly hazy, since it happened during the point where I was kind of busy finishing up my PhD. It’s possible that the discussion was actually about zfs rather than dtrace, but the distinction isn’t terribly important when it comes to the decision making process. But yes, my recollection is that it was raised as a hypothetical issue and dealt with based on our understanding of how the code could be implemented in Linux in relation to the legal situation regarding derived works of the kernel. There’s only a finite number of legal hacks you can get away with (and Ubuntu exploits most of them – you almost certainly don’t want to look at how linux-restricted-modules is implemented), and the opinion at the time was that we couldn’t see any way CDDLed code could be used as a functional part of Linux. If Sun’s lawyers feel otherwise, a published opinion would be a wonderful thing. Like I said, I think everyone would benefit as a result.
I’ve no idea if you’re at OSCon this year, but I’m supposed to get in on the evening of the 22nd and I’ll be in SF on the 29th and 30th. I’d be happy to discuss this in a situation where ascii’s emotional failings don’t make it sound like we’re at each others throats. As a bonus, the current exchange rate makes it practical for me to actually buy competent quantities of beer.
Matthew,
I won’t be at OSCON this year, but I’ll be around SF on the 29th and 30th. And while rather busy trying to get a product out the door, it would nonetheless be interesting to have a conversation about this — drop me a line and we’ll get something set up…
Hello everyone,
an interesting discussion brews about what Linux dtrace is or might be and how it will break the GPL etc etc.
The Linux port is pure source code – no object code (and has to be because Linux binary compatibility amongst kernel releases is near impossible).
The dtrace implementation is a set of changes/mods/hacks (however you want to call them) against the SUn source, released as part of CDDL (because I cannot change the Sun licensing terms). I see no issue with dtrace-for-linux.
If a distro decides to package dtrace with linux – thats a different and debatable issue – one for which I cannot answer, but if they were to provide a source package that compiles when you install it – that sounds acceptable to me.
For Linux/dtrace to be successful requires solving some technical challenges and my goal/mindset is to avoid GPL by not touching the kernel source. (I am referring to the kernel source so I can figure out how to branch into the kernel and/or patch it).
So – you – the general audience – need to decide for yourself whether the approach is sound.
If it isnt, feel free to talk to me.
I am accepting of Dtrace as a technological sound product which I *want to use*.
Grab the source source.
This is a part time project. After 3 months, I can trace system calls and maybe other stuff. I have a number of things to enhance/fix before I let this loose on a production system.
In operational server environments it doesn’t hurt OpenSolaris that Linux, due to the GPL, can’t get out of its own way to bring DTrace and ZFS in. Frankly, there are a lot of advantages in the Solaris kernel that Linux doesn’t have and which we’d never have been exposed to if we hadn’t be forced to look at OpenSolaris to get things like DTrace. That being said I’d fully love to see DTrace standardized (ISO/ANSI?). Cross-platform adoption can only help folks understand all platforms better. There’s plenty of times I’d wished DTrace was on Windows even.
Jason J. W. Williams: like what things are advantages in the Solaris kernel which you wouldn’t have been exposed to? I take it you come from the Linux world then?