After writing about Oracle’s port of DTrace to OEL, I wanted to take it for a spin. Following the directions that Wim Coekaerts spelled out, I installed and configured a VM to run OEL with Oracle’s nascent DTrace port. Setting up the system was relatively painless.
Here’s my first DTrace invocation on OEL:
[root@screven ~]# uname -a
Linux screven 2.6.32-201.0.4.el6uek.x86_64 #1 SMP Tue Oct 4 16:47:00 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
[root@screven ~]# dtrace -n 'BEGIN{ trace("howdy from linux"); }'
dtrace: description 'BEGIN' matched 1 probe
CPU ID FUNCTION:NAME
0 1 :BEGIN howdy from linux
^C
Then I wanted to see what was on the system:
[root@screven ~]# dtrace -l | wc -l 574
Are you kidding me? For comparison, my Mac has 154,918 probe available and our illumos-derived Delphix OS has 77,320 (Mac OS X has many probes pre-created for each process). It looks like this beta only has the syscall provider, but digging around I can see that Wim didn’t mention that the profile provider is also there:
[root@screven ~]# modprobe profile [root@screven ~]# dtrace -l | wc -l 587
Sweet.
[root@screven ~]# dtrace -n profile:::profile-997 dtrace: failed to enable 'profile:::profile-997': Failed to enable probe
Not that sweet.
At least I can run my favorite DTrace script:
[root@screven ~]# dtrace -n syscall:::entry'{ @[execname] = count(); }'
dtrace: description 'syscall:::entry' matched 285 probes
^C
pickup 9
abrtd 11
qmgr 17
rsyslogd 25
rs:main Q:Reg 35
master 52
tty 60
dircolors 80
hostname 92
tput 92
id 198
unix_chkpwd 550
auditd 599
dtrace 760
bash 1515
sshd 8327
I wanted to trace activity when I connected to the system using ssh… but ssh logins fail with all probes enabled. To repeat: ssh logins fail with DTrace probes enabled. I’d try to debug it, but I’m too dejected.
Evaluation
While I’d like to give this obviously nascent port the benefit of the doubt, its current state is frankly embarrassing. It’s very clear now why Oracle wasn’t demonstrating this at OpenWorld last week: it doesn’t stand up to the mildest level of scrutiny. It’s fine that Oracle has embarked on a port of DTrace to the so-called unbreakable kernel, but this is months away from being usable. Announcing a product of this low quality and value calls into question Oracle’s credibility as a technology provider. Further, this was entirely avoidable; there were other DTrace ports to Linux that Oracle could have used as a starting point to produce something much closer to functional.
This is not DTrace
So, OEL users, know that this is not DTrace. This is no better than one of the DTrace knockoffs and in many ways much worse. What Oracle released is worse than worthless by violating perhaps the most fundamental tenet of DTrace: don’t damage the system. And, to the OEL folks, I’m sure you’ll get there, but how about you take down your beta until it’s ready? As it is, people might get the wrong impression about what DTrace is.
13 Responses
Stay classy, Adam.
Steve, help me out. I don’t think it’s reasonable to announce DTrace for OEL and then roll out this — even as a beta. Do you object to my calling this effort lazy? to asking for its removal until it’s in a useable state? to pointing out serious and obvious problems?
Just ignore him. Steve is obviously a SystemTap advocate in disguise. 😉
Maybe we should send some FreeBSD developers down to Oracle HQ to show ’em how it’s done.
Sure, or Apple, or QNX, or maybe just over Paul Fox with the code he’s written for Linux which is already more functional.
Thanks for trying it out. I was expecting a lot more to be functional, and certainly not expecting it to be harmful. Were there any other providers besides syscall and profile? I was hoping for fbt and maybe proc and io, at least.
I’m very happy for Oracle to be attempting to port DTrace to Linux. But I’m confused that Wim wrote “The word evaluate is key here” (http://blogs.oracle.com/wim/entry/trying_out_dtrace), with the current state as it is. I hope that’s misspoken: the words “don’t evaluate yet” should be key! I think _that_ would be in the best interests of OEL and DTrace, since evaluating this early release of code may scare people away from what will be an amazing technology, once fully ported, functional (dtrace test suite) and safe. Evaluate then, not now.
No other providers: just s subtly broken syscall provider and an obviously broken profile provider.
But the kernel is unbreakable right? Can’t complain when you have an unbreakable kernel now, can you 😉
Oracle’s port looks more alpha than beta, and Oracle should make that clear when they talk about it.
I think its good that Oracle have announced that they are working on this, but then its inevitable that everyone wants to see what progress they are making. Well now we know, and a little disappointed that they are not further along.
But I’m not too surprised, since I’m sure it’s very difficult challenge that has been set. From reading Paul Fox’s blog I get the impression that DTrace on Linux, is quite a bit more difficult to get to work reliably, than DTrace for Solaris. Not helped by limitations forced by the CDDL/GPL licencing issues.
I wonder how long Oracle have been working on this? I wonder who at Oracle IS working on this? Lets see what rate of progress they can make in the coming months.
I would say to Oracle, publish your code to git, so the community can contribute. Publish you project plan, and be ‘open’. Don’t just restrict this to Oracle support subscribers. Unfortunately I’ve a feeling that Oracle will not do that, but I hope I’m wrong.
I would encourage anyone interested in DTrace for Linux to have a look at Paul Fox’s work and provide feedback and assistance to Paul.
You can’t possibly be as surprised as you’re pretending to be here. If you’d stuck around long enough to experience more of the pain and suffering that is working for Oracle, you’d have seen this pattern in action several times. While some poor intern undoubtedly deserves the blame for this shoddy “port”, it’s Oracle’s vaunted executive team that deserves the blame for anyone seeing it. Their combination of “fire, ready, aim” product announcements, exclusion of engineering from the decision-making process, and frequent purges of anyone expressing independent thought makes it impossible for them to tell good ideas from bad, beta-ready products from breadboards, or fundamentally sound engineering from hazards to their customers’ businesses. Now that something’s been announced, Oracle will proceed to apply its “million interchangeable parts” approach to software engineering; somewhere a big team of mediocre engineers — probably a sad sack mix of H-1 holders and underwater mortgagors who owe child support — who first read about their new product on Slashdot are being whipped and bludgeoned into creating a product their bosses can sell as the one they announced.
Does that “[call] into question Oracle’s credibility as a technology provider”? It should, but Wall Street analysts will keep toeing the line and CIOs will keep playing golf with Larry, and nothing much will change. They certainly don’t care what you write or what I write or even what the truth is; they have the money to make the “truth” whatever they want it to be. Oracle’s nasty little secret is that it’s one of the industry’s worst-managed companies, up there with Yahoo and HP. They keep getting a pass because they keep reporting nicely-managed earnings, a strategy that worked well for GE, until it didn’t. I would love for this not-DTrace DTrace to be its undoing, but it’s going to take more than that to bring these guys down. Perhaps the best we can hope for is to get the message out to everyone reading that this lousy DTrace port is actually very representative of the way Oracle creates and ships products. There’s a reason almost all of their product lines came from acquisitions: they have absolutely no idea how to do anything themselves, and they boil off any acquired talent that does.
I challenge Oracle to produce the engineering architects who reviewed this piece of software, who approved its announcement. I challenge them to explain why they thought it was ready and how that decision was made. I issue this challenge because I know that no such review took place and no such architects were asked their opinion (if indeed they even exist). That’s just not the way Oracle does business, because Oracle simply is not an engineering company.
Just to add to the conversation (a little). If you havent tried my dtrace, please feel free to do so. Its very functional – there are a few things missing (kernel SDT probes, and work needs to be done on user space probes/process grabbing). But theres a lot in there and works on a wide variety of old and new kernels.
I will try to avoid being drawn into the rights/wrong/potential of Oracles work – thats for others to partake in. Whatever Oracle does, I feel is beneficial. Even if that work gets aborted, its a win for various reasons.
http://crtags.blogspot.com for the blog and link to the dtrace code.
So far, from this conversation, Oracle are dipping toes in the water. I can hazard a guess why sshd would die on tracing, and hopefully Oracle will resolve that issue (and I hope they enjoy the land of GCC vs the Sun C compiler issues, which I have written about many times).
Imagine if they made automobiles or aeroplanes with so many missing or non-functional parts. I guess you could drag them around and take the customers for a ride just the same.
It’d be one thing to try and hold an industry powerhouse to a standard it has no interest in meeting. It’s quite another to put out an announcement that implies way, way, WAY more than it delivers. Shameless promotion of things that don’t work, I think, is something any software engineer has an obligation to be ashamed for producing and an obligation, as a peer, to call it out for the farce that it is.