Oracle’s port: this is not DTrace
After writing about Oracle’s port of DTrace to OEL, I wanted to take it for a spin. Following the directions that Wim Coekaerts spelled out, I installed and configured a VM to run OEL with Oracle’s nascent DTrace port. Setting up the system was relatively painless.
Here’s my first DTrace invocation on OEL:
[root@screven ~]# uname -a
Linux screven 2.6.32-201.0.4.el6uek.x86_64 #1 SMP Tue Oct 4 16:47:00 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux
[root@screven ~]# dtrace -n 'BEGIN{ trace("howdy from linux"); }'
dtrace: description 'BEGIN' matched 1 probe
CPU ID FUNCTION:NAME
0 1 :BEGIN howdy from linux
^C
Then I wanted to see what was on the system:
[root@screven ~]# dtrace -l | wc -l 574
[root@screven ~]# modprobe profile [root@screven ~]# dtrace -l | wc -l 587
[root@screven ~]# dtrace -n profile:::profile-997 dtrace: failed to enable 'profile:::profile-997': Failed to enable probe
[root@screven ~]# dtrace -n syscall:::entry'{ @[execname] = count(); }'
dtrace: description 'syscall:::entry' matched 285 probes
^C
pickup 9
abrtd 11
qmgr 17
rsyslogd 25
rs:main Q:Reg 35
master 52
tty 60
dircolors 80
hostname 92
tput 92
id 198
unix_chkpwd 550
auditd 599
dtrace 760
bash 1515
sshd 8327
Evaluation
This is not DTrace
In: DTrace · Tagged with: DTrace, OEL, Oracle
DTrace for Linux
Yesterday (October 4, 2011) Oracle made the surprising announcement that they would be porting some key Solaris features, DTrace and Zones, to Oracle Enterprise Linux. As one of the original authors, the news about DTrace was particularly interesting to me, so I started digging.
I should note that this isn’t the first time I’ve written about DTrace for Linux. Back in 2005, I worked on Linux-branded Zones, Solaris containers that contained a Linux user environment. I wrote a coyly-titled blog post about examining Linux applications using DTrace. The subject was honest — we used precisely the same techniques to bring the benefits of DTrace to Linux applications — but the title wasn’t completely accurate. That wasn’t exactly “DTrace for Linux”, it was more precisely “The Linux user-land for Solaris where users can reap the benefits of DTrace”; I chose the snappier title.
I also wrote about DTrace knockoffs in 2007 to examine the Linux counter-effort. While the project is still in development, it hasn’t achieved the functionality or traction of DTrace. Suggesting that Linux was inferior brought out the usual NIH reactions which led me to write a subsequent blog post about a theoretical port of DTrace to Linux. While a year later Paul Fox started exactly such a port, my assumption at the time was that the primary copyright holder of DTrace wouldn’t be the one porting DTrace to Linux. Now that Oracle is claiming a port, the calculus may change a bit.
What is Oracle doing?
Even among Oracle employees, there’s uncertainty about what was announced. Ed Screven gave us just a couple of bullet points in his keynote; Sergio Leunissen, the product manager for OEL, didn’t have further details in his OpenWorld talk beyond it being a beta of limited functionality; and the entire Solaris team seemed completely taken by surprise.
What is in the port?
Leunissen stated that only the kernel components of DTrace are part of the port. It’s unclear whether that means just fbt or includes sdt and the related providers. It sounds certain, though, that it won’t pass the DTrace test suite which is the deciding criterion between a DTrace port and some sort of work in progress.
What is the license?
While I abhor GPL v. CDDL discussions, this is a pretty interesting case. According to the release manager for OEL, some small kernel components and header files will be dual-licensed while the bulk of DTrace — the kernel modules, libraries, and commands — will use the CDDL as they had under (the now defunct) OpenSolaris (and to the consernation of Linux die-hards I’m sure). Oracle already faces an interesting conundum with their CDDL-licensed files: they can’t take the fixes that others have made to, for example, ZFS without needing to release their own fixes. The DTrace port to Linux is interesting in that Oracle apparently thinks that the CDDL license will make DTrace too toxic for other Linux vendors to touch.
Conclusion
Regardless of how Oracle brings DTrace to Linux, it will be good for DTrace and good for its users — and perhaps best of all for the author of the DTrace book. I’m cautiously optimistic about what this means for the DTrace development community if Oracle does, in fact, release DTrace under the CDDL. While this won’t mean much for the broader Linux community, we in the illumos community will happily accept anything of value Oracle adds. The Solaris lover in me was worried when it appeared that OEL was raiding the Solaris pantry, but if this is Oracle’s model for porting, then I — and the entire illumos community I’m sure — hope that more and more of Solaris is open sourced under the aegis of OEL differentiation.
10/10/2011 follow-up post, Oracle’s port: this is not DTrace.
In: DTrace · Tagged with: DTrace, illumos, Linux, OEL, OOW, OpenSolaris, Oracle, Solaris
On RAID-6 recovery
RAID algorithms have become a particular fascination of mine, and I recently read a very interesting paper that describes an optimization for RAID reconstruction (by Xiang, Xu, Lui, Chang, Pan, and Li). Before writing double- and triple-parity RAID algorithms for ZFS, I spent a fair bit of time researching the subject and have stayed interested since. Before describing the reconstruction optimization, some preamble is required. RAID algorithms can be divided into two buckets: one-dimensional algorithms, and multi-dimensional algorithms (terms of my own choosing; I haven’t seen this distinction discussed in literature).
One-dimensional RAID
A one-dimensional algorithm is one in which all data in a single RAID stripe is used to compute all parities. The RAID algorithm used by ZFS falls into this category as do most algorithms derived from Reed-Solomon coding. For a given RAID stripe’s set of data blocks, 
, we can compute the nth parity block with some function 
. For example, ZFS, roughly, uses the formula
![\[ p(D,n) = \sum_{i=1}^{\left|{D}\right|} 2^{(i-1)(n-1)} \cdot D_{i} \]](http://dtrace.org/blogs/ahl/wp-content/ql-cache/quicklatex.com-43f0a95ef45d13ed08b8556aac63c922_l3.png "Rendered by QuickLaTeX.com")
Here, addition and multiplication are defined over a Galois Field – the explanation would be far longer than it would be interesting or relevant so I’ll omit it from this post. It is worth noting that this particular approach only works for three parity disks or fewer, but that too is an entirely different subject (albeit an interesting one). Reconstruction of a missing block in a one-dimensional algorithm requires reading the available data, and performing some computation; each stripe may be reconstructed separately (and thus, in parallel).
Multi-dimensional RAID
A multi-dimensional algorithm is one in which parts of multiple logical RAID stripes may contribute to parity calculation. Examples of this include IBM’s EVENODD and NetApp’s slight simplification, Row-Diagonal Parity (RDP). These are most easily conveyed through diagrams:
EVENODD
RDP
With both EVENODD and RDP, calculation of the first parity block simply XOR the data blocks in that RAID stripe. The second parity block is calculated by a simple XOR of data values across RAID stripes more or less diagonally. Both of these techniques place constraints on the width of a RAID stripe.
Optimizing RAID reconstruction for fewer reads
The paper, A Hybrid Approach of Failed Disk Recovery Using RAID-6 Codes: Algorithms and Performance Evaluation, describes a optimization for reconstruction under multi-dimensional RAID algorithms. The key insight is that with parity calculations that effectively overlap, a clever reconstruction algorithm can use fewer blocks, thus incurring fewer disk reads. As described in the paper, normally when a given disk fails, all remaining data blocks and blocks from the first parity disk are used for reconstruction:
simple recovery
It is, however, possible to read fewer total blocks by taking advantage of the fact that certain blocks can be multiply used. In the reconstruction below, blocks with circles are used for “row” reconstruction, and blocks with squares are used for “diagonal” reconstruction.
optimized recovery
Note that the simple approach requires reading 36 blocks (none from disk 7) whereas the reconstruction described in the paper requires reading only 27 blocks. This applies generally: the new approach requires 25% fewer blocks to perform the same reconstruction. And the paper includes a method of balancing the reads between disks.
Disappointing results
Unfortunately, optimizing for fewer reads didn’t translate to significant performance improvements in the overall recovery. For RDP it was about 12% better in the best case, but typically closer to 7%. For EVENODD the improvement was less than 5% in all cases. Why? The naïve recovery algorithm streams data off the healthy hard drives, performs a simple computation, and streams good data onto the replacement drive. Streaming is what drives do best – 3.5” or 2.5”, 7200, 10K, or 15K RPM; SATA, SAS, or FC they all stream pretty well. There may be some contention for I/O resources, but either that contention isn’t severe or the “skips” in the I/O patterns interrupt the normal streaming efficiency.
Applicability in flash
There’s another medium, however, that has throughput and IOPS to spare where this RAID reconstruction approach could be highly effective: flash. With SSDs, it’s possible to see throughput that strains the limits of I/O systems; reading less data could be a significant improvement, and the non-contiguous read patterns wouldn’t degrade performance as they do with HDDs. For all-flash arrays, this sort of optimization may be one of many in its class; with a surplus of IOPS, compute, and memory, the RAID algorithms designed for slow disks, slow CPUs, and a dearth of DRAM, may need to be scrapped and rethought.
In: Software · Tagged with: EVENODD, flash, RAID, RAID-6, RDP, ZFS
Flash news I wish I could read
NetApp has Hybrid Aggregate drives coming, with data moved automatically in real time between flash located next to the spinning disks. The company now says that this is a better technology than PCIe flash approaches.
NetApp, presenting at an Analyst Day event in New York on 30 June, said that having networked storage move as it were into the host server environment was disadvantageous. This was according to Stifel Nicolaus analyst Aaron Rakers.
Manish Goel, NetApp’s product ops EVP, said SSDs used as hard disk drive replacements were not as interesting as using flash at the disk layer in a Hybrid Aggregate drive approach – and this was coming.
Flash Cache in the controller is a straightforward array read I/O accelerator. PCIe flash in host servers is a complementary technology but will not decentralise the storage market and move networked storage back into the host servers.
NetApp has a patent in this Hybrid Aggregate disk drive area called “Mechanisms for moving data in a Hybrid Aggregate”.…
Surely this is what we all understand as auto-placement of data in a virtual storage pool comprising SSD and fast disk tiers, such as Compellent’s block-level Data Progression? Not so, according to a person close to the situation: “It’s much more automatic, real-time and granular. Compellent needs policies and is not real-time. [NetApp] will be automatic and always move data real-time, rather than retroactively.”
In: Flash · Tagged with: Compellent, EMC, flash, HSP, NetApp, Oracle, SSD, Sun, TheRegister, ZFS
Engineers and customers
Today I took the train out to Long Island to meet up with our New York sales team for a visit with a prospective customer. You never know with an initial meeting, but this one was great. I thought I’d share a bit about what made these guys so excited which is the same stuff that gets me excited about what we’re doing at Delphix.
First though, there are some engineers who have never spoken with a customer. There are some engineering organizations in which requirements are collected from customers, correlated by product managers, handed to engineering managers, and given to engineers. It’s a fine workflow, but this needs to be balanced against engineers engaging directly with customers, hearing their issues, and brainstorming solutions technologist to technologist. Engineers talking to a small number of customers may miss broad trends or fail to connect certain dots, but it’s a complementary activity and part of being a holistic engineer.
I’ve heard software engineers groan that the right technical decision was trumped by business concerns. Those people might be good engineers, but they aren’t great ones. Engineering can’t stop at the boundaries of software; it must necessarily consider the whole ecosystem of the product and the company. Yes, we might not have architected the feature this way if we didn’t have legacy customers support, but we do (and we should be happy for it). (And, of course, this logic can be taken to the other extreme with equally bad results.) This doesn’t mean that a great engineer collects all data first hand, but the whole system must be considered, and walking into a customer’s office from time to time is a reality check.
In today’s meeting, the customer was learning about Delphix for the first time. And they got it right away. As with many enterprises, they have a initiative around virtualization to enable more self-service and more empowerment of their developers. The data in their relational databases is a big anchor weighing down those efforts; the time and effort required to copy and provision databases is a huge drag. Smart guys, they oscillated between how Delphix works — a super-smart, database-optimized storage gateway — and what Delphix does — virtualizing their Oracle databases, bringing the agility and cost-savings of other virtualization technologies. And the slide-ware made real through a demo of the product GUI elicited an terse expression of comprehension: “That’s cool.”
And maybe the best reason for engineers to get into the field is to witness customers who get how cool the product is.
In: Delphix, Software · Tagged with: Delphix
Another reason Apple misses ZFS
Apple recently announced a new iMac model — in itself, only as notable as the seasons — but with an interesting option: users can choose to have both an HDD and an SSD. Their use of these two is absolutely pedestrian, as noted on the Apple store:
If you configure your iMac with both the solid-state drive and a Serial ATA hard drive, it will come preformatted with Mac OS X and all your applications on the solid-state drive. Then you can use the hard drive for videos, photos, and other files.
Fantastic. This is hierarchical storage management (HSM) as it was conceived by Alan Turing himself as he toiled against the German war machine (if I remember my history correctly). The onus of choosing the right medium for given data is completely on the user. I guess the user who forks over $500 for this fancy storage probably is savvy enough to copy files to and fro, but aren’t computers pretty good at automating stuff like that?
Back at Sun, we built the ZFS Hybrid Storage Pool (HSP), a system that combines disk, DRAM, and, yes, flash. A significant part of this is the L2ARC implemented by Brendan Gregg that uses SSDs as a cache for often-used data. Hey Apple, does this sound useful?
Curiously, the new iMacs contain the Intel Z68 chipset which provides support for SSD caching. Similar to what ZFS does in software, the Intel chipset stores a subset of the data from the HDD on the SSD. By the time the hardware sees the data, it’s stripped of all semantic meaning — it’s just offsets and sizes. In ZFS, however, the L2ARC knows more about the data so can do a better job about retaining data that’s relevant. But iMac users suffer a more fundamental problem: the SSD caching feature of the Z68 doesn’t appear to be used.
It’s a shame that Apple abandoned the port of ZFS they had completed ostensibly due to “licensing issues” (DTrace in Mac OS X uses the same license — perhaps a subject for another blog post). Fortunately, Ten’s Complement has picked up the reins. Apple systems with HDDs and SSDs could be the ideal use case ZFS in the consumer environment.
In: ZFS · Tagged with: Apple, HSP, L2ARC, ZFS
Delphix Next
It’s rare to get software right the first time. I’m not referring to bugs in implementation requiring narrow fixes, but rather places in a design that simply missed the mark. Even if getting it absolutely right the first time were possible, it would be prohibitively expensive (and time-consuming) so we make the best decisions we can, hammer it out, and then wait. Users of a product quickly become experts on its strengths and weaknesses for them. Customers aren’t beta testers — again, I’m not talking about bugs — but rather they expose use cases you never anticipated, and present environments too convoluted to ever conceive at a whiteboard.
When I worked in the Fishworks group at Sun, we learned more about our market in the first three months after shipping 1.0 than we had in the 30 months we spent developing it. We found the product both struggling in unanticipated conditions, and being used to solve problems we could have never predicted. Some of these we might have guessed earlier given more time, but some will never come to light until you ship. That you need to ship 1.0 before you can write 2.0 is a deeper notion than it appears.
I joined Delphix a couple of weeks before our formal launch at the DEMO conference. Since then, we’ve engaged in more proofs-of-contept (PoCs) and more customers have rolled us into use, and we’ve continued to learn of new use cases for Delphix Server, and found the places where we needed to rethink our assumptions. And we knew this would be the case — you can’t get it right the first time. Over the past several months, we in Delphix engineering have been writing the second version of the most critical components in our stack, incorporating the lessons learned with our customers. The team has enjoyed the opportunity to revisit design decisions with new information; it’s fun to feel like we’re not just getting it done, but getting it right.
When building 1.0, you make a mental list of the stuff you’d fix if only you had the time. In the next release, you figure out all the whizzy stuff you now can build on the stable foundation. We’re excited for the forthcoming 2.6 release — more so even for new ideas we found along the way that will be the basis for our future work. We’ve got a great team working on a great product. Check in on the Delphix blogs in the coming months for details on the 2.6 release and the other stuff we’ve got in the works.
In: DTrace · Tagged with: Delphix, software
Delphix welcomes Matt Ahrens and George Wilson
This week it was my pleasure to welcome my former Sun colleague Matt Ahrens and George Wilson to Delphix. Matt and I studied computer science together at Brown and then joined Sun in 2001. Matt joined Jeff Bonwick to start ZFS while I worked on DTrace. George joined Sun in 1996, and worked in a variety of roles, joining the ZFS team in 2006 (just as I was leaving to help start Fishworks).
George and Matt bring an amazing knowledge of ZFS — the lower, and upper halves respectively — and are also just great engineers who are already contributing tangibly to the the success of Delphix. You can take a look at Matt’s old blog, or watch George in a bunch of videos (including one of him being interviewed by a muppet).
In: Delphix · Tagged with: Delphix, GeorgeWilson, MattAhrens, ZFS
Delphix CEO joins the blogosphere
Welcome to Jed Yueh, our CEO at Delphix, who recently posted his first blog entry.
…
With three clicks from our intuitive, consumer-grade interface, you can cut through a decade of frustration and redundancy in enterprise datacenters. And customers are quickly seeing the value of the gun vs. the sword: in our first two quarters of sales alone, we added Fortune 1000 customers across several industries, including financial services, telecommunications, high technology, retail, manufacturing, travel, consumer goods, and SaaS.
…
Stay tuned to the Delphix blog for more on what we’re up to.
In: Delphix · Tagged with: Delphix, JedYueh
ZIL analysis from Chris George
Chris George from DDRdrive put together a great presentation at the OpenStorage summit looking at the ZFS intent log (ZIL), and how their product is particularly well-suited as a ZIL device. Chris did a particularly interesting analysis of the I/O pattern ZFS generates to ZIL devices (using DTrace of course). With writes to a single ZFS dataset, writes are almost 100% sequential, but with activity to multiple datasets, writes become significantly more non-sequential. The ZIL was initially designed to accelerate performance with a dedicated hard drive, but the Hybrid Storage Pool found a significantly better ZIL device in write-optimized, flash SSDs.
In the 7000 series, the performance of these SSDs — called Logzillas — aren’t particularly sensitive to random write patters. Less sophisticated, cheaper SSDs are more significantly impacted by randomness in that both performance and longevity can suffer.
Chris concludes that NV-DRAM is better suited than flash for the ZIL (Oracle’s Logzilla built by STEC actually contains a large amount of NV-DRAM). I completely agree; further, if HDDs and commodity SSDs continue to be target ZIL devices, ZFS could and should do more to ensure that writes are sequential.
In: ZFS · Tagged with: ChrisGeorge, DDRdrive, HSP, OpenStorageSummit, ZFS, ZIL
