Category: Software

I just ran across this interview with Linus. If you’ve read my blog, you know I’ve talked before about OS innovation, particularly with regards to Linux and Solaris. So I found this particular Linus quote very interesting:

There’s innovation in Linux. There are some really good technical features that I’m proud of. There are capabilities in Linux that aren’t in other operating systems. A lot of them are about performance. They’re internal ways of doing things in a very efficient manner. In a kernel, you’re trying to hide the hard work from the application, rather than exposing the complexity.

This seems to fly in the face of his previous statements, where he claimed that there’s no innovation left at the OS level. Some people have commented on my previous blog entry that Linux cannot innovate simply because of its size: with so many hardware platforms to support, it becomes impossible to integrate large projects. What you’re left with is “technical features” to improve performance, which (in general) are nothing more than algorithm refinement and clever tricks¹.

I also don’t buy the “supporting lots of platforms means no chance for innovation” stance. Most of the Solaris 10 innovations (Zones, Greenline, Least Privileges) are completely hardware independent. Those projects which are hardware dependent develop a platform-neutral infrastructure, and provide modules only for those platforms which they suppot (FMA being a good example of this). Testing is hard. It’s clear that Linux testing often amounts to little more than “it compiles”. Large projects will always break things; even with our stringent testing requirements in Solaris, there are always plenty of “follow on” bugfixes to any project. If Linux (or Linus) is afraid of radical change, then there will be no chance for innnovation².

As we look towards OpenSolaris, I’m intrigued by this comment by Linus:

I am a dictator, but it’s the right kind of dictatorship. I can’t really do anything that screws people over. The benevolence is built in. I can’t be nasty. If my baser instincts took hold, they wouldn’t trust me, and they wouldn’t work with me anymore. I’m not so much a leader, I’m more of a shepherd. Now all the kernel developers will read that and say, “He’s comparing us to sheep.” It’s more like herding cats.

Stephen has previously questioned the importance of a single leader for an open source project. In the course of development for OpenSolaris, we’ve looked at many open source models, ranging from the “benevolent dictatorship” to the community model (and perhaps briefly, the pornocracy). I, for one, question Linus’ scalability. Too many projects like crash dumps, kernel debugging, and dynamic tracing have been rejected by Linus largely because Linux debugging is “as good as it needs to be.” Crash dumps and kernel tracing are “vendor features,” because real kernel developers should understand the code well enough to debug a problem from a stack trace and register dump. I love the Solaris debugging tools, but I know for a fact that there are huge gaps in our capabilities, some of which I hope to fill in upcoming releases. I would never think of Solaris debugging as perfect, or beyond reproach.

So it’s time for Space Ghosts’s “something…. to think about….”:

Is it beneficial to have one man, or one corporation, have the final say in the development of an open source project?

¹ZFS is an example of truly innovative performance improvements. Once you chuck the volume manager, all sorts of avenues open up that nobody’s ever though about before.

²To be clear, I’m not an ardently anti-Linux. But I do believe that technology should speak for itself, irregardless of philosophical or “religious” issues. Obviously, I’m part of the Solaris kernel group, and I believe that Solaris 10 has significant innovations that no other OS has. But I also have faith in open source and the open development model. If you know of Linux features that are truly innovative (and part of the core operating system), please leave a comment and I’ll gladly acknowledge them.

The other day on vacation, I ran across a Slashdot article on UNIX branding and GNU/Linux. Tne original article was mildy interesting, to the point where I actually bothered to read the comments. Now, long ago I learned that 99% of Slashdot comments are worthless. Very rarely to you find thoughtful and objective comments; even browsing at +5 can be hazardous to your health. Even so, I managed to find this comment, which contained in part some perspective relating to my previous post on Linux innovation:

Clearly, this person is not the sharpest tool in the shed when in comes to Operating Systems. But it begs the question: How widespread is this point of view? We love innovation, and it shows in Solaris 10. We have yet to demo Solaris 10 to a customer without them being completely blown away by at least one of the many features. DTrace, Zones, FMA, SMF, and ZFS are but a few reasons why Solaris won’t have “joined IRIX, Digital UNIX, and VMS” in a few years.

Part of this is simply that people have yet to experience real OS innovation such as that found in Solaris 10. But more likely this is just a fundamental disconnection between OS developers and end users. If I managed to get my mom and dad running Java Desktop System on Solaris 10, they wouldn’t never know what DTrace, Zones, or ZFS is, simply because it’s not visible to the end user. But this doesn’t mean that it isn’t worthwhile innovation: as with all layered software, our innovations directly influence our immediate consumers. Solaris 10 is wildly popular among our customers, who are mostly admins and developers, with some “power users”. Even though these people are a quantitatively small portion of our user base, they are arguably the most important. OS innovation directly influences the quality of life and efficiency of developers and admins, which has a cascading effect on the rest of the software stack.

This cascading influence tends to be ignores in arguments over the commoditization of the OS. If you stand at any given layer, you can make a reasonable argument that the software two layers beneath you has become a commodity. JVM developers will argue that hardware is a commodity, while J2EE developers will argue that the OS is a commodity. Even if you’re out surfing the web and use a web service developed on J2EE, you’re implicitly relying on innovation that has its roots at the OS. Of course, the further you go from the OS the less prominent the influence is, but its still there.

So think twice before declaring the OS irrelevant. Even if you don’t use features directly provided by the OS, your quality of life has been improved by having them available to those that do use them.

In a departure from recent musings on the inner workings of Solaris, I thought I’d examine one of the issues that Bryan has touched on in his blog. Bryan has been looking at some of the larger issues regarding OS innovation, commoditization, and academic research. I thought I’d take a direct approach by examining our nearest competitor, Linux.

Bryan probably said it best: We believe that the operating system is a nexus of innovation.

I don’t have a lot of experience with the Linux community, but my impression is that the OS is perceived as a commodity. As a result, Linux is just another OS; albeit one with open source and a large community to back it up. I see a lot of comments like “Linux basically does everything Solaris does” and “Solaris has a lot more features, but Linux is catching up.” Very rarely do I see mention of features that blow Solaris (or other operating systems) out of the water. Linus himself has said:

A lot of the exciting work ends up being all user space crap. I mean, exciting in the sense that I wouldn’t car [sic], but if you look at the big picture, that’s actually where most of the effort and most of the innovation goes.

So Linus seems to agree with my intuition, but I’m in unfamiliar territory here. So, I pose the question:

Is the Linux operating system a source of innovation?

This is a specific question: I’m interested only in software innovation relating to the OS. Issues such as open source, ISV suport, and hardware compatibility are irrelevant, as well as software which is not part of the kernel or doesn’t depend on its facilities. I consider software such as the Solaris ptools as falling under the purview of the operating system, because they work hand-in-hand with the /proc filesystem, a kernel facility. Software such as GNOME, KDE, X, GNU tools, etc, are all independent of the OS and not germane to this discussion. I’m also less interested in purely academic work; one of the pitfalls of academic projects is that they rarely see the light of day in a real-world commercial setting. Of course, most innovative work must begin as research before it can be viable in the industry, but certainly proven technologies make better examples.

I can name dozens of Solaris innovations, but only a handful of Linux ones. This could simply be because I know so much about Solaris and so little about Linux; I freely acknowledge that I’m no Linux expert. So are there great Linux OS innovations out there that I’m just not aware of?

In a departure from my usual Solaris propaganda, I thought I’d try a little bit of history. This entry is aimed at all of you C programmers out there that enjoy the novelty of Obfuscated C. If you think you’re a real C hacker, and haven’t heard of the obfuscated C contenst, then you need to spend a few hours browsing their archives of past winners¹.

If you’ve been reading manpages on your UNIX system, you’ve probably been using some form of troff². This is an early typesetting language processor, dating back to pre-UNIX days. You can find some history here. The nroff and troff commands are essentially the same; they are built largely from the same source and differ only in their options and output formats.

The original troff was written by Joe F. Ossanna in assembly language for the PDP-11 in the early 70s. Along came this whizzy portable language known as C, so Ossana rewrote his formatting program. However, it was less of a rewrite and more of a direct translation of the assembly code. The result is a truly incomprehensible tangle of C code, almost completely uncommented. To top it off, Ossana was tragically killed in a car accident in 1977. Rumour has it that attempts were made to enhance troff, before Brian Kernighan caved in and rewrote it from scratch as ditroff.

If you’re curious just how incomprehensible 7000 lines of uncommented C code can be, you can find a later version of it from The Unix Tree, an invaluable resource for the nostalgic among us. To begin with, the files are named n1.c, n2.c, etc. To quote from ‘n6.c’:

setch(){
register i,*j,k;
extern int chtab[];
if((i = getrq()) == 0)return(0);
for(j=chtab;*j != i;j++)if(*(j++) == 0)return(0);
k = *(++j) | chbits;
return(k);
}
find(i,j)
int i,j[];
{
register k;
if(((k = i-'0') >= 1) && (k <= 4) && (k != smnt))return(--k);
for(k=0; j[k] != i; k++)if(j[k] == 0)return(-1);
return(k);
}

If this doesn’t convince you to write well-structured, well-commented code, I don’t know what will. The scary thing is that there are at least 18 bugs in our database open against nroff or troff; one of the side-effects of promising full backwards compatibility. Anyone who has the courage to putback nroff changes earns a badge of honor here – it is a dark place that has claimed the free time of a few brave programmers³. Whenever an open bug report includes such choice phrases as this, you know you’re in trouble:

I’ve seen this problem on non-Sun Unix as well, like Ultrix 3.1 so the problem likely came from Berkeley. The System V version of *roff (ditroff ?) doesn’t have this problem.

¹One of my personal favorites is this little gem, a 2000 winner ‘natori’. It should be a full moon tomorrow night…

#include <stdio.h>
#include <math.h>
double l;main(_,o,O){return putchar((_--+22&&_+44&&main(_,-43,_),_&&o)?(main(-43,
++o,O),((l=(o+21)/sqrt(3-O*22-O*O),l*l<4&&(fabs(((time(0)-607728)%2551443)/
405859.-4.7+acos(l/2))<1.57))[" #"])):10);}

²On Solaris, most manpages are written in SGML, and can be found in /usr/share/man/sman*.

³I’d like to think that the x86 disassembler is a close second, but maybe that’s just because I’m a survivor.

Before I start looking at some of problems we’re addressing in Solaris, I want to step back and examine one of the more fundamental problems I’ve been seeing in the industry. In order to develop more powerful software quickly, we insert layers of abstraction to distance ourselves from the actual implmentation (let someone else worry about it). There is nothing inherently wrong with this; no one is going to argue that you should write your business critical web service in assembly language instead of Java using J2EE. The problem comes from the disturbing trend that programmers are increasingly less knowledgeable about the layers upon which they build.

Most people can sit down and learn how to program Java or C, given enough time. The difference between an average programmer and a gifted programmer is the ability to truly understand the levels above and below where one works. For the majority of us¹, this means understanding two things: our development platform and our customers. While understanding customer needs is a difficult task, a more tragic problem is the failure of programmers to understand their immediate development environment.

If you are a C programmer, it is crucial that you understand how virtual memory works, what exactly happens when you write to a file descriptor, and how threads are implemented. You should understand what a compiler does, how the dynamic linker works, and how the assembly code the compiler generates really works. If you are a Java programmer, you need to understand how garbage collection works, how java bytecodes are interpreted, and what JIT compiling really does. Just because you don’t need to know the OS works doesn’t mean you shouldn’t. Knowing your environment encourages good software practice, as well as making you more effective at solving problems when they do occur.

Unfortunately, everything in the world is not somebody else’s fault. As these new layers are added, things tend to become less and less observable. Not to mention that poor documentation can ruin an otherwise great tool. You may understand how a large number of cross calls can be the product of a misbehaving application, but if you can’t determine where they’re coming from, what’s the point? We in the Solaris group develop tools to provide useful layers of abstraction, as well as tools that rip the hood off² so you can see what’s really happening inside.

Before I start examining some of these tools, I just wanted to point out that there is a large human factor involved that is out of our hands. The most powerful tool in the world can be useless in the hands of someone without a basic understanding of their system. Hopefully by providing these tools we can simultaneously expose the inner workings while sparking desire to learn about these inner workings. So take some some time to read a good book once in a while.

¹For us kernel developers, the landscape is a little different. We have to work with a multitude of difference hardware (development platforms) in order to provide an effectively limitless number of solutions for our customers. I like to think that kernel engineers are pound-for-pound the best group of programmers around because of this, but maybe that’s just my ego talking.

²Scott is in the habit of talking about how we sell cars, not auto parts. Does that mean we also provide the Jaws of Life to save you after you crash your new Enzo?