KVM on illumos

A little over a year ago, I came to Joyent because of a shared belief that systems software innovation matters — that there is no level of the software stack that should be considered off-limits to innovation. Specifically, we ardently believe in innovating in the operating system itself — that extending and leveraging technologies like OS virtualization, DTrace and ZFS can allow one to build unique, differentiated value further up the software stack. My arrival at Joyent also coincided with the birth of illumos, an effort that Joyent was thrilled to support, as we saw that rephrasing our SmartOS as an illumos derivative would allow us to engage in a truly open community with whom we could collaborate and collectively innovate.

In terms of the OS technology itself, as powerful as OS virtualization, DTrace and ZFS are, there was clearly a missing piece to the puzzle: hardware virtualization. In particular, with advances in microprocessor technology over the last five years (and specifically with the introduction of technologies like Intel’s VT-x), one could (in principle, anyway) much more easily build highly performing hardware virtualization. The technology was out there: Fabrice Bellard‘s cross-platform QEMU had been extended to utilize the new microprocessor support by Avi Kivity and his team with their Linux kernel virtual machine (KVM).

In the fall of last year, the imperative became clear: we needed to port KVM to SmartOS. This notion is almost an oxymoron, as KVM is not “portable” in any traditional sense: it interfaces between QEMU, Linux and the hardware with (rightful) disregard for other systems. And as KVM makes heavy use of Linux-specific facilities that don’t necessarily have clean analogs in other systems, the only way to fully scope the work was to start it in earnest. However, we knew that just getting it to compile in any meaningful form would take significant effort, and with our team still ramping up and with lots of work to do elsewhere, the reality was that this would have to start as a small project. Fortunately, Joyent kernel internals veteran Max Bruning was up to the task, and late last year, he copied the KVM bits from the stable Linux 2.6.34 source, took out a sharpened machete, and started whacking back jungle…

Through the winter holidays and early spring (and when he wasn’t dispatched on other urgent tasks), Max made steady progress — but it was clear that the problem was even more grueling than we had anticipated: because KVM is so tightly integrated into the Linux kernel, it was difficult to determine dependencies — and hard to know when to pull in the Linux implementation of a particular facility or function versus writing our own or making a call to the illumos equivalent (if any). By the middle of March, we were able to execute three instructions in a virtual machine before taking an unresolved EPT violation. To the uninitiated, this might sound pathetic — but it meant that a tremendous amount of software and hardware were working properly. Shortly thereafter, with the EPT problem fixed, the virtual machine would happily run indefinitely — but clearly not making any progress booting. But with this milestone, Max had gotten to the point where the work could be more realistically parallelized, and we had gotten to the point as a team and a product where we could dedicate more engineers to this work. In short, it was time to add to the KVM effort — and Robert Mustacchi and I roped up and joined Max in the depths of KVM.

With three of us on the project and with the project now more parallelizable, the pace accelerated, and in particular we could focus on tooling: we added an mdb dmod to help us debug, 16-bit disassembler support that was essential for us to crack the case on the looping in real-mode, and DTrace support for reading the VMCS. (And we even learned some surprising things about GNU C along the way!) On April 18th, we finally broke through into daylight: we were able to boot into KMDB running on a SmartOS guest! The pace quickened again, as we now had the ability to interactively ascertain state as seen by the guest and much more quickly debug issues. Over the next month (and after many intensive debugging sessions), we were able to boot Windows, Linux, FreeBSD, Haiku, QNX, Plan 9 and every other OS we could get our hands on; by May 13th, we were ready to send the company-wide “hello world” mail that has been the hallmark of bringup teams since time immemorial.

The wind was now firmly at our backs, but there was still much work to be done: we needed to get configurations working with large numbers of VCPUs, with many VMs, over long periods of time and significant stress and so on. Over the course of the next few months, the bugs became rarer and rarer (if nastier!), and we were able to validate that performance was on par with Linux KVM — to the point that today we feel confident releasing illumos KVM to the greater community!

In terms of starting points, if you just want to take it for a spin, grab a SmartOS live image ISO. If you’re interested in the source, check out the illumos-kvm github repo for the KVM driver itself, the illumos-kvm-cmd github repo for our QEMU 0.14.1 tree, and illumos itself for some of our KVM-specific tooling.

One final point of validation: when we went to integrate these changes into illumos, we wanted to be sure that they built cleanly in a non-SmartOS (that is, stock illumos) environment. However, we couldn’t dedicate a machine for the activity, so we (of course) installed OpenIndiana running as a KVM guest on SmartOS and cranked the build to verify our KVM deltas! As engineers, we all endeavor to build useful things, and with KVM on illumos, we can assert with absolute confidence that we’ve hit that mark!

Posted on August 15, 2011 at 7:00 am by bmc · Permalink
In: Uncategorized

44 Responses

Subscribe to comments via RSS

  1. Written by Alasdair Lumsden
    on August 15, 2011 at 7:54 am
    Permalink

    From all of the OpenIndiana team, I’d like to offer a heartfelt thank you to Joyent! Bringing KVM to illumos brings yet another great technology to the stack and helps demonstrate the commitment of those behind it.

  2. Written by Michael Widmann
    on August 15, 2011 at 8:07 am
    Permalink

    WOW WOW WOW – GREAT – WONDERFUL – WOW.

    YOU’RE WONDERFUL really …

    thanks a lot the community will thank you all a lot – really really really !

    Michael

  3. Written by Nick Zivkovic
    on August 15, 2011 at 8:25 am
    Permalink

    Congratulations!
    Glad to see that the Illumos project is carrying on (Sun) Solaris’s legacy as a nexus of systems innovation.

    I see that kvm (and qemu) are licensed under the GPL. Does the fact that illumos-kvm is a loadable kernel module (and not part of the illumos kernel itself) immunize it to the viral nature of the GPL?

    Great work, I’ll give it a spin soon!

    • Written by bmc
      on August 15, 2011 at 9:09 am
      Permalink

      Nick,

      Thanks for the kind words! And no, there are no licensing issues here: KVM is GPL’d, but the kernel is no way a derived work of it. Looking forward to you giving it a spin!

      • Written by Ken MacLeod
        on August 16, 2011 at 6:57 am
        Permalink

        The KVM GPL license doesn’t change the license of the kernel.

        However, when you distribute the KVM module combined with the kernel, the KVM license says the combined work has to be distributed under the GPL (or compatible) license or you don’t have the right to distribute the GPL-licensed work.

  4. Written by KVM on illumos « Joyeur
    on August 15, 2011 at 8:59 am
    Permalink

    [...] post originally appeared on DTrace.org and can also be found on the new SmartOS.org [...]

  5. Written by Geoff Flarity
    on August 15, 2011 at 9:10 am
    Permalink

    Woot!

    Thanks!

  6. Written by Sunil Mushran
    on August 15, 2011 at 9:44 am
    Permalink

    Does it handle live migrations (aka vmotion)? If so, what’s your storage story.

    • Written by bmc
      on August 15, 2011 at 11:55 am
      Permalink

      Migration is much more a QEMU-level issue than a KVM-level issue. Looking into the future, we do intend to integrate QEMU migration with ZFS remote replication; stay tuned.

  7. Written by Skipp_OSX
    on August 15, 2011 at 10:24 am
    Permalink

    It’s Haiku, not HaikuOS

    • Written by bmc
      on August 15, 2011 at 11:57 am
      Permalink

      Updated — thank you! (And have I mentioned the strange affection that I have developed for Haiku over the course of this project?)

      • Written by UX-admin
        on August 16, 2011 at 2:56 pm
        Permalink

        Is it because Haiku is based upon BeOS, which in turn draws from AmigaOS?

  8. Written by Thijs
    on August 15, 2011 at 12:19 pm
    Permalink

    Guys, good work!, even the story was catching… I see a bright future ahead!

  9. Written by ZFS Stammtisch - Seite 49 - Forum de Luxx
    on August 15, 2011 at 1:11 pm
    Permalink

    [...] [...]

  10. Written by Andreas Nilsson
    on August 15, 2011 at 2:30 pm
    Permalink

    Fantastic work!

    Will you pursue writing a qemu-equivalent frontend which can/will better integrate with things like zonecfg and friends?

    • Written by bmc
      on August 15, 2011 at 11:37 pm
      Permalink

      Yes, stay tuned — we’re going to be fleshing out SmartOS with enough components of our larger SmartDataCenter systems to (for example) be able to provision a zone with a KVM instance.

  11. Written by KVM makes its way to Illumos
    on August 15, 2011 at 3:28 pm
    Permalink

    [...] was present in Xen. Well today I was pleasantly surprised to see that several folks from Joyent ported KVM to Illumos! This is way cool, and I can’t wait to give this a spin! matty on August 15, 2011 [...]

  12. [...] multi-tenant environment built on a reliable, enterprise-grade storage stack.” Yes, they have ported the KVM virtualization facility from Linux to [...]

  13. Written by Henkis
    on August 15, 2011 at 5:35 pm
    Permalink

    Fantastic work! I’m thrilled to see that several of you who made a real difference to Solaris before the acquisition is providing new innovations to illumos.

    I’m working with large scale deployment of zones myself so I am especially thrilled to see that KVM can with benefit be run inside of a zone. The added security comes at a good time after the “breaking out of KVM” presentation at Blackhat (ironically by a Oracle employee).

  14. Written by Dave
    on August 15, 2011 at 10:25 pm
    Permalink

    This is like Christmas.

    • Written by bmc
      on August 15, 2011 at 11:39 pm
      Permalink

      Dave, It’s our privilege to play the role of Santa. ;)

  15. Written by TS
    on August 15, 2011 at 11:52 pm
    Permalink

    Hi, Brian:

    Congrats on the release. Two things:

    1. Can you clarify if Joyent SmartOS now is Illumos only? I mean is it 100% clean from Oracle’s closed code base?

    2. Can Joyent release a SmartOS operating system user guide pdf file? It took me nearly 10 minutes to find admin/admin and root/root are the login passwd. And I haven’t found the commands to do a HDD install from the livecd.iso released today.

    Thanks for the hard work and contribution to Illumos base.

    • Written by Jason Harley
      on August 16, 2011 at 9:10 am
      Permalink

      TS -

      This guide (by Ryan Nelson of Joyent) should help in the mean time:

      http://www.ryan.net/smartos-disk-blogpost/real_disk_smartos.html

    • Written by bmc
      on August 16, 2011 at 10:25 am
      Permalink

      I would start with Ryan’s guide, but yes, we know that this is — in the immortal words of Bart Smaalders — all blade and no handle. We have lots of planned improvements coming; thank you for your words of encouragement, and stay tuned!

  16. Written by Jeff Brue
    on August 16, 2011 at 12:31 am
    Permalink

    Better than christmas

  17. Written by Chris
    on August 16, 2011 at 2:57 am
    Permalink

    This is really great news, but … how do I install smartos to hd.

  18. Written by The Observation Deck » KVM on illumos | VT Tech
    on August 16, 2011 at 3:54 am
    Permalink

    [...] Subscribe to comments via RSS [...]

  19. Written by fil
    on August 16, 2011 at 4:46 am
    Permalink

    What about the GPL? Isn’t that an issue that you’re taking code from inside the Linux kernel to integrate to a kernel that is not distributed under the GPL? I know Linux was never able to integrate ZFS because of the license disparity (in the other direction, distributing ZFS under the GPL would not be possible). How were you able to work around this issue?

    • Written by bmc
      on August 16, 2011 at 10:32 am
      Permalink

      No, our KVM port remains GPL and its own work (and lives in its own repo) — the illumos kernel is CDDL but is in no way a derived work of our KVM port.

  20. Written by Jon Strabala
    on August 16, 2011 at 9:21 am
    Permalink

    Fantastic news, I use VirtualBox right now but seeing that is is now somewhat closed off by Oracle and despite several months of efforts I can not even purchase licences or support for it (Oracle doesn’t understand how to quote software support or licenses).

    The ability to switch to something else that is “native” illumOS is happy news indeed.

    I look forward to the ability to live migrations. I will be giving this a spin in a few days and look forward to see if it can import OVF, VDI or VMDK images.

    • Written by bmc
      on August 16, 2011 at 10:30 am
      Permalink

      Jon, Yep — it’s all coming. Also — and at the risk of upselling you ;) — you might be interested in our SmartDataCenter product, which gives you the ability to deploy, manage and observe a cloud of SmartOS-based compute nodes. (Our 6.5 version of the product (due out in early October) will have KVM support.)

  21. Written by Ben Taylor
    on August 16, 2011 at 9:55 am
    Permalink

    I remember discussing Qemu with you years ago. I’m somehow not surprised you managed to do this port. :-)

  22. Written by UX-admin
    on August 16, 2011 at 10:41 am
    Permalink

    Where is the documentation?

    pkgtrans(1M) is present, but pkgadd(1M) and /var/sadm/install/contents is missing.

    What is used as a software management subsystem?

  23. Written by UX-admin
    on August 16, 2011 at 10:47 am
    Permalink

    In the ensuing excitement, I almost forgot: congratulations!

    When you wrote “we finally broke through into daylight” it was a reminder of the engineering toil (and excitement!) that we as a team are also going through; at that moment, I knew exactly how that must have felt.

    You did what others only tried to do.

  24. Written by DavidH
    on August 16, 2011 at 2:10 pm
    Permalink

    Does KVM work under SPARC?

    Will we be able to use KVM (QEMU) under SPARC to provide access to x86, x64, or ARM systems, possibly running Solaris?

  25. Written by Adam Leventhal
    on August 16, 2011 at 3:15 pm
    Permalink

    Great stuff, Bryan. Great contribution to illumos and the industry.

  26. Written by Mike La Spina
    on August 16, 2011 at 8:38 pm
    Permalink

    You rock! Thanks for all your incredible efforts!

  27. Written by SmartOS
    on August 17, 2011 at 7:44 am
    Permalink

    [...] und schnüre daraus eine eigene Server-Distribution. Klingt smart und heißt auch SmartOS. Und ja, sie haben KVM auf den SmartOS/Illumos-Kernel portiert. Wer es testen möchte kann sich die Live-CD herunterladen und/oder das System selber bauen. Viel [...]

  28. [...] SmartOS, основанной на кодовой базе проекта IllumOS. KVM дополняет систему виртуализации основанного на Solaris ядра, Zones, [...]

  29. Written by Keith Thomson
    on August 17, 2011 at 9:45 pm
    Permalink

    I tried your SmartOS live disc here as a trial on my system. It booted up, but claims that there’s no support in my hardware for KVM. I run KVM on it though linux on this hardware all the time. Have you tested this at all with AMD64 based chips, or was all of the work done with Intel chips?

    • Written by sang-suan gam
      on August 18, 2011 at 9:08 pm
      Permalink

      HI,

      it hung at when booting up, at “stage 2 …” for me ..
      what are the hw pre-req for running the live/ or installations ?

      thanks, and this is a really great step forward!
      sam

  30. Written by KVM is coming to Illumos… « Dave Koelmeyer
    on August 23, 2011 at 5:55 pm
    Permalink

    [...] Update: Bryan Cantrill makes is official: http://dtrace.org/blogs/bmc/2011/08/15/kvm-on-illumos/ [...]

Subscribe to comments via RSS