As had been rumored for a while, Oracle effectively killed Solaris on Friday. When I first saw this, I had assumed that this was merely a deep cut, but in talking to Solaris engineers still at Oracle, it is clearly much more than that. It is a cut so deep as to be fatal: the core Solaris engineering organization lost on the order of 90% of its people, including essentially all management.

Of note, among the engineers I have spoken with, I heard two things repeatedly: “this is the end” and (from those who managed to survive Friday) “I wish I had been laid off.” Gone is any of the optimism (however tepid) that I have heard over the years — and embarrassed apologies for Oracle’s behavior have been replaced with dismay about the clumsiness, ineptitude and callousness with which this final cut was handled. In particular, that employees who had given their careers to the company were told of their termination via a pre-recorded call — “robo-RIF’d” in the words of one employee — is both despicable and cowardly. To their credit, the engineers affected saw themselves as Sun to the end: they stayed to solve hard, interesting problems and out of allegiance to one another — not out of any loyalty to the broader Oracle. Oracle didn’t deserve them and now it doesn’t have them — they have been liberated, if in a depraved act of corporate violence.

Assuming that this is indeed the end of Solaris (and it certainly looks that way), it offers a time for reflection. Certainly, the demise of Solaris is at one level not surprising, but on the other hand, its very suddenness highlights the degree to which proprietary software can suffer by the vicissitudes of corporate capriciousness. Vulnerable to executive whims, shareholder demands, and a fickle public, organizations can simply change direction by fiat. And because — in the words of the late, great Roger Faulkner — “it is easier to destroy than to create,” these changes in direction can have lasting effect when they mean stopping (or even suspending!) work on a project. Indeed, any engineer in any domain with sufficient longevity will have one (or many!) stories of exciting projects being cancelled by foolhardy and myopic management. For software, though, these cancellations can be particularly gutting because (in the proprietary world, anyway) so many of the details of software are carefully hidden from the users of the product — and much of the innovation of a cancelled software project will likely die with the project, living only in the oral tradition of the engineers who knew it. Worse, in the long run — to paraphrase Keynes — proprietary software projects are all dead. However ubiquitous at their height, this lonely fate awaits all proprietary software.

There is, of course, another way — and befitting its idiosyncratic life and death, Solaris shows us this path too: software can be open source. In stark contrast to proprietary software, open source does not — cannot, even — die. Yes, it can be disused or rusty or fusty, but as long as anyone is interested in it at all, it lives and breathes. Even should the interest wane to nothing, open source software survives still: its life as machine may be suspended, but it becomes as literature, waiting to be discovered by a future generation. That is, while proprietary software can die in an instant, open source software perpetually endures by its nature — and thrives by the strength of its communities. Just as the existence of proprietary software can be surprisingly brittle, open source communities can be crazily robust: they can survive neglect, derision, dissent — even sabotage.

In this regard, I speak from experience: from when Solaris was open sourced in 2005, the OpenSolaris community survived all of these things. By the time Oracle bought Sun five years later in 2010, the community had decided that it needed true independence — illumos was born. And, it turns out, illumos was born at exactly the right moment: shortly after illumos was announced, Oracle — in what remains to me a singularly loathsome and cowardly act — silently re-proprietarized Solaris on August 13, 2010. We in illumos were indisputably on our own, and while many outsiders gave us no chance of survival, we ourselves had reason for confidence: after all, open source communities are robust because they are often united not only by circumstance, but by values, and in our case, we as a community never lost our belief in ZFS, Zones, DTrace and myriad other technologies like MDB, FMA and Crossbow.

Indeed, since 2010, illumos has thrived; illumos is not only the repository of record for technologies that have become cross-platform like OpenZFS, but we have also advanced our core technologies considerably, while still maintaining highest standards of quality. Learning some of the mistakes of OpenSolaris, we have a model that allows for downstream innovation, experimentation and differentiation. For example, Joyent’s SmartOS has always been focused on our need for a cloud hypervisor (causing us to develop big features like hardware virtualization and Linux binary compatibility), and it is now at the heart of a massive buildout for Samsung (who acquired Joyent a little over a year ago). For us at Joyent, the Solaris/illumos/SmartOS saga has been formative in that we have seen both the ill effects of proprietary software and the amazing resilience of open source software — and it very much informed our decision to open source our entire stack in 2014.

Judging merely by its tombstone, the life of Solaris can be viewed as tragic: born out of wedlock between Sun and AT&T and dying at the hands of a remorseless corporate sociopath a quarter century later. And even that may be overstating its longevity: Solaris may not have been truly born until it was made open source, and — certainly to me, anyway — it died the moment it was again made proprietary. But in that shorter life, Solaris achieved the singular: immortality for its revolutionary technologies. So while we can mourn the loss of the proprietary embodiment of Solaris (and we can certainly lament the coarse way in which its technologists were treated!), we can rejoice in the eternal life of its technologies — in illumos and beyond!

Last Tuesday, several months of preparation came to fruition in the inaugural Systems We Love. You never know what’s going to happen the first time you get a new kind of conference together (especially one as broad as this one!) but it was, in a word, amazing. The content was absolutely outstanding, with attendee after attendee praising the uniformly high quality. (For guided tours, check out both Ozan Onay’s excellent exegesis and David Cassel’s thorough New Stack story — and don’t miss Sarah Huffman’s incredible illustrations!) It was such a great conference that many were asking about when we would do it again — and there is already interest in replicating it elsewhere. As an engineer, this makes me slightly nervous as I believe that success often teaches you nothing: luck becomes difficult to differentiate from design. But at the risk of taunting the conference gods with the arrogance of a puny mortal, here’s some stuff I do think we did right:

• Emphasizing the love. When a technologist is explaining their love for a technology, there is a level at which it’s irrefutable: even if you yourself abhor the technology being described, you can’t deny the presenter’s affection for it. Indeed, that you find a love perplexing may make it that much more compelling — and many of the most interesting moments at Systems We Love came when presenters described some delightfully strange wart of a technology (e.g., record locators!). This is the breakthrough that is due to Papers We Love: people want to talk and hear about the things that uplift them and the ideas that inspire them! And when this desire is the unifying principle (rather than the use of a particular technology or the fealty to a particular company), the hallway track becomes much more diverse and fascinating, as attendees have shared values but not necessarily shared experiences.
• Presenting the technologies of others. Many (most?) conferences consist of presenters explaining their own technologies (or that of their company or whatever) — which inevitably has a sales-y undercurrent, even for technical or academic presentations. (And I myself am certainly guilty of this!) To avoid this implied superiority of particular systems or ideas, we deliberately guided submissions away from one’s own technology and rather towards one’s own narrative: why do you love this system? This allowed presenters just enough distance to speak earnestly of a technology, giving their love that much more weight.
• Single track, single day. I (personally) like single-track conferences not only for the obvious reason that it doesn’t force painful choices for the attendee (a choice that I always seem to make incorrectly myself), but also for the slightly more subtle reason that it creates a shared experience among attendees: everyone sees everything, providing for ample discussion fodder for the hallway track. This is also the reason that I prefer a single day: with the conference only running for one day, everyone makes an effort to be there — and when people go for drinks after the event, everyone is relaxed: no one is apprehensive about the talk they need to give in the morning! (Or, if you’re like me, the slides that still must be written!) So while this decision constrained the amount of content (and forced very hard decisions on the part of the program committee), I think it was a huge win.
• Twenty minute talks. I was apprehensive about the length of the timeslot, for a reason that I acknowledge is personal and somewhat petty: I’m a bit of a TED talk malcontent. Yes, I like the spirit of TED talks (and there are certainly some great ones!) — but to my way of thinking, they often come across as just a little too pat, leaving the listener more with an illusion of understanding than a desire to ask more questions. As it turns out, I needn’t have worried; the twenty minute length worked perfectly, with many attendees explicitly praising it — and I loved that many (many!) of the talks ended with suggestions for further reading! So as it turns out, twenty minutes actually is enough time to get to depth in a way that is captivating and compelling yet open-ended — and it’s short enough to encourage presenters to be tight and to allow for a wide variety of content.
• Big, diverse, awesome program committee. A program will reflect its program committee, so carefully crafting a program committee is essential. Systems We Love was blessed with an absolutely outstanding program committee, joined by a love of systems but otherwise representing many different perspectives and backgrounds. Because the talk submissions were pretty short, everyone could review everything — and because we did everything online, there was no need to meet (in person or virtually). This allowed for the PC to grow essentially without bounds, which paid off with not only diverse perspectives, but also diverse submissions. In terms of composing it, I decided to shoot the moon and simply ask the people who I thought would comprise my dream PC — and I was pleasantly surprised when everyone I asked accepted! I hope that members of the PC found it to be an uplifting experience — though the high quality of submissions and the low acceptance rate definitely induced some amount of mental anguish!
• Double-blind selection process. One thing that I felt strongly about going into Systems We Love was having a double-blind selection process. I have personally witnessed how an unblinded process can lead to terrible decisions, and a welcome change in the past five years in academic computer science has been that many more conferences have gone double-blind. Just in case I had any doubt about being double-blind, it was obliterated by hearing of the experiences of Kathryn McKinley: she has personally seen the difference it has made in the diversity of accepted work, and she makes an emphatic and compelling case that anything less than double-blind is irresponsible. For Systems We Love, I think being double-blind worked very well, and it had its intended effect in that some of the selected talks were from new faces with new perspectives. (It also must be said that when we de-blinded as a final check, we saw some of the rejected proposals were from some well-known names; everyone needs to bring their A-game to a double-blind process!)
• Mechanics. Mechanics can’t make a conference, but they can certainly break one: if the venue is terrible or the food doesn’t show up or the vibe of the space doesn’t match that of the conference, it can have a serious adverse effect. Thanks to the hard work of Brittany Berry at Joyent, I think we did well on this front: modulo an early morning coffee crisis with a vendor, all went smoothly: the space was great (even when packed), the A/V team very accommodating (especially given our monster of a schedule!), and the food was tasty and plentiful. (I also particularly like that Brittany found a way to donate the excess food at the end of the day!) We also made the decision to livestream the conference and (perhaps more importantly) to leave the entire livestream up; while not inexpensive, it allowed many more people to feel the love — and also allowed all of our content to be online from the moment it was presented.

Okay, so that’s a pretty long list of things that worked; what didn’t work so well? I would say that there was basically only a single issue: the packed schedule. We had 19 (!!) 20 minute talks, and there simply wasn’t time for the length or quantity of breaks that one might like. I think it worked out better than it sounds like it would (thanks to our excellent and varied presenters!), but it was nonetheless exhausting and I think everyone would have appreciated at least one more break. Still, there were essentially no complaints about the number of presentations, so we wouldn’t want to overshoot by slimming down too much; perhaps the optimal number is 16 talks spread over four sessions of four talks apiece?

So where to go from here? We know now that there is a ton of demand and a bunch of great content to match (I’m still bummed about the terrific submissions we turned away!), so we know that we can (and will) easily have this be an annual event. But it seems like we can do more: maybe an event on the east coast? Perhaps one in Europe? Maybe as a series of meetups in the style of Papers We Love? There are a lot of possibilities, so please let us know what you’d like to see!

Finally, I would like to reflect on the most personally satisfying bit of Systems We Love: simply by bringing so many like-minded people together in the same room and having them get to know one another, we know that lives have been changed; new connections have been made, new opportunities have been found, and new journeys have begun. We knew that this would happen in the abstract, but in recent days, we have seen it ourselves: in the new year, you will see new faces on the Joyent engineering team that we met at Systems We Love. (If it needs to be said, the love of systems is a unifying force across Joyent; if you find yourself captivated by the content and you’re contemplating a career change, we’re hiring!) Like most (if not all) of us, the direction of my life has been significantly changed by meeting or hearing the right person at the right moment; that we have helped facilitate similar changes in our own small way is intensely gratifying — and is very much at the heart of what Systems We Love is about!

We’ve been overwhelmed by the positive response to Systems We Love! As simple as this concept is, Systems We Love — like Papers We Love, !!Con and others that inspired it — has tapped into a current of enthusiasm. Adam Leventhal captured this zeitgeist in a Hacker News comment:

What catches our collective attention are systems we hate, systems that suck, systems that fail–or systems too new to know. It’s refreshing to consider systems established and clever enough to love. There are wheels we don’t need to reinvent, systems that can teach us.

Are you tantalized by Systems We Love but you don’t know what proposal to submit? For those looking for proposal guidance, my advice is simple: find the love. Just as every presentation title at !!Con must assert its enthusiasm by ending with two bangs, you can think of every talk at Systems We Love as beginning with an implicit “Why I love…” So instead of a lecture on, say, the innards of ZFS (and well you may love ZFS!), pick an angle on ZFS that you particularly love. Why do you love it or what do you love about it? Keep it personal: this isn’t about asserting the dominance of one system — this is about you and a system (or an aspect of a system) that you love.

Now, what if you don’t think you love anything at all? Especially if you write software for a living and you’ve been at it for a while, it can be easy to lose the love in the sluice of quotidian sewage that is a deployed system. But I would assert that beneath any sedimented cynicism there must be a core of love: think back to when you were first discovering software systems as your calling and to your initial awe of learning of how much more complicated these systems are than you realized (what a colleague of mine once called “the miracle of boot”) — surely there is something in that awe from which you draw (or at least, drew) inspiration! I acknowledge that this is the exception rather than the rule — that it feels like we are more often disappointed rather than pleasantly surprised — but this is the nature of the job: our work as software engineers takes us to the boundaries of systems that are emerging or otherwise don’t work properly rather than into the beautiful caverns deep below the surface. To phrase this in terms of an old essay of mine, we spend our time in systems that are grimy or fetid rather than immaculate — but Systems We Love is about the inspiration that we derive from those immaculate systems (or at least their immaculate aspects).

Finally, don’t set the bar too high for yourself: we are bound to have a complicated relationship with any system with which we spend significant time, and just because you love one aspect of a system doesn’t mean that other parts don’t enrage, troll or depress you! So just remember it’s not Systems We Know, Systems We Invented or Systems We Worship — it’s Systems We Love and we hope to see you there!

One of the exciting trends of the past few years is the emergence of Papers We Love. I have long been an advocate of journal clubs, but I have also found that discussion groups can stagnate when confined to a fixed group or a single domain; by broadening the participants and encouraging presenters to select papers that appeal to the heart as well as the head, Papers We Love has developed a singular verve. Speaking personally, I have enjoyed the meetups that I have attended — and I was honored to be given the opportunity to present on Jails and Zones at Papers We Love NYC (for which, it must be said, I was flattered by the best introduction of all time). I found the crowd that gathered to be engaged and invigorating — and thought-provoking conversation went well into the night.

The energy felt at Papers We Love is in stark contrast to the academic venues in which computer science papers are traditionally presented, which I accentuated in a candid keynote at the USENIX Annual Technical Conference, pointing to PWL as a model that is much more amenable to cross-pollination of ideas between academics and practitioners. My keynote was fiery, and it may have landed on dry tinder: if Rik Farrow’s excellent summary of my talk is any indicator, the time is right for a broader conversation about how we publish rigorous work.

But for us practitioners, however well they are discussed, academic work remains somewhat ancillary: while papers are invaluable as a mechanism for the rigorous presentation of thinking, it is ultimately the artifacts that we develop — the systems themselves — that represent the tangible embodiment of our ideas. And for the systems that I am personally engaged in, I have found that getting together to them is inspiring and fruitful, e.g. the quadrennial dtrace.conf or the more regular OpenZFS developer summit. My experiences with Papers We Love and with these system-specific meetings caused me to ask on a whim if there would be interest in a one-day one-track conference that tried to capture the PWL zeitgeist but for systems — a “Systems We Love.”

While I had thrown this idea out somewhat casually, the response was too clear to ignore: there was most definitely interest — to the point of expectation that it would happen! And here at Joyent, a company for which love of systems is practically an organizing principle, the interest quickly boiled into a frothy fervor; we couldn’t not do this!

It took a little while to get the logistics down, but I’m very happy to report that Systems We Love is on: December 13th in San Francisco! To determine the program, I am honored to be joined by an extraordinary program committee: hailing from a wide-range of backgrounds, experience levels, and interests — and united by a shared love of systems. So: the call for proposals is open — and if you have a love of systems, we hope that you will consider submitting a proposal and/or joining us on December 13th!

Early this afternoon, I had just recorded a wide-ranging episode of Arrested DevOps with the incomparable Bridget Kromhout and noticed that I had a flurry of Twitter mentions, all in reaction to this tweet of mine. There was just one problem: I didn’t tweet it. With my account obviously hacked, I went into fight-or-flight mode and (thanks in no small part to Bridget’s calm presence) did the obvious things: I changed my Twitter password, revoked the privileges of all applications, and tried to assess the damage…

Other than the tweet, I (thankfully!) didn’t see any obvious additional damage: no crazy DMs or random follows or unfollows. In terms of figuring out where the malicious tweet had come from, the source of the tweet was “Twitter for Android” — but according to my login history, the last Twitter for Android login was from me during my morning commute about two-and-a-half hours before the tweet. (And according to Twitter, I have only used the one device to access my account.) The only intervening logins were two from Quora about an hour prior to the tweet. (Aside: WTF, Quora?! Revoked!)

Then there was the oddity of the tweet itself. There was no caption — just the two images from what I gathered to be Germany. Looking at the raw tweet, however, cleared up its source:

{
  "created_at": "Mon Sep 12 17:56:31 +0000 2016",
  "id": 775392664602554400,
  "id_str": "775392664602554369",
  "text": "https://t.co/pYKRhaAdvC",
  "truncated": false,
  "entities": {
    "hashtags": [],
    "symbols": [],
    "user_mentions": [],
    "urls": [],
    "media": [
      {
        "id": 775378240244449300,
        "id_str": "775378240244449280",
        "indices": [
          0,
          23
        ],
        "media_url": "http://pbs.twimg.com/media/CsKyZsBWgAAHgVq.jpg",
        "media_url_https": "https://pbs.twimg.com/media/CsKyZsBWgAAHgVq.jpg",
        "url": "https://t.co/pYKRhaAdvC",
        "display_url": "pic.twitter.com/pYKRhaAdvC",
        "expanded_url": "https://twitter.com/MattAndersonBBC/status/775378264772775936/photo/1",
        "type": "photo",
        "sizes": {
          "medium": {
            "w": 1200,
            "h": 1200,
            "resize": "fit"
          },
          "large": {
            "w": 2048,
            "h": 2048,
            "resize": "fit"
          },
          "thumb": {
            "w": 150,
            "h": 150,
            "resize": "crop"
          },
          "small": {
            "w": 680,
            "h": 680,
            "resize": "fit"
          }
        },
        "source_status_id": 775378264772776000,
        "source_status_id_str": "775378264772775936",
        "source_user_id": 1193503572,
        "source_user_id_str": "1193503572"
      }
    ]
  },
  "extended_entities": {
    "media": [
      {
        "id": 775378240244449300,
        "id_str": "775378240244449280",
        "indices": [
          0,
          23
        ],
        "media_url": "http://pbs.twimg.com/media/CsKyZsBWgAAHgVq.jpg",
        "media_url_https": "https://pbs.twimg.com/media/CsKyZsBWgAAHgVq.jpg",
        "url": "https://t.co/pYKRhaAdvC",
        "display_url": "pic.twitter.com/pYKRhaAdvC",
        "expanded_url": "https://twitter.com/MattAndersonBBC/status/775378264772775936/photo/1",
        "type": "photo",
        "sizes": {
          "medium": {
            "w": 1200,
            "h": 1200,
            "resize": "fit"
          },
          "large": {
            "w": 2048,
            "h": 2048,
            "resize": "fit"
          },
          "thumb": {
            "w": 150,
            "h": 150,
            "resize": "crop"
          },
          "small": {
            "w": 680,
            "h": 680,
            "resize": "fit"
          }
        },
        "source_status_id": 775378264772776000,
        "source_status_id_str": "775378264772775936",
        "source_user_id": 1193503572,
        "source_user_id_str": "1193503572"
      },
      {
        "id": 775378240248614900,
        "id_str": "775378240248614912",
        "indices": [
          0,
          23
        ],
        "media_url": "http://pbs.twimg.com/media/CsKyZsCWEAA4oOp.jpg",
        "media_url_https": "https://pbs.twimg.com/media/CsKyZsCWEAA4oOp.jpg",
        "url": "https://t.co/pYKRhaAdvC",
        "display_url": "pic.twitter.com/pYKRhaAdvC",
        "expanded_url": "https://twitter.com/MattAndersonBBC/status/775378264772775936/photo/1",
        "type": "photo",
        "sizes": {
          "small": {
            "w": 680,
            "h": 680,
            "resize": "fit"
          },
          "thumb": {
            "w": 150,
            "h": 150,
            "resize": "crop"
          },
          "medium": {
            "w": 1200,
            "h": 1200,
            "resize": "fit"
          },
          "large": {
            "w": 2048,
            "h": 2048,
            "resize": "fit"
          }
        },
        "source_status_id": 775378264772776000,
        "source_status_id_str": "775378264772775936",
        "source_user_id": 1193503572,
        "source_user_id_str": "1193503572"
      }
    ]
  },
  "source": "Twitter for Android",
  "in_reply_to_status_id": null,
  "in_reply_to_status_id_str": null,
  "in_reply_to_user_id": null,
  "in_reply_to_user_id_str": null,
  "in_reply_to_screen_name": null,
  "user": {
    "id": 173630577,
    "id_str": "173630577",
    "name": "Bryan Cantrill",
    "screen_name": "bcantrill",
    "location": "",
    "description": "Nom de guerre: Colonel Data Corruption",
    "url": "http://t.co/VyAyIJP8vR",
    "entities": {
      "url": {
        "urls": [
          {
            "url": "http://t.co/VyAyIJP8vR",
            "expanded_url": "http://dtrace.org/blogs/bmc",
            "display_url": "dtrace.org/blogs/bmc",
            "indices": [
              0,
              22
            ]
          }
        ]
      },
      "description": {
        "urls": []
      }
    },
    "protected": false,
    "followers_count": 10407,
    "friends_count": 1557,
    "listed_count": 434,
    "created_at": "Sun Aug 01 23:51:44 +0000 2010",
    "favourites_count": 2431,
    "utc_offset": -25200,
    "time_zone": "Pacific Time (US & Canada)",
    "geo_enabled": true,
    "verified": false,
    "statuses_count": 4808,
    "lang": "en",
    "contributors_enabled": false,
    "is_translator": false,
    "is_translation_enabled": false,
    "profile_background_color": "C0DEED",
    "profile_background_image_url": "http://abs.twimg.com/images/themes/theme1/bg.png",
    "profile_background_image_url_https": "https://abs.twimg.com/images/themes/theme1/bg.png",
    "profile_background_tile": false,
    "profile_image_url": "http://pbs.twimg.com/profile_images/618537697670397952/gW9iQsvF_normal.jpg",
    "profile_image_url_https": "https://pbs.twimg.com/profile_images/618537697670397952/gW9iQsvF_normal.jpg",
    "profile_link_color": "0084B4",
    "profile_sidebar_border_color": "C0DEED",
    "profile_sidebar_fill_color": "DDEEF6",
    "profile_text_color": "333333",
    "profile_use_background_image": true,
    "has_extended_profile": false,
    "default_profile": true,
    "default_profile_image": false,
    "following": false,
    "follow_request_sent": false,
    "notifications": false
  },
  "geo": null,
  "coordinates": null,
  "place": {
    "id": "5a110d312052166f",
    "url": "https://api.twitter.com/1.1/geo/id/5a110d312052166f.json",
    "place_type": "city",
    "name": "San Francisco",
    "full_name": "San Francisco, CA",
    "country_code": "US",
    "country": "United States",
    "contained_within": [],
    "bounding_box": {
      "type": "Polygon",
      "coordinates": [
        [
          [
            -122.514926,
            37.708075
          ],
          [
            -122.357031,
            37.708075
          ],
          [
            -122.357031,
            37.833238
          ],
          [
            -122.514926,
            37.833238
          ]
        ]
      ]
    },
    "attributes": {}
  },
  "contributors": null,
  "is_quote_status": false,
  "retweet_count": 2,
  "favorite_count": 9,
  "favorited": false,
  "retweeted": false,
  "possibly_sensitive": false,
  "possibly_sensitive_appealable": false,
  "lang": "und"
}

Note in particular that the media has a source_status_id_str of 775378264772775936; it’s from this tweet roughly an hour before mine from Matt Anderson, the BBC Culture editor who (I gather) is Berlin-based.

Why would someone who had just hacked my account burn it by tweeting an innocuous (if idiosyncratic) photo of campaign posters on the streets of Berlin?! Suddenly this is feeling less like I’ve been hacked, and more like I’m the victim of data corruption.

Some questions I have, that I don’t know enough about the Twitter API to answer: first, how are tweets created that refer to media entities from other tweets? i.e., is there something about that tweet that can give a better clue as to how it was generated? Does the fact that it’s geolocated to San Francisco (albeit with the broadest possible coordinates) indicate that it might have come from the Twitter client misbehaving on my phone? (I didn’t follow Matthew Anderson and my phone was on my desk when this was tweeted — so this would be the app going seriously loco.) And what I’m most dying to know: what other tweets refer to the photos from the tweet from Matthew? (I gather that DataSift can answer this question, but I’m not a DataSift customer and they don’t appear to have a free tier.) If there’s a server-side bug afoot here, it wouldn’t be surprising if I’m not the only one affected.

I’m not sure I’m ever going to know the answers to these questions, but I’m leaving the tweet up there in hopes that it will provide some clues — and with the belief that the villain in the story, if ever brought to justice, will be a member of the shadowy cabal that I have fought my entire career: busted software.

Something that got a little lost in the excitement of Samsung’s recent acquisition of Joyent was dtrace.conf(16), our quadrennial (!) unconference on DTrace. The videos are up, and in the spirit of Adam Leventhal‘s excellent wrap-ups from dtrace.conf(08) and dtrace.conf(12), I wanted to provide a survey of the one-day conference and its content.

Once again, it was an eclectic mix of technologists — and once again, the day got kicked off with me providing an introduction to dtrace.conf and its history. (Just to save you the time filling out your Cantrill Presentation Bingo Card: you can find me punching myself at 16:19, me offering unsolicited personal medical history at 20:11, and me getting trolled by unikernels at 38:25.)

Some at the conference (okay, one) had never seen or used DTrace before, so to get brains warmed up, Max Bruning gave a quick introduction to DTrace — featuring a real problem. (The problem that Max examined is a process running on LX-branded zones on SmartOS.)

Next up was Graeme Jenkinson from the University of Cambridge on distributed tracing featuring CADETS, a system with a DTrace-inspired query language called Event Query.

We then started a troika of presentations on core DTrace improvements, starting with George Neville-Neil on core D language improvements, some of which have been prototyped and others of which represent a wish list. This led into Matt Ahrens (of ZFS fame) on D syntactic sugar: language extensions that Matt implemented for D to make it easier to write more complicated (and more maintainable) scripts. A singular point of pride for me personally is how much DTrace was used to implement and debug ZFS: Matt has used DTrace as much as anyone, and anything that he feels he needs to make his life easier is something that will almost certainly benefit all DTrace users. First among these: support for “if”/”else”, a change that since dtrace.conf has gone through code review and is poised to integrate into DTrace! The final presentation in this segment on core improvements was Joyent engineer Robert Mustacchi on CTF everywhere, outlining work that Robert and Adam did in 2013 to bring user-level CTF understanding to DTrace.

The next group of presentations focused more on how DTrace is used, kicking off with Eric Saxby on DTracing applications, with a particular focus on instrumenting Ruby programs using Chris Andrews‘ excellent libusdt. When instrumenting upstack, we found that it’s useful for DTrace to pull apart JSON — and Joyent engineer Joshua Clulow presented next on the DTrace JSON subroutine that he implemented a few years ago. (And because I know you’re dying to know: Josh’s presentation is on vtmc, terminal-based presentation software unsurprisingly of his own creation.) Wrapping up this section was Dan McDonald talking about the challenges of DTrace-based dynamic asserts: because of the ubiquity of asserts, we really need to add is-enabled probes to the kernel to support dynamic asserts — an improvement that is long overdue, and that we will hopefully implement before 2020!

In the penultimate group of presentations, we got into some system-specific instrumentation and challenges, starting with James Nugent on DTrace and Go. The problem he refers to as preventing “Go and DTrace from working very well together” is the fact that Go doesn’t preserve frame pointers on x86 — but the good news is that this has changed and frame pointers will be preserved starting in 1.7, making DTrace on Go much more useful! After James, Joyent engineer Dave Pacheco described his experiences of using DTrace and Postgres. For our Manta object storage system, Postgres is a critical component and understanding it dynamically and in production has proved essential for us. George Neville-Neil then took the stage again to discuss performance improvements with always-on instrumentation. (Notable quote: “this is being recorded, but I’ll say it anyway.”) Gordon Marler from Bloomberg discussed the challenges of instrumenting massive binaries with thousands of shared objects, consisting of multiple languages (C, C++ and — pause for effect — Fortran 77) and millions of symbols (!!) — which necessitated DTrace ustack performance improvements via some custom (and optimized) postprocessing.

The final group of presentations kicked off with Joyent engineer Alex Wilson and me presenting DTrace in the zone and the DTrace privilege model, which is an important (if ominous) precursor for a very interesting presentation: security researcher Ben Murphy describing his diabolically clever work on
DTrace exploitation.

As the last presentation of the day (as we felt it would be a good segue to drinks), George Neville-Neil led a brief discussion on what he calls OpenDTrace — but is really about sharing the DTrace code more effectively across systems. (DTrace itself is entirely open source, so “OpenDTrace” is something of a redundancy.) George kicked off an OpenDTrace organization on GitHub and it currently holds scripts and the DTrace toolkit, with the aspirations of potentially mirroring the OpenZFS effort to encourage cross-platform development and collaboration.

After George wrapped up, we celebrated the passing of another quadrennial in traditional DTrace fashion: with cans of Tecate and exhilarating rounds of Fishpong. And for you, dear reader, we have a bonus for you for managing to read this far: if you weren’t able to make it to the conference, we have a few extra dtrace.conf(16) t-shirts. To get one of these, e-mail us your size, address, and maybe a sentence or two on how you use or have used DTrace. Supplies are (obviously) limited; if you miss out, you’ll have to wait until the next dtrace.conf in 2020!