Project Bardiche: Introduction

I just recently landed Project Bardiche into SmartOS. The goal of Bardiche has been to create a more streamlined data path for layer two networking in illumos. While the primary motivator for this was for KVM guests, it’s opened up a lot of room for more than just virtual machines. This bulk of this project [...]

Posted on March 20, 2014 at 4:01 pm by rm · Permalink · Comments Closed
In: Bardiche, illumos, joyent, KVM, smartos

A New Challenge

About two years ago Joyent began offering Linux instances, running under KVM, stored on ZFS, and secured by Zones (“double hull virtualization”). Since then, I’ve been doing more and more work on Linux performance as customers deploy on these instances. It’s been fascinating to work on both the illumos and Linux kernels at the same [...]

Posted on March 5, 2014 at 5:44 pm by Brendan Gregg · Permalink · Comments Closed
In: Personal

The Total Cost of Unmasked Data

Data breaches make headlines at a regular cadence. Each is a surprise, but they are not, as a whole, surprising. While the extensive and sophisticated Target breach stuck in the headlines, a significant breach at three South Korean credit card companies happened around the same time. The theft of personal information for 20m subscribers didn’t [...]

Posted on February 12, 2014 at 3:01 am by ahl · Permalink · Comments Closed
In: Delphix, Masking

Another 10 Performance Wins

Following on from my earlier 10 performance wins post, here is another group of 10 I have worked on. # Target Analyzed Key Tool Fixed Specific Improvement 11 redis System DTrace System scheduler 41% 12 rsync System DTrace Application app config 5x 13 mongoperf System DTrace System ZFS tuning up to 8x 14 backups System [...]

Posted on February 11, 2014 at 11:08 am by Brendan Gregg · Permalink · Comments Closed
In: performance, wins

The OpenZFS write throttle

In my last blog post, I wrote about the ZFS write throttle, and how we saw it lead to pathological latency variability on customer systems. Matt Ahrens, the co-founder of ZFS, and I set about to fix it in OpenZFS. While the solution we came to may seem obvious, we arrived at it only through [...]

Posted on February 9, 2014 at 8:55 pm by ahl · Permalink · Comments Closed
In: MattAhrens, OpenZFS, performance, ZFS

Benchmarking the Cloud

Benchmarking, and benchmarking the cloud, is incredibly error prone. I provided guidance though this minefield in the benchmarking chapter of my book (Systems Performance: Enterprise and the Cloud); that chapter can be read online on the InformIT site. I also gave a lightning talk about benchmarking gone wrong at Surge last year. In this post, [...]

Posted on January 10, 2014 at 4:05 pm by Brendan Gregg · Permalink · Comments Closed
In: benchmarking, Cloud, performance

ZFS fundamentals: the write throttle

It’s no small feat to build a stable, modern filesystem. The more I work with ZFS, the more impressed I am with how much it got right, and how malleable it’s proved. It has evolved to fix shortcomings and accommodate underlying technological shifts. It’s not surprising though that even while its underpinnings have withstood the [...]

Posted on December 26, 2013 at 5:40 pm by ahl · Permalink · Comments Closed
In: latency, MattAhrens, OpenZFS, performance, Throughput, ZFS

Understanding DTrace ustack helpers

or, everything you ever wanted to know about stack traces I promised this post over a year ago, and now that someone’s actually working on a new ustack helper, I thought it was finally time to write about what ustack helpers are, how they work, and how I went about building one for Node.js. Only [...]

Posted on November 20, 2013 at 4:50 pm by dap · Permalink · Comments Closed
In: DTrace, joyent, Node.js, smartos

Userland CTF in DTrace

We at Joyent use DTrace for understanding and debugging userland applications just as often as we do for the kernel. That is part of the reason why we’ve worked on things like flamegraphs, the Node.js ustack helper, and the integration of libusdt in node module’s like restify and bunyan. I’ve just put back some work [...]

Posted on November 14, 2013 at 12:54 pm by rm · Permalink · Comments Closed
In: Miscellaneous

Cloud Performance Training

Next week (Nov 18-22) I’m teaching a 5 day class on Cloud Performance, based on my book Systems Performance: Enterprise and the Cloud. I’ve taught this twice internally, and this will be the first class available publicly. I hope to teach it again, but one never knows, this may be the first and last chance [...]

Posted on November 13, 2013 at 3:36 pm by Brendan Gregg · Permalink · Comments Closed
In: Training