Response time – or latency – is crucial to understand in detail, but many of the common presentations of this data hide important details and patterns. Latency heat maps are an effective way to reveal these. I often use tools that provide heat maps directly, but sometimes I have separate trace output that I’d like to convert into a heat map. To answer this need, I just wrote trace2heatmap.pl, which generates interactive SVGs.

I explained how latency heat maps work in the 2010 article “Visualizing System Latency” (ACMQ, CACM). I’ve previously shared interesting examples in: Rainbow Pterodactyl, Icy Lake, ZFS L2ARC.

Problem

I whipped up a simple example to explain this, using disk I/O latency (I have plenty of real-world examples, but explaining them can get sidetracked). This is a single disk system, with a single process performing a sequential synchronous write workload.

Using iostat(1M) to examine average latency (asvc_t):

$ iostat -xnz 1
[...]
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0  220.0    0.0 9635.8  0.0  1.0    0.0    4.6   0  99 c1d0
                    extended device statistics
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0  203.0    0.0 8976.2  0.0  1.0    0.0    5.1   0  99 c1d0

I could plot average latency (as many monitoring products do), but the average is seriously misleading, and doesn’t explain what’s really happening. And since latency is so important for performance, I want to know exactly what is happening.

I had “iosnoop -Dots” running, which collected two minutes of per-I/O latency and other details:

# iosnoop -Dots > out.iosnoop
^C
# more out.iosnoop
STIME(us)   TIME(us)    DELTA(us) DTIME(us) UID   PID D    BLOCK   SIZE      COMM PATHNAME
9339835115  9339835827  712       730       100 23885 W 253757952 131072    odsync <none>
9339835157  9339836008  850       180       100 23885 W 252168272   4096    odsync <none>
9339926948  9339927672  723       731       100 23885 W 251696640 131072    odsync <none>
[...15,000 lines truncated...]

I/O latency is the “DELTA(us)” column. This file was thousands of lines long – too much to read.

Latency Histogram: With Outliers

The latency distribution can be examined as a histogram (using R, and a subset of the trace file):

This shows that the average has been dragged up by latency outliers: I/O with very high latency.

This is a fairly common occurrence, and it’s very useful to know when it has occurred. Those outliers may be individually causing problems, and can be easily be plucked from the trace file for further analysis; eg:

# awk '$3 > 50000' out.iosnoop_marssync01
STIME(us)   TIME(us)    DELTA(us) DTIME(us) UID   PID D    BLOCK   SIZE      COMM PATHNAME
9343218579  9343276502  57922     57398      0      0 W 142876112  4096      sched <none>
9343218595  9343276605  58010     103        0      0 W 195581749  5632      sched <none>
9343278072  9343328860  50788     50091      0      0 W 195581794  4608      sched <none>
[...]

Most of the I/O in the histogram was in a single column on the left.

Latency Histogram: Zoomed

Zooming in, by generating a histogram of the 0 – 2 ms range:

The I/O distribution is bimodal. This also commonly occurs for latency or response time in different subsystems. Eg, the application has a “fast path” and a “slow path”, or a resource has cache hits vs cache misses, etc.

But there is still more hidden here. The average latency reported by iostat hinted that there was per-second variance. This histogram is reporting the entire two minutes of iosnoop output.

Latency Histogram: Animation

I rendered the iosnoop output as per-second histograms, and generated the following animation (a subset of the frames):

Not only is this bimodal, but the modes move over time. This had been obscured by rendering all data as a single histogram.

Heat Map

Using trace2heatmap.pl to generate a heat map from the iosnoop output.

Click for an interactive SVG version, and compare to the animation above.

The command used was:

$ awk '{ print $2, $3 }' out.iosnoop | ./trace2heatmap.pl --unitstime=us \
    --unitslatency=us --maxlat=2000 --grid > heatmap.svg

Without “–grid”, the grid lines are not drawn (making it more Tufte-friendly); see the example.

trace2heatmap.pl gets the job done, but it’s probably a bit buggy – I spent three hours writing it (and more than three hours writing this post about it!), really for just the trace files I don’t already have heat maps for.

Heat Maps Explained

It may already be obvious how these work. Each frame of the histogram animation becomes a column in a latency heat map, with the histogram bar height represented as a color:

Click for higher resolution.

Production Use

If you want to add heat maps to your monitoring solution, then great! However, note that tracing per-event latency can be expensive to perform. DTrace minimizes the overheads as much as possible using per-CPU buffers and asynchronous kernel-user transfers; other tools (eg, strace, tcpdump) are expected to have higher overhead. This can cause problems for production use: you want to understand the overhead, including when using DTrace, before tracing events.

Heat maps have been used successfully in production – and recorded at a one-second granularity 24x7x365 – by some products built upon DTrace. These use the DTrace aggregating feature to pass a quantized summary of latency, instead of every event, to user-level, reducing the data transfer by a large factor (eg, 1000x). This summary may consist of a per-second array with about 200 elements for different latency ranges, each containing the count of events, and is from the DTrace aggregating actions @quantize, @lquantize, or @llquantize (best). This array is then resampled (downsampled) to the resolution desired for the heat map (usually down to 30 or so levels).

Examples of products that can generate heat maps in a production-efficient way, are the Oracle ZFS Storage Appliance, Joyent Cloud Analytics, and Circonus.

There is also Tracelytics, although I can’t comment on its efficiency, as I don’t know its internals yet.

Other Uses

Heat maps (and trace2heatmap.pl) can be used to examine metrics other than latency, such as offset, I/O size, and utilization. For examples, see the heat maps in Visualizing Device Utilization, and Subsecond Offset Heat Maps.

Background

Bryan and I developed latency heat maps in 2008 for the ZFS Storage Appliance. For more background, see Visualizing System Latency.

Thanks to Deirdre for helping with another post!

I started working with flash in 2006 — fortunate timing as flash was just starting to take hold in the enterprise. I started asking customers I’d visit about flash. I’ll always remember the response from an early adopter when I asked about how he planned on using the new, expensive storage, “We just bought it, and we have no idea.” It was a solution in search of a problem — the garbage can model at play.

Flash has evolved significantly since then from a raw material used on its own to a component in systems of increasing complexity. I wrote recently about the various techniques being employed to get the most out of flash; all share the basic idea of trading compute and IOPS (abundant commodities) for capacity (still more expensive for flash than hard drives). The ideal use cases are the ones that benefit most from that trade-off, ones where compression and dedup consume cheap compute cycles rather than expensive space on the NAND flash. Flash storage is best with data that contains high degrees of redundancy that clever software can squeeze out. With those loose criteria, it’s been amazing to me how flash storage vendors have flocked to the VDI use case. It’s certainly well-suited — big on IOPS with nearly identical data from different Windows installs that’s easily compressed and deduped — but seemingly every flash vendor has decided that it’s one part — if not the part — of the market they want to address. Take a look at the information on VDI from various flash storage vendors: Fusion, Nimble, Pure Storage, Tegile, Tintri, Violin, Virident, Whiptail — the list goes on and on.

I worked extensively with flash until leaving Oracle in 2010 when I decided to leave for a start up. I ended up not sticking with flash precisely because it was — and is — such a crowded space. I’d happily bet on the space, but it was harder to pick one winner. One of the things that drew me to Delphix though was precisely its compatibility with flash. At Delphix we create virtual database copies by sharing blocks; think of it as dedup before the fact, or dedup but without the runtime tax. Creating a virtual copy happens almost instantaneously saving tremendous amounts of administration time, unblocking developers, and accelerating projects — hence our credo of agile data. Unlike storage-based snapshots, Delphix virtual copies are database aware, provisioning is fully integrated and automated. Those virtual copies also take up much less physical space, but with as many or more IOPS hitting the aggregate of those virtual copies. Sound familiar yet? One tenth the capacity with the same workload — let’s call it 10x greater IOPS intensity — is ideally suited for flash storage.

Flash storage is best when clever software can squeeze out redundancies; Delphix is that clever software for databases. Delphix customers are starting to combine our product with their flash storage purchases. An all-flash array that’s 5x the $/TB as disk storage suddenly becomes half the price of disk storage when combined with Delphix — with substantially better performance. We as an industry still haven’t realized the full potential of flash storage. Agile data through Delphix fills in another big piece of the flash picture.

Applications are nothing without data. Enterprise applications have data anchored in infrastructure, tied down by production requirements, legacy architecture, and security regulations. But projects demand fresh data under the control of their developers and testers, requiring processes to work around these impediments. The suboptimal result leads to cost overruns, schedule delays, and poor quality.

Agile development requires agile data. Agile data empowers developers and testers to control their data on their schedule. It unburdens IT by efficiently providing data where it is needed independent of underlying infrastructure. And it accelerates application delivery by providing fresh and complete data whenever necessary. It grants its users super powers.

Many technologies can solve part of the agile data problem, but a partial solution still leaves you with suboptimal processes that impede your business. A complete agile data solution must embrace the following attributes.

Non Disruptive Synchronization

Production data is sensitive. The environment has been highly optimized and secured, and its continued operation is critical to the success of the business – introducing risk is unacceptable. An agile data solution must automatically synchronize with production data such that it can provide fresh and relevant data copies, but it cannot mandate changes to how the production environment is managed, nor can its operation jeopardize the performance or success of business critical activities.

Service Provisioning

Data is more than just a sequence of bits. Projects access data through relational databases, NoSQL databases, REST APIs, or other APIs. An agile data solution must move beyond copying the physical representation of the data by instantiating and configuring the systems to access that data. Leaving this process to the end users induces delays and amplifies risk.

Source Freedom

Data is pervasive. Efforts to mandate a single data representation, be it a particular relational or NoSQL system, rarely succeed and limit the ability of projects to choose the data representation most appropriate for their needs. As projects needs diversify the data landscape, the ability to manage all data through a single experience becomes essential. This unified agile experience necessitates a solution not tied to a single data source.

Platform Independence

The premier storage, virtualization, and compute platforms of today may be next year’s legacy architecture. Solutions limited to a single platform inhibit the ability of organizations to capitalize on advances in the industry, be it a high performance flash array or new private cloud software. Agility over time requires a solution that is not tied to the implementation of a particular hardware or software platform.

Efficient Copies

Storage costs money, and time costs the business. Agile development requires a proliferation of data copies for each developer and tester, magnifying these effects. Working around the issue with partial data leads to costly errors that are caught late in the application lifecycle, if at all. An agile solution must be able to create, refresh, and rollback copies of production data in minutes while consuming a fraction of the space required for a full copy.

Workflow Customization

Each development environment has its own application lifecycle workflow. Data may need to be masked, projects may need multiple branches with different schemas, or developers may need to restart services as data is refreshed. Pushing responsibility to the end user is error prone and impedes application delivery. An agile solution must provide stable interfaces for automation and customization such that it can adapt to any development workflow.

Self Service Data

Developers and testers dictate the pace of their assigned tasks, and each task affects the data. Agile development mandates that developers have the ability to transform, refresh, and roll back their data without interference. This experience should shield the user from the implementation details of the environment to limit confusion and reduce opportunity for error.

Resource Management

Each data copy consumes resources through storage, memory, and compute. Once developers experience the power of agile data, they will want more \copies, run workloads on them for which they were not designed, and forget to delete them when they are through. As these resources become scarce, the failure modes (such as poor performance) become more expensive to diagnose and repair. Combatting this data sprawl requires visibility into performance and capacity usage, accountability through auditing and reports, and proactive resource constraints.

Delphix is the agile data platform of the future. You can sync to your production database, instantly provision virtual databases where they are needed using miniscule amount space, and provide each developer their own copy of the data that can be refreshed and rolled back on demand. This platform will become only more powerful over time as we add new data sources, provide richer workflows targeting specific applications and use cases, and streamline the self service model. An enterprise data strategy without Delphix is just a path to more data anchors, necessitating suboptimal processes that continue to slow application development and your business.

Some companies go big or go home when it comes to hackathons – week long activities, physical hacks, etc. We wanted to preserve freedom but be a little more targeted. The directive was simple: spend a day doing something unrelated to your normal day job that in some way connects to the business. People volunteered ideas and mentorship ahead of time so that even the newest engineers could meaningfully participate. The result was a resounding success. Whether people were able to give a demo, sketch on a whiteboard, or just speak to their ideas and the challenges they faced, everyone pushed themselves in new directions and walked away having learned something through the experience.

The set of activities covered a wide swath of engineering, including:

• Using D3.js for visualizing analytics data
• “zero copy” iSCSI in illumos
• web portal for customer data analysis
• “zpool dump” to store pool metadata for offline zdb(1M) use
• Real time engineering dashboard to aggregate commits, bugs, reviews, and more
• “D++” DTrace syntactic sugar: function elapsed time, unrolled while loops, callers array
• Mobile application to monitor Delphix alerts and faults
• Global symbol tab completion for MDB
• Network performance tool
• Speeding up unit tests
• Browser usage analytics
• ‘zfs send’ to a POSIX filesystem
• BTrace++ (a.k.a. CATrace) to make java tracing safe and easy
• New V2P (virtual to physical) mechanisms in Delphix
• Tools to more easily deploy changes to VMs

For myself, I put together a prototype of a hosted SSH/HTTP proxy for use by our support organization. This was my first real foray into the world of true PaaS cloud software – running node.js, redis, and cloudAMQP in a heroku instance, and it’s been incredibly interesting to finally play with all these tools I’ve read about but never had a reason to use. I will post details (and hopefully code) once I get it into slightly better shape.

Only a fraction of these are really what I would consider a contribution to the product itself, which is where our initial trepidation around a hackathon went awry. No matter how complex your product or how high the barriers to entry , engineers will find a way to build cool things and try out new ideas in a hackathon setting. Everything that people did, from learning how to make changes to our OS to improving our quality of life as engineers to testing new product ideas, will provide real value to the engineering organization. On top of that, it was incredibly fun and a great way to get everyone working together in different ways.

It’s something we’ll certainly look at doing again, and I’d recommend that every company, organization, or group, find some way to get engineers together with the express purpose of working on ideas not directly related to their regular work. You’ll end up with some cool ideas and prototypes, and everyone will learn new things while having fun doing it.

Wikipedia defines systems software in the narrowest terms: the stuff that interacts with hardware. But that covers a tiny fraction of modern systems. So what is systems software? It depends on when you’re asking the question. At one time, the web server was the application; now it’s the systems software on which many web-facing applications are built. At one time a database was the application; now it’s systems software that supports a variety of custom and off-the-shelf applications. Before my time, shells were probably considered a bleeding edge application; now they’re systems software on which some of the lowest-level plumbing of modern operating systems are built.

Any layer on which people build applications of increasing complexity is systems software. Most software that endures the transition to systems software does so whether its authors intended it or not. People in the software industry often talk about standing on the shoulders of giants; the systems software accumulated and refined over decades are those giants.

Stable interfaces define systems software. The programs that consume those interfaces expect the underlying systems software to be perfect every time. Initially innovation might happen in the interfaces themselves — the concurrent model of Node.js is a great example. As software matures, the interfaces become commodified; innovation happens behind those stable interfaces. Systems is only “dead” at its edges. Interfaces might be flexible and well-designed, or sclerotic and poorly designed. Regardless, new or improved systems software can increase performance, enhance observability, or simply fit a different economic niche.

There are a few different types of systems software. First there’s supporting systems software, systems software written as necessary foundation for some new application. This is systems software written with a purpose and designed to solve an unsolved — or poorly solved — problem. Chronologically, examples include UNIX, compilers, and libraries like jQuery. You write it because you need it, but it’s solving a problem that’s likely not unique to your particular application.

Then there’s accidental systems software. Stick everything from Apache to Excel to the Bourne shell in that category. These didn’t necessarily set out to be the foundation on which increasingly complex software would be written, but they definitely are. I’m sure there were times when indoctrination into systems-hood was painful, where the authors wanted to change interfaces, but good systems software respects its consumers and carries them forward. Somewhat famously make preserved its arcane syntax because two consumers already existed. JavaScript started as a glorified web design tool; now it sits several layers beneath complex client-side applications. Even more recently, developers of Node.js (itself  JavaScript-based) changed a commonly used interface that broke many applications. Historical mistakes can be annoying to live with, but — as the Node.js team determined — compatibility trumps cleanliness.

The largest bucket is replacement systems software. Linux, Java, ZFS, and DTrace fall into this category. At the time of their development, each was a notionally compatible replacement for something that already existed. Linux, of course, reimplemented the UNIX design to provide a free, compatible alternative. Java set about building a better runtime (the stable interface being a binary provided to customers to execute) designed to abstract away the operating system entirely. ZFS represented a completely new way of thinking about filesystems, but it did so within the tight constraints of POSIX interfaces and storage hardware. DTrace added new and unique observability to most of the stable interfaces that applications build on.

Finally, there’s intentional systems software. This is systems software by design, but unlike supporting systems software, there’s no consumer. Intentional systems software takes an “if you build it, they will come” approach. This is highly prone to failure — without an existence proof that your software solves a problem and exposes the right interfaces, it’s very difficult to know if you’re building the right thing.

Why define these categories? Knowing which you’re working with can inform your decisions. If you’ve written accidental systems software that has had systems-ness thrust upon it, realize that future versions need to respect the consumers — or willfully cast them aside. When writing replacement systems software, recognize the constraints on the system, and know exactly where you’re constrained and where you can innovate (or consider if you don’t want to use the existing solution). If you’ve written supporting systems software, know that others will inevitably need solutions to the same problems. Either invest in maintaining it and keeping it best of breed; resign to the fact that it will need to be replaced as others invest in a different solution; or open source it and hope (or advocate) that it becomes that ubiquitous solution.

TL;DR?

What’s systems software? It is the increasingly complex, increasingly capable, increasingly diverse foundation on which applications are built. It’s that long and growing tail of the corpus of software at large. The interfaces might be static, but it’s a rich venue for innovation. As more and more critical applications build on an interface, the more value there is in improving the systems software beneath it. Systems software is defined by the constraints; it’s a mission and a mindset with unique challenges, and unique potential.

Software products are born of many different types of organizations, and even within similar organizations roles might have different names. Here’s a generic example with some names on the roles. New products and features start with product managers. Their role is to talk to customers and sales, educate themselves on the market, and determine the right product or enhancement. The handoff to engineering takes the form of a product requirements document (PRD) — it might sound like jargon, but the term is more or less universal. Software engineers execute against that PRD; QA engineers design tests that assert conformance to the PRD while developers steer the product from point A to point B as described by product management. Documentation writers and learning services take the PRD and the software to generate collateral that teaches customers how to use it. Product marketing makes the PowerPoints; sales presents them to customers.

And that’s where babies come from.

It’s not a perfect process, but it’s birthed many successful products. The shortcoming is that it can bury engineers under filters. Instead of learning about actual customer problems, engineers hear some processed form of what the customer said. Instead of raw critique of a new feature, engineers hear a softened and truncated form. The more technical the product and the market, the more those filters impede innovation and hamper the trajectory of the product.

The holistic engineer augments the jobs of those specialists, participating in each phase of product development. They join in those early conversations with customers, and share the responsibility of market comprehension. They partner to construct the requirements and design that those engineers will then implement. Along the way, engineers of course validate decisions with sales and customers — this is Agile writ large — but engineers also participate in the outbound documentation, training, and marketing activities.

From start to finish, the process is designed to fuel innovation by arming creative engineers with data and understanding. Customers often tell you what they want; they rarely tell you what they need. The more technical or disruptive the product, the more value an engineer has in those conversations, extracting the essence of the problem from the noise of preconceptions. The relationships with customer and the full context around their problems keeps engineers grounded as the inevitable gaps emerge in the product specification. Holistic engineers also help to educate the rest of the company and the rest of the world about new products and features. The process of explaining technology advises the way engineers design and build products. When we’re having a hard time explaining a feature or presenting a product, we need to revise our design. We’ve all heard engineering accused of building a product that was too complicated for the market, or engineers complain that a product failed because it was poorly marketed; both are symptoms of poor coordinating. Giving engineers holistic responsibility guards against this problem — if the product is failing the onus is on them to solve it.

Most important though are the feelings of ownership and agency associated with the whole-body approach. The holistic engineer is explicitly tasked with making a product succeed. That’s not to say that he or she goes it alone — specialists in all functions have major roles — rather the engineer is empowered to move the product through all stages; the other side of that coin is that there’s no opportunity to shrug off a responsibility as belonging to someone else.

In this model, everyone in every role at the company has the opportunity to engage in product management. Indeed, there’s still value in explicit product management. Channels of communication need to be easy and open for people with ideas to connect to people who will distill them into implementation. And it’s not enough to just create the right environment; hiring processes need to identify broad thinkers, and mentorship needs to nurture and reward holistic execution. Not every engineer can — or wants to — take on those additional responsibilities, but the best thrive with market and technology awareness, unencumbered by filters. They want responsibility and authority to make their ideas succeed.

The idea of the holistic engineer isn’t theoretical, it’s the model we stumbled into in the Solaris Kernel Group, and later implemented deliberately at Fishworks. There, a small team took on wide ranging responsibilities to build a product that’s now doing $400m/year for Oracle. At Delphix we’re again inculcating and hiring for holistic thinking. At all three I’ve seen engineers develop new products and features that address customer needs that would have otherwise never emerged from customers’ initial requests. It’s not easy to find the right kind of engineers, but if a company can empower the right engineers in the right ways — and they can live up to the responsibility — the payoff is a better product, built more efficiently.

There are variations with how these can be configured, and details in this table may vary. At the very least, this can serve as a checklist of characteristics to confirm, which may also be helpful if you are considering other technologies (eg, VMWare). Wikipedia also has a table of general characteristics.

The three in this table represent different types: OS Virtualization (Zones), and Hardware Virtualization of both Type 1 (Xen) and Type 2 (KVM) varieties.

The delivered performance of these is critical. In general, we use fast server hardware, 10 GbE networks, ZFS for all file systems, DTrace for performance analysis, and Zones wherever possible. We also performed our own port of KVM to illumos, and run KVM instances inside Zones, providing additional resource controls than can be applied, and improved security (“double-hulled virtualization”).

There are many characteristics I’d like to discuss in more detail. In this post, I’ll look at the I/O path (network, disk) and its overhead.

I/O Path

How does I/O differ between traditional Unix and Zones?

Performance is exactly the same – there is no overhead. Zones partition the OS in the same way that chroot isolates a process in the file system. There isn’t necessarily an extra layer in the software stack to make this work.

Now for Xen and KVM (simplified!):

GK is Guest Kernel, and domU on Xen runs the guest OS. Some of these arrows are indicating the control-path, where components inform each other, either synchronously or asynchronously, that more data is ready to transfer. The data-path may be implemented in some cases by shared memory and ring buffers. There are also different ways this can be configured. For example, Xen can use Isolated Driver Domains (IDD), or stub-domains, to run the I/O proxies in isolation.

With Xen, the hypervisor performs CPU scheduling for the domains, and then each domain has its own OS kernel for thread scheduling. The hypervisor supports different CPU scheduling classes, including Borrowed Virtual Time (BVT), Simple Earliest Deadline First (SEDF), and Credit-Based. The domains use the OS kernel scheduler, and whatever regular scheduler classes and policies they provide.

The extra overhead of multiple schedulers costs performance. Having multiple schedulers can also create complex issues with how they interact, adding CPU latency in the wrong situations. Debugging this can be very difficult, especially since the Xen hypervisor is running out of reach of the usual OS performance tools (try xentrace instead).

Sending I/O via the I/O proxy processes (which are usually qemu) involves context-switching and more overhead. There has been lots of work to minimize this, including shared memory transports, buffering, I/O coalescing, and paravirtualization drivers.

With KVM, the hypervisor is a kernel module (kvm) which is scheduled by the OS scheduler. It can be tuned using the usual OS kernel scheduler classes, policies and priorities. The I/O path takes fewer steps than Xen. (The original Qumranet KVM paper described it as five steps vs ten, although this description isn’t including paravirtualization.)

With Zones, there’s no comparison. The I/O path – which for high-speed networking is very sensitive – has none of these extra steps. While this has been well known in the Solaris community for years (Zones being a Solaris technology), and also the FreeBSD community (as Zones are based on FreeBSD jails), the Linux community is still learning about them and developing their own version: Linux Containers. Glauber Costa described them in his talk “The failure of Operating Systems, and how we can fix it” for Linuxcon 2012, and listed various use cases where KVM was currently used. Many of the use cases could be served by Containers, and didn’t actually need KVM.

Sometimes you (and our customers) really do need Hardware Virtualization, as their applications depend on a particular version of the Linux kernel, or Windows. We provide this with KVM (we’ve phased out Xen).

Internals

Some deeper insights into how these work (often using DTrace).

Network I/O, Zones

The following two stack traces show how a network packet is transmitted from the global zone (the host, which is the same as a bare-metal install) and from a zone (the guest):

  Global Zone:                            Zone:
  mac`mac_tx+0xda                         mac`mac_tx+0xda
  dld`str_mdata_fastpath_put+0x53         dld`str_mdata_fastpath_put+0x53
  ip`ip_xmit+0x82d                        ip`ip_xmit+0x82d
  ip`ire_send_wire_v4+0x3e9               ip`ire_send_wire_v4+0x3e9
  ip`conn_ip_output+0x190                 ip`conn_ip_output+0x190
  ip`tcp_send_data+0x59                   ip`tcp_send_data+0x59
  ip`tcp_output+0x58c                     ip`tcp_output+0x58c
  ip`squeue_enter+0x426                   ip`squeue_enter+0x426
  ip`tcp_sendmsg+0x14f                    ip`tcp_sendmsg+0x14f
  sockfs`so_sendmsg+0x26b                 sockfs`so_sendmsg+0x26b
  sockfs`socket_sendmsg+0x48              sockfs`socket_sendmsg+0x48
  sockfs`socket_vop_write+0x6c            sockfs`socket_vop_write+0x6c
  genunix`fop_write+0x8b                  genunix`fop_write+0x8b
  genunix`write+0x250                     genunix`write+0x250
  genunix`write32+0x1e                    genunix`write32+0x1e
  unix`_sys_sysenter_post_swapgs+0x14     unix`_sys_sysenter_post_swapgs+0x14

I spent (way) too much time double-checking that I didn’t switch these two stacks by accident, since they are identical. The stack on the right shows the same code path taken.

You could configure Zones in a way that it does have overhead, just like on a normal system. For example, enabling a firewall for network I/O, or mounting file systems via lofs instead of directly. These are optional, and may be worth the extra performance overhead for certain use cases.

Network I/O, KVM

The full code path for performing network I/O is complex.

The first part is the guest process writing to its driver. In this case, I’m demonstrating a Linux Fedora guest with DTrace-for-Linux, and tracing the paravirt driver:

guest# dtrace -n 'fbt:virtio_net:start_xmit:entry { @[stack(100)] = count(); }'
dtrace: description 'fbt:virtio_net:start_xmit:entry ' matched 1 probe
^C
[...]
              kernel`start_xmit+0x1
              kernel`dev_hard_start_xmit+0x322
              kernel`sch_direct_xmit+0xef
              kernel`dev_queue_xmit+0x184
              kernel`eth_header+0x3a
              kernel`neigh_resolve_output+0x11e
              kernel`nf_hook_slow+0x75
              kernel`ip_finish_output
              kernel`ip_finish_output+0x17e
              kernel`ip_output+0x98
              kernel`__ip_local_out+0xa4
              kernel`ip_local_out+0x29
              kernel`ip_queue_xmit+0x14f
              kernel`tcp_transmit_skb+0x3e4
              kernel`__kmalloc_node_track_caller+0x185
              kernel`sk_stream_alloc_skb+0x41
              kernel`tcp_write_xmit+0xf7
              kernel`__alloc_skb+0x8c
              kernel`__tcp_push_pending_frames+0x26
              kernel`tcp_sendmsg+0x895
              kernel`inet_sendmsg+0x64
              kernel`sock_aio_write+0x13a
              kernel`do_sync_write+0xd2
              kernel`security_file_permission+0x2c
              kernel`rw_verify_area+0x61
              kernel`vfs_write+0x16d
              kernel`sys_write+0x4a
              kernel`sys_rt_sigprocmask+0x84
              kernel`system_call_fastpath+0x16
             2015

That’s the Linux 3.2.6 network transmit path.

Control is passed by KVM to the qemu I/O proxy, which then transmits it on the host OS via the usual means (native driver). Here is the SmartOS stack in this case:

host# dtrace -n 'fbt::igb_tx:entry { @[stack()] = count(); }'
dtrace: description 'fbt::igb_tx:entry ' matched 1 probe
^C
[...]
              igb`igb_tx_ring_send+0x33
              mac`mac_hwring_tx+0x1d
              mac`mac_tx_send+0x5dc
              mac`mac_tx_single_ring_mode+0x6e
              mac`mac_tx+0xda
              dld`str_mdata_fastpath_put+0x53
              ip`ip_xmit+0x82d
              ip`ire_send_wire_v4+0x3e9
              ip`conn_ip_output+0x190
              ip`tcp_send_data+0x59
              ip`tcp_output+0x58c
              ip`squeue_enter+0x426
              ip`tcp_sendmsg+0x14f
              sockfs`so_sendmsg+0x26b
              sockfs`socket_sendmsg+0x48
              sockfs`socket_vop_write+0x6c
              genunix`fop_write+0x8b
              genunix`write+0x250
              genunix`write32+0x1e
              unix`_sys_sysenter_post_swapgs+0x149
             1195

Both of these stacks are pretty complex to begin with. Then there is the stuff in-between the Linux kernel and the illumos kernel, which gets even more complicated and involved. Basically, the paravirt code paths allow the two kernel stacks to make intimate love.

When Robert Mustacchi of Joyent last investigated these code paths in detail, he drew up some wonderful ASCII diagrams like the following:

/*
 *                  GUEST                        #       QEMU
 * #####################################################################
 *                                               #
 *    +----------+                               #
 *    |  start_  | (1)                           #
 *    |  xmit()  |                               #
 *    +----------+                               #
 *         ||                                    #
 *         ||       +-----------+                #
 *         ||------>|free_old_  | (2)            #
 *         ||------>|xmit_skbs()|                #
 *         ||       +-----------+                #
 *         \/                        (3)         #
 *    +---------+        +-------------+     + - #--- PIO write to VNIC
 *    |  xmit_  |------->|virtqueue_add|     |   #    PCI config space (6)
 *    |  skb()  |------->|_buf_gfp()   |     |   #
 *    +---------+        +-------------+     |   #
 *        ||                                 |   # +- VM exit
 *        ||         +- iff interrupts       |   # |  KVM driver exit (7)
 *        \/         |  unmasked (4)         |   # |
 *    +---------+    |     +-----------+(5)  |   # |  +---------+
 *    |virtqueue|----*---->|vp_notify()|-----*---#-*->| handle  | (8)
 *    |_kick()  |----*---->|           |-----*---#-*->|PIO write|
 *    +---------+          +-----------+         #    +---------+
 *        ||                                     #        ||
 *        ||   (13)                              #        ||
 *        **-----+ iff avail ring                #        \/      (9)
 *        ||       capacity < 20                 # +-----------------+
 *        ||       else return                   # |virtio_net_handle|
 *        ||                                     # |tx_timer()       |
 *        \/   (14)                              # +-----------------+
 *    +----------+                               #  ||
 *    |netif_stop|                               #  ||             (10)
 *    |_queue()  |                               #  ||   +---------+
 *    +----------+                               #  ||-->|qemu_mod_|
 *        ||                                     #  ||-->|timer()  |
 *        ||     (15)                 (16)       #  ||   +---------+
 *    +----------------+     +----------+        #  ||
 *    |virtqueue_enable|---->|unmask    |        #  ||              (11)
 *    |_cb_delayed()   |---->|interrupts|        #  ||  +------------+
 *    +----------------+     +----------+        #  |+->|virtio_     |
 *        ||                   ||                #  +-->|queue_set_  |
 *        || (18)              ||       (17)     #      |notification|
 *        ||  +-return   +-------------------+   #      +------------+
 *        ||  | iff ---->|check if the number|   #       |
 *        **--+ is false |of unprocessed used|   #       |  disable host
 *        ||             |ring entries is >  |   #       +- interrupts
 *        ||             |3/4s of the avail  |   #          (12)
 *        \/   (19)      |ring index - the   |   #
 *   +-----------+       |last freed used    |   #
 *   |free_old_  |       |ring index         |   #
 *   |xmit_skbs()|       +-------------------+   #
 *   +-----------+                               #
 *        ||                                     #
 *        ||     (20)                            #
 *        **-----+ iff avail ring                #
 *        ||       capacity is                   #
 *        ||       now > 20                      #
 *        \/                                     #
 *   +-----------+                               #
 *   |netif_start| (21)                          #
 *   |_queue()   |                               #
 *   +-----------+                               #
 *        ||                                     #
 *        ||                                     #
 *        \/  (22)               (23)            #
 *   +------------+      +----------+            #
 *   |virtqueue_  |----->|mask      |            #
 *   |disable_cb()|----->|interrupts|            #
 *   +------------+      +----------+            #
 *                                               #
 *                                               #
 */
		  Figure II: Guest / Host Packet TX Part 1

I included this diagram just to give you a sense of what happens. And that’s only part 1.

In brief, this uses ring buffers in shared memory to transfer the data, and a notification mechanism to inform when data is ready to transfer. When everything is working as intended, performance can be quite reasonable. It isn’t bare-metal fast (or Zones fast), but it isn’t terrible either. I’ve included some numbers later in this post.

The CPU overhead and reduced network performance is one thing. Another is the complexity this introduces, which hampers analysis and performance investigations. With Zones, there is one kernel TCP/IP stack to study and tune. Given its complexity, one is more than enough! With KVM, there are two different kernel TCP/IP stacks, plus KVM and paravirt. Investigating performance can take ten times longer, or so long that it becomes prohibitive. This is why I included “Observability” as a key characteristic in my comparison table. If it’s harder to see, it’s harder to tune.

Network I/O, Xen

The guest transmit and I/O proxy transmit stacks are the same. The in-between bit gets more complex. The hypervisor can’t be inspected using OS observability and debugging tools, since it’s running on bare-metal directly. There is xentrace, which looks pretty useful, as it instruments many event types in the Xen scheduler using static probes. (Even if it isn’t real-time and programmatic like DTrace, and, requires me to learn Yet Another Tracer.)

/proc, Zones

While the I/O path may have zero extra overhead by default, there are some overheads with OS Virtualization, usually for administration or observability, and not in the CPU or I/O “hot path”.

For example, a Zone cannot see other guests on the same system via /proc, as read by prstat(1M), top(1), etc. This is implemented in usr/src/uts/common/fs/proc/prvnops.c:

static int
pr_readdir_procdir(prnode_t *pnp, uio_t *uiop, int *eofp)
{
[...]
        /*
         * Loop until user's request is satisfied or until all processes
         * have been examined.
         */
        while ((error = gfs_readdir_pred(&gstate, uiop, &n)) == 0) {
                uint_t pid;
                int pslot;
                proc_t *p;

                /*
                 * Find next entry.  Skip processes not visible where
                 * this /proc was mounted.
                 */
                mutex_enter(&pidlock);
                while (n < v.v_proc &&
                    ((p = pid_entry(n)) == NULL || p->p_stat == SIDL ||
                    (zoneid != GLOBAL_ZONEID && p->p_zone->zone_id != zoneid) ||
                    secpolicy_basic_procinfo(CRED(), p, curproc) != 0))
                        n++;
[...]

The full list of processes are scanned, and just the local Zone’s processes are returned. This might sound a bit inefficient – couldn’t a linked list be added to proc_t so that Zone processes could be walked directly? Sure, but let’s be data driven.

Here’s the time to read /proc from a Zone by the prstat(1M) command, measuring using DTrace:

# dtrace -n 'fbt::pr_readdir_procdir:entry /execname == "prstat"/ {
    self->ts = timestamp; } fbt::pr_readdir_procdir:return /self->ts/ {
    @["ns"] = avg(timestamp - self->ts); self->ts = 0; }'
dtrace: description 'fbt::pr_readdir_procdir:entry ' matched 2 probes
^C
  ns                                                           544584

On average, that’s 544 us (microseconds).

Now with an extra 1000 processes in another Zone (which represents a typical dozen extra guests):

# dtrace -n 'fbt::pr_readdir_procdir:entry /execname == "prstat"/ {
   self->ts = timestamp; } fbt::pr_readdir_procdir:return /self->ts/ {
   @["ns"] = avg(timestamp - self->ts); self->ts = 0; }'
dtrace: description 'fbt::pr_readdir_procdir:entry ' matched 2 probes
^C
  ns                                                           594254

That added 50 us. For a /proc read – which shouldn’t be hot path. If it is, and 50 us matters, we can look at it then.

(While I was here, I also checked pidlock, which is, ahem, global. It is not currently a problem. This was also checked using DTrace.)

Network Throughput Results

I try not to share performance testing results without triple checking numbers, and I don’t have time for that right now (this was just supposed to be a quick blog post). I can share some previous numbers from a few months ago, when I did have the time to test carefully and perform Active Benchmarking.

This was a series of network throughput and IOPS tests using iperf, to test differences with default installations of 1 Gbyte SmartOS Zones and CentOS KVM instances (Xen wasn’t tested). The client and server were in the same datacenter, but not on the same physical host, so that the full network stack was used.

I should make it clear that these results are not a “max config” for our cloud. It’s a minimum config (1 Gbyte instances). If this were a marketing activity, I’d probably be compelled to test the max config. Which, for our SmartOS kernel, will be a lot of work, as it can drive multiple 10 GbE ports at line rate, which requires a lot of load-generating clients to perform.

For these results, YMMV based on workload, platform kernel type, and tuning. If you are to use them, think carefully about how they would apply, and to what degree. If you workload is CPU- or File System-bound, then you are probably better off testing their performance than using these network results.

A typical invocation on the server:

iperf -s -l 128k

And on the client:

iperf -c server -l 128k -P 4 -i 1 -t 30

The thread count (-P) was varied to investigate limits. The final result – the average over 30 seconds – was used.

Throughput

Searching for the highest Gbits/sec:

The peak Zones performance was 4.54 Gbits/sec with 4 threads. More threads hit the CPU caps for the 1 Gbyte (small) instance, with the CPU scheduler latency causing TCP breakdown. Larger SmartOS instances have higher CPU caps, and should be able to take performance further.

For the KVM test, these were default CentOS instances. I know that with a more modern Linux kernel with network stack tuning, we can improve throughput much further. The most I’ve reached is around 900 Mbits/sec for 1 VCPU KVM Linux (this was after we tuned KVM up from 110 Mbits/sec using a lot of DTrace analysis). Even at 900 Mbits/sec, it’s still 5x slower than Zones.

Note the “suspected limiter” column. This is essential to confirm what was actually tested, and comes from Active Benchmarking. It means I did performance analysis for every single result (including those not listed here to save room). In case you are wondering, it took a full day to perform all tests and analyze each result (again, using DTrace).

IOPS

Searching for the highest packets/sec:

In this case, Zones is 4x the packet rate of KVM. As before, the limiting factor becomes the cloud CPU limits, and I was only testing small 1 Gbyte servers. Bigger servers get higher CPU quotas, and all of these numbers should scale higher.

Conclusion

In this post, I summarized performance characteristics of three virtualization technologies – Zones, Xen, and KVM – and then investigated the I/O path in more detail. Zones add no overhead, whereas Xen and KVM do, which could limit network throughput to a quarter of what it could be.

By default we encourage customers to deploy on Zones, for reasons of performance, observability, and simplicity (debuggability). This may mean compiling their applications for SmartOS (our illumos-based OS which hosts the Zones) if they aren’t already in the repo. In cases where they absolutely must have Linux or Windows, and the applications can’t run elsewhere, then it’s Hardware Virtualization (KVM).

There are more performance characteristics to consider that I didn’t explore here, except briefly in the summary table, including how CPU allocation and VCPUs work, how memory allocation works and file system caches, and more. These could be topics for follow up posts.

This post wasn’t supposed to be so much about DTrace, but it’s the essential tool in so much of our high-performance work that it would be hard not to mention. We use it to improve overall performance for Zones and KVM, to track down latency outliers, explain benchmark results, study the effects of multi-tenancy, and to improve the performance of applications and the OS.

My background with ZFS includes leading various performance work for the world’s first ZFS-based storage appliance at Sun Microsystems and later Oracle, and now further analysis and tuning as Joyent’s lead performance engineer where we run a public cloud on ZFS. Given the risk of other tenants (noisy neighbors) interfering with your performance, I can’t imagine running a cloud on anything else. This talk includes the tools and tuning we use to make sure ZFS runs smoothly.

The video is on youtube:

The slides are available on slideshare and as a PDF:

During the talk I referenced several DTrace scripts for ZFS analysis. These are either from the File System chapter of the DTrace book, and are online at dtracebook.com, or from the fs directory of the dtrace-cloud-tools project, online at github/brendangregg.

Thanks to Deirdré for organizing a great conference, and filming and editing my video, and to all who spoke, attended, and helped out. For more about zfsday, see Adam’s summary. I’m looking forward to the next zfsday!

In this post, I’ll provide an example of a USE-based metric list for use within a SmartOS SmartMachine (Zone), such as those provided by the Joyent Public Cloud. These use the illumos kernel, and so this list should also be mostly relevant for OmniOS Zones, and to a lesser degree (due to some missing features) Solaris Zones. This is primarily intended for users of the zones. For the system administrators of the physical systems (via the Global Zone), also see the Solaris checklist, which has greater visibility.

Cloud limits (software resource controls) are listed first, as they are usually encountered before the physical limits.

Cloud Limits

These cover CPU, memory, disk I/O (file system), and network.

• For the Joyent Public Cloud, the CPU cap is the bursting limit. You are bursting when your CPU usage is over the kstat “caps::cpucaps_zone*:baseline”. If everyone bursts at the same time, your minimum usage should be the baseline, which is provided by the Fair Share Scheduler (FSS).
• sm-cpuinfo is from the smtools package.

Storage devices (disks) are not listed, since limits for storage I/O are imposed at the file system layer.

Physical Resources

Since Zones are OS-Virtualization (OS partitioning), the physical resources are not emulated or virtualized, and many of the observability tools will show you the entire physical system. This can be both good – you can really understand what’s going on, and confusing – why are the resources busy when my system is idle? (it’s someone else; you can’t see their process address space).

• For Joyent SmartMachines, Cloud Analytics (both the GUI and API) provide additional details from the physical system (global zone) that are not directly visible from within the SmartMachine.
• CPU utilization: a single hot CPU can be caused by a single hot thread, or mapped hardware interrupt. Relief of the bottleneck usually involves tuning to use more CPUs in parallel.
• nicstat may already be available in the SmartMachine; if not, there are both C and Perl versions (either may need a little tweaking to work properly).
• vmstat "r": this is coarse as it is only updated once per second.
• Memory capacity utilization: interpreting vmstat's "free" has been tricky across different Solaris versions (we documented it in the Perf & Tools book), due to different ways it was calculated, and tunables that affect when the system will kick-off the page scanner. It'll also typically shrink as the kernel uses unused memory for caching (ZFS ARC).
• Be aware that kstat can report bad data (so can any tool); there isn't really a test suite for kstat data, and engineers can add new code paths and forget to add the counters.

Software Resources

• plockstat often drop events due to load; I often roll my own to avoid this using the DTrace plockstat provider (examples in the DTrace book).
• File descriptor utilization: while other OSes have a system-wide limit, Solaris doesn't (at least at the moment, this could change; see my writeup about it).

What's Next

See the USE Method for the follow-up strategies after identifying a possible bottleneck. If you complete this checklist but still have a performance issue, move onto other strategies: drill-down analysis and latency analysis.

Also see the Solaris Performance Checklist if you have access to the physical host (global zone).

The video of the talk is on the LISA site, and the slides are below, also available as a PDF.

I’ve summarized the methodologies in the talk below.

Methodology Summaries

Blame-Someone-Else Anti-Method:

1. Find a system or environment component you are not responsible for
2. Hypothesize that the issue is with that component
3. Redirect the issue to the responsible team
4. When proven wrong, go to 1

Streetlight Anti-Method:

1. Pick observability tools that are
familiar
found on the Internet
found at random
2. Run tools
3. Look for obvious issues

Ad Hoc Checklist Method:

1. ..N. Run A, if B, do C

Problem Statement Method:

1. What makes you think there is a performance problem?
2. Has this system ever performed well?
3. What has changed recently? (Software? Hardware? Load?)
4. Can the performance degradation be expressed in terms of latency or run time?
5. Does the problem affect other people or applications
(or is it just you)?
6. What is the environment? What software and hardware is used? Versions? Configuration?

Scientific Method:

1. Question
2. Hypothesis
3. Prediction
4. Test
5. Analysis

Workload Characterization Method:

1. Who is causing the load? PID, UID, IP addr, …
2. Why is the load called? code path
3. What is the load? IOPS, tput, type
4. How is the load changing over time?

Drill-Down Analysis Method:

1. Start at highest level
2. Examine next-level details
3. Pick most interesting breakdown
4. If problem unsolved, go to 2

Latency Analysis Method:

1. Measure operation time (latency)
2. Divide into logical synchronous components
3. Continue division until latency origin is identified
4. Quantify: estimate speedup if problem fixed

USE Method:

For every resource, check:

1. Utilization
2. Saturation
3. Errors

Stack Profile Method:

1. Profile thread stack traces (on- and off-CPU)
2. Coalesce
3. Study stacks bottom-up