Busy Week: ZFS throttling, DTrace, node.js and Cloud Analytics

Last week was busy here at Joyent, with a number of important announcements. They included:

ZFS I/O Throttling

Bill Pijewski posted about ZFS I/O throttling, which he and Jerry Jelinek have been working on to manage disk throughput for Joyent smartmachines (based on Solaris zones). Filesystem I/O resource controls are crucial to allow multiple applications (and cloud tenants) to coexist peacefully, and to scale further on the same disk I/O resources.

It’s also amazing to see Bill and Jerry create, code, and demonstrate a solution in a small number of months after joining Joyent. This has been a long-standing issue, which I first wrote about in 2005 when I created the Zones Examples website (which I later moved to the solarisinternals.com wiki). I had included a section on resource controls:

    Resource Fine Control Coarse Control
    CPU shares with FSS Processor Sets
    Memory projects and rcapd Memory Sets
    Disk Size ZFS, SVM soft partitions Separate Disks or Slices
    Disk Throughput   Separate Disks or Controllers
    Network IPQoS Separate NICs
    Swap   Swap Sets

Features in italic don’t exist yet, but these or similar such features may appear in future updates to Solaris 10.

Apart from the missing pieces in italics, I had nothing at all for fine control of disk throughput (or swap). ZFS I/O throttling will give us such a fine control. It’s also latency-based, not throughput-based, which makes sense since throughput (Mbytes/sec) makes for a poor resource control metric (random vs. sequential I/O is often not equal, even if their Mbytes/sec are).

Bill also mentioned he’d write more about a new tool he wrote called “vfsstat”, and included some sample output:

[root@z02 ~]# vfsstat -M 5
[...]
   r/s    w/s   Mr/s   Mw/s wait_t ractv wactv read_t writ_t  %r  %w zone
 149.8    0.0    1.2    0.0    0.0   3.0   0.0   20.0    0.0  99   0 z02
   r/s    w/s   Mr/s   Mw/s wait_t ractv wactv read_t writ_t  %r  %w zone
 153.3    0.0    1.2    0.0    0.0   3.0   0.0   19.6    0.0 100   0 z02
   r/s    w/s   Mr/s   Mw/s wait_t ractv wactv read_t writ_t  %r  %w zone
 156.8    0.0    1.2    0.0    0.0   3.0   0.0   19.1    0.0  99   0 z02
   r/s    w/s   Mr/s   Mw/s wait_t ractv wactv read_t writ_t  %r  %w zone
 162.3    0.0    1.3    0.0    0.0   3.0   0.0   18.5    0.0 100   0 z

This looks like iostat (which summarizes data about as well as you can in one text line) but is measuring at the VFS interface. This is important for understanding the real pain applications feel when using the filesystem, despite what the disks are doing.

DTrace probes for node.js

Next, Robert Mustacchi posted about DTrace probes for node v0.4.x, adding DTrace USDT probes for HTTP server and client requests, socket operations and garbage collection. The probe list shows request/response and start/done pairs that can be traced for calculating latency:

rm@devel ~ $ dtrace -l -n node*:::
   ID   PROVIDER      MODULE                          FUNCTION NAME
 4331  node12090        node _ZN4node14dtrace_gc_doneEN2v86GCTypeENS0_15GCCallbackFlagsE gc-done
 4332  node12090        node _ZN4node15dtrace_gc_startEN2v86GCTypeENS0_15GCCallbackFlagsE gc-start
 4333  node12090        node _ZN4node26DTRACE_HTTP_CLIENT_REQUESTERKN2v89ArgumentsE http-client-request
 4334  node12090        node _ZN4node27DTRACE_HTTP_CLIENT_RESPONSEERKN2v89ArgumentsE http-client-response
 4335  node12090        node _ZN4node26DTRACE_HTTP_SERVER_REQUESTERKN2v89ArgumentsE http-server-request
 4336  node12090        node _ZN4node27DTRACE_HTTP_SERVER_RESPONSEERKN2v89ArgumentsE http-server-response
 4337  node12090        node _ZN4node28DTRACE_NET_SERVER_CONNECTIONERKN2v89ArgumentsE net-server-connection
 4338  node12090        node _ZN4node22DTRACE_NET_SOCKET_READERKN2v89ArgumentsE net-socket-read
 4339  node12090        node _ZN4node23DTRACE_NET_SOCKET_WRITEERKN2v89ArgumentsE net-socket-write
 4340  node12090        node _ZN4node21DTRACE_NET_STREAM_ENDERKN2v89ArgumentsE net-stream-end

… It was for this work that Robert found and fixed a DTrace bug regarding -L.

Cloud Analytics

Lastly, Dave Pacheco posted about Cloud Analytics which launched on Joyent’s no.de cloud services. Later that night I gave a short talk on this at the node.js community event with Bryan Cantrill:

Which included demonstrating node.js HTTP latency heatmaps, using Robert’s DTrace probes:

… and included filesystem latency heat maps:

In a way, this is the heatmap version of Bill’s vfsstat.

At the same community event, Ryan Dahl talked about what’s new in node.js 0.4. node.js is powering the Cloud Analytics you see above and is doing a terrific job, especially considering this is aggregating data across multiple cloud smartmachines and displaying the summary in realtime.

Print Friendly
Posted on March 8, 2011 at 3:18 am by Brendan Gregg · Permalink
In: Uncategorized