Busy Week: ZFS throttling, DTrace, node.js and Cloud Analytics
Last week was busy here at Joyent, with a number of important announcements. They included:
ZFS I/O Throttling
Bill Pijewski posted about ZFS I/O throttling, which he and Jerry Jelinek have been working on to manage disk throughput for Joyent smartmachines (based on Solaris zones). Filesystem I/O resource controls are crucial to allow multiple applications (and cloud tenants) to coexist peacefully, and to scale further on the same disk I/O resources.
It’s also amazing to see Bill and Jerry create, code, and demonstrate a solution in a small number of months after joining Joyent. This has been a long-standing issue, which I first wrote about in 2005 when I created the Zones Examples website (which I later moved to the solarisinternals.com wiki). I had included a section on resource controls:
Resource Fine Control Coarse Control CPU shares with FSS Processor Sets Memory projects and rcapd Memory Sets Disk Size ZFS, SVM soft partitions Separate Disks or Slices Disk Throughput Separate Disks or Controllers Network IPQoS Separate NICs Swap Swap Sets Features in italic don’t exist yet, but these or similar such features may appear in future updates to Solaris 10.
Apart from the missing pieces in italics, I had nothing at all for fine control of disk throughput (or swap). ZFS I/O throttling will give us such a fine control. It’s also latency-based, not throughput-based, which makes sense since throughput (Mbytes/sec) makes for a poor resource control metric (random vs. sequential I/O is often not equal, even if their Mbytes/sec are).
Bill also mentioned he’d write more about a new tool he wrote called “vfsstat”, and included some sample output:
[root@z02 ~]# vfsstat -M 5 [...] r/s w/s Mr/s Mw/s wait_t ractv wactv read_t writ_t %r %w zone 149.8 0.0 1.2 0.0 0.0 3.0 0.0 20.0 0.0 99 0 z02 r/s w/s Mr/s Mw/s wait_t ractv wactv read_t writ_t %r %w zone 153.3 0.0 1.2 0.0 0.0 3.0 0.0 19.6 0.0 100 0 z02 r/s w/s Mr/s Mw/s wait_t ractv wactv read_t writ_t %r %w zone 156.8 0.0 1.2 0.0 0.0 3.0 0.0 19.1 0.0 99 0 z02 r/s w/s Mr/s Mw/s wait_t ractv wactv read_t writ_t %r %w zone 162.3 0.0 1.3 0.0 0.0 3.0 0.0 18.5 0.0 100 0 z
This looks like iostat (which summarizes data about as well as you can in one text line) but is measuring at the VFS interface. This is important for understanding the real pain applications feel when using the filesystem, despite what the disks are doing.
DTrace probes for node.js
Next, Robert Mustacchi posted about DTrace probes for node v0.4.x, adding DTrace USDT probes for HTTP server and client requests, socket operations and garbage collection. The probe list shows request/response and start/done pairs that can be traced for calculating latency:
rm@devel ~ $ dtrace -l -n node*::: ID PROVIDER MODULE FUNCTION NAME 4331 node12090 node _ZN4node14dtrace_gc_doneEN2v86GCTypeENS0_15GCCallbackFlagsE gc-done 4332 node12090 node _ZN4node15dtrace_gc_startEN2v86GCTypeENS0_15GCCallbackFlagsE gc-start 4333 node12090 node _ZN4node26DTRACE_HTTP_CLIENT_REQUESTERKN2v89ArgumentsE http-client-request 4334 node12090 node _ZN4node27DTRACE_HTTP_CLIENT_RESPONSEERKN2v89ArgumentsE http-client-response 4335 node12090 node _ZN4node26DTRACE_HTTP_SERVER_REQUESTERKN2v89ArgumentsE http-server-request 4336 node12090 node _ZN4node27DTRACE_HTTP_SERVER_RESPONSEERKN2v89ArgumentsE http-server-response 4337 node12090 node _ZN4node28DTRACE_NET_SERVER_CONNECTIONERKN2v89ArgumentsE net-server-connection 4338 node12090 node _ZN4node22DTRACE_NET_SOCKET_READERKN2v89ArgumentsE net-socket-read 4339 node12090 node _ZN4node23DTRACE_NET_SOCKET_WRITEERKN2v89ArgumentsE net-socket-write 4340 node12090 node _ZN4node21DTRACE_NET_STREAM_ENDERKN2v89ArgumentsE net-stream-end
… It was for this work that Robert found and fixed a DTrace bug regarding -L.
Cloud Analytics
Lastly, Dave Pacheco posted about Cloud Analytics which launched on Joyent’s no.de cloud services. Later that night I gave a short talk on this at the node.js community event with Bryan Cantrill:
Which included demonstrating node.js HTTP latency heatmaps, using Robert’s DTrace probes:
… and included filesystem latency heat maps:
In a way, this is the heatmap version of Bill’s vfsstat.
At the same community event, Ryan Dahl talked about what’s new in node.js 0.4. node.js is powering the Cloud Analytics you see above and is doing a terrific job, especially considering this is aggregating data across multiple cloud smartmachines and displaying the summary in realtime.
