Node.js in production: runtime log snooping
This post also appeared on the Joyeur blog. This post is the one of several about how we operate Node.js in production at Joyent. Most of my experience comes from working on our Manta Storage Service, which currently comprises a few thousand Node processes across three datacenters. Logging is one of the most primitive but [...]
Read more...The USE Method: Unix 7th Edition Performance Checklist
PDP 11/70 front panel (similar to the 11/45) Out of curiosity, I’ve developed a USE Method-based performance checklist for Unix 7th Edition on a PDP-11/45, which I’ve been running via a PDP simulator. 7th Edition is from 1979, and was the first Unix with iostat(1M) and pstat(1M), enabling more serious performance analysis from shipped tools. [...]
Read more...In: performance, Unix, usemethod
The USE Method: FreeBSD Performance Checklist
In this post, I’ll provide an example USE Method-based performance checklist for FreeBSD, for identifying common bottlenecks and errors. This is intended to be used early in a performance investigation, before moving onto more time consuming methodologies. This should be helpful for anyone using FreeBSD, especially system administrators. This was developed on FreeBSD 10.0 alpha, [...]
Read more...In: FreeBSD, performance, usemethod
The USE Method: Mac OS X Performance Checklist
This is my example USE Method-based performance checklist for the Apple Mac OS X operating system, for identifying common bottlenecks and errors. This draws upon both command line and graphical tools for coverage, focusing where possible on those that are provided with the OS by default, or by Apple (eg, Instruments). Further notes about tools [...]
Read more...In: DTrace, Mac OS X, macosx, performance, usemethod
OpenZFS: the next phase of ZFS development
I’ve been watching ZFS from moments after its inception at the hands of Matt Ahrens and Jeff Bonwick, so I’m excited to see it enter its newest phase of development in OpenZFS. While ZFS has long been regarded as the hottest filesystem on 128 bits, and has shipped in many different products, what’s been most [...]
Read more...In: FreeBSD, illumos, Linux, macosx, MattAhrens, OpenZFS, ZFS
Delphix plus three years
Today marks my third anniversary of joining Delphix. Joining a startup, I knew there would be lots to learn — indeed there’s been a lesson nearly once-a-day. Here are my top three lessons from my first three years at a startup. Even if the points themselves should have been obvious to me, the degree of [...]
Read more...Happy 10th Birthday, DTrace!
Ten years ago this morning — at 10:27a local time, to be exact — DTrace was integrated. On the occasion of DTrace’s fifth birthday a half-decade ago, I reflected on much of the drama of the final DTrace splashdown, but much has happend in the DTrace community in the last five years; some highlights: DTrace [...]
Read more...Topics in post-mortem debugging
A couple of weeks ago, Joyent hosted A Midsummer Night’s Systems meetup, a fun event with talks ranging from Node.js fatwas to big data for Mario Kart 64. My colleague Jeremy Jones had recently done some amazing work, perfect for the meetup, but with his first child less than a day old, Jeremy allowed me [...]
Read more...Memory Leak (and Growth) Flame Graphs
Memory Leak Memory Flame Graph Your application memory usage is steadily growing, and you are racing against time to fix it. This could either be memory growth due to a misconfig, or a memory leak due to a software bug. For some applications, performance can begin to degrade as garbage collection works harder, [...]
Read more...In: flamegraphs, memory, performance, visualizations
Kartlytics: Applying Big Data Analytics to Mario Kart 64
This post also appears on the Joyeur blog. If you missed it, Joyent recently launched Manta, a web-facing object store with compute as a first-class operation. Manta makes it easy to crunch on Big Data in the cloud, and we’ve seen it used by both ourselves and others and others to solve real business problems [...]
Read more...