JavaScript Lint on SmartOS


Photo by Sam Fraser-Smith

Back at Fishworks, we used a tool called JavaScript Lint (JSL) for static code analysis. You may know that lint was originally written to identify potential semantic problems in C code, like use of uninitialized variables or blocks of unreachable code. Lint warnings are usually static checks that could reasonably have been compiler warnings, but for whatever reasons those checks didn’t make it into the compiler.

JSL helps us catch similar errors in JavaScript: undeclared variables, variables hiding other variables in the same scope, etc. There exist several JavaScript linters out there, including Crockford’s JSLint and Google’s Closure Linter. But there are two relatively unique properties about JSL:

  • It does not conflate style with lint. Style refers to arbitrary code formatting rules (like leading whitespace rules). Lint refers to actual program correctness issues (like missing “break” statements inside a switch). The line is certainly fuzzy, as in the case of JavaScript semicolon style, but that’s why:
  • It’s highly configurable. Each individual warning can be turned on or off, and warnings can be overridden for individual lines of code. This is essential for cases where potentially dangerous behavior is being deliberately used (carefully, of course).

We’ve been using JSL on Cloud Analytics since day 1, but until recently we were using a hacked-up build I created back in November just to make forward progress. As we started using it more, it became clear that we needed to be able to build JSL reliably, which was not trivial on SmartOS because the old version of SpiderMonkey that JSL bundles doesn’t build on Solaris 10 or later out of the box. I worked out how to build it (see below), but in doing so I decided it wasn’t worth maintaining the complexity of the existing build system.  JSL hasn’t been changed much in the last many months, so I created a github fork of JSL where I removed everything that clearly wasn’t necessary for JSL and replaced the whole build system with a couple of Makefiles. The result is much less portable, but it builds on SmartOS and I expect it can be made to build on MacOS (note: see update below) and Linux with few modifications. If you want to build the existing JavaScript Lint subversion tree on Illumos, here’s what you have to do:

  • Since Sun Studio is no longer available, you’ll want to build with gcc.
  • Remove the file spidermonkey/src/lock_SunOS.s.  According to the comments, this file is only needed with Sun Studio, and it won’t build with the GNU assembler.  If you don’t remove the file, it will be picked up by a Makefile wildcard (not the line in Makefile.ref that appears to refer to it!).
  • Copy the file below into spidermonkey/src/config/SunOS5.11_i86pc.mk.
  • In the root, run “python setup.py build”.
  • This should work except for the very last step in the build.  The build system runs something like this:
    gcc -shared build/temp.solaris-2.11-i86pc-2.4/javascriptlint/pyspidermonkey/pyspidermonkey.o build/temp.solaris-2.11-i86pc-2.4/javascriptlint/pyspidermonkey/nodepos.o -Lbuild/spidermonkey -ljs -o build/lib.solaris-2.11-i86pc-2.4/javascriptlint/pyspidermonkey.so

    On my system, this produces hundreds of linker errors because gcc is trying to use the GNU linker instead of the OS linker. If you replace the “gcc” in that line with “ld” and run it by hand, it should work.

Here’s the contents of SunOS5.11_i86pc.mk:

#
# Config stuff for SunOS5.11
#

AS = as
CC = gcc
CCC = g++
CFLAGS +=  -Wall -Wno-format
RANLIB = echo
OS_CFLAGS = -DXP_UNIX -DSVR4 -DSYSV -DSOLARIS -DHAVE_LOCALTIME_R
OS_LIBS = -lsocket -lnsl -ldl
HAVE_PURIFY = 1
MKSHLIB = $(LD) -G
# Use the editline library to provide line-editing support.
JS_EDITLINE = 1

Of course, these instructions are very specific to the build environment, so YMMV. It took me a while to figure out the right settings in SunOS5.11_i86pc.mk, so I wanted to make that available in case anyone else is trying to build JSL on SmartOS, Illumos or other Solaris-based systems.  That said, if you don’t care about remaining close to the original source, you may as well just use my fork on github. Even if it doesn’t build out of the box in your environment, the Makefiles should be far easier to understand and modify.

Update: The github fork now builds on MacOSX, too, though you need to install Python first because the one shipped with OSX doesn’t include headers.

Distributed Web Architures @ SF Node.js Meetup

At the Node Meetup here at Joyent‘s offices a few weeks ago I gave a brief talk about Cloud Analytics as a distributed web architectureMatt Ranney of Voxer and Curtis Chambers of Uber also spoke about their companies’ web architectures. Thanks to Jackson for putting the videos together. All around it was a great event!

Presenting at OSCON Data 2011

Brendan‘s and my proposal for OSCON Data 2011 has been accepted! The talk is called “Design and Implementation of a Real-Time Cloud Analytics Platform“. We’ll dive into the implementation of Cloud Analytics with particular focus on the design considerations that allow it to work in real-time on a large production cloud.

We’ll also cover some real-world case studies of people using CA to understand application behavior. If you’ve used no.de‘s Analytics to look at your application, we’d love to hear from you! Leave a comment, tweet @dapsays, or email dap at joyent.

Example: HTTP request latency and garbage collection

A few weeks ago I posted about Cloud Analytics on no.de. I described the various metrics we included in the initial launch and how they’re relevant for Node applications. Today I’ve got a small example using HTTP request latency and garbage collection. I wrote a small HTTP server in Node.js that simply echoes all requests, but in doing so deliberately creates a bunch of garbage. That is, it allocates a bunch of extra memory while processing each request but doesn’t save it anywhere. The goal is just to observe garbage collection in action, and here’s what it looks like when hammered with an HTTP client (also written in Node.js):

The dark band at the bottom of the first heatmap indicates that most of the HTTP requests were serviced within a few milliseconds. But there are a few outliers: some HTTP requests took much longer to complete (around 30ms). I’ve highlighted the different request URLs in different colors so we can see that there’s no obvious correlation between latency and URL. Looking at the second heatmap, we can see that these slow requests correlate almost exactly with GC events. Since we know that V8 suspends Javascript while running GC, we can conclude that GC is the source of the extra HTTP request latency. If this weren’t a simple demo, the next step would be to understand why the application is generating so much garbage.

The other interesting bit about this data is that a line graph of average latency would not have revealed the outliers at all. Without seeing the whole distribution, you might think performance was great and not realize that a small percentage of users’ requests were significantly slower than the rest, let alone be able to understand why.

While pretty simple, this example shows how you can use CA to observe a performance problem and quickly test a couple of hypotheses. We’d love to hear about your real-world experiences with CA. Feel free to share below.