mpstat videos
I’ve been filming short videos on performance topics, and previously posted a series of videos on vmstat. Here I discuss mpstat(1M), which is useful for investigating CPU performance per CPU.
The videos that follow are for mpstat(1M) on Solaris-based operating systems. All are available here: mpstat videos, and embedded below:
mpstat: Key Fields
mpstat: All the Fields
mpstat: Digging Deeper
In the digging deeper video I show how investigate the mpstat statistics with other tools to identify their origin, including:
- CPU cross calls (“xcal”): using DTrace.
- Kernel mutex spins (“smtx”): using lockstat(1M).
- Interrupts (“intr”/”ithr”): using intrstat(1M) and mdb(1).
- Individual hot CPU in system time (“sys”): using prstat(1M) and DTrace.
Most of this deeper investigation requires system administrator privileges (typically root) to run the tools, and access to various DTrace providers. Many of these capabilities are not currently available inside virtualized contexts including Solaris Zones and Joyent’s SmartOS. These users still benefit from these capabilities since the global system administrators can investigate and solve issues on their behalf. Be aware that mpstat(1M) is currently system-wide, so any issues shown may be hurting other tenants and not the zone you are looking at.
For more details on mpstat and DTrace, see their man pages, source code behind the metrics, and the books: Solaris Performance and Tools and DTrace.
In: Performance · Tagged with: mpstat, video
on June 20, 2011 at 7:00 am
Permalink
Hi Brendan,
I’m trying to run lockstat on my Joyent Smartmachine as I follow the last video. I get the following running that command:
sudo lockstat -n1000000 -s8 -o out.locks sleep 5
lockstat: can’t load kernel symbols: No such file or directory
on June 20, 2011 at 7:05 am
Permalink
intrstat doesn’t seem to work on Joyent Smartmachines either:
sudo intrstat 1
intrstat: invalid program: probe description :::interrupt-start does not match any probes
Any chance these are global zone only commands? That’s cool, it would just be nice to know which I can and can’t expect to run on Joyent Smartmachines.
Cheers,
GF
on June 20, 2011 at 4:20 pm
Permalink
Yes, sorry about that, they are currently global only commands since they inspect the kernel (which has access to other tenant data). If it really looked like there was an interrupt or kernel lock issue, then the Joyent operation team can run these commands to see what’s up; and if that became common, we’d see if it could be made easier – such as by adding that instrumentation to Cloud Analytics for SmartMachine users to run themselves.
on June 24, 2011 at 8:50 am
Permalink
[...] as mpstat(1M) for examining activity [...]