Over the years, we have been keeping an eye on the LSOF (LiSt Open Files) utility written by Vic Abell. It’s a well respected utility that provides indispensible information that cannot be found any other way. We have tried to emulate some aspects of lsof, but we still fall short in several categories. As part of the RAS group, I’m curious to know what observability features our system is lacking. This is a huge open-ended question, so I’m trying to limit the scope by performing a direct comparison to lsof. If you’re not familiar with lsof, then consider it a black box that will answer any questions about open files or connections on your system.
We have started to get better at this. The fuser(1M) and pfiles(1) commands are a decent start, especially since pfiles started having path information in Solaris 10. These two utilities still fail to address one major area of concern: systemic problems. We can tell you who has a given file open, and we can tell you which files a process has open, but we can’t tell you who has NFS files open on your system or which processes are bound to a particular port. Running pfiles on every process in the system is not an acceptable solution, if only because it involves stopping every single process on your system. Dtrace can identify problems as they occur, but it can’t give you a snapshot of the current state of the system.
I am not looking to clone lsof, but I do think we should be able to answer some of the questions that our customers are asking. This is not the first time this has come up within the Solaris group, but it is something I want to explore as we look beyond Solaris 10. I don’t need to know all of lsof’s features; I can read the manpages easily enough. So my question to you admins and developers out there is this: What specific questions does lsof allow you to answer that you otherwise cannot determine with the stock Solaris tools, and how does it make your life easier?
4 Responses
I use lsof -i all the time on Solaris 8/x86 to find out what process is listening on what port.
The single utility I would most like to have is a versionof top/prstat that lists processes sorted by I/Os to a physical device. I read your past article and I realize it can be done with Dtrace, but you should still package this as a standalone utility for usability’s sake.
BTW, Windows NT’s performance monitor has the ability to list disk I/O (reas/write/other bytes/ops per second) per procss, which is more than Solaris gives me today.
lsof -i . You happen to need to know who is using a port. Especially if many users install Apaches / Tomcats on a single server. This is the main reason for which I installed lsof (I know it can do much more but that’s what I use it for).
And yes, it is annoying to have to install a third party package to get this. BTW: Linux netstat can print this info so another reason why you should make it available
And yes again the new pfiles path information is useful.
As for I/O I think the yet to be released DTrace io provider will do much more than NT can get close to. You want a package ? I’m sure there will be DTrace recipes available for whatever you would like to know.
True, a utility to gather this info for you out of the box wouldn’t heart either.
A 3rd vote for something like linux’s netstat -p. That would really alleviate 90% of my use of lsof (or scary pfiles solutions).
These are not so directly related to lsof, but they would be useful.
Some sort of per-process vmstat/mpstat/iostat would be handy to see which process is page faulting at a glance, or how much CPU a particular process is using across all the CPU’s.
Most of my debuggng happens with truss, but that only provides so much visability. If the it’s application problem (like in the VM or some 3rd party thing that I don’t have much debugging info for) though, truss output is of course not good enough. Sounds like DTrace can do some of this, and I’m looking forward to it.
However, my company’s official Solaris image is 8 right now, and probably won’t be 10 for years. Just like the new manageability stuff in Java 1.5 (sorry, 5.0), it all sounds great, but the fact that it’s years away is a bummer.
I’m guessing that something like FreeBSD’s sockstat utility would be most useful, judging by the above comments. Certainly, it’s saved me a couple of times.
http://www.freebsd.org/cgi/man.cgi?query=sockstat&manpath=FreeBSD+4.10-RELEASE&format=html
What I find really annoying about Linux’s “netstat -p” is that it needs root privs, which is very irritating.