Debugging dynamic library dependencies on illumos

In this short follow-up to my post on illumos process tools, I’ll expand a bit on ldd and pldd, which print the dynamic linking dependencies of binaries and processes, respectively, and crle, which prints out the runtime linker configuration. These tools are available in most illumos distributions including SmartOS.

Understanding builds (and broken builds in particular) can be especially difficult. I hate running into issues like this one:

$ ffmpeg
ld.so.1: ffmpeg: fatal: libavdevice.so.53: open failed: No such file or directory
Killed

You can use ldd to see the dynamic library dependencies of a binary:

$ ldd $(which ffmpeg)
        libavdevice.so.53 =>     (file not found)
        libavfilter.so.2 =>      (file not found)
        libavformat.so.53 =>     (file not found)
        libavcodec.so.53 =>      (file not found)
        libswresample.so.0 =>    (file not found)
        libswscale.so.2 =>       (file not found)
        libavutil.so.51 =>       (file not found)
        libsocket.so.1 =>        /lib/libsocket.so.1
        libnsl.so.1 =>   /lib/libnsl.so.1
        libvpx.so.0 =>   /opt/local/lib/libvpx.so.0
        libm.so.2 =>     /lib/libm.so.2
        libbz2.so.0 =>   /opt/local/lib/libbz2.so.0
        libz.so.1 =>     /lib/libz.so.1
        libc.so.1 =>     /lib/libc.so.1
        libmp.so.2 =>    /lib/libmp.so.2
        libmd.so.1 =>    /lib/libmd.so.1
        libpthread.so.1 =>       /lib/libpthread.so.1
        librt.so.1 =>    /lib/librt.so.1
        libgcc_s.so.1 =>         /opt/local/lib/libgcc_s.so.1

In this case, the problem is that I installed ffmpeg into /usr/local, but the ffmpeg build appears not to have used the -R linker flag, which tells the runtime linker where to look dynamic libraries when the program is loaded. As a result, ffmpeg doesn’t know where to find its own libraries. If I set LD_LIBRARY_PATH, I can see that it will work:

$ LD_LIBRARY_PATH=/usr/local/lib ldd $(which ffmpeg)
        libavdevice.so.53 =>     /usr/local/lib/libavdevice.so.53
        libavfilter.so.2 =>      /usr/local/lib/libavfilter.so.2
        libavformat.so.53 =>     /usr/local/lib/libavformat.so.53
        libavcodec.so.53 =>      /usr/local/lib/libavcodec.so.53
        libswresample.so.0 =>    /usr/local/lib/libswresample.so.0
        libswscale.so.2 =>       /usr/local/lib/libswscale.so.2
        libavutil.so.51 =>       /usr/local/lib/libavutil.so.51
        ...

I resolved this by rebuilding ffmpeg explicitly with LDFLAGS += -R/usr/local.

ldd only examines binaries, and so can only print out dependencies built into the binary. Some programs use dlopen to open libraries whose name isn’t known until runtime. Node.js add-ons and Apache modules are two common examples. You can view these with pldd, which prints the dynamic libraries loaded in a running process. Here’s the output on a Node program with the node-dtrace-provider add-on:

$ pfexec pldd $(pgrep -x node)
32113:  /usr/local/bin/node /home/snpp/current/js/snpp.js -l 80 -d
/usr/local/bin/node
/lib/libz.so.1
/lib/librt.so.1
/lib/libssl.so.0.9.8
/lib/libcrypto.so.0.9.8
/lib/libdl.so.1
/lib/libsocket.so.1
/lib/libnsl.so.1
/lib/libkstat.so.1
/opt/local/lib/libstdc++.so.6.0.16
/lib/libm.so.2
/opt/local/lib/libgcc_s.so.1
/lib/libc.so.1
/usr/sfw/lib/libgcc_s.so.1
/home/dap/node_modules/dtrace-provider/build/Release/DTraceProviderBindings.node

If you want to see where the system looks for dynamic libraries, use crle, which prints or edits the runtime linker configuration:

$ crle
Configuration file [version 4]: /var/ld/ld.config
  Platform:     32-bit LSB 80386
  Default Library Path (ELF):   /lib:/usr/lib:/opt/local/lib
  Trusted Directories (ELF):    /lib/secure:/usr/lib/secure  (system default)
Command line:
  crle -c /var/ld/ld.config -l /lib:/usr/lib:/opt/local/lib

Of course, for more information on any of these tools, check out their man pages. They’re well-documented. If you find yourself debugging build problems, you’ll probably also want to know about nm, objdump, and elfdump, which are available on many systems and well documented elsewhere.

4 thoughts on “Debugging dynamic library dependencies on illumos

  1. Pingback: illumos» Blog Archive » illumos News: Sept 20, 2012

  2. “or edits the runtime linker configuration:”
    please never ever do this!!!!!

    Rebuild you stuff to use -R or if you have to set LD_LIBRARY_PATH (not global though)

  3. Holzi, what’s wrong with using crle to expand the linker’s search path? Often you’re using code someone else has built, so relinking isn’t an option, and using LD_LIBRARY_PATH is a horrible hack that always ends up in having to edit multiple .profiles or, worse, wrap binaries in shell scripts.

    IMO, crle(1) was invented for a reason, and it isn’t something to be avoided. It’s clean, global, and permanent.

  4. ever tried to fix a system where /lib and /usr/lib are missing in crle config? :) Which seems to happen quite often when people mess with it. So I preach not to mess with it.
    Was just speaking out a warning here.

    Sure LD_LIBRARY_PATH is a hack so is every other change to the linker config. Thats what -R was invented for.

    If I use code that someone else build I demand that it finds the libs it needs otherwise it’s broken software.

Comments are closed.