As Adam noted in the Solaris Top 11-20, watchpoints are now much more useful in Solaris 10. Before I go into specific details regarding Solaris 10 improvements, I thought I’d give a little technical background on how watchpoints actually work. This will be my second highly technical entry in as many days; in my next post I promise to tie this into some real-world applications and noticeable improvements in S10.
The idea of watchpoints has been around for a long time. The basic idea is to allow a debugger to set a watchpoint on a region of memory within a process. When that region of memory is accessed, the debugger is notified in order to take appropriate action. This typically serves two purposes. First, it’s useful for interactive debuggers when determining when a region of memory gets modified. Second, it can be used as a protection mechanism to avoid buffer overflows (more on this later).
As with most modern operating systems, Solaris implements a virtual memory system. A complete explanation of how virtual memory works is beyond the scope of a single blog post. The simplest way to explain it is that each process refers to memory by a virtual address, which corresponds to a physical piece of memory. Each piece of memory is called a page, which can be either mapped (resident in RAM) or unmapped (possibly stored on disk). The operating system has control over when and how pages get mapped in or out of memory. If a program tries to access memory that is unmapped, the OS will map in the necessary pages as needed. Once pages are mapped, accesses will be handled directly in hardware until the OS decides to unmap the memory1. There are many benefits of this, including the ability for processes to see a unified flat memory space, inability to access other processes’ memory, and the ability to store unused pages on disk until needed.
To implement watchpoints, we need a way for the operating system to intercept accesses to a specific virtual page within a process. If we leave pages mapped, then accesses will be handled in hardware and the OS will have no say in the matter. So we keep pages with watchpoints unmapped until they are actually accessed. When the process tries to read/write/execute from the watched page, the OS gets notified via a trap2. At this point, we temporarily map in the page and single step over the instruction that triggered the trap. If the instruction touches a watched area (note that there can be more than one watched area within a page), then we notify the debugger through a SIGTRAP signal. Otherwise, the instruction executes normally and the process continues.
Things become a little more complicated in a multithreaded program. If we map in a page for a single thread, then all other threads in the process will be able to access that memory without OS intervention. If another thread accesses the memory while we’re stepping over the instruction, we can miss triggering a watchpoint. To avoid this, we have to stop every thread in the process while we step over the faulting instruction. This can be very expensive; we’re looking into more efficient methods. I won’t spend too much time discussing how the debugger communicates with the OS when setting and reacting to watchpoints. Most of this information can be found in the proc(4) manpage.
With my next post I’ll examine some of the specific enhancements made to watchpoints in Solaris 10.
1This is obviously a very simplistic view of virtual memory. Curious readers should try a good OS textbook or two for more detailed information.
2Traps are quite an interesting subject by themselves. On Solaris SPARC, you can see what traps are ocurring with the very cool trapstat(1M) utility that Bryan wrote.