Dynamic Tracing Guide
Chapter 42

Tracing Hardware Virtual Machines

Traditionally understanding what is going on inside of a Hardware Virtual Machines has been a black box. With DTrace you can begin to open up that black box and understand what is going on. This chapter focuses on debugging virtual machines that utilize extensions to the x86 instruction set architecture and assumes some familiarity with hardware virtual machine architecture.

Virtualization Background

In this chapter various terms have specific meanings that they don't in other contexts. These terms are:

  • host refers to the operating system that is doing the virtualization.

  • guest refers to the operating system instance that is being subject to virtualization.

  • Virtual Machine Monitor (VMM) refers to the piece of software that runs inside of the host operating system and provides the virtualization framework for guest oprating systems. It is composed of both a kernel module and a user land process. The VMM is responsible for emulating hardware such as the i8254, i8259, PCI buses, ethernet and disk controllers.

  • Virtual CPU (VCPU) refers to a virtualized instance of a CPU that the guest operating system is exposed to and interacts with.

  • Virtual Machine Control Structure (VMCS) is a structure that describes the operation of a virtual machine. There is one VMCS per logical processor per guest.

vmregs[] Array

The vmregs[] array enables you to access the registers of the currently active VMCS. These registers allow you to see the guest virtual machine state. The following tables list a subset of the indices into the vmregs[] array corresponding to each of the supported CPU architecture extensions. The full list is available in /usr/lib/dtrace/regs.d.

Intel VMX vmregs[] Constants
















%es base


%es base


%es base


%es base


%es base

The vmregs[] can be combined with other providers to provide insight into what a guest virtual machine without having to access it. The following script profiles what is running on a per-guest basis using the value of %cr3. On x86 systems, that register contains the root of the page table and thus is unique on a per-process basis.

/execname == "qemu-kvm"/
	@[pid, vmregs[VMX_GUEST_CR3]] = count();

	printa("%05d 0x%x %@d\n", @);

Running this yields the following output:

65197 0xffffff02d5ed3000 1
65197 0xffffff02d5ee7000 2
65197 0x77bda000 896

Looking at output like this indicate to us that we primariliy had one process that was accessing running on CPU inside of the guest during the ten second windows.