Kernel Debugging with VirtualBox

Virtualization readily lends itself to debugging of low-level code that is difficult to analyze in conventional environments. It is also convenient for kernel debugging which would otherwise require two separate systems or at least a separate serial terminal.

OS/2

Setting up kernel debugging in OS/2 is quite simple: the standard retail kernel (OS2KRNL) on the target system (i.e. system being debugged) is overwritten by the debug kernel, typically distributed as OS2KRNLB, which includes the kernel debugger (KD). It is highly advisable, though optional, to also copy selected symbol (.SYM) files alongside their corresponding binaries. The symbol files are typically included on OS/2 installation media and available separately for fix packs.

The OS/2 kernel debugger assumes that a terminal is attached to a serial port, normally COM1. A terminal normally need not be attached to boot the system, but if a crash occurs, OS/2 will stop and wait for input from the debug terminal.

In VirtualBox, setting up a serial terminal is very simple. The host pipe mode of serial port emulation is usually suitable. It is possible to connect two VMs via emulated serial ports over a host pipe; such a setup is required for Windows NT (WinDbg) or Windows 98 (Rterm98) kernel debugging. However, for debugging OS/2 or Windows 3.x VMs a second VM may not be necessary.

On Windows hosts, everyone’s favorite terminal program PuTTY can be used. Assuming a VM uses a serial port attached to a host pipe called \\.\pipe\os2kd, PuTTY should be started as follows:

putty -serial \\.\pipe\os2kd

When a VM with kernel debugging enabled boots up, some amount of debug output should be visible on the debug terminal. Execution can be typically interrupted with Ctrl+C (in the debug terminal, not the target!). The g command continues execution. The beginning of a debug session might look as follows:

In the above sample, there are no symbols near the location where the debugger stopped. It turns out that the E10h selector was mapping the BIOS ROM and OS/2 called the APM idle function which executed a HLT instruction just prior to the location where the debugger stopped.

The OS/2 kernel debugger contains basic command reference accessible with ? and .? commands. OS/2 debugging is very extensively documented in the excellent four-volume OS/2 Debugging Handbook, still available from IBM.

Windows 3.x

Kernel debugging of Windows 3.x guests is only slightly more complicated. Rather than containing a built-in kernel debug facility, Windows 3.x uses a separate WDEB386.EXE debugger. WDEB386 is a successor to Microsoft’s older SYMDEB debugger and in many ways very similar to the OS/2 kernel debugger (which likewise can’t deny SYMDEB ancestry).

As with OS/2, symbol files are optional but highly recommended. For debugging VxDs, a debug version of WIN386.EXE is essential. Getting the debug components for Windows 3.x is somewhat more difficult than on OS/2. The standard installation media contain no debug support. The basic debugging components are provided in the SDK, but for VxD debugging one must obtain the appropriate DDK.

WDEB386 also doesn’t automatically load symbol files like the OS/2 KD and must be told explicitly which .SYM files to load. An invocation of WDEB386 from the \WINDOWS directory might look as follows:

WDEB386 /S:SYSTEM\KRNL386.SYM /S:SYSTEM\USER.SYM ..\WIN.COM

That would load symbols for KRNL386.EXE and USER.EXE and launch WIN.COM. The debug terminal would look similar to the following (with more symbols loaded):

The debug version of Windows 3.1 prints an identifier in the lower right corner of the screen much like contemporary Windows versions:

Like the OS/2 KD, WDEB386 uses COM1 by default. Unlike the older SYMDEB, WDEB386 defaults to a serial terminal and not the console.

A word of warning is in order. WDEB386.EXE as distributed with the Windows 3.1 development kits does not run on any Pentium on later processor. WDEB386, as well as some old versions of the OS/2 KD, reads the 386/486 test registers which are not available on later CPUs. The MOV to/from TR instructions were removed and cause undefined opcode exceptions. Microsoft’s Q125635 article provides advice on how to patch the WDEB386 executable to avoid the problem.

Basic WDEB386 documentation can be found in Microsoft Knowledge Base articles KB75252 and KB72379. It is worth noting that although WDEB386 contains special features to support Windows debugging, it is a general purpose debugger and can be used for debugging standard real mode DOS executables. That said, there are much better DOS debuggers than WDEB386.

Last but not least, a sample VirtualBox serial port settings shot:

Note that on Windows hosts, the pipe name must be in \\.\pipe\pname format but pname may be freely chosen as long as it doesn’t conflict with an existing pipe.