64-bit trouble with Solaris 10 U4 and earlier

On many newer systems, Solaris 10 releases up to and including Solaris 10 8/07 (that is Update 4, also known as S10U4) behave in an unhelpful manner. The installer can be launched from CD or DVD and the system can be installed, but the first time the new OS is booted, it almost immediately reboots the system or hangs. This affects both physical and virtualized systems.

The proximate cause of the crash is that when the Solaris kernel switches to PAE (Physical Address Extension) page tables, the tables are built incorrectly and do not properly map the system’s address space. This causes a page fault very shortly after. The OS tries to handle the fault, but fails (because it can’t access memory properly), and soon triggers a triple fault and CPU reset. If left unattended, the system will attempt to boot again and fail in the same way, stuck in an endless crash loop.

The ultimate cause of the problem is much simpler: Solaris incorrectly detects the CPU capabilities based on the processor model. The Solaris engineers had the misfortune of taking Intel’s processor family and model classification seriously, which turned out to be a bad idea.

In the good old days of the Pentium and Pentium Pro, the world was simple. The Pentium was a family 5 processor, Pentium Pro was family 6, and the older i486 was family 4. The Pentium II and III were still family 6 (so-called P6 family) as they did not differ significantly from the Pentium Pro in terms of basic capabilities.

With the Pentium 4, things got interesting. Rather than reporting itself as family 7, the Pentium 4 was family 15, entirely skipping families 7 to 14. But that wasn’t the end of it. When the Core Solo/Duo processors were released, they went back to family 6, since the Core was derived from the P6 micro-architecture. Suddenly the processor family jumped backwards, even though the processors were more capable.

To drive home the message that the model/family designations are best ignored, Intel at one point arbitrarily extended the family and model numbers with additional bits. This change affected the latter Core 2 models and following Core i3/i5/i7 processors. Those were also the CPUs causing trouble for Solaris 10 Update 4.

Without analyzing in exact detail what went wrong, it appears that Solaris 10 might think the processor is not PAE capable, even though every 64-bit CPU by definition must be. The PAE page tables are built incorrectly with fairly disastrous effects; the system is in such a state that it can’t even handle a kernel panic and therefore can’t provide any meaningful diagnostic information.

Needless to say, the problem was noticed as soon as someone attempted to install Solaris 10 or the then in-development Solaris 11 (Nevada) on one of the new processors. The issue was fixed in Solaris 10 Update 5 (officially known as Solaris 10 5/08) and Nevada build 72. The relevant bug number was 6574102, fixed by patch 127112-04.

Fixing the problem in a VM

How to fix the issue, especially in a VM? The easiest solution is using a newer version of Solaris. Solaris 10 Update 5 and later does not have the problem and will run happily on current 64-bit Intel processors. But folks running the older S10 versions may well have a reason to do so.

A sure-fire way to avoid the issue is forcing Solaris to use the 32-bit kernel. The exact steps to do that depend on the boot method, either the old Solaris boot loader in the initial Solars 10 release or GRUB in Update 1 and later. In either case, kernel/unix must be loaded rather than the default kernel/amd64/unix.

For the old loader, enter b kernel/unix when Solaris asks whether to (b)oot or enter the (i)nterpreter. For the GRUB-based loaders, add kernel/unix to the line starting with kernel.

Once Solaris is booted, the eeprom command may be used to change the default kernel to 32-bit. Simply run the following: eeprom boot-file=kernel/unix

When Solaris is successfully booted in 32-bit mode, patches may be applied to fix the 64-kernel, among other things.

When running Solaris 10 in VirtualBox, there is an interesting alternative. The CPUID information may be faked in such a way that Solaris will detect a CPU type close enough to what it needs to see. CPUID leaf 1 contains the relevant information, but only the EAX register value should be modified in order to change only just enough to placate Solaris.

First, the VBoxManage list hostcpuids command should be run to query the host’s CPUID information. The beginning of the information might look as follows:

Host CPUIDs:
Leaf no. EAX      EBX      ECX      EDX
00000000 0000000d 756e6547 6c65746e 49656e69
00000001 000206a7 06100800 17bae3ff bfebfbff
00000002 76035a01 00f0b2ff 00000000 00ca0000

In the example, the host’s CPUID leaf 1 contains 000206a7 in EAX, i.e. extended family/model bits are used. That will definitely upset old 64-bit Solaris 10 releases. To fix that, the EAX value needs to be modified; faking a Pentium D class processor is appropriate. To do that, run the following:

VBoxManage modifyvm <vmname> --cpuidset 1 00000f47 06100800 17bae3ff bfebfbff

Note that only the EAX contents are modified while EBX/ECX/EDX are unchanged. With the modification in place, Solaris 10 Update 4 and older should be able to boot in 64-bit mode.

Lo and behold, the original Solaris 10 release with no patches running on a modern CPU.

It is worth noting that once again, this bug belongs to a class of errors not discoverable by testing. At the time the Solaris CPU detection code was written, it almost certainly worked correctly with all processors available at the time. Only after several years, new processor models broke assumptions made by the Solaris kernel. Defensive programming helps, but it cannot entirely avoid such problems.