LOADALL Strikes Again

A minor mystery recently popped up while running IBM’s OS/2 1.1 (1988), the first OS/2 version with the Presentation Manager GUI. While Microsoft’s and IBM’s releases of OS/2 were fully compatible from application perspective, there were differences in the drivers supplied and the kernel itself was slightly different.

One of the differences was that while all Microsoft’s OS/2 kernels (starting with the May 1987 beta) detected a 386 CPU and used 386-specific instructions to switch from protected back to real mode, IBM’s OS/2 1.0 and 1.1 kernels did not. IBM only used the 286-specific method (which MS’s OS/2 kernels also used, but only on 286s) of mode switching: triple fault the CPU, trigger a shutdown cycle, let the motherboard reset the CPU, and ask the BIOS to skip the POST code and jump to a specified address soon after the CPU restarts execution.

This method is not supported by most virtualization products (Virtual PC, VMware) and may not work with some recent BIOSes. But that’s not the interesting part. The really interesting part is that when running a DOS box and hence executing in real mode, IBM OS/2 1.0 and 1.1 in some cases executes the infamous 286 LOADALL instruction. In OS/2 1.0 this appears to be fairly rare, in version 1.1 LOADALL is typically executed within seconds of switching to the DOS box.

Why would OS/2 do this? It is clearly a performance optimization. OS/2 is executing in real mode but needs to access some memory beyond 1MB. The LOADALL instruction is used to load the in-CPU descriptor cache that will allow a segment register (usually ES) address up to 64K of memory anywhere in the 16MB 286 address space.

Memory past the 1MB boundary is of course normally accessed in protected mode, but that has numerous disadvantages. The machine state must be completely changed, real-mode code can (typically) no longer be executed, interrupts must be disabled for relatively long periods of time. Worst of all, returning back to real mode takes quite a while; resetting the CPU in order to switch from protected to real mode works, but it is not fast.

Using LOADALL neatly sidesteps most of these issues. The CPU never leaves real mode and although interrupts are still dangerous (reloading a segment register will overwrite its “magic” contents set up by LOADALL), the entire operation takes far less time and it is even possible to execute memory copies “speculatively” with interrupts enabled.

It is well known that Microsoft’s HIMEM.SYS uses LOADALL on 286 systems for exactly these reasons. It is also well known that although 386s and perhaps some later CPUs implemented a LOADALL instruction, it used a different opcode and was not compatible with the 286 variant. For completeness it should be added that the opcode used by 286 LOADALL (0Fh 05h) was later reused for the SYSCALL instruction.

With these facts in mind, one might wonder how IBM’s OS/2 1.0/1.1 ever worked on 386 and later CPUs, or if it ever did? Work it certainly did, but how then?

IBM sold 386 systems at the time, initially the PS/2 Model 80, eventually joined by Model 70 and others. OS/2 1.0/1.1 “naturally” includes specific support for these systems and does not execute LOADALL. However, generic 386 AT-compatibles can also run OS/2 1.0/1.1. On those systems, IBM’s OS/2 will triple fault the CPU to get back to real mode and it will execute LOADALL. That should not work, but it does. Why?

The answer lies in a typical 386+ BIOS. The BIOS installs an invalid opcode handler, detects attempts to execute a 286 LOADALL instruction, and emulates it well enough to satisfy questionably written software such as IBM OS/2 1.0/1.1.

The BIOS cannot fully emulate all aspects of LOADALL, but can reliably emulate the typical use case of LOADALL: The caller is executing in real mode, wishes to remain in real mode, and wants to modify the descriptor cache of one or more segments such that the base is above 1MB. It is assumed that CS and SS remain normal real-mode segments. The GDT/LDT/IDT/TSS may be modified.

In theory, software could use LOADALL in order to switch from real to protected mode, but that is somewhat pointless because such a switch can be achieved without the use of undocumented instructions. Using LOADALL to switch from protected to real mode is not possible at all, and utilizing LOADALL in protected mode is probably more work than achieving the same effect through documented means. These limitations allow a BIOS to emulate the common LOADALL uses.

It was previously mentioned that the 286 LOADALL opcode was later recycled for use by the SYSCALL instruction. Does that mean systems with newer CPUs cannot utilize the BIOS LOADALL emulation? On Intel processors, SYSCALL must be explicitly enabled and even then is only recognized in 64-bit mode, so it cannot possibly conflict with software written for the 286. On AMD processors, SYSCALL is not limited to 64-bit mode but must still be enabled via EFER.SCE, which again won’t be the case with software designed to run on 286s. Therefore SYSCALL does not conflict with 286 LOADALL emulation.

It would be interesting to know what Intel thought about Microsoft’s use of LOADALL. It’s fairly certain that Intel supplied Microsoft with non-public LOADALL documentation, but it’s unknown whether Intel sanctioned LOADALL use in retail software. Especially since Microsoft only started using the 286 LOADALL instruction after the 386 was already on the market and did not support it.