386MAX and EISA DMA

A few weeks back I was reminded that the source code to the 386MAX (later Qualitas MAX) memory manager was released in 2022 on github. Back in the 1990s I used primarily EMM386 and QEMM, but I have some experience with 386MAX as well. It’s quite comparable with QEMM in terms of features and performance.

The source code itself is quite interesting. The core memory manager is of course written 100% in assembler. However, it is very well commented assembly code, written in uniform style, and it generally gives the impression of a well maintained code base (unlike, say, MS-DOS).

The github release also includes enough binaries that one can experiment with 386MAX without building a monster code base.

I was able to get enough of it installed to have a functional 386MAX setup. Then I thought that for easier experimentation, perhaps I should set up a boot floppy with 386MAX. And that’s where trouble struck.

My boot floppy had a nasty tendency to horribly crash and burn (the technical term is ‘triple fault’) when loading additional drivers in some configurations. This is what the CONFIG.SYS looked like:

FILES=30
BUFFERS=10
DOS=HIGH,UMB
DEVICE=386MAX.SYS USE=E000-EFFF NOWIN3
DEVICEHIGH=POWER.EXE
DEVICEHIGH=VIDE-CDD.SYS /D:CD001

After some experimentation, I narrowed the problem to the following combination: 386MAX provides “high DOS memory” (RAM between 640K and 1M), DOS has UMB support enabled (via DOS=UMB), drivers are loaded into UMBs via DEVICEHIGH statement, and the system (VM really) has more than 16 MB RAM. The same configuration worked fine when loading from hard disk, or with 16 MB RAM (or less).

That gave me enough clues to suspect problems with DMA. In a setup using traditional IDE (no bus mastering), there’s no DMA involved. But when booting from a floppy, the data is transferred using DMA. If the DMA controller can’t reach memory because it’s beyond 16 MB, there will be trouble.

386MAX needs a DMA buffer to deal with situations when memory involved in a DMA transfer is paged, and the pages are not contiguous and/or are out of the DMA controller’s reach. But what if the DMA buffer itself is above 16 MB?

I already had a small utility to print information about a VDS (Virtual DMA Services) provider. It was trivial to add a function to print the physical address of the DMA buffer. There is typically one DMA buffer that can be allocated through VDS and it may also be used by the memory manager to deal with floppy DMA transfers. Sure enough, in a configuration with 32 MB RAM, 386MAX had a DMA buffer close to 32 MB! That would definitely not work with floppy transfers. But why?!

It was time to trawl through the source code. It was not hard to find the logic for allocating the DMA buffer. Because 386MAX supports old and exotic systems, the logic is fairly involved. The most unusual is an Intel Inboard 386 PC — a 386 CPU inside a PC or PC/XT system. In that case, the DMA buffer must be allocated below 1 MB (really below 640 KB). On a PC/AT compatible machine, it must lie below 16 MB. Except on old Deskpro 386 machines, the last 256 or so KB aren’t accessible through the DMA controller and the DMA buffer has to be placed a little below 16 MB. Finally on EISA systems, the 8237 compatible DMA controller can address the entire 32-bit space, and the DMA buffer can be anywhere.

But VirtualBox VMs don’t pretend to support EISA. Why would 386MAX think it’s running on an EISA system? Of course the source code had the answer. When 386MAX detects a PCI system (i.e. system with a PCI BIOS), it assumes EISA style DMA support!

Why oh Why?!

At first glance that makes no sense. But on closer look it does, sort of. Early PCI machines were generally Pentium or 486 systems with Intel chipsets. These were almost guaranteed to use the SIO or SIO.A southbridge (82378IB or 82378ZB SIO, or 82378AB SIO.A). And those chipsets did in fact include EISA DMA functionality with full 32-bit addressing!

In 1996, Intel released the 82371FB (PIIX) and 82371SB (PIIX3) southbridge. The PIIX/PIIX3 southbridge no longer included EISA DMA functionality and the 8237 compatible DMA controller could only address 16 MB RAM.

It is hard to be certain but the developement of 386MAX probably effectively stopped sometime around 1997. At that time, PCI machines were becoming common, but not necessarily with more than 16 MB RAM and running DOS. The problem may have slipped under the radar, especially because it probably in practice only affected booting with 386MAX from floppy.

Was it Always Wrong?

That said… I believe that placing the DMA buffer above 16 MB is always wrong in a system with ISA slots. On an EISA (or early PCI) system, the 8237 DMA controller can indeed address the entire 32-bit address space, and that takes care of floppy transfers. But there are also ISA bus-mastering adapters, such as some SCSI HBAs or network cards. And those cannot address more than 16 MB.

If a driver for an ISA bus-mastering adapter uses VDS to allocate a DMA buffer, it expects that the buffer will be addressable by the adapter. That will not be the case when 386MAX allocates the DMA buffer above 16 MB.

Again, the scenario may be exotic enough that it could have slipped through the cracks. Owners of EISA systems would be likely to use EISA storage and network adapters, not ISA. Likewise PCI machines would presumably utilize PCI storage and network adapters. Thus the problem would not be visible.

At the same time the system would actually need to have more than 16 MB installed for the problem to show up. That was quite uncommon in the days of 386MAX, let’s say 1995 and earlier. Machines with more RAM did exist, but were vastly more likely to run NetWare, OS/2, Windows NT, or some UNIX variant. At the same time, a PCI machine with 32 MB RAM or more and the older SIO or SIO.A southbridge would actually work, at least for floppy transfers, with the DMA buffer anywhere in memory.

It should be noted that the VDS specifications does not explicitly take this situation into account. It mentions that the VDS scatter/gather lock function may return memory not addressable by DMA, and in that case drivers are supposed to use the dedicated DMA buffer. The DMA buffer is assumed to be always addressable through DMA, but no mention is made of the 16 MB limit on ISA systems. Given that the VDS specification was initially written in 1990 and last updated in 1992, it is entirely plausible that machines with more than 16 MB RAM were not high on the spec writers’ priority list.

Workarounds?

386MAX has a NOEISADMA option which forces the DMA buffer to be allocated below 16 MB. This option does not appear to be well documented, but finding it was not difficult when reviewing the 386MAX source code. This at least partially solves the problem.

Of course a much better solution would be fixing 386MAX to not assume EISA style DMA on PCI systems. Although the assumption was true on early PCI systems, it is not true in general.

But Wait, There’s More!

Later I realized that there’s another factor in the whole mess, the NOWIN3 keyword. The NOWIN3 keyword removes support for Windows 3.x, thus saving memory, and also makes it unnecessary to have 386MAX.VXD in the same directory as 386MAX.SYS (useful when booting from a floppy).

But… of course… the NOWIN3 keyword does more than that. And that’s where the well commented 386MAX source code was really helpful. By default, when 386MAX supports Windows 3.x, the memory manager makes sure that UMB memory is physically below 16 MB. A comment says:

; If we're providing Win3 support, ensure that the PTEs
; we're to use to high DOS memory are below 16MB physical memory
; If there's possible DMA target mapped into high DOS, Windows
; won't check the address and fails.

Aha… so by default, UMBs are below 16 MB, and hence within reach of a PC/AT compatible DMA controller. With the NOWIN3 keyword, the restriction is removed and the UMB physical memory can end up above 16 MB.

When loading drivers into UMBs, 386MAX probably does not use the dedicated DMA buffer because the memory is contiguous and (it believes) addressable by the DMA controller.

By default, that is actually true. I managed to create a configuration (NOWIN3 keyword, PCI system incorrectly identified as EISA DMA capable, loading drivers from floppy to UMBs) where DMA goes wrong.

Now it’s clearer why the problem went unnoticed. Let’s recap the conditions necessary to trigger the bug:

• Machine with more than 16 MB RAM (common now, not in early to mid-1990s)
• Machine with a 1996 or later PCI chipset (incorrectly assumed to be EISA DMA capable)
• 386MAX providing high memory aka UMBs (typical)
• DOS using UMBs (common)
• Loading drivers into UMBs via DEVICEHIGH/LOADHIGH (common)
• 386MAX Windows 3.x support disabled via NOWIN3 (very uncommon?)
• Booting from floppy (not common)

This scenario, although easy to reproduce, is just exotic enough that it’s entirely plausible that it wouldn’t have been encountered when 386MAX was popular.