DOS 4.0

The apparent cul-de-sac

IBM DOS 4.0

One of the seemingly insignificant yet deeply meaningful changes brought by DOS 4.0 was a name change. No longer Personal Computer DOS, the release sold with IBM systems was now called IBM DOS. Version 4.0, like 3.3, was developed by IBM with minimal involvement from Microsoft; many former DOS engineers at Microsoft were busy building OS/2. However, while 3.3 was a relatively minor incremental upgrade, DOS 4.0 was intended to usher a new era of DOS with a redesigned user interface and reworked internals. From a marketing perspective, DOS was now something of a second class system, suitable for older machines, but only intended as a stepping stone towards OS/2 on newer hardware.

Unfortunately, though not surprisingly, IBM prioritized its own objectives, which often did not match the needs and expectations of the wider DOS market. As a result, DOS 4.0 was a failure from a marketing point of view, the only DOS version to which the majority of users refused to upgrade.

IBM DOS 4.0 was announced on July 19, 1988 with immediate availability. The price was raised again, this time to $150 for a new license and $95 for an upgrade. It is worth noting that the upgrade package was not any different and in fact had the same part number; the only difference was price. There were again two packages, one with 3½” (720K) and one with 5¼” (360K) floppy installation media.

While in development, DOS 4.0 was known as DOS 3.4. In retrospect, version 3.4 might have been more accurate for the DOS kernel, but the entire package with the DOS Shell, large disk support, and a new installer probably deserved a new major version number.

The code name for DOS 4.0 at IBM was Tugboat, clearly in line with the OS/2 code names (Trimaran for OS/2 1.1, Sloop for 1.2, Cruiser for 2.0).

New Internals

Perhaps the most significant change in DOS 4.0 was the introduction of 32-bit logical sector numbers and the consequent breaking of the 32MB partition size barrier. That change wasn’t strictly speaking new, having been first introduced in Compaq’s DOS 3.31 in late 1987. However, beginning with DOS 4.0, every OEM version (starting with IBM’s) supported large partitions.

When a larger than 32MB partition was in use, DOS required the SHARE.EXE utility to be installed. If it wasn’t, a warning message (“WARNING: SHARE should be loaded for large media”) was printed on the console. SHARE was not strictly required for DOS operation, but was needed for applications using old-style FCBs (File Control Blocks). On large partitions, DOS could not use FCBs directly and FCBs had to be translated to SFT (System File Table) operations. The FCB to SFT translation logic was implemented in SHARE.EXE.

The new IFSFUNC.EXE module was IBM’s attempt to better abstract the DOS redirector interface. The redirector interface (INT 2Fh/11h) introduced in DOS 3.0 was functional, but heavily relied on internal DOS structures. That made it unnecessarily difficult to modify and improve DOS, even if such changes had no bearing on redirector functionality.  And if changes did have to be made, redirectors needed to be updated.

Unfortunately IBM again did not publicly document the IFSFUNC interface; because DOS 4.0 never held a majority of the DOS market, redirectors typically supported the older redirector interface, with relatively minor changes for DOS 4.0. In DOS 5.0, IFSFUNC was removed again. It is possible that IBM planned to support filesystems such as HPFS through IFSFUNC, but such support never materialized.

DOS 4.0 was more forward-looking than the previous versions and designed with the new generation of 286- and 386-based systems in mind. Full 640KB of base memory was more or less expected, and there was some built-in support for expanded and extended memory.

DOS 4.0 was the most memory-hungry release of DOS when it came to conventional memory consumption. It was perhaps designed with the idea that a new system with 640KB of conventional memory running DOS 4.0 would still have at least as much free conventional memory as a 512KB system running DOS 3.3. But due to market dynamics, the assumption turned out to be wrong. Application writers started assuming a system with 640KB of conventional memory running DOS 3.x, and suddenly DOS 4.0 was too fat and bloated.

IBM made another mistake in assuming that it was free to change many undocumented aspects of DOS. When DOS 4.0 was released, many popular utilities and networking software packages turned to be incompatible with the new version. Coupled with several well publicized bugs, this severely hampered the adoption of DOS 4.0.

Installation

DOS 4.0 was the first version which came with a dedicated installer, significantly better than the rudimentary SELECT utility in DOS 3.x. The new SELECT was displayed in soothing blue color (at least on color displays) and showed the big IBM logo. This design had been already used with IBM OS/2 1.0 and numerous other IBM products of the late 1980s and early 1990s.

The SELECT utility was a menu-driven installation tool which allowed the user to easily select numerous installation options. It could install DOS on a fixed disk or onto a set of floppies.

SELECT let the user to choose between maximizing functionality, minimizing memory usage, or balancing the two. Based on user’s decision, SELECT enabled or disabled functionality such as FASTOPEN and adjusted the number of various DOS buffers.

Some users found the choices provided by SELECT confusing, although it’s not entirely clear how IBM could have made that better without triggering complaints from other users that DOS didn’t give them enough choice.

New User Interface

DOS 4.0 was part of the push towards OS/2 and graphical user interfaces. The command line interface was officially obsoleted by the new full-screen DOS Shell, compliant with the new CUA (Common User Access) guidelines and somewhat similar to the (upcoming) OS/2 File Manager. The DOS Shell sported menus, dialogs, push buttons, scroll bars, and on-line help.

The DOS Shell was customizable and users could add their own menu entries to easily launch arbitrary programs and utilities. The Shell could be controlled either by a mouse or by a keyboard. There was naturally an easy way to “shell out” to the traditional DOS command prompt.

A File System utility was a prominent part of the DOS Shell. This presented the directory structure of a disk in the classic tree form, allowing easy navigation. The File System part of the DOS Shell was very similar to the File System applet in OS/2’s Presentation Manager (released a few months after DOS 4.0).

The DOS Shell was met with a mixed reception. Some users liked it, as it made the PC more accessible to beginners. Others complained that the DOS Shell was incompatible with popular TSRs—often because the TSRs could not correctly handle applications running in graphics mode. The solution was forcing the DOS Shell to use text mode.

The DOS Shell itself was of course entirely optional and users who found no need for it were not forced to use the shell.

More Memory—Or Less

DOS 4.0 was the first version with built-in support for EMS (expanded memory) conforming to the Lotus-Intel-Microsoft (LIM) EMS 4.0 specification. DOS 4.0 supported EMS on two fronts, first by providing EMS to applications and second by utilizing EMS within DOS itself.

Since IBM DOS 4.0 was designed for IBM systems, it included support for IBM memory adapters. EMS support was implemented in two layers, a low-level hardware-specific driver, and the XMA2EMS.SYS driver providing the actual EMS interface.

On 386-based PS/2 systems (initially Model 80), DOS 4.0 could emulate an IBM expanded memory adapter using the XMAEM.SYS driver. The XMAEM.SYS driver used the virtual 8086 mode of the 386 CPU. It was somewhat similar to EMM386 in concept, although simpler. Like many early 386 memory managers, XMAEM.SYS did not provide VCPI (let alone DPMI) support and was thus incompatible with DOS extenders or other 386 control software.

DOS 4.0 could place disk buffers in expanded memory, using the BUFFERS /X command in CONFIG.SYS. Because up to 10,000 buffers (sector-sized) were supported, DOS 4.0 used hashing instead of linear lists to manage disk buffers. The BUFFERS command could be used instead of a traditional disk cache (such as SMARTDRV), with somewhat different performance characteristics.

The expanded memory buffers caused significant problems for IBM. One of the problems was that DOS 4.0 was initially only tested on IBM systems, and IBM’s expanded memory implementations always offered at least six EMS handles, more than the LIM EMS standard requires. When users installed IBM DOS 4.0 on non-IBM systems, the EMS implementation often only offered 4 EMS handles and the DOS buffers implementation failed, with potentially data-destroying consequences. The problem was fixed in an update (CSD) from June 1989, and was also corrected in MS-DOS 4.01.

The FASTOPEN.EXE extension could likewise utilize expanded memory with the /X switch. FASTOPEN was a very specialized disk cache. FASTOPEN only cached file searches and file opens, and was much more efficient than disk buffers. When a typical application searched for a file (such as PATH searches), it needed to read directory sectors and FAT sectors which contained largely irrelevant information. In a typical case with a relatively small number of disk buffers (10 or so), disk searches had a tendency to force out all useful file buffers and replace them with relatively unimportant directory/FAT buffers, causing increased disk I/O. This was an issue especially with larger disks (in the 100MB+ range). FASTOPEN needed much less memory to cache the same information and thus increased the efficiency of disk buffers.

Additionally, in DOS 4.0 FASTOPEN also implemented so-called FASTSEEK functionality. This was yet another specialized cache for fast access file extent information. As files became large and fragmented, locating various portions of a file required a potentially significant number of disk accesses to read and process the FAT, with expensive seeks moving between the FAT (at the beginning of a disk) and the data area. FASTSEEK was not especially useful for sequential access, but very effective for random access files, especially if they were fragmented.

A new MEM.EXE utility was added to display the amount of available memory. This functionality was previously—in a rather idiosyncratic manner—available through CHKDSK. However, CHKDSK only displayed conventional memory statistics while MEM also showed extended and expanded memory data. In addition, MEM supported the /DEBUG and /PROGRAM options which displayed detailed conventional memory usage statistics suitable for determining what exactly was using how much memory.

The MODE command was enhanced to support a console with more than 25 lines, taking advantage of EGA and VGA adapters with programmable fonts. The TREE command was changed to display the directory tree in a semi-graphical format, far easier to read than the line-by-line output of earlier versions.

A new feature of DOS 4.0 was the ability to report a different DOS version to applications, a so-called “lie list”. The feature was not documented and the list was hardcoded in IBMDOS.COM, but it was located at the very end of the file and could be potentially modified by a system administrator. The predefined entries included e.g. IBMCACHE.SYS or WIN200.BIN and made DOS 4.0 report version 3.40 to these applications, although the the mechanism was not restricted to any specific range of faked version numbers.

Reception

DOS 4.0 was relatively poorly received. Some of the criticism leveled against DOS 4.0 was justified, and some of it wasn’t; however, it stuck and DOS 4.0 never managed to shake the bad name.

Among the more widespread problems with DOS 4.0 was incompatibility with leading disk utilities. This was natural and inevitable. Since DOS 4.0 supported large disks with 32-bit sector addressing, the disk format had to be different. Utility vendors soon updated their software, but not before the trade press was awash with stories about DOS 4.0 being incompatible with this or that disk utility.

Some problems were issues of perception. IBM sold DOS 4.0 to anyone but never claimed that IBM DOS 4.0 was compatible with anything other than IBM hardware. One of the worst such issues was caused by the way DOS 4.0 used EMS. DOS assumed that six EMS pages were available, which was the case with IBM memory expansion boards, but not guaranteed by the LIM EMS standard and not the case with third-party adapters.

The graphical DOS Shell caused problems with TSRs which could not handle graphics modes. This was easily avoided by forcing the shell to run in text mode, but the only real solution was updating the TSRs to correctly function when the foreground application used a graphics mode. Again this generated bad press for DOS 4.0.

When DOS 4.0 was configured for higher performance with many buffers and caches, the amount of available conventional memory was noticeably lower than in DOS 3.3. This was a legitimate complaint, if perhaps somewhat overblown.

In addition, there were no wonderful new applications requiring DOS 4.0. Because of that, users and vendors were not forced to work out compatibility issues, and many simply decided to stick with DOS 3.3 and ignore the issue altogether.

Last but not least, IBM wasn’t very open about admitting problems with DOS 4.0. In mid-1988 there was a quiet update to version 4.01 fixing several critical bugs, but IBM never publicly announced the release, and the new packaging was not clearly marked. Only the floppies contained in the package were labeled as 4.01. DOS itself continued to report version 4.00, although the timestamps of many DOS files were newer.

Later Developments

DOS 4.0 was on the market for several years (roughly 1988-1991), significantly longer than previous versions of DOS. Not surprisingly, it continued to be adapted and modified in that period.

In early 1990, Microsoft released a Russian version of MS-DOS 4.01 for the Soviet Union. This was one of the first Western software products designed for the USSR (and also one of the last, as the Soviet Union ceased to exist in 1991). The Russian MS-DOS 4.01 featured a prominent anti-piracy warning displayed every time the system started; it is highly doubtful that the warning had any effect.

Russian MS-DOS 4.01 leveraged the internationalization technology built into DOS by Microsoft and IBM, which included keyboard layout and display font switching. The only significant non-standard feature was hot-key switching between Russian and Latin keyboard layouts. The base system was fully localized, but the DOS Shell was not.

Also in 1990, IBM released the PS/1 line of home computers. The PS/1 systems came with a specially adapted version of DOS which was built into the PS/1 ROM. This was DOS 4.02, with a 1990 copyright message.

This version of DOS still identified itself as DOS 4.00, even in releases including files with mid-1991 timestamps. IBM’s ROM DOS 4.0 appears to have used somewhat different technology than Microsoft’s ROM DOS 5.0.

An Opening for Competition

The poor reception of DOS 4.0, together with the relative lack of attention given to DOS by Microsoft and IBM, created an opening for Microsoft’s long-time competitor, Digital Research. DR DOS 3.x gained favor with some OEMs by virtue of being significantly cheaper than MS-DOS. However, by 1990 Digital Research offered DR DOS 5.0 (since DOS 4.0 had a bad name, DR went from 3.41 straight to 5.0) which was widely accepted to be superior to MS-DOS 3 and 4.

This caused some serious anxiety among high-ranking Microsoft officials. DOS was Microsoft’s cash cow, which funded expansion into other markets. If DOS income were to be severely reduced, it would have caused big financial headaches for Microsoft.

Microsoft naturally realized that and scrambled to release DOS 5.0, “accidentally” matching DR DOS feature for feature. The release of DOS 5.0 in 1991 spelled a quick death for the unpopular DOS version 4.0.

The Vista of DOS

There are interesting parallels between Windows Vista and DOS 4.0. Like Windows Vista, DOS 4.0 was highly unpopular for many reasons. Like Vista, DOS 4.0 was often considered less preferable than the previous version (DOS 3.3/Windows XP) until the successor version (DOS 5.0/Windows 7), not radically different, won the hearts and minds of the press and users.

As with Vista, the software compatibility issues were hammered out by vendors, and by the time the next version came out, everything just worked. Not so much because the new version magically solved all problems, but rather because incompatibilities had been quietly fixed while the world wasn’t paying attention.

And much like with Windows 7, the resource requirements of DOS 5.0 seemed modest in 1991, while similar requirements of DOS 4.0 had been deemed outrageous in 1988. High memory support in DOS 5.0 relieved much of the conventional memory pressure, but it also required at least a 286. By the time DOS 5.0 was released, the the typical system was in fact a 286 or better, but that had not been the case when DOS 4.0 was released.

References

Developing Applications Using DOS, Ken W. Christopher, Jr., Barry A. Feigenbaum, Shon O. Saliga, 1990, John Wiley & Sons, ISBN 0-471-52231-7
Incompatibilities Hinder Useful DOS 4.0 Features, InfoWorld, August 15, 1988, page 1
IBM Doesn’t Want To Come Clean on DOS 4.0, InfoWorld, August 29, 1988, page 74
Early DOS 4.0 Users Say ‘Stay Away’, InfoWorld, September 5, 1988, page 5
IBM announcement letter archive