As previously mentioned, the OS/2 Museum adapted the Phantom redirector example from the second edition of Undocumented DOS to demonstrate that the redirector interface was already fully implemented in the August, 1984 release of PC DOS 3.0, a fact apparently unknown to most authors of DOS literature.
The Phantom 3.0 redirector is now available for download in the form of a 360K floppy image. Kudos to the authors of Undocumented DOS for writing it in the first place. Needless to say, the code is provided as is, and although bug reports are welcome, no promises are being made as to if or when any issues might be fixed.
Bug Fixes
This project turned out to be a bit more involved than it should have been, because the Phantom redirector as delivered with Undocumented DOS contained a surprising number of bugs. Following is a list of the issues discovered and fixed:
- Crash on startup. In set_up_xms_disk(), Phantom called the dos_ftime() routine to get the current time in a format suitable for storing in a FAT directory entry. Unfortunately the dos_ftime() routine was only meant to be called from within the INT 2Fh handler and used uninitialized data during startup. This did not cause visible harm on DOS 5.0/6.x but caused hangs with DOS 3.x/4.0.
- Truncated directory searches. Phantom contained an embarrassing bug which caused directory searches to be limited to 32 entries (one 1,024-byte sector). This was trivially fixed in find_next_entry() by resetting the index variable after exhausting each directory sector.
- Problems with larger directories. Indirectly related to the previous bug, there was a really embarrassing blooper in get_dir_start_sector(). A spurious semicolon after an if statement caused Phantom to fail finding a subdirectory if it wasn’t located in the first sector of its parent directory.
- Inability to create directories on non-empty drive. A slightly less embarrassing bug in Phantom resulted in 16-bit instead of 32-bit arithmetic being used in get_dir_start_sector() to calculate sector offsets for directory searches. As a result, directory creation and searching worked well on an empty disk, but failed horribly after a few files had been copied onto the Phantom disk.
- Bad CHKDSK/LABEL behavior on DOS 4.0. Some DOS commands, including CHKDSK and LABEL, detect non-local drives via INT 21h/4409h. On DOS 4.0, that causes the ‘IFS header’ pointer in the CDS to be dereferenced. Because Phantom left this pointer as NULL, DOS was examining memory in the interrupt vector table (IVT) and returned unpredictable results. Worse yet, this could cause unpredictable memory corruption. This was fixed by providing an IFS header with contents crafted to placate DOS.
- Spurious file write errors. Phantom was not updating the open mode in the fill_sft() routine, which caused some files to be non-writable on DOS 3.x. When a file is created via INT 21h/3Ch, it should be opened for reading and writing (regardless of the attributes given). This was fixed by forcing the open mode in the SFT to allow writing when a file was created this way.
- Incorrect CD behavior on DOS 3.0. This problem is perhaps not so much a bug in Phantom as the documentation it was based on. On DOS 3.0, the symptom was ‘CD FOO’ resulting in the current directory being set to ‘FOO\FOO’. This was fixed by not updating the CDS in the redirector in function cd() and letting DOS do it. See below for details.
- Incorrect CD behavior with deep directories. Changing into directories more than two levels deep, such as ‘\A\B\C’, failed to work and ended up in \A instead. The problem was that Phantom modified the pathname buffer in the SDA, which clearly wasn’t supposed to happen. Fixed by restoring the buffer before returning to DOS.
- Incorrect findfirst error code. The TREE utility in DOS 4+ failed to traverse the entire directory tree, stopping after the first leaf directory. Fixed by adjusting error codes returned by findfirst (INT 2Fh/11Bh). More details below.
It is apparent that Phantom was never properly tested, especially not with older DOS versions (3.x/4.0), even though Phantom contained code to explicitly support those versions. Given that, it is likely that some bugs in Phantom still remain. Nevertheless, Phantom remains an invaluable example of DOS redirector programming, and the known bugs were fairly easy to fix once the problems were understood.
One of the more interesting misconceptions in Phantom was clearly caused by a lack of any official documentation. The INT 2Fh/1105h redirector callback is executed by DOS when changing directories. The Phantom authors (and RBIL contributors) clearly thought that it was the redirector’s job to update the CDS, but in fact the callback is only meant to validate that the target directory exists; DOS subsequently updates the CDS. This caused the most visible problems on DOS 3.0, but the fix (not updating the CDS in INT 2Fh/1105h callback) works well on all DOS version.
Another interesting issue was the handling of findfirst/findnext error codes. This was causing obvious errors with TREE (on DOS 4+) where the display stopped after traversing into the first leaf directory. The DOS findfirst function (INT 21h/4Eh) is documented to return three possible error codes, 2 (file not found), 3 (path not found), and 18 (no more files). Microsoft’s official documentation is poor and does not explain which error is returned when. TREE obviously expects error 18 (no more files) to be returned when searching for subdirectories and none are found; Phantom was returning error 2 (file not found), confusing TREE.
The problem was in part caused by Phantom not creating the ‘.’ and ‘..’ entries in each subdirectory; on a typical DOS disk, a findfirst for entries with the directory attribute bit set and a ‘*.*’ search pattern will always find at least those entries in each subdirectory. For Phantom, that was not the case. The solution was to return error 18 (no more files) from findfirst if the search pattern contains wildcards and no matching entry was found. If findfirst finds nothing but the search pattern did not contain wildcards, error 2 (file not found) is returned.
During testing, it was found that DOS 3+ does not appear to return error code 2 from findfirst (INT 21h/4Eh), although it could perhaps happen under unusual circumstances. Typically the returned code will be 3 (if search pattern is invalid, such as ‘C:\FOO\’) or 18 (if search pattern is valid but no matching entry was found).
Some DOS references claim that findfirst returns either error 2 or 18, which is demonstrably untrue with DOS 3+. However, it was probably true for DOS 2.x. The behavior documented for DR-DOS is not what was observed with MS-DOS/PC DOS; namely MS-DOS/PC DOS always returns error 18 on failed searches regardless of whether the search pattern included wild-cards or not. Microsoft’s documentation is, as usual, silent on the topic; findfirst is documented to return error codes 2, 3, and 18, but no hint whatsoever is given as to which error code might be returned when (the RBIL is, sadly, no better). Which would of course be fine if DOS utilities didn’t depend on specific error codes being returned from findfirst…
Modifications for DOS 3.0
Adapting Phantom to work with PC DOS 3.0 was a straightforward process. Almost all of the modifications needed to account for differences in the layout of internal DOS structures, but the redirector interface itself needed no changes.
The DOS data segment in DOS 3.0 was laid out differently from later versions, requiring changes to the List of Lists processing code and a different SDA layout. In addition, the SFT layout in DOS 3.0 was slightly different, with the filename being offset by one byte (DOS 3.0 stored a 16-bit offset into a directory sector rather than the 8-bit index of the relevant directory index).
As mentioned above, the change directory logic had to be modified to work correctly on DOS 3.0, but this was strictly speaking a fix to better conform to the DOS redirector interface rather than a version-specific modification.
Impressions
While working with Phantom, it was inevitable to become acquainted with certain aspects of internal DOS operation. It soon became apparent that as far as file and directory functionality is concerned, the differences between DOS 3.0 (1984) and DOS 6.0 (1993) are not nearly as big as one might be led to believe. While there are numerous minor differences, the basic structure remained constant across all versions. It certainly didn’t look like any kind of “rewrite” happened at any point between 3.0 and 6.x.
It may not be unjustified to say that DOS 3.0 was the last truly major version of DOS, with the following releases only bringing relatively minor evolutionary changes which did not fundamentally affect the core DOS operation and structure. In a way that should be expected: MS-DOS 4.0 (the near-mythical multi-tasking MS-DOS 4) was where significant changes occurred after 1984. And DOS 5.0, later released as OS/2, was where truly major changes in DOS occurred after 1985-1986, while DOS 3.x was receiving only very minor updates.
For all the hoopla surrounding the “regular” DOS 4.0 (1988) and 5.0 (1991), neither of those versions brought anything like the changes between DOS 1.x and 2.0 or 2.x and 3.0. MS-DOS 6.0 was widely known at the time it was released (1993) not to include any significant changes to the core of DOS, however important a release it may have been for other reasons. Again not surprisingly, since at that time DOS was so entrenched and simultaneously obsolete that major changes did not make sense. Windows 95 was a different story of course…
Would you be so kind and point me where I could download content of Undocumented DOS disk (especially from the 2nd edition)? Mine copy got lost many years ago. I’m also looking for a disk from “Personal Computer from the Inside Out: The Programmer’s Guide to Low-Level PC Hardware and Software” book, and especially the debugger that was on that floppy.
Please give me a few days and I’ll try to put the disk images up for download. Googling around might unearth something too, but I’m not sure.
Sure, take all the time you need. I’ve been googling around for those disk for some time already but no luck. I’ve considered buying another copy of the book but to find one with a floppy (assuming it will be still readable) is also a bit tricky.
Thank you in advance 🙂
Ask and ye shall receive (at least something): 1st Edition and 2nd Edition. I threw in the DOS Internals disk too. For that one I also have the original disk and hence disk image, but the files are probably more interesting. The “Personal Computer from the Inside Out” book I do not have.
Thank you! That was quick. If you like I have a copy of files from Undocumented PC book (probably 2nd edition).
The “PC Inside Out” book had a debugger on disk which was used to port Windows into protected mode. The story is being told in that particular book too.
Thanks, I have both of the Undocumented PC disks. I’ve been toying with a thought of maintaining an errata list for the (quite useful) utilities 🙂
Ah, that was the SST debugger? David Weise and Murray Sargent?
SST it was. The idea for an errata is a very good idea. I’ve tested some of the code from the Undocumented DOS disks under DosBox and this is great demonstration of how emulation is far from original MS-DOS 5. On the other hand games never used a lot of dos internal structures in the first place so this result was kind of suspected.
Anyway poking around DosBox with different utilities is a lot more fun than reading its source code. I just wish DosBox would support debug registers.
Do you have any more info on the SST debugger ? It is an interesting part of the history of the development of Windows but there isn’t a lot of info out there about it.
DosBox isn’t very good as it was only meant to run DOS games. Something like the in-browser PCjs emulator is actually much better – http://www.pcjs.org
DOSBox is very interesting but as you say, it’s definitely not DOS. In my experience, most games used DOS as a loader and filesystem, not much more, and DOSBox reflects that.
Fortunately running real MS-DOS in an emulator/hypervisor is not so hard either.
For just about anything other than a game NTVDM is much better than DOSBox. Of course if you’re running 64-bit Windows then you’re out of luck. Obviously the best option is to have a small DOS partition at the beginning of your hard disk and run real DOS though if a system doesn’t provide a compatibility legacy BIOS then that becomes impossible. I don’t know if it is common for UEFI systems to not provide legacy BIOS support (at least not yet anyway).
Unfortunately I don’t have any more info than what you can find in that book but I guess since it made it to a book there must be some other sources as well. It is also kind of interesting to see how MS own debugging tools were sometimes lagging behind commercial software available at the time. One reason for I can come out with for this situation is: access to source code. Why SoftICE has been developed for DOS and Windows and not for Linux in first place? The lack of source code push the whole industry into peeking into internals.
Obviously SoftICE and other debuggers were developed for DOS and Windows because they were the dominant operating systems for the PC. Linux (or any flavor of Unix) was never dominant, isn’t dominant now and never will be dominant.
At least for DOS, people peeked into its internals because many key functions were undocumented and also because it was small and interested people (like myself) could easily study it.
There aren’t many companies that are going to make their source code freely available. Publishing source code generally means losing control. Consider the PC BIOS, does anyone really believe that Compaq BIOS engineers didn’t look at the BIOS source code in the PC Technical Reference manuals ? Once Compaq, Phoenix and others duplicated the PC BIOS that opened the door for a plethora of clones which was good for the PC industry overall but not so good for IBM.
I agree that NTVDM does a good job… except… it’s not there in 64-bit Windows versions (CPU limitation, V86 tasks are not supported when the CPU is in 64-bit mode).
I have also seen nasty problems with NTVDM in Windows 10. Lots of DOS-extended programs were crashing on me when the same stuff ran fine on Windows 8.1 and earlier.
I think SoftICE predates Linux. It certainly predates Linux as a useful OS, because until at least the late 1990s installing Linux was far beyond the average user’s capabilities.
Developing for Linux has its own challenges. Sure, there’s source, but no debugger as powerful as Soft-ICE (or WinDbg). Sure, there’s source — all 5,000 kernel versions plus vendor patches. Too much of a good thing etc.
Windows 10 NTVDM has problems that the earlier version don’t ? That sounds like they removed or reduced support for something like DPMI or VCPI. For the last few years Microsoft has been dropping any lingering 16-bit support from their products. For example MASM 10 and later can no longer assemble 16-bit code and all of the related directives (e.g. .8086, .8087, .286, .286p) have been removed.
Linux came out in 1991 so most DOS software predates it. I’m not sure I would describe Linux as a useful OS in regards to the PC as it never gained anything remotely approaching a significant foothold in the PC market. OS/2 obviously had the same problem but it’s more significant than Linux as far as PCs go because it was meant as a successor to DOS and was developed by Microsoft and IBM.
Hardcore Linux fans would likely argue that GDB is a powerful debugger but it has to be the most cryptic debugger ever made. Several years ago I had to use GDB in the Cygwin environment and it wasn’t a pleasant experience though nothing about Cygwin was particularly pleasant. Like pretty much everything else in the Unix world GDB seems to be cryptic just for the sake of being cryptic which is another reason why Linux or Unix never caught on.
Regarding the original article and changes to DOS: while functionally DOS 5 has few changes from previous versions there were significant structural changes to support loading in the HMA. There had to be a true separation of code and data because the data areas for both IBMBIO.COM/IO.SYS and IBMDOS.COM/MSDOS.SYS remain in low memory while the code for both could be loaded in the HMA. Prior to DOS 5 CS overrides were often used to access data and that had to be changed.
None of the DOS 6.x series brought about any significant changes and the DOS API was unchanged after DOS 5. I would of course argue that PC DOS 7 brought a lot of changes due to cleanup and optimizations.
Found the SST debugger –
http://www.mmnt.net/db/0/0/89.179.18.66/oldftp/TOOLS/SOFTWARE/Lang/DEBUG/SST
Great find but I don’t think it was a version I’ve been using – on the other hand it was so long ago that I might be wrong.
By the way if you’re interested in DOS debuggers then give this a look –
https://sites.google.com/site/pcdosretro/enhdebug
It does everything the standard DOS DEBUG does and much more but uses less memory. It supports all instructions (including undocumented ones) through Pentium Pro and there is an extended version which can debug DPMI programs.
I had never used the original version but tried recently the version by Japeth for some DPMI based code debugging (using his HX DOS extender as DPMI host).
Back in a DOS days I’ve been using Periscope and Softice actively and now I in the processes of rebuilding my old DOS toolkit. I already got Quaid Analyser and Sourcer. There are just few things missing – including those early version of PS and SST.
I took a better look at site dosfan pointed and there is a great article regarding DOS Int 21h evolution including disassembly of interrupt handler. Great read for all those DOS fans that like to read asm code or analyze DOS internals.
https://sites.google.com/site/pcdosretro/dosdispatcher
If you’re using the older Japheth version of the enhanced DEBUG (1.25 or earlier) then you should definitely update. All known bugs were fixed, new features were added and the code was optimized and cleaned up. Also Japheth disappeared awhile ago so all of his software including his excellent MASM-compatible assembler JWasm are no longer maintained.
Why do you want an older version of the Periscope software ? 5.31 should serve all of your needs though keep in mind it only recognizes up to the 486 and can only assemble 8086 and 286 instructions.
I’m well aware of Japeth disappearance more then a year ago. I hope he is doing fine whatever else he is doing today. Guys at masm board try to resurrect JWasm development. I hope someone will pickup HX development as well eventually.
As for Periscope I just would like to have as much versions as possible. Right now I am testing 5.31 which actually turns out to be 5.0. The 5.40 I have is working fine but I can’t make run.com load executables. It always ends up with wrong pss version. The easy bypass is to use pskey to redirect int 3 to PS and than insert CC opcode at the beginning of debugging file but it adds unnecessary work. Unless I am doing something very wrong the only logical reason I see for this is that the archive I’ve got has mixed up files from different versions.
That does sound like a version mismatch. I have Periscope 5.31 and 5.40 but the original distribution files are on floppies and I no longer have a functioning floppy drive.
I doubt anyone will do anything further with the HX DOS Extender. Interest in DOS seems to have disappeared and there generally isn’t anything else to add to DOS software at this point. I considered putting up some additional DOS programs on PC DOS Retro but I never did since there is little or no interest.
Look at traffic at FreeDOS list and Dos ain’t dead I would say there are at least 3 more people interested in DOS still besides us 😉
As for HX I guess there are still some bugs to be fixed – it is quite complex piece of software, it even outgrown TNT Dos Extender and if you consider the fact that Japeth was also behind JEMM, JWasm, JWLink he actually ofered toolkit quite similar to the Phar Lap one.
But I agree on one point: in many cases there isn’t much to add in terms of functionality to existing DOS software. Many limits had been pushed and moved long time ago. Since HX was the last dos extender under active development/support we can say that Dos Extender era finally come to an end.
I finally added a page about SYSVARS –
https://sites.google.com/site/pcdosretro/sysvars
Great 🙂 I remember when my software stopped working after upgrade to new DOS version. I had to spend some time with debugger (my “beloved” Periscope) in order to fix it until I’ve realized that fields inside List of list/SYSVARS has been rearranged.
Looking at how simple DOS was I must say that List of list was really a bright idea. It also allowed loading device drivers outside of config.sys: a very handy trick.
By the way the table is officially called SYSVARS though INT 21h function 52h is referred to as Get_In_Vars. It was never actually called “list of lists”.
True, I believe the name List of lists came from Undocumented DOS book, at least this is the most famous place I can think of right now. Personally I think that List of list name is better describing the actual structure.
List of lists is a misnomer since not everything in SYSVARS is a pointer to another table. Of course without seeing the DOS source code there’s no way you would know what the structure was actually called.
I have searched many times for the Undocumented DOS book source code, without ever finding a link. Then today I was searching for info on the “PC Inside Out” book, found this page, read the comments, and saw your download links for the Undocumented DOS book source code.
Google seems unable to associate the Undocumented DOS book title with your source code links. Thanks for keeping them alive. I’m glad to finally find them. The books are easy to find, but the source code, not so much. For most people DOS may be dead, but as they say, those who don’t know history are doomed to repeat it.
I know that problem. Used books are not that hard to find (and typically cheap), but the media is usually long gone. Glad you found this site helpful!
I found a link to the source code of the Undocumented PC book:
https://www.codebus.net/d-FsPC.html
You do need to register (free) to download.
I am still looking for the source code to the book DOS and Windows Protected Mode – Programming with DOS Extenders in C by Al Williams.
Has any got that disk and is willing to share that?
NTVDM compatibility on Windows 10 significantly improves when legacy console is enabled. The new console (the one that supports mouse text selection and what not) indeed causes problems.