Several years ago, I found out the hard way that old versions of DOS have trouble with hard disks with more than 17 sectors per track. To recap, DOS versions older than 3.3 may hang when booting from a hard disk with more than 17 sectors per track, but the exact behavior is somewhat complex.
Old DOS versions required IO.SYS to be contiguous and attempted to load it with a single disk read if possible, but perhaps to keep the code size small, the loading code isn’t very sophisticated and reads sectors from the beginning of IO.SYS until the end of the disk track. With 17 sectors per track, there can never be a problem. With lots of sectors (more than 30 can be a problem) DOS can hang, but the behavior depends on exactly where IO.SYS begins relative to the end of the disk track it’s on. Even on a disk with maximum (63) sectors per track, old DOS versions may boot fine if IO.SYS is sufficiently close to the end of the track.
Where exactly IO.SYS is depends on the size of the FAT tables and the root directory size. In practice the root directory size is more or less fixed, but the FAT size very much depends on the size of the DOS partition. Needless to say, the behavior is unpredictable to users and highly undesirable.
The problem was fixed in 1986 or early 1987, which implies that it was already known. Now I have a good idea why it was known.
Standard ST-506 MFM drives used with the IBM PC/XT and PC/AT always had 17 sectors per track. That was a function of the disk spinning at 3,600 RPM , the drive controller delivering data at 5 Mbps, and the sectors being 512 bytes long. But in the mid-1980s, another disk interface started gaining popularity: ESDI, or Enhanced Small Device Interface.
ESDI was an evolutionary step between ST-506 on one end of the spectrum and SCSI/IDE on the other. The ESDI interface was considerably more intelligent than ST-506, but just like ST-506 (and unlike SCSI/IDE), required a separate controller.
ESDI started out with 10 Mbps data rates and over time achieved over 20 Mbps. ESDI drives were thus inherently much faster than ST-506 drives, but also had more sectors per track. It is obvious that if an ESDI drive rotated at the same 3,600 RPM and used the same 512-byte sectors but moved data at twice the bit rate of ST-506, it had to be able fit more sectors on a track.
And sure enough, ESDI drives commonly had 34 or 35 sectors per track. There were also ESDI drives with more than 1,024 cylinders. And all of that caused trouble.
Translation Required
The problems were clearly known, even if they were not well explained. For example, the documentation for WD1007 ESDI controllers says MS-DOS 3.3 needs to be used, although it is not clear on why. It also says that “some of the older operating systems can only recognize 17 sectors per track”, but does not name any names.
Compaq was much clearer: “MS-DOS 3.1 or earlier supports drive type 25 only”. The 130 MB ESDI disk in the DeskPro 386 could be configured as either type 25 (17 sectors per track) or type 35 (34 sectors per track) in the BIOS. Compaq firmly pointed the finger at DOS.
Note that elsewhere the same document states: “MS-DOS 3.2 or later supports drive type 25 or 35. Depending on the application, there may be some performance improvement by using drive type 35.” Also note that the original DeskPro 386 shipped with DOS 3.1, and Compaq does not appear to have used DOS 3.2 much at all, although it did exist. (DOS 3.2 was generally a fairly short-lived product.)
It is clear from the Compaq and WD documentation that ESDI controllers supported geometry translation. In the early days of ESDI, many users were probably forced to translate the sectors per track from 34 to 17.
This was never a limitation of the PC BIOS, only a limitation of PC operating systems written in an era when only hard disks with 17 sectors per track existed. In the case of DOS, it was clearly not even a conscious decision but a bug that could not be discovered through testing at the time.
ESDI controllers were also capable of working around actual BIOS limitations, notably the 1,024 cylinder limit. The WD1007 controllers supported translations with 63 sectors per track, same as the “maximal” geometry used by IDE drives. The WD1007V-SE2 supported a maximum BIOS geometry of 63 SPT, 16 heads, and 1024 cylinders—the familiar 504 MB limit of the BIOS INT 13h service.
It perhaps does not say anything positive about the PC industry that problems which were known in 1986 or so took a good ten years to solve.
> The ESDI interface […], unlike SCSI/IDE, required a separate controller.
I’m pretty sure SCSI required a separate controller? Or maybe I’m misunderstanding what you mean by “separate”?
A controller external to the drive.
HDDs used to come as FDDs: no board bolted to the drive. That board is
the actual controller.
{ATA,IDE} “controllers” — in common parlance — are more like adapters
that interface w/ the actual controller, which nowadays is bolted to the
drive. Hence “SCSI adapter”. SCSI folks (like me) tended to get quite
indignified by the suggestion that the “SCSI host adapter” (the board
that goes into the {ISA,PCI,…} slot) is a mere controller.
And indeed: SCSI adapters are a lot more capable than {ATA,IDE} ones.
But the latter are still adapters.
I meant a separate disk controller. SCSI hard disks need a separate host bus controller but not a separate disk controller. I realize that was not entirely clear.
There were even ESDI to SCSI adapters, you plugged in an ESDI disk and the controller made it look like a SCSI drive.
IDE comes down to moving the actual controller to the drive, making it
more “integrated”, and in the process moving the {ST-506,ESDI…}
interface to the disk assembly itself.
That makes it necessary to have some other interface between the drive
assembly and the main board. For plain IBM pee-cees, that interface is
now known as “ATA”. ATA (in its “parallel” form) is, as the name
suggests, not really a disk interface as much as it is a subset of the
ISA bus. (IIRC an early 8-bit form of ATA was employed in some Amstrad
machines). Confusingly: “ATA” was originally known as simply “IDE”
(a term which ought to refer to SCSI drives as well). In theory, plain
ATA drives appear to the host as if there was never any ATA in between.
SCSI (in its classic form), the more expensive alternative, is also not
a pure disk interface. Unlike ATA, however, SCSI is based on a
(more-or-less) machine-neutral message exchange and not particularly
tied to pee-cees of any kind. Also, unlike non-disk (or rather:
non-storage) ATA devices (which are rare to nonexistent): SCSI devices
take many forms, and some have even taken SCSI as an attempt to provide
for a generic system bus (but that’s another story).
SCSI survived by decoupling the message exchange from the physical
interface. ATA “survived” as SATA (which, in classic pee-cee style, has
at least 2 different (and AFAIK incompatible) host interface modes).
As Necasek pointed out: converters do exist and thus muddy the picture
somewhat further 🙂
As others wrote, the old ST506 interface required an actual “controller card”. The parallel data cable (the wide one) transferred bits between the drive and the card. The other cable (the narrow one not shared between the two drives that a controller could support) carried data like stepper motor on/off commands for moving the heads and signals to turn heads on/off for reading and writing. The card was, in a very real sense, controlling the drive.
Which is why you could (often) increase the capacity of an MFM hard drive by replacing its controller with an RLL controller (and then low-level formatting the drive). If the drive’s hardware was physically capable of handling the more precise timing required by RLL, doing so would increase capacity by 50%. This worked because the interface cables had absolutely no knowledge of disk sectors – they just connected the drive hardware to the controller card.
ESDI was similar to this. Again, putting all of the responsibility for controlling the drive on that interface card.
Later technologies (including SCSI and IDE), moved all of this controller hardware onto the drive’s own circuitry. So the cables are now carrying higher-level data like requests to read/write specific blocks, leaving it up to the drive’s on-board controller to move and activate the heads appropriately.
This is why a SCSI or IDE interface card is not actually a controller card but a “host bus adapter”. It serves to connect the computer to the data bus used by the drive, but doesn’t control the drive’s hardware. Which is why these buses can be used to connect all kinds of devices including optical drives, Zip drives, scanners and tape drives, not just hard drives.
Back in the day when SCSI was somewhat popular, it was not unusual to see a bus with many attached devices. I personally have an old PC (a Micron dual Pentium Pro system upgraded with Pentium II overdrive processors) that has a hard drive, a zip drive, a DAT drive and a CD burner all connected to a single SCSI adapter. This only works because each drive is controlling its own hardware in response to high level commands relayed between it and the operating system via the SCSI adapter.
Yes, the old ATA lives on in SATA. It’s not called Serial ATA for nothing. As for the two different host interface modes, that existed with ATA too but was not widespread; see for example Pacific Digital’s ADMA (different, incompatible command interface) or Dell CERC ATA100 (IDE RAID controller that looks like a SCSI HBA to the host).
ESDI was definitely similar to ST506, even used the same cabling, but one big difference was that the data separator was on the drive and not in the controller. So no more playing with the data signal and getting 50% more thanks to RLL. ESDI was also a lot more intelligent than ST506 in that the drives could accept a range of simple commands. ESDI also specified things like defect lists, which are standardized in SCSI drives but not IDE drives (that is to say, IDE drives have defect lists, but no standardized way to read them).
ESDI was clearly an improvement over ST506 but it just didn’t go far enough. Between IDE and SCSI, there was no room for it and it vanished pretty quickly. I’m pretty sure that by the time the official ANSI ESDI standard was ratified (1990), ESDI was already on the way out. With the ATA standard, things went differently.
A SCSI HBA is definitely different from a ST506 or ESDI controller in that it’s not at all disk specific, it is “just” a fairly generic data pump. Nowadays USB has taken over a lot of the role SCSI used to play.
By the way, are you the same David Charlap who wrote an article about a minesweeper game in the OS/2 Electronic Developer Magazine? I remember reading that and trying to write my own based on that article.
Not to be confused with the habit of many more modern OSes of treating
even ATA hardware as SCSI 🙂
Or {,S}ATA CD drives. They use a SCSI-over-ATA emulation protocol called
“ATAPI”. To make it even more confusing…
Wasn’t the “data” part of the ST-506 interface essentially analog…?
ESDI kind of filled a similar niche that VLB did.
Many storage devices present themselves as SCSI even over USB (how many
incompatible protocols does the “USB Mass Storage class” provide for,
anyway?).
Yes, it was analog, just like on floppies. The data separator was in the controller. ESDI sent digital NRZ data between the drive and controller, I believe because they didn’t think the ST506 approach would work with double or quadruple the bit rate.
USB MSD specifies some crazy number of command sets, but as far as I know pretty much everything uses SCSI commands.
IIRC, me once had a USB->ATA adapter that presented the drive as ATA
over USB. In fact, me recalls the lunix kernel (me ran lunix back then)
failing to work w/ an old 1.06G Samsung HDD connected that way, for the
stated reason that the drive didn’t support ATA-2. That must’ve been
’round 2005. Me memory of that period is somewhat hazy for some reason,
so don’t shoot me if me recalls wrong.
Every USB->{,S}ATA adapter me’s had since then has presented the drive
as SCSI. Good thing, too…
>early 8-bit form of ATA was employed in some Amstrad machines
and Tandy 1000 (TL/2, TL/3, RL and RLX according to wiki), and Olivetti Prodest PC1, and probably many more.
VWestlife recently did a vid on XT IDE. What is curious is CompactFlash cards have 8bit mode, but usually dont work plugged into XT IDE. Except Simone Riminucci published a tutorial on modding Prodest PC1 in 2014 https://www.ti99iuc.it/web/_upload/image/PC1-OnePage/Docs/GuidaCFModPC1HD/pc1CF-IDE-Guide.pdf
I dont understand this tutorial, there are some straightforward things like cutting reset line, or rerouting A2 line (missing in Olivetti?), but there are also weird ones like pulling ‘IOCHRDY’ to ‘Chip select 3P’. Would love to have a second opinion on what this does and how its making CF work.
DOS 3.3 came with the IBM PS/2, which shipped with an ESDI controller as an option. It also supported multiple 32MB partitions as well.
@rasz_pl:
Me of course figured that more machines would have XTA, but me never
realized it was *that* widespread.
As for CompactFlash: it’s really a shame that ATA was never really used
for more than disks (and tape drives). It would’ve made a large part of
the PCMCIA spec redundant, and it would’ve reduced slot pressure (both
mechanical && allocative).
Missed opportunities…
I would not say XTA was widespread — as evidenced by the scarcity of XT compatible IDE drives. But it was definitely a thing in the latter XT days, and a number of vendors used it.
The “trouble” with PCMCIA was that they adopted ATA quite early, before ATA-1 was even finished. Once the devices were out there, there was probably very little enthusiasm for making major changes.
Certainly. Compaq could finally drop their ENHDISK.SYS 🙂
@Michal Necasek: Yes, I am the same David Charlap that wrote a Minesweeper app for OS/2. I’m surprised anybody still remembers it after all those years.
I developed OS/2 software professionally pretty much up until IBM pulled the plug on it, in the late 90’s. Today, I’m developing embedded software for hardware devices my employer designs. Sometimes based on Linux, sometimes based on microcontroller OS’s like FreeRTOS, and sometimes on a microcontroller’s “bare metal” API.
My memory is weird, that’s all I can say.
I worked on OS/2 drivers professionally after IBM pulled the plug on it, in the early 2000s. If I weren’t doing virtualization nowadays, I’d likely be doing embedded software too.
Long long time ago, there also was the term “AT-Bus”.. AT-Bus HDDs essentially were IDE Hard Disks. Or should I say Fixed-Disks? 😉
To me, “AT bus” just means “ISA”. Not “IDE”. Though as me explained: the
latter is essentially a subset of the former.
Hence also XTA, which is a subset of the (8-bit) XT bus. (Dunno if
there’s any diff between the latter and the original IBM PC one; didn’t
only the slot spacing get changed…?)
It was… complicated. The IDE interface certainly is mostly ISA bus in a different form factor. Mostly.
IDE as such is a nearly meaningless term; SCSI drives have “integrated drive electronics” too. The committee working on the ATA standard struggled with the naming because the interface had no real name. Circa 1987, Compaq called it the “COMPAQ 16-Bit interface” in its technical references. The committee couldn’t use that, the term “IDE” was considered too nebulous, and in the end they arrived at “AT Attachment” without ever explaining whether “AT” stood for anything.
In the late 1980s, IDE drives were often advertised as “AT-bus”. That changed to “IDE” in the early 1990s.
And yes, “AT bus” and “ISA” are IMO interchangeable terms.
Don’t forget EIDE. I think the term was invented by Western Digital in 1994 or so.
Ah yes, EIDE vs. Fast ATA, and no one could quite understand what the difference was.
Did the documentation for RLL controllers ever state that they might not work with DOS versions 3.2 or older?
David C: Some slight mixups in your generally correct description of MFM. The narrow cables transfer data while the wider cable does things like stepping. It’s not a coincidence that the wider cable has the same number of pins as a floppy drive. Although the pinout differs, most of the signals are the same. I.e. they both have signals for the index sensor, write gate, step, step direction, four drive select signals and head select. There is no motor on/off on MFM, and there are three/four head select signals, and of course read and write data uses the narrower cable. In principle you could build a computer that uses the same interface for all those signals both for a floppy disk and an IDE drive, and just have two different parts that process the actual data (or possibly even run the same part at two different speeds). Not sure if that ever did happen, probably not. Older computers tended to use a dedicated floppy controller chip which did drive more or less all the signals. At the time it was cost effective to have a generic I/O chip to drive everything except read data, write data and write gate, like the Amiga did, MFM disks didn’t look like a part of the future. Also those few generic PIO lines did cost so little to implement as compared to the price of a hard disk that I doubt that anyone would had wanted to save that little money on something as expensive as a hard disk setup.
rasz_pl: Compact flash implements the regular 16-bit “IDE” ATA protocol, and an 8-bit variation of it. “XT-IDE” drives seem to derive from some MFM controller (afaik from Seagate) commonly used in XT class PCs. The major differences in addition to the bus width is that ATA uses three address lines for eight possible registers while the “XT-IDE” uses two address lines for four possible registers, and the commands differ too.
The mod for the Italian PC you link to seems cool (although I can’t read Italian). Judging by the pictures it seems like the mod adds the missing address line and does some other minor things (inverting the reset line and whatnot), but the most important part seems to be that it changes the built in BIOS to use the ATA protocol rather than the “XT-IDE” protocol.
So it effectively reuses the existing interface hardware to build an interface for a different type of drive.
If sacrificing half of the disk capacity would be ok then that mod could also use regular ATA/IDE disks and just throw away half of the data bits. (It would probably be wise to add some pull-up resistors to the otherwise not connected lines in order to not leave them floating). Given that both a CF card and the adapter between CF and 40-pin ribbon cable actually cost some money, although not much, while an ATA/IDE disk can be had almost for free, it might make sense to use a mechanical disk this way. For an XT class machie I doubt that it would be a problem if you could only use half of the disk capacity when for example the leftover disk from an XBox upgrade is 8 or 10GB. (The 10GB disk used in later XBoxes are also super quiet, nice for those machines that don’t have a noisy fan).
If my analysis is correct, the problem needed 30+ sectors to occur, so it shouldn’t happen with RLL drives.