Semantic Differences, Microsoft v. Microsoft

While comparing the behavior of various versions of old Microsoft C compilers, I tried building a trivial hello-world type program with CL.EXE from Microsoft C/C++ 7.0 (March 1992) running on top of a 32-bit Windows Server 2003. This seemingly trivial task failed:

Command line error D2018 : cannot create linker response file

A quick search revealed that rather predictably, I wasn’t the first person hitting this problem. And as usual, there were only questions, not answers.

MS C/C++ 7.0 was Microsoft’s first compiler with C++ support included, but that’s not what made it special. It was Microsoft’s first DOS/Windows 3.x compiler which required a 386 host and used a 32-bit DOS extender.

Looking at the C/C++ 7.0 executables a bit closer, it’s apparent that these are in fact 32-bit PE modules, with a DOS extender in their respective DOS stub. The catch is that due to their age (released more than a year before NT 3.1), the PE modules are different enough that no NT release can use them. Hence the compiler must be run through NTVDM as if it were a normal DOS executable.

Now, given that when NT 3.1 was released (1993), Microsoft C/C++ 7.0 was still a current product, one would expect that the compiler would run under NT 3.1… and indeed it did! MS C/C++ 7.0 also works under NT 3.50 (1994), but it no longer functions under NT 3.51 (1995). The compiler fails with the above D2018 error message in NT 3.51 and all subsequent releases. At that point, Visual C++ 1.0/1.5 had taken over and C/C++ 7.0 was more or less obsolete, so perhaps the problem slipped under the radar, but why did the compiler stop working at all? What changed?

The reason why the compiler fails on newer NT versions is its questionable use of the seek function (INT 21h/42h). In DOS, seeking to negative offsets succeeds, although subsequent read/write operations will fail. In NT (Win32 API), seeking to negative offsets fails immediately.

The Win32 API behavior has been consistent since NT 3.1; however, the NTVDM behavior has not. In NT 3.1 and 3.50 NTVDM, seeking to negative offsets succeeds, while in NT 3.51 and later it fails.

Clearly this is a bug which somehow crept into NTVDM, since NTVDM is supposed to emulate DOS semantics. Why Microsoft C/C++ 7.0 wants to seek to negative offsets (on a temporary file) is not clear, but it’s apparent that this is why MS C/C++ 7.0 fails on NT 3.51 and later.

Seeking to negative offsets is a questionable practice and its semantics changed over the years. It is especially problematic when seeking is implemented via the classic lseek() routine:

off_t lseek(int fildes, off_t offset, int whence);

While it makes good sense to for the offset argument to be negative when seeking backwards, there simply are no negative offsets in a regular disk file. Hence the return value indicates a failure when the final file position is negative. If seeking to negative offsets is allowed, it becomes difficult to distinguish between failures and successful seeks to negative offsets.

But it wasn’t always so. The Open Group standard provides some hints, and DOS is of course very old—the seek functionality first appeared in DOS 2.0 and was implemented sometime in 1982 or so, with the semantics being derived from then-current XENIX (likely UNIX Version 7). Moreover, the DOS API indicates errors through the carry flag, which means that there is no trouble distinguishing failed seeks from successful seeks to negative offsets.

There does not appear to be any fix for MS C/C++ 7.0. It’s unclear why the NTVDM semantics changed in NT 3.51, but it is understandable that C/C++ 7.0 was no longer important; extremely few applications would have been affected by the change, though it’s ironic that Microsoft’s own compiler was one of them.

The RBIL notes the difference in behavior between DOS and NT when seeking to negative offsets, although it incorrectly implies that all NT versions behave the same. The OS/2 MVDM, needless to say, behaves correctly and executes Microsoft’s CL.EXE compiler driver without trouble.

MS C/C++ 7.0, a victim of MS C/C++ 7.0?

But wait… there’s more to the story. Microsoft was always aware of the semantic differences between DOS and other operating systems (which included OS/2). The lseek() routine in older Microsoft compilers—at least in C 4.0 (1986), C 5.1 (1988), and C 6.0 (1990)—explicitly guarded against seeking to negative offsets. When a backward seek was attempted, the library first checked whether it would end up at a negative offset, and if so, the call failed.

A new feature of the C/C++ 7.0 run-time library was the LSEEKCHK.OBJ file which, when linked in, forced the lseek() routine to return an error if an attempt was made to seek to a negative file offset on DOS, just like earlier versions of Microsoft C. The exact same error checking in the lseek() routine was present, but the big difference from the previous compiler versions was it was now turned off by default. The Microsoft Visual C++ 1.0 and 1.5 run-time libraries behaved the same as MS C/C++ 7.0.

At this point, it is only a speculation that CL.EXE shipped with Microsoft C/C++ 7.0 used the compiler’s own library with changed lseek() semantics. Detailed analysis was not made, but it seems at least likely.

Why MS C/C++ 7.0 introduced this change at all is unclear, and the documentation does not provide any hints. It seems odd in light of the fact that neither OS/2 nor NT allowed seeking to negative offsets anyway, and older versions of Microsoft’s run-time library forced the more portable semantics. But it is certainly ironic that the CL.EXE tool shipped with C/C++ 7.0 itself ran afoul of these semantic differences and bugs introduced in another Microsoft product.