Complications, Complications

The other day someone asked how hard it would be to modify the Open Watcom linker, wlink, to properly support exports from IOPL segments in OS/2 LX modules. Not terribly hard it turned out, all it needed was to emit a different “bundle type” for exports from IOPL segments. Rather than a regular 32-bit export, it needs to be a special 16-bit call gate export.

A very slight complication is that the 16-bit call gate export is naturally limited to 16-bit offsets, so the linker needs to error out if an attempt is made to export an entry at an offset 65,536 bytes or more into the segment/object. That is easy enough. There are also complications when calling into IOPL segments from the same module, but that’s a separate topic.

When I tried to do rudimentary testing of whether the linker now produces sensible output, I ran into an unexpected problem. For calling into any IOPL segment, in both NE and LX modules, OS/2 uses call gates. The Intel 286/386 call gate mechanism has a provision for the CPU to automatically copy a certain number of parameters (either words or dwords, depending on whether the call gate is 16-bit or 32-bit) when switching stacks. In OS/2, these call gates are always 16-bit, so there are parameter words optionally copied.

The NE and LX executable formats both use the top 5 bits of a flag byte to specify the number of parameter words (those five bits get copied into the CPU-defined call gate descriptor, which also has 5 bits for the parameter count). The Microsoft and IBM linkers (LINK/LINK386), sensibly enough, accept the number of parameter words through the EXPORTS directive.

Now, the Watcom linker already does that too… but things were not adding up. I realized that the Watcom IOPL_WORD_SHIFT macro is not defined as 3 as I expected, but rather as 2. So the number or parameter words in the resulting executable is wrong. Except… hang on a sec.

The wlink documentation is actually pretty clear that the linker takes the number of bytes, not words, to copy—unlike MS/IBM tools. So if you wanted 2 parameter words to be copied, you tell wlink to copy 4 bytes. Instead of emitting 2 << 3 as expected, the linker does 4 << 2 instead, which produces exactly the same result. Except not always because the user can specify, say, 5 parameter bytes, and then a reserved bit will be set in the flag byte.

It gets worse. In the linker documentation one can find the following error message:

MSG 2042: too many IOPL words in EXPORT directive
The maximum number of IOPL words for a 16-bit executable is 63

The error message gives the impression that the EXPORT directive takes words, when the actual directive documentation says bytes. Furthermore, the maximum number of parameter bytes is 63, and even that is wrong—it’s five bits, so the maximum is 31 words, which would be 62 bytes. Clearly people previously working on the compiler were just as confused by this as I was.

That is further evidenced by the fact that internally the linker keeps the size of call gate parameters in a variable named iopl_words, even though the variable actually holds bytes.

Get Me Out of Here!

It would be so much better if wlink accepted the number of parameter words rather than bytes, because that is what the hardware does, therefore that is what the executable format uses, and therefore that is what other tools take as input.

But can I just change wlink to do that? People might be producing 16-bit NE modules with IOPL segments and exports with non-zero parameter word count. If they expect that the linker takes the number of bytes, their code may break in rather nasty ways if the linker behavior suddenly changes.

And therein lies the dilemma: Is the set of such people non-empty? IOPL segments are rather exotic on OS/2, and that’s quite aside from the fact that OS/2 itself is pretty exotic these days. Changing the linker to be more compatible with other tools seems like a good thing, but breaking existing code is not. And determining whether existing code is a concern at all is quite difficult.

Searching the Internet for existing code turned out to be fruitless, but of course that does not prove anything. Finding existing code would have settled the matter, but finding nothing is not meaningful.

MS2WLINK Offers Insight?

Then it occurred to me that perhaps I should check the ms2wlink utility that ships with the Open Watcom compilers. It takes .DEF files intended for Microsoft/IBM linkers and produces command files suitable for wlink.

Checking the ms2wlink source code revealed that it is similarly confused with regard to range checking, but contains this crucial comment:

// convert iopl value to a byte value

The ms2wlink utility expects parameter words on input (same as MS/IBM linkers, as it should) and multiplies the value by two to produce wlink compatible output.

Hmm, so that would be a reason not to change the wlink behavior. Of course I could change both wlink and ms2wlink, but that could lead to quite interesting results when mixing and matching different wlink and ms2wlink versions.

Hitting Paydirt

OK, so search engines got me nowhere, but there is one more thing to check—the Open Watcom source tree itself. And sure enough, I found something. A makefile (\bld\trap\par\os2v2.pio\makefile) with the following, suddenly relevant comment:

Once WLINK properly supports 16-bit IOPL segments in 32-bit DLLs,
the port I/O can be built into the parallel trap file/server modules,
which is where it belongs in the first place.

I’m almost certain that comment was written by me about 20 years ago, and now I’m adding that support for IOPL segments in 32-bit DLLs. Nearby I found what I’d been looking for:

extra_linker = segment 'port' IOPL export INPORT 2, OUTPORT 4

Sure enough, the INPORT function takes 2 bytes or one word, OUTPORT takes 4 bytes or two words. So, someone does rely on that, and never mind that I completely forgot what I’d written aeons ago.

How Did This Happen?

I couldn’t help but wonder if this word/byte mess was merely an oddity or a bug. After reviewing old Watcom C versions, I’m certain it was a bug. The tools behavior and the usage help printed by wlink has not changed since Watcom C/C++ 10.5 (1995).

However, in Watcom C/C++ 10.0 (1994) and older versions, the wlink online help claimed that the EXPORT directive takes IOPL words, not bytes. The catch is that the linker behavior wasn’t any different. So in version 10.5, the linker usage help was adjusted to match what the linker already did.

The linker documentation in version 10.0 (older versions are harder to check because only hardcopy documentation was provided) quite clearly stated that the EXPORT directive took IOPL words, and incorrectly claimed that the maximum was 63 words. The version 10.5 documentation simply changed ‘words’ to ‘bytes’.

There was also a change in the ms2wlink utility behavior. In Watcom C version 9.0 (1992), ms2wlink did not change the number of IOPL words specified in Microsoft/IBM .def files. Someone must have noticed that this produced incorrect results and in Watcom C version 9.5 (1993), ms2wlink multiplied the IOPL words by two.

The oldest version of IOPL-capable wlink that I have been able to examine is version 7.0 (1991) shipped with Watcom C 8.5. It is no different from the later versions. The only even older wlink version I have is 4.1 from 1988, but that is not capable of generating OS/2 executables.

My best guess is that the Watcom linker was always intended to take IOPL words and not bytes as input, as evidenced by the linker source code as well as pre-1995 documentation. Somehow the original authors messed it up, probably by getting the IOPL_WORD_SHIFT macro wrong. Users noticed that to get the expected result, they must feed the linker the number of bytes rather than words. A few releases down the road, the Watcom developers first fixed the ms2wlink utility to account for the linker behavior, and later made the same decision I did—rather than breaking people’s code, document what the linker actually does.

A curious thing about this bug is that although the linker clearly didn’t do what it was supposed to do, by trivially massaging the linker command input users were able to achieve the desired results. When the error was discovered years later, changing the documentation to account for the buggy behavior was the lesser evil.

Conclusion

Fixing past mistakes is sometimes impractical. I can’t make the wlink behavior more sane without breaking existing code. All things considered, living with the odd syntax seems less bad than blowing other people’s code up (even if “other people” might be me).

In the end I decided to leave the “IOPL byte” syntax the way it was. It’s strange and unfortunately different from other tools, but it actually does work, and it’s been that way for quite a long time.

However, I improved the error checking so that the linker complains if an attempt is made to supply an odd number of parameter bytes, and fixed the range checks to allow up to 31 words or 62 bytes. While I was at it, I also changed the IOPL_WORD_SHIFT macro to do what the name suggests, and changed the input processing in the linker such that the iopl_words variable really holds the count of words and not bytes.

Problem solved, about as well as it could be.