Unintended Side Effects

I’ve been lately trying to understand and improve the idling behavior of DOS programs and one of my guinea pigs has been the Open Watcom vi editor.

While the 16-bit real-mode DOS variant of the editor idles nicely, the 32-bit DOS extended version that’s actually shipped with the Open Watcom tools just refuses to idle. What’s notable is that it won’t idle with DOS POWER.EXE, with DOSIDLE, and it also won’t idle when run under Windows 9x or XP. And it doesn’t idle with my DR-DOS $IDLE$ driver either.

So I decided to track down what’s going on in the DR-DOS case. What I could see was that the DRIDLE.SYS driver never went into idle mode because every time it decremented the INT 16h idle counter, it was already reset to the initial (maximum) value by a timer tick.

In other words, as far as the DRIDLE.SYS was concerned, the editor wasn’t idle because it was only calling INT 16h very rarely. But I know that the editor polls INT 16h in a tight loop!

While trying to characterize the problem, I also found another oddity: The refusal to idle happens with the (default) DOS/4GW extender. But when the same vi.exe program is run using the CauseWay or DOS/32A DOS extenders (both also shipped with Open Watcom), the editor idles just fine!

Initially I thought there could be some problem specific to the protected-mode version of the editor, since the code is necessarily slightly different from the real-mode variant. But if it were the editor itself, it would behave the same with all DOS extenders, wouldn’t it?

The only option was to step through the code and see what actually happens. With a profiler I could see that yes, INT 16h does get invoked many, many times per second. So I put a breakpoint on the protected-mode INT 16h entry point, which is what gets executed when a protected-mode program executes the INT 16h instruction.

What I found was… unexpected. DOS/4GW, like all DOS extenders, installs its own INT 16h handler which calls down to the real-mode handler, and provides whatever input and output parameter conversion is necessary.

Only the INT 16h handler in DOS/4GW does a little bit more than that. It has special handling for INT 16h functions 00h and 10h (read keystroke), plus functions 01h and 11h (check if keystroke available). It is the latter pair that’s used by applications polling the keyboard.

The logic is roughly as follows. When INT 16h function 01h or 11h (polling) is called, DOS/4GW checks if the timer has ticked (by examining whether the byte at 40:6C is different from its last saved value) or if an internal flag is set. When neither is true, the INT 16h handler returns to the caller without passing down to the real-mode INT 16h handler and without checking the BDA to see if a keystroke is available.

When INT 16h function 00h or 10h is called, DOS/4G decrements the internal copy of the timer tick low byte at 40:6c, and passes down to the real-mode handler.

As far as I can see, this is some sort of optimization which avoids frequent transitions between 32-bit and 16-bit code, and depending on host configuration, also avoids frequent switching between protected and real mode. Especially in cases where switching between real and protected mode is required, this optimization likely greatly reduces interrupt latency (by significantly reducing the periods of time where the CPU is unable to take interrupts).

The optimization has a minor side effect in that a newly arriving keystroke won’t be seen until the next timer tick. However, if keystrokes arrive quickly, they will also be processed quickly (because every time a keystroke is read, the next poll will call into the BIOS). I can’t say I ever noticed this delay which can be up to about 1/18th of a second.

Unfortunately, the DOS/4G logic also has another side effect in that it completely defeats idle detection based on frequent INT 16h polling. Because the real-mode INT 16h handler is almost never executed when polling, idle detection fails because the program does not appear very idle at all.

This of course also explains why other DOS extenders behave differently: Their protected-mode INT 16h handler does not avoid calling into real mode, and therefore idle detection does not get bypassed.

Is There a Workaround?

Remember how I mentioned that there’s some kind of internal flag that controls whether INT 16h functions 00h/10h should be skipped or not? Presumably there must be some way to set it…

And sure enough, there is. It is in fact documented in a file called DOS4GW.DOC, which is conveniently shipped with the Open Watcom tools. Except the explanation, frankly, does not make a lot of sense.

To disable the optimized INT 16h polling behavior in DOS/4G(W), one has to set the DOS16M environment variable as follows:

SET DOS16M=^0x04

Yes, that’s a caret, which may have a special meaning in some command processors like 4DOS. With this setting, the Open Watcom protected-mode vi.exe magically idles just fine!

But the explanation for this setting is rater strange. According to DOS4GW.DOC, the flag does this:

0x04  directly pass down keyboard status calls -- When this option is
      set, status requests are passed down immediately and
      unconditionally.  When disabled, pass-downs are limited so the
      8042 auxiliary processor does not become overloaded by keyboard
      polling loops.

What’s strange about that? In a standard PC/XT/AT BIOS, INT 16h function 01h/11h normally does not touch any keyboard hardware at all. It only checks whether the keyboard buffer head and tail pointers in the BDA are the same or not. And besides, if the 8042 keyboard controller really were overloaded by frequent polling, why would it not happen in real mode as well?

The explanation in DOS4GW.DOC has all the hallmarks of a desperate fix for a problem that is not fully understood. There’s no doubt that skipping the INT 16h polling call-downs to real-mode BIOS did something, on some machine, at some point in time. But I rather doubt that it worked for the reasons the authors imagined it did.

The fix was almost certainly implemented at a time before POWER.EXE was common, and the power save disabling side effect was not noticed. Even on an old laptop with power management, the problem could well have gone undetected, given the absence of a loud fan that aggressively kicks up. However, the fact that the behavior could be overridden using an environment variable suggests that the implementors strongly suspected that skipping calls to the BIOS might cause trouble.

I’m not entirely sure when this fix was implemented. A quick survey reveals that it was already there in DOS/4GW version 1.8, shipped with Watcom C 8.5 (1992). The behavior remains unchanged at least up to and including DOS/4G version 2.60 (1997). In all cases, the workaround using the DOS16M environment variable also applies.

All in all, another curious mystery solved.

Update: In Windows XP, the caret is a special character. To set the environment variable correctly, one must type:

SET DOS16M=^^0x04

If only a single caret is used, it will be left out entirely and the environment variable will have no effect.