Disabling Quick Edit Mode

Last week I decided to finally solve a minor annoyance that’s been pestering me since I switched my main development machine from Windows 7 to Windows 10 over a year ago. As it is with these things, a certain threshold of pain needs to be reached before delving into the unknown, because one never knows how many rabbit holes it will lead to.

The problem: Every now and then, I use the Open Watcom tools, often for cross-development to DOS or OS/2. When I do that, I tend to use the console-based tools, including the vi editor and wd debugger, in part because they work the same across platforms. Both tools can work with a mouse, and especially the debugger is just much productive when used with a mouse. On Windows 7, that works fine. Not so much on Windows 10, because instead of sending mouse events to the application, Windows tries to select text in the console window. Not helpful:

Anyone who has fought with this problem probably knows the answer: Quick Edit Mode. When Quick Edit is enabled, mouse selects text from the console window, Enter copies, right-click pastes. Great for a shell, much less so in a mouse-driven text mode UI. On Windows 10, Quick Edit mode is on by default, and that is what caused me grief.

So, is Quick Edit Mode some kind of Windows 10 novelty? Far from it! Quick Edit has been with us for quite some time, the only change in Windows 10 was that it’s on by default. In other words, the Open Watcom tools have always had the same problem, but no one really noticed.

Programmatically Turning Quick Edit Mode Off

Turning off Quick Edit mode is slightly non-obvious. It can be done through the obvious enough SetConsoleMode API, but as Microsoft documents, it is accomplished through not passing the ENABLE_QUICK_EDIT_MODE flag and passing ENABLE_EXTENDED_FLAGS.

If that sounds a little fishy, it should—and it gets worse.

The Quick Edit Mode has been present in Windows NT ever since the original version 3.1 (1993):

The ENABLE_QUICK_EDIT_MODE flag did not turn up in the NT platform SDKs until many, many years later, in the SDK for Windows Vista (2007). However, since Windows NT 3.51, the ENABLE_QUICK_EDIT_MODE was supported, but undocumented and hidden.

It’s confusing enough that a quick timeline may be in order:

• 1993: Windows NT 3.1 released, supports Quick Edit mode. Applications cannot control Quick Edit mode through GetConsoleMode/SetConsoleMode.
• 1995: Windows NT 3.51 adds ENABLE_QUICK_EDIT_MODE flag, allowing applications to disable it through SetConsoleMode. However, together with Insert Mode, the Quick Edit functionality is undocumented and partially hidden through a flag eventually called ENABLE_EXTENDED_FLAGS.
• 2007: The Windows Vista SDK defines ENABLE_QUICK_EDIT_MODE, ENABLE_INSERT_MODE, and ENABLE_EXTENDED_FLAGS; the functionality is documented but in a woefully misleading way.
• 2015: Windows 10 enables Quick Edit Mode by default, forcing innocent console applications to deal with the complication.

So how does this work in detail? Let’s go through the changes.

First of all, note that when changing the console mode, SetConsoleMode will fail with ERROR_INVALID_PARAMETER if it is passed any flags it does not understand. That complicates writing Win32 applications designed to work across NT versions (we will ignore Windows 9x here, as that has its own set of complexities in the Win32 console department).

In the times of Windows NT 3.1 and 3.5, things were simple. Applications could neither query nor set the Quick Edit Mode state. Starting with NT 3.51, things became much more interesting.

The ENABLE_QUICK_EDIT_MODE and ENABLE_INSERT_MODE flags were present, but considered private, and guarded by ENABLE_EXTENDED_FLAGS. The semantics were so strange that people writing Microsoft’s documentation apparently didn’t understand them either.

It would perhaps be logical if ENABLE_EXTENDED_FLAGS were required to set ENABLE_QUICK_EDIT_MODE and/or ENABLE_INSERT_MODE through SetConsoleMode, and GetConsoleMode would return ENABLE_EXTENDED_FLAGS if either ENABLE_QUICK_EDIT_MODE or ENABLE_INSERT_MODE is set.

But that’s not how it works at all. SetConsoleMode does not need ENABLE_EXTENDED_FLAGS to set ENABLE_QUICK_EDIT_MODE or ENABLE_INSERT_MODE, but does require ENABLE_EXTENDED_FLAGS in order to clear both of the other flags.

GetConsoleMode is even stranger. It may return ENABLE_EXTENDED_FLAGS when neither ENABLE_QUICK_EDIT_MODE nor ENABLE_INSERT_MODE is set, but it may also return none of the three flags even when ENABLE_QUICK_EDIT_MODE and/or ENABLE_INSERT_MODE is in fact set.

In the Windows Vista (NT 6.0) SDK, Microsoft defined ENABLE_QUICK_EDIT_MODE, ENABLE_INSERT_MODE, and ENABLE_EXTENDED_FLAGS in wincon.h. The “new” flags were also documented, but the documentation is misleading. It claims that ENABLE_EXTENDED_FLAGS must be set when setting either ENABLE_QUICK_EDIT_MODE or ENABLE_INSERT_MODE (wrong), and makes no mention that attempting to set these flags on old NT versions or on Windows 9x will cause SetConsoleMode to fail. It is quite possible that people writing the documentation did not understand just how tricky the API is.

As an aside, the Vista SDK also defined the ENABLE_AUTO_POSITION flag but did not document it at all. Testing shows that ENABLE_AUTO_POSITION behaves like ENABLE_QUICK_MODE and ENABLE_INSERT_MODE in that it is tied to ENABLE_EXTENDED_FLAG.

For the sake of completion, it should be mentioned that Windows 10 added and even documented a new ENABLE_VIRTUAL_TERMINAL_INPUT flag. To complete the confusion, this new flag does not depend on ENABLE_EXTENDED_FLAG.

How The Heck…

The behavior described above is probably straight from the How Not To Design an API book. How did Microsoft achieve that?

The ENABLE_EXTENDED_FLAGS bit is actually stored separately in the console state. To demonstrate what havoc that wreaks, let’s consider the following sequence of events:

1. Call SetConsoleMode with ENABLE_QUICK_EDIT_MODE. That enables the Quick Edit Mode and internally sets ENABLE_EXTENDED_FLAGS whether or not the caller specified ENABLE_EXTENDED_FLAGS.
2. Call GetConsoleMode. It will return ENABLE_EXTENDED_FLAGS and ENABLE_QUICK_EDIT_MODE flags both set.
3. Call SetConsoleMode with none of ENABLE_EXTENDED_FLAGS, ENABLE_QUICK_EDIT_MODE, or ENABLE_INSERT_MODE flags set. That could be any application written using SetConsoleMode with flags publicly documented before 2007. This will not turn off ENABLE_QUICK_EDIT_MODE, but it will clear ENABLE_EXTENDED_FLAGS in the console state.
4. Call GetConsoleMode. It will return neither ENABLE_EXTENDED_FLAGS nor ENABLE_QUICK_EDIT_MODE, even though Quick Edit Mode is in fact still enabled.
5. Call SetConsoleMode with ENABLE_INSERT_MODE but none of ENABLE_EXTENDED_FLAGS or ENABLE_QUICK_EDIT_MODE. Now Insert Mode will be enabled, and Quick Edit Mode will be disabled.

Why would Microsoft do such a thing? Most likely there was some reason. The best guess is that some existing applications did not restore the original console state on exit and would turn off both Quick Edit Mode and Insert Mode when run. That could explain why ENABLE_EXTENDED_FLAGS was required to turn off Quick Edit Mode and Insert Mode.

But it does not explain why ENABLE_EXTENDED_FLAGS itself can be turned off. Especially when e.g. with NT 3.51, the default console state is ENABLE_QUICK_EDIT_MODE off, ENABLE_INSERT_MODE off, but ENABLE_EXTENDED_FLAGS on!

Using GetConsoleMode and SetConsoleMode safely

Given the above, it should be apparent that using GetConsoleMode and SetConsoleMode 100% safely is not possible. The reason for that is simple: An application cannot be certain that it queried the true console state on startup, and therefore cannot necessarily restore the state on termination.

Fortunately, in most cases ENABLE_EXTENDED_FLAGS will be set on NT 3.51 and later. When GetConsoleMode sees that flag set, it can be certain that the ENABLE_QUICK_EDIT_MODE and ENABLE_INSERT_MODE bits are also correct. However, if ENABLE_EXTENDED_FLAGS is clear, the status of ENABLE_QUICK_EDIT_MODE and ENABLE_INSERT_MODE is indeterminate.

As long as the application never sets any of ENABLE_EXTENDED_FLAGS, ENABLE_QUICK_EDIT_MODE, and ENABLE_INSERT_MODE, all will be well—and that may be the use case Microsoft was trying to cover. But once any of those bits are set, the state will be modified and cannot be safely restored.

In general, applications should strive to preserve the ENABLE_EXTENDED_FLAGS bit whenever it’s set because that is the only way to make sure GetConsoleMode can fully read the console state.

This mess was directly caused by Microsoft first hiding the undocumented functionality (why?) and later documenting it (because it’s too useful and necessary!). And needless to say, people have been wondering about it before, although no one appears to have tracked how the functionality evolved over time.

Wait, There’s Even More In Windows 10!

Windows 10 now makes things worse for console applications by intercepting the Ctrl+V key combination by default.

Much like the mess caused by Microsoft’s use of the forward slash, Ctrl+V was an idea stolen elsewhere, causing clashes down the road. As we all know, the Ctrl+C and Ctrl+V shortcuts were stolen from the Mac, except in the original they were Command+C and Command+V, and modern Macs have a separate Ctrl key so there is no clash. (To be fair, the shortcuts may have originally been Ctrl+C and Ctrl+V on Xerox PARC machines, but that’s likely not where Microsoft got them from.)

The PC had no Command key and Microsoft used the Control key in Windows. The trouble is that Ctrl+C is a very longstanding key combination used with terminals, going back to the 1960s, and it is in fact something that DOS supported since before Windows and NT supported from the beginning as well.

Windows users have been trained to use Ctrl+C and Ctrl+V for copy and paste, instead of the old IBM CUA Ctrl+Ins and Shift+Ins shortcuts, which perhaps require more finger-twisting but have the major advantage that they don’t clash with anything. (Note that generally the CUA shortcuts work just fine in Windows 10, including in console windows.)

And so we now have Microsoft repeating the stupidity that Linux users have long been cursing, with GUI terminal applications randomly deciding to intercept certain keys and combinations, leaving them unavailable to the software running inside the terminal. If a console application uses Ctrl+V, it may be out of luck on Windows 10 and never see that key combination. This is clearly a clash of expectations where users who never left the Windows island are convinced that Ctrl+V must always mean “paste”, and users who come from different backgrounds are upset that they can’t use Ctrl+V for what they need to.

Apparently hand-editing JSON files now considered to be the solution.

Happy Mousing

Whether Ctrl+V is working or not, I can now use the mouse again with the console wd debugger. Quick Edit Mode gets automatically turned off:

The mouse works, left clicking triggers menus and selects symbols, right clicking brings up context menus:

I have managed to scratch this particular itch, but the rabbit hole that the fixing led to turned out to be really deep.