AVX support disrupts WoW64 debugging

Sometimes, the old and the new intersect in unexpected ways. After upgrading to a Sandy Bridge based system (Core i7) and Windows 7 SP1 64-bit some time ago, I noticed that debugging exceptions in 32-bit user programs didn’t quite work right. Recently the issue has been brought to my attention again and I had to do some digging.

The symptom of the problem is that when a debugger stops on a second chance exception, the context of the (32-bit) process being debugged is damaged and the debugger stops in a bogus location, namely ZwRaiseException or NtRaiseException (same thing with two different names). That is a huge problem when the debugged process handles certain exceptions. The debugger can’t stop on every first chance exception, because those occur in the normal flow of execution. Yet if a real bug pops up and causes an unhandled exception, the debugger will not show where the problem is!

The issue is 100% reproducible… but only on some systems. The ingredients are: 64-bit Windows 7 SP1, a 32-bit process being debugged, and a recent CPU. The process doesn’t matter, the type of exception doesn’t matter, the debugger doesn’t matter (both 64-bit and 32-bit debuggers are affected). Software installed on the system doesn’t matter beyond the base OS.

Problem

A log of a short debugging session with WinDbg/NTSD illustrates the problem:

(7b8.4d4): WOW64 breakpoint - code 4000001f (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
ntdll32!LdrpDoDebuggerBreak+0x2c:
77000fab cc              int     3
0:000:x86> g (7b8.4d4): Integer divide-by-zero - code c0000094 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
*** WARNING: Unable to verify checksum for crashm.exe
crashm!main+0x10:
00401020 f7f9            idiv    eax,ecx                           <----- correct
0:000:x86> k
ChildEBP RetAddr
0018ff48 004010e5 crashm!main+0x10 [crash.c @ 7]
0018ff88 75dd339a crashm!mainCRTStartup+0xb4
0018ff94 76f99ef2 kernel32!BaseThreadInitThunk+0xe
0018ffd4 76f99ec5 ntdll32!__RtlUserThreadStart+0x70
0018ffec 00000000 ntdll32!_RtlUserThreadStart+0x1b
0:000:x86> g
(7b8.4d4): Integer divide-by-zero - code c0000094 (!!! second chance !!!)
ntdll32!ZwRaiseException+0x12:
76f815de 83c404          add     esp,4                             <----- wrong
0:000:x86> k
ChildEBP RetAddr
0018fa7c 76f7014d ntdll32!ZwRaiseException+0x12
0018fa8c c0000094 ntdll32!KiUserExceptionDispatcher+0x29
WARNING: Frame IP not in any known module. Following frames may be wrong.
0018ff48 004010e5 0xc0000094
0018ff88 75dd339a crashm!mainCRTStartup+0xb4
0018ff94 76f99ef2 kernel32!BaseThreadInitThunk+0xe
0018ffd4 76f99ec5 ntdll32!__RtlUserThreadStart+0x70
0018ffec 00000000 ntdll32!_RtlUserThreadStart+0x1b
0:000:x86> g
(7b8.4d4): Unknown exception - code 00000000 (first chance)
(7b8.4d4): Unknown exception - code 00000000 (!!! second chance !!!)
ntdll32!ZwRaiseException+0x12:
76f815de 83c404          add     esp,4
0:000:x86> g
WARNING: Continuing a non-continuable exception
(7b8.4d4): Unknown exception - code 00000000 (first chance)
(7b8.4d4): Unknown exception - code 00000000 (!!! second chance !!!)
ntdll32!ZwRaiseException+0x12:
76f815de 83c404          add     esp,4

The first chance exception is reported correctly, but as mentioned earlier, stopping on first chance exceptions may not be feasible. The second chance exception is erroneously reported in ZwRaiseException, which is completely bogus. The stack is damaged enough that execution cannot be continued at all, which compounds the problem.

It’s worth pointing out that on the damaged stack, the first bogus frame (going from bottom to top as printed) shows the location as 0xC0000094. That certainly does not look like a valid address, but it looks exactly like the exception code of a division by zero. If ever something looked like a coincidence… this is not it.

Yet that only shows Windows is doing something wrong, not what or why.

For reference, this was the test program:

#include <stdio.h>

int main( int argc, char **argv )
{
    int i;

    i = 33 / (argc - 1);
    return( i );
}

The testcase was written in a slightly convoluted way to defeat compiler optimizations and ensure a division error will happen at run-time.

The compiler and linker used doesn’t matter, and the bug can be reproduced with a tiny assembler program as well.

Clues

While investigating the problem, I naturally tried to reproduce it in a VM, as it would have been easier to analyze the issue in a virtual environment. But… the problem didn’t happen in a VM. The Windows 7 version was exactly the same, the host CPU was the same, the software used was the same. But what failed on the host stubbornly worked in a VM, no matter how the host and guest were configured (memory size, number of CPUs, etc.).

I knew that was a clue, but couldn’t interpret it. One possibility was that virtualization masked some new-fangled CPU feature and that hid the problem. That could explain why the bug showed up on Intel Core i7 and i5 systems, but not Intel Core 2 or AMD Phenom machines.

A few days later, a coworker mentioned in passing that the AVX instruction set requires a larger memory area to save the state. That got me wondering, because in earlier investigation I noticed that on the broken systems, there seemed to be more data on the exception stack.

Microsoft documents that AVX is indeed supported and enabled by default on Windows 7 SP1. AVX state is indeed saved when exceptions are dispatched. Hmm, could that be it?

To confirm or disprove the theory, I’d need a way to disable AVX. That cannot be done in the system’s BIOS. But hey… Microsoft documents how to disable AVX in Windows as a workaround for a completely unrelated problem. The xsavedisable switch is not properly documented, but maybe it works…

Solution

I executed

bcdedit /set xsavedisable 1

from an elevated command prompt and rebooted the system. Re-ran the debugger testcase and lo and behold, now exception handling works properly!

The problem is definitely a bug in Windows 7 SP1 on AVX-enabled systems; however, it seems to be specific to the debug support in the WoW64 component. A 32-bit version of Windows 7 SP1 behaves correctly, and so does debugging of 64-bit processes. A 32-bit process can handle its own exceptions correctly when running under WoW64, it’s only when a 32-bit process is run under a debugger within WoW64 that trouble strikes.

I can only guess that with AVX on, the kernel saves AVX context on the exception stack in the user process (after continuing a first chance exception), but WoW64 does not expect the additional data when a debugger is attached and messes up the stack. That of course does no good and the user process is toast.

For now, disabling AVX is an acceptable workaround for me. Hopefully the bug will be eventually fixed and it will be possible to both use AVX and debug 32-bit processes properly.