It has only recently been brought to my attention that Intel’s SYSENTER/SYSEXIT instructions have rather unusual past, and their origin is shrouded in mystery and confusion. One facet of the usage of these instructions is also a little unorthodox.
Depending on who you ask, the SYSENTER/SYSEXIT instruction pair was introduced either in the Pentium Pro or in the Pentium II. But that makes absolutely no sense on the face of it, right? Either the Pentium Pro supported these instructions or it didn’t. Well…
The Pentium Pro (aka P6) was introduced in November 1995 and it was the first member of a tremendously successful family of processors which destroyed all of x86’s RISC rivals in the late 1990s, and quite possibly saved Intel after the Itanium and NetBurst debacles. The Pentium Pro and its immediate successors, Pentium II and Pentium III, turned out to be immensely scalable and between 1995 and 2001, raised the clock speed from 150 MHz (original Pentium Pro) all the way to 1.4 GHz (Pentium III-S and other Tualatin PIII variants).
Undocumented?
The Pentium Pro also introduced a few new instructions, such as CMOV/FCMOV (conditional moves) or RDPMC, and a number of architectural features (global pages, page address extensions, machine check exceptions, MTRRs). The Pentium Pro documentation from December 1995 (Pentium Pro Family Developer’s Manual, Volume 2: Programmer’s Reference Manual, Intel order no. 242691-001) certainly made no mention of SYSENTER or associated MSRs.
On the other hand, Intel’s U.S. Patent no. 5948097 (Method and apparatus for changing privilege levels in a computer system without use of a call gate, Glew et al.) filed in August 1996 describes SYSENTER in detail and says: “For one embodiment, SYSENTER and SYSEXIT are assembly-language instructions that may be executed on an Intel architecture processor, such as the Pentium Pro processor.”
The book Pentium Pro and Pentium II System Architecture (Shanley) claims that SYSENTER/SYSEXIT were introduced in the Pentium II but notes that “these instructions are not documented in the currently-released (as of 8/7/97) Intel manuals”. The book was published in December 1997. The book also says that SYSENTER/SYSEXIT capability is indicated by the SEP bit in the CPUID feature mask, but only if the processor is family 6, model 3, stepping 3 or greater. That is quite unusual and goes against the logic of CPUID.
The Pentium II Processor Developer’s Manual from October 1997 (Intel order no. 243502-001) contains absolutely no mention of SYSENTER or SYEXIT, and neither does the Intel Architecture Software Developer’s Manual from December 1997 (Intel order numbers 243190-001, 243191-001, 243192-001).
On the other hand Intel’s Application Note AP-485, Intel Processor Identification and the CPUID Instruction from June 1997 (Intel order no. 241618-007) is slightly older and does mention SYSENTER and SYSEXIT, but only explains how to detect the presence of the instructions, not how to use them. It is likely that AP-485 is the source of information in Pentium Pro and Pentium II System Architecture.
The second edition of the Intel Architecture Software Developer’s Manual is from April 1999 (Intel order numbers 243190-002, 243191-002, 243192-002) also covers the Pentium III and does properly document SYSENTER and SYSEXIT—as instructions introduced in the Pentium II. The instruction set reference gives the following pseudocode for detecting SYSENTER presence (already seen in the AP-485 CPUID note):
IF (CPUID SEP bit is set)
IF (Family == 6) AND (Model < 3) AND (Stepping < 3)
THEN
Fast System Call NOT supported
FI
ELSE Fast System Call is supported
FI
It also says: “The Pentium Pro processor (Model = 1) returns a set SEP CPUID feature bit, but does not support the SYSENTER/SYSEXIT instructions.” Curiously, the SYSENTER MSRs are not listed in the MSR reference (Appendix B of Volume 3: System Programming) at all.
It is notable that the current (July 2017) Intel SDM lists the SYSENTER MSRs as architectural MSRs supported on the Pentium Pro and later (not Pentium II and later as one might think).
Documented Errata
In October 1998, the Pentium II Specification Update (Intel order no. 243337-019) added erratum A62, SYSENTER/SYSEXIT instructions can implicitly load “null segment selector” to SS and CS registers. The erratum does not affect a typical operating system and wouldn’t be very noteworthy on its own.
What is quite noteworthy is that the same erratum (SYSENTER/SYSEXIT instructions can implicitly load “null segment selector” to SS and CS registers) was also added as erratum 82 to the October 1998 edition of the Pentium Pro Processor Specification Update (Intel order no. 242689-032). The obvious question is how that is possible, if the Pentium Pro supposedly does not support SYSENTER/SYSEXIT?
Some have claimed that erratum 82 is in fact the reason why SYSENTER support should be disabled on Pentium Pros. That is extremely difficult to believe when the Pentium II has the exact same erratum.
Out in the Wild
It wasn’t until the early 2000s when SYSENTER/SYSEXIT started being used; given that public documentation may have only surfaced in 1999, that should not be surprising. Windows XP (2001) added optional support for SYSENTER and SYSEXIT. In 2003, Linux introduced SYSENTER support in the 2.6.0 kernel.
And immediately there was trouble. The initial Linux code caused system crashes on Pentium Pro CPUs. The reason? Linux implemented the model/stepping version check documented by Intel in the SDM instruction reference.
To this day (July 2017), the Intel SDM instruction reference claims that the check for CPUs which have the SEP bit but do not support SYSENTER is ‘IF (Family = 6) and (Model < 3) and (Stepping < 3)’.
But sometime in 2000 or so, the CPUID application note was changed to list the SYSENTER support condition to ‘IF (Family == 6) AND (ModelStepping < 0x33)’ which is certainly not the same! Notably for later-model Pentium Pros with model 1, stepping 9, the two conditions give different results.
As Linux users discovered, the condition given in the instruction reference is incorrect and SYSENTER (or is it SYSEXIT?) does not behave as expected even on later Pentium Pros.
An interesting question is the Pentium Pro OverDrive, in reality a Deschutes Pentium II. The PPro OverDrive identifies itself as model 3, stepping 2 (in reality it is a close relative of model 5 Pentium IIs). According to the CPUID test, it does not support SYSENTER. According to common sense, it really should.
So What Happened?
The unanswered question is what really happened (and even Andy Glew, the author of the SYSENTER patent and designer of the instruction doesn’t know). SYSENTER certainly was part of the Pentium Pro design. It seems likely that Intel discovered some problem with the SYSENTER instruction very shortly before the chip was released. And instead of documenting the problems and fixing them, Intel decided not to document the instruction at all.
Given that existing software had no expectations about SYSENTER, that didn’t cause trouble. Only after the Pentium II came out and SYSENTER was documented, it became apparent that it had been present on the Pentium Pro all along, but was either buggy or functioned differently. It is not obvious why Intel chose not to fix it in later Pentium Pro steppings.
So… what does SYSENTER and SYSEXIT actually do on Pentium Pro CPUs? How is it different from the later models? And what about the Pentium Pro OverDrive? So many questions.
> The Pentium Pro (aka P6) was introduced in November 1995 and it was the first member of a tremendously successful family of processors which destroyed all of x86’s RISC rivals in the late 1990s, and quite possibly saved Intel after the Itanium and NetBurst debacles. The Pentium Pro and its immediate successors, Pentium II and Pentium III, turned out to be immensely scalable and between 1995 and 2001, raised the clock speed from 150 MHz (original Pentium Pro) all the way to 1.4 GHz (Pentium III-S and other Tualatin PIII variants).
And much more than that too – Pentium M was a lightly refreshed P6, Core 2 was a massively overhauled and improved design, but still P6 derived, and IIRC, everything after Core 2 is a descendant of that. So yeah, Kaby Lake is a (distant) descendent of the Pentium Pro. When you consider Intel’s last from-scratch x86 designs were Bonnell and NetBurst, that might be a good thing!
This is now keeping me wondering almost as much as the question what the perpetually “reserved” CR1 ever did.
At least with Pentium Pro’s SYSENTER implementation, we have a chance of finding out through experimentation (I consider it unlikely that they disabled the instruction, given both the timeframe and that it probably would have been easier to clear the flag in CPUID).
From what I read, the instruction exists and is not disabled on PPros (that’s what Linux users found out when 2.6 kernel enabled SYSENTER use). But it does not behave the same as on PII and later. So yes, it should be possible to discover what it does through experimentation.
Yes, the Pentium M was an updated/reworked P6, hybridized with the NetBurst front side bus and optimized for lower power usage. It is also my understanding that Core/Core 2 and the current crop of CPUs are still P6 derivatives, but I can’t judge if the similarity is still significant or not. But I think there’s no question that the Pentium Pro was a one of the most significant microprocessor designs.
If it was an intermittent bug, it might be impossible to figure out what SYSENTER does on the Pentium Pro unless someone at Intel releases information describing a working case. I expect that there was a case that worked at least some of the time or Intel would have completely blocked the instructions.
Note that Intel went through two Pentium II designs before getting a quad processor replacement for the Pentium Pro so it wouldn’t be completely impossible that the initial Pentium II fixes interfered with multi-socket designs.
In practice, not documenting the instructions was just as good as blocking them.
The thing with late multi-socket PIIs was interesting. The black Pentium Pro with 1M L2 cache lasted quite a while.
> As Linux users discovered, the condition given in the instruction reference is incorrect and SYSENTER (or is it SYSEXIT?) does not behave as expected even on later Pentium Pros.
SYSEXIT. The LKML archive link states that SYSENTER worked fine and the system call executed successfully, only to double fault at SYSEXIT later. This suggests that SYSENTER to call and IRET to return could’ve worked and still be faster than INT + IRET.
Anyone with a Pentium Pro machine and kernel debugging knowledge to test what’s actually going on with SYSEXIT around?
I have both, but currently not the time to set up an experiment. It should be possible to figure out just what exactly goes wrong on the Pentium Pros. I also want to check a Pentium Pro OverDrive because I’d be shocked if SYSENTER didn’t work there.
I suppose back in the day, no one cared enough because when Linux introduced SYSENTER support, Pentium Pros were already quite outdated (there were probably ~3 GHz P4s on the market, and 1+ GHz PIII-based Xeons for servers were already old news).
> there were probably ~3 GHz P4s on the market
It was 2002, so Northwoods were brand new parts, and since it’s NetBurst, they weren’t much faster than an average abacus despite the clock speed. But yeah, point taken, PPro was already 7 years old technology and everyone was on PIII or P4, so nobody bothered.
The Linux 2.6 trouble post that I found is actually from August 2003. Either way, most of the world at the time was likely running at least a Pentium II, which itself was 5 years old in 2007 and far surpassed in performance in 2002/2003. That plus the Pentium Pro was never all that widespread to begin with. I bet there were lots more Pentium MMXs than Pentium Pros in operation in 2003.
I wonder whether it was something on Socket 8 boards that caused SYSEXIT to misbehave. That would certainly explain why Pentium II OverDrives, despite being Deschutes chips, reported a CPUID of 0x1632, ie. the kind of CPUID that would make software not use SYSENTER and SYSEXIT.
That doesn’t make much sense to me (it should be a purely CPU/software thing, not socket/board related), but who knows. The OverDrive CPUID was AFAICT what it was because intel specified back in 1995 or 1996 what it was going to be. That is why it didn’t reflect a Deschutes CPU when the PII OverDrive finally materialized. The guidelines for SYSENTER use were again set by Intel, but the reasons for sorting the CPUs that way the are less than obvious. I think the reality is that by the time SYSENTER was in common use, Socket 8 boards were all but gone, and they were never all that common to begin with.
On some rainy day I’m going to play with a Socket 8 board and a few CPUs.
Pingback: Relationship between system calls API, syscall instruction and exception mechanism (interrupts) – Developer FAQs
Michal, is there any chance you could investigate what was going on with SYSENTER/SYSEXIT on those CPUs?
There is, but it will take me some time. I have the hardware, but am in the middle of a move. If you can think of some good testcase, please let me know.
I have a quad-socket Pentium Pro (part SL22Z, 200 MHz) that I can test on. Sounds interesting.
I have now formulated a hypothesis as to what might have went on with SYSENTER/SYSEXIT on the Pentium Pro. It is based on two pieces of evidence:
1. The reports that Linux experienced kernel panics when they attempted to use these instructions on the Pentium Pro;
2. These instructions causing the exact same symptom (Linux kernel panicking and Windows XP BSoD’ing) on 86Box and PCem until it was fixed on PCem by making the STI instruction delay interrupts by one instruction.
What I think happened is that Intel added SYSENTER/SYSEXIT to the Pentium Pro, but did not make STI delay interrupts, and that caused problems that Intel likely got test reports of, and since it was too late to fix the problem, Intel just pretended the instructions to do not exist. I also suspect the problem carried over to the very first Kalamth Pentium II’s (those with CPUID 0x0632), hence why Intel instructed developers to not use those instructions on CPUID 0x0632, either. And it had to be fixed by CPUID 0x0633.
What exactly is the sequence there? STI followed immediately by SYSEXIT? STI has always blocked interrupts for the next instruction, but it’s plausible SYSENTER/SYSEXIT could have broken that.
I don’t know exactly what’s wrong with the PPro/early PII implementation but I agree that it’s almost certainly something that Intel discovered late in the game, couldn’t fix the hardware, and figured that simply not documenting those instructions was the quickest and cheapest solution that didn’t hurt anyone (until much later when Intel documented the instructions).
I should add that I can easily imagine that after interrupts have been disabled for a while, the likelihood that there is an interrupt pending when the STI gets executed is not negligible. Probably doesn’t take long at all for things to go boom.
From what I know, STI did not delay interrupts until after the next instruction at least on the 8088 (when I made STI delay interrupts on my emulated 8088, the PCjr BIOS broke), and until SYSENTER/SYSEXIT, there was no real reason to delay them because they couldn’t cause much of a problem. It’s with SYSENTER/SYSEXIT that it can be problematic, as an interrupt between STI and SYSEXIT is going to cause another SYSENTER/SYSEXIT pair, which will then result in the SYSEXIT of that pair followed by the SYSEXIT of the previous pair, making the second SYSEXIT GPF(0) because it would get called with CPL = 3 (because SYSEXIT sets the CPL to that), which is invalid. Which is exactly what was happening on the emulator before STI was fixed to delay interrupts.
That’s why I wonder if that might have been the case on real hardware as well, until it was fixed on Pentium II’s with CPUID 0633. A good way to shed more light on this would be to find out when exactly Intel made STI delay interrupts until after the next instruction.
I just wanted to add that I have no idea how the interrupt delay is implemented in the real microcode. Perhaps just a flag is set that the next instruction has to honor, in which case, it could simply be that SYSEXIT originally did not honor the flag, which could explain the problems even if the interrupt delay had been added on earlier CPU’s.
STI absolutely did delay interrupts from the beginning of x86. It delayed interrupts only if interrupts were actually being enabled (so a long sequence of STIs does not prevent interrupts from being delivered). Starting out with interrupts disabled, a STI/CLI sequence will not allow interrupts. Some 486s reportedly have a bug where a 1-cycle instruction between STI and CLI may also prevent interrupts from being recognized.
Quoting Intel iAPX 86,88 User’s Manual, August 1981: “STI (Set Interrupt-enable flag) sets IF to 1, enabling processor recognition of maskable interrupt requests appearing on the INTR line. Note however, that a pending interrupt will not actually be recognized until the instruction following STI has executed.”
And yes, delaying interrupts is really critical with instructions like SYSEXIT or SWAPGS which do not nest. I believe the original idea was to allow an interrupt service routine to run with interrupts disabled and return with STI/RETF rather than IRET. The delaying effect of STI is clearly far older than PPro.
Yes, conceptually STI sets an internal flag “enable IF at the beginning of next instruction”, and interrupts are checked after the instruction executes. But for a long time, Intel CPUs have been pipelining/fusing/whatever instructions in various ways, and the following instruction can erroneously affect the behavior of a preceding one (e.g. 386 POPA bug). It could be that the SYSEXIT microcode was simply buggy, or there was some more complex microarchitectural interplay involved. AFAIK Intel never documented the erratum because as far as they were concerned, the broken PPros have no SYSENTER/SYSEXIT.
The intel document “Addendum – Intel Architecture Software Developer’s Manual Volume 2: Instruction Set Reference”, document number 243689-001 with copyright 1996/97 documents SYSENTER / SYSEXIT as available in Pentium II.
Well, I just came across this PDF of Pentium Pro errata: https://www.sandyflat.net/digerati/pentiumpro/24268935.pdf . And it mentions one erratum with regards to SYSENTER/SYSEXIT – that they do not GPF when a null segment selector is loaded to CS and/or SS. This confirms that the instructions are indeed present on the Pentium Pro, but are buggy.
I think it is well known(?) that the PPro implemented SYSENTER/SYSEXIT. The erratum actually sounds pretty harmless, the OS just needs to not put fishy values into the relevant MSRs.
The more interesting question is why Intel accepted and published an erratum for an instruction that was not documented and supposedly could not be used in the Pentium Pro!