What does Xenix have to do with the Enigma machine? Perhaps surprisingly, there is a clear connection…
When reconstructing 386 Xenix 2.2.3, the libmdep.a library proved to be a particularly tough nut to crack. That is because it’s one of the three files in a standard Xenix install (/xenix, /etc/getty, /usr/sys/mdep/libmdep.a) which need to be “branded”. Let’s back up a bit and describe the SCO Xenix copy protection.
The Xenix disks did not use any kind of physical copy protection. The media could be backed up and copied without any technical difficulties. The copy protection consisted of a serial number and an activation key which needed to be used in order to “brand” a Xenix installation before it could be used. The branding process wrote a scrambled copy of the serial number into the files, and for files other than the kernel, also decrypted the data which was shipped encrypted on the installation disks.
The installation disks were no good without the serial number and activation key, and if files installed on hard disk were copied, they would contain the serial number and could thus be traced to the source. The mechanism was just complex enough that it was not trivial to defeat, although from a modern perspective the cryptography used was laughably weak.
The Xenix kernel was special in that it was branded but not encrypted, but without a valid serial number severely restricted the number of concurrent processes. It was just good enough to install the OS from floppies but not enough to run a multi-user system.
Now back to libmdep.a. On the restored disks, one sector near the beginning of the library was missing. It turned out that reconstructing the data was not that hard: The missing 512 bytes covered part of the symbol directory and part of the OMF header of the first object file. The OMF header was recreated by copying it from another file and tweaking slightly, and ar had no trouble recreating the symbol directory. So we had a working libmdep.a but we needed an encrypted library which the installation process can decrypt.
Now, Xenix comes with /etc/brand which does what the name suggests. It writes the serial number into files and decrypts files which are shipped encrypted. But when renamed to debrand, the exact same binary turns around and encrypts files (as it happens, the encryption is symmetric). So obviously the debrand utility can be used to encrypt the reconstructed library for placement on the installation media. But just as obviously, debrand needs some kind of key or password, and that is not the serial number or activation key. Rather, it is a separate 3-character code. Which we don’t have.
By the way: Astute readers may be wondering how the serial number branded into the kernel gets propagated when the kernel is rebuilt (using the so-called Link Kit). That’s precisely where the libmdep.a library comes in. As it turns out, libmdep.a contains an object module which has the serial number branded into it as well, and that’s how a freshly built kernel still contains a serial number.
Brute Force
So, how hard can it be to break that 3-character code by brute force? Not that hard. 3 characters is really not much. The approach was very simple. Run debrand with successive iterations of the code on a an encrypted file (essentially libmdep.a or getty). Check if the first kilobyte or so matches the encrypted file on the distribution disk. Automate with a shell script.
After a long while of writing the script (courtesy of Michael) and a short while of running it in a VM, bingo! The 3-character code is ‘bkb’. We can encrypt the reconstructed library and copy the missing sector to the N4 disk.
It is worth noting that any given set of Xenix floppies is encrypted with a given 3-character code. There is a large number of serial number + activation key combinations which are tied to the same 3-character code and can unlock the installation media. Obviously SCO didn’t want to produce different disks to go with each serial number. This way many serial numbers could be handed out for a given product, yet keys for one product could not be used to unlock a different one.
Digging Deeper
But how does the encryption really work? It was fairly clear that it processes 256-byte blocks and that it wasn’t the DES encryption traditionally used for Unix passwords. Did SCO invent something of their own? That would be very un-Unix-like, and of course they didn’t. What they did was take the crypt command and adapt the encryption algorithm slightly. Note that crypt was not shipped with Xenix due to U.S. export regulations, but was available on request, and SCO obviously had the source code.
One clue was that at least some versions of /etc/brand were built from source files that among others included crypt.c. And after some digging, this turned up. “A one-rotor machine designed along the lines of Enigma but considerably trivialized”, with a 256-byte “rotor”.
Xenix uses the same algorithm with two noteworthy modifications. It always works on 256-byte blocks, and it uses the 3-character code (extended to 5 characters with two constant characters) directly in place of ‘buf’ in the setup code.
With the algorithm in hand, we can construct a much better key finder. When dealing with encrypted files like libmdep.a, how does /etc/brand know the serial number + activation key combination is the right one? That is actually a two-step process.
First, the given serial number and activation key are slightly transformed and a checksum is used to verify that they form a valid pair. The 3-character code is effectively part of the activation key.
The second step is decrypting the first 256-byte data block, calculating its checksum, and comparing it with the checksum stored in the encrypted file. If the checksum matches, the code is considered valid and the rest of the file is decrypted.
Now it’s trivial to try various 3-character codes (lowercase only!), decrypt 256 bytes, calculate the checksum, and see if it matches. On a modern PC, it takes less than a second to go through the entire 3-character key space. And interestingly, there always seem to be two valid 3-character codes which can unlock a given file. That may be a weakness in the algorithm used.
Final Touch
The last hurdle was generating an activation key given a serial number and a 3-character code. As it turns out, there are multiple valid activation keys. The first three characters of the activation key correspond to the 3-character code, the second three are usually unused, and the last two are derived from a checksum which covers the serial number and the first 6 characters of the activation key. The activation key is then scrambled using a relatively simple (but non-obvious) algorithm which nicely obscures the 3-character code.
With an activation key calculator at hand, it is possible to decrypt those 386 PS/2 Xenix disks, for example, even though the kind person who uploaded the images didn’t provide the serial number and activation key. By the way, the 3-character code for those disks is, very fittingly, ‘ibm’.
An unusual Xenix serial number
It is also possible to produce personalized license plates, as seen above.
What Have We Learned?
If there’s any lesson to be learned, it’s probably that 30-year old copy protection is relatively easy to break using tools and computing power that did not exist 30 years ago.
What’s that ‘Z’ doing there?
When Xenix boots up, it prints a couple of different letters on the screen. The final ‘Z’ stays. I think it’s basically a progress indicator and ‘Z’ means the kernel is fully initialized.
{M$’,SCO’s} idea of a twirling baton?
The article says “(which is ??? shipped with Xenix)”. Was that meant to be a placeholder until you figured out whether crypt was shipped or not (then replacing the three ? with the word “not”), or was it actually shipped with Xenix and you are very surprised by that?
“If there’s any lesson to be learned, it’s probably that 30-year old copy protection is relatively easy to break using tools and computing power that did not exist 30 years ago.”
It’s also fun because of this. Tou don’t have to be a hardcore reverse engineering expert to crack it. I’ve done a keygen for Citrix 1.8 a while ago and it was easy with IDA – nothing was obfuscated, minimum optimizations – the C code generated by the decompiler was perfectly readable.
It was meant to be a placeholder until I verify that Xenix shipped with the crypt utility. But then it got complicated and I forgot to get back to it. Thanks for reminding me!
The ‘Z’ was the first hint I had 10 years ago that the kernel was actually running when trying to install under any emulation before the diskette density errors were being reported. That’s when I made a disk image of an installed system, and had it running on Qemu which was for a while the only way to get it running.
Hard to believe it’s been nearly 10 years!
So, what are the magic 512 bytes needed to repair the N4 install disk image? That is the only missing sector in the main OS install set I wasn’t able to pattern match with something else. Might be able to work through the instructions above, but it would save me a bunch of time.
The only other sector that I could not find was on the optional International Supplement disk 3. But that is not really needed.
Is that the corrupted libmdep.a? I just fixed that one by hand… I’m unfortunately in the middle of a move right now so I can’t get to my work files easily. I should be able to get back to you in within a week.
Michal,
Can you make the corrected disk image set available to us? I am in the process of trying to restore an Intel 320 and may be able to get it to boot SCO XENIX, with a lot of work. Be nice to have a working system on virtualbox to develop the load disks with.
Thanks
Bill
Re: letters and the final “z”
The letters spit up at boot time were sort of a progress bar, but they were useful when bringing up a kernel with problems or weird hardware and you knew how to read those tea leaves. They’re printed on the same line with a carriage return but no newline, so mostly they get overwritten with only the “z” line remaining if the kernel proper actually booted. This is really early in boot so very little of the system can be trusted, so it’s really rudimentary telemetry, mostly useful to kernel devs and only a few steps above a blinking light. They’re more apparent if you’re booting on an actual serial terminal where everything happens in slow motion and I think the newlines may be ignored. Somewhere in the boot logs you can actually see the full messages. /usr/adm/messages, perhaps? Each driver used to do its own thing in xxinit() (the driver constructor called at boot) but eventually printcfg was created to standardize the formatting. That was probably around 3.2v2 or 3.2v4. (3.2v3 never shipped.)
It’s been a long time and I may be mixing various XENIX and UNIX (OpenDesktop/OpenServer) versions, but the type of information is something like:
E was the FPU init. There was support for 287 with a 386 (Tandy 4000 had such an configuration in the early runs), there was Weitek and other configurations. If there was no hardware FPU, a software emulation would be used.
Fnn was the exciting one. There was a big loop calling every driver’s fooinit() entry point and nn was an incrementing indication that the system survived. Hardware would be probed. Things that were statically configured would be done here and printcfg would announced I/O, IRQ, and such. If your system hung here, it was PROBABLY a driver problem. Because the boot floppy didn’t have access to the symbol table, the driver name was a static entry in a table of devswitch that had the entry point to fooinit and an char[8] of “foo” (whatever) so that the driver name could be printed here. Drivers that literally copied sample code and had the wrong names were common in the 2.2/2.3 XENIX era, so Fnn crashes or hangs could be a bit of a snipe hunt as they weren’t checked for uniqueness.
G was when the PICs were configured and enabled. If the system hung here, there was probably interrupt issues. (Remember, this was the era of jumpers and dip switches and craziness of trying to mix level and edge sensitive interrupts on the same IRQ lines.) The system was running without interrupts until this point. Prior to this point, driver interrupts are simulated by calling the xxintr() entry point manually. (It does that during panic dumps, too…)
H was where the downstream half of the drivers were called and configured. That’s where you’d probe the HBA configuration and announce cylinders, heads, sectors or the number of attached serial ports on an intelligent serial adapter or the type of media found on an ethernet driver or such. Memory pools would be divvied up based on the found hardware.
I don’t remember much past this. Maybe I never had to debug anything that late in boot. I really think the “Z” was just a final “hurray!” that the kernel had basically finished initializing and was ready to call /etc/init and start doing UNIX-y things.
This lettering became so institutionalized and recognized by SCO users that when they went to the SVR3.2 UNIX product and later, the SVR4.2 kernel that became UnixWare 7 they shoehorned it back in even when it didn’t make much sense. On a 286-based Tandy 3000 in 88, announcing your disk drive’s configuration when found was handy, but on a system with hotplug HBAs and RAID arrays that can show up months after the system has booted, it was a weird fit.
I might be a bit off in some of the recollection, but hopefully this helps people understand the SCO boot letter alphabet soup telemetry.
Pingback: New top story on Hacker News: Tales from the Xenix Crypt (2017) – Hckr News
OMG! Nostalgia heavy. Reading this has brought back many memories of being at SCO and working with Xenix, ODT and Unixware. Most notably a set of characters popped into my head which I can only assume was the serial key widely used internally when installing SCO UNIX…muscle memory.
Thanks for the comment. When were you there? Late 1980s/early 1990s? I never thought of that but yeah, SCO would have had to have some internally used serial numbers and activation keys.
Pingback: Fox in the Crypt | OS/2 Museum
The version of Xenix x86 I used did not have a serial number, but to install it, or do anything with floppies, it required *perfect* media. Cleaned the drives before using.
It was SCO 2.3.2, and after getting it all installed, ( runtime, and dev tools) we did some C programming homework on it, and it worked rather well. ( Turbo XT, 2x20Mb ), so we loaded on the source for Hack, and tried to compile it. After about 10 .a files, it choked on something, blew up the file system into an unrecoverable mess. ( would not boot into single user, no fsck…)
We copied all the disks ( 2 times ), and all the documentation, all lost to the sands of time.
I will be looking up to see if the SCO 386 version runs Rogue…
I did hear rumors that the x86 version ran on a Compaq 386, but the 286 version would only run on an IBM.
“Note that crypt was not shipped with Xenix due to U.S. export regulations”
Note that this was before 40-bit RC2 and RC4 was officially allowed to be exported in 1992.