What does Xenix have to do with the Enigma machine? Perhaps surprisingly, there is a clear connection…
When reconstructing 386 Xenix 2.2.3, the libmdep.a library proved to be a particularly tough nut to crack. That is because it’s one of the three files in a standard Xenix install (/xenix, /etc/getty, /usr/sys/mdep/libmdep.a) which need to be “branded”. Let’s back up a bit and describe the SCO Xenix copy protection.
The Xenix disks did not use any kind of physical copy protection. The media could be backed up and copied without any technical difficulties. The copy protection consisted of a serial number and an activation key which needed to be used in order to “brand” a Xenix installation before it could be used. The branding process wrote a scrambled copy of the serial number into the files, and for files other than the kernel, also decrypted the data which was shipped encrypted on the installation disks.
The installation disks were no good without the serial number and activation key, and if files installed on hard disk were copied, they would contain the serial number and could thus be traced to the source. The mechanism was just complex enough that it was not trivial to defeat, although from a modern perspective the cryptography used was laughably weak.
The Xenix kernel was special in that it was branded but not encrypted, but without a valid serial number severely restricted the number of concurrent processes. It was just good enough to install the OS from floppies but not enough to run a multi-user system.
Now back to libmdep.a. On the restored disks, one sector near the beginning of the library was missing. It turned out that reconstructing the data was not that hard: The missing 512 bytes covered part of the symbol directory and part of the OMF header of the first object file. The OMF header was recreated by copying it from another file and tweaking slightly, and ar had no trouble recreating the symbol directory. So we had a working libmdep.a but we needed an encrypted library which the installation process can decrypt.
Now, Xenix comes with /etc/brand which does what the name suggests. It writes the serial number into files and decrypts files which are shipped encrypted. But when renamed to debrand, the exact same binary turns around and encrypts files (as it happens, the encryption is symmetric). So obviously the debrand utility can be used to encrypt the reconstructed library for placement on the installation media. But just as obviously, debrand needs some kind of key or password, and that is not the serial number or activation key. Rather, it is a separate 3-character code. Which we don’t have. Continue reading →
