Release security

In April I announced that release packaging is now signed. Today all EXEs and DLLS in w64devkit are code-signed with my key. My signing key established a good reputation thanks to thousands of unique signatures observed on the order of ~100,000 hosts — a pleasant side effect from including ~300 binaries in a release. Users should have fewer problems these days with security software. MSYS2 has also adopted my signing tool, aas-sign, which is now included in w64devkit releases.

Builds are now automated by GitHub Actions, triggered when I (and only I) push a new tag. That process code-signs and creates the release. Every step of the release process is transparent, derived strictly from source in the repository. Nowhere does it go behind a curtain and permit secret tampering.

But that’s not all. I enabled release immutability: On publish, release artifacts are locked in and nobody, not even me, can modify them. You can tell by the presence of a release attestation at the end of the artifacts listing. Nobody involved in the project down the road can go rogue and sneak something into an old release.

Toolchain changes

The x64 release is now a “multilib” toolchain. That is it can compile programs for 32-bit Windows, like a superset of the x86 release. The only purpose of the x86 release is to run w64devkit on older hardware or older operating systems. Like the x86 release, it targets Windows XP by default and requires a CPU supporting SSE2 (i.e. at least Pentium 4).

To target x86, pass -m32 when compiling and linking. Or better, use tools prefixed with the i686-w64-mingw32 architecture triple. The latter is easier in general because different tools require different switches, and the prefixed tools are aliases that automatically do the right thing.

$ x86_64-w64-mingw32-gcc -o hello64.exe hello.c
$ i686-w64-mingw32-gcc -o hello32.exe hello.c

Adding multilib was cheaper and easier than I anticipated, and it’s thanks to Peter0x44. It’s already been convenient for me for several projects.

In the past you needed w64devkit’s bin/ on your $PATH in order to use the compiler, because that’s how GCC found the other tools it needs. This is no longer the case, and you can invoke path/to/gcc.exe from scripts and such without touching your path. This required a small GCC patch.

The standard COFF object format supports up to 65,535 sections. When the format was designed this probably seemed like more than enough, but modern C++ programs can easily exceed it, particularly debug builds of programs using lots of lambda functions. As C++ projects grow they eventually hit a threshold where builds fail unless you ask for the bigobj COFF format, which supports over 4 billion sections. This is annoying, and the tools ought to deal with this automatically.

A couple years ago I patched Bintutils to produce bigobj by default so the old limit would never affect a build. The catch is that Binutils lacks an interface to request standard COFF, so you can’t downgrade if needed. Why would you need to? When interacting some simpler linkers like the official Go toolchain (gc)! Some linkers can’t consume bigobj. It also breaks some builds that auto-detect the toolchain’s object format, as some detection scripts don’t know about bigobj (ex. libbacktrace at the time).

Ideally Binutils would quietly upgrade to bigobj as needed. Only large programs see bigobj. Naive linkers and detection scripts only deal with standard COFF. Well, thanks to Peter0x44 that’s now what Bintutils does upstream! My bigobj hack is no longer necessary, and cgo works again.

This past year I’ve decided to put less emphasis on making w64devkit as small as possible, and I’m willing to increase the installation and distribution size for worthwhile features. This is embodied by switching from -Os to -O2 for static runtimes and tools that are computationally expensive. Everything is now a bit bigger and a bit faster. Programs that spend significant time in the standard library will be a little faster, too. Programs already designed for performance will see no difference.

New tools

CMake, Ninja, and a new, graphical CMake debugger are all now included. They require at least Windows 7. I’ve patched CMake to default to Ninja, as CMake’s default is to look for a Visual Studio installation. You can still -G MinGW Makefiles for GNU Make instead of Ninja, but I don’t recommend it. Ninja is faster and more robust.

$ cmake -B build       # configure for Ninja
$ cmake --build build  # build with Ninja

ccmake is included (a Peter0x44 suggestion), a TUI front-end to examine and modify CMake configurations. It’s been useful, more than I expected. You can pass -j for parallel builds, but I strongly recommend using the environment instead, e.g. in your .profile:

export CMAKE_BUILD_PARALLEL_LEVEL="$(nproc)"
export CTEST_PARALLEL_LEVEL=$(nproc)

Yes, parallel testing, too! I also recommend multi-config if that makes sense for you:

$ cmake -B build -G 'Ninja Multi-Config'
$ cmake --build build --config Debug
$ cmake --build build --config Release

I was tempted to make this default, but it changes the build tree layout (Debug/ and Release/ directories) and a shocking percentage of real world CMakeLists.txt break under multi-config because they’re written incorrectly. The world needs a good CMake linter, which could catch most cases statically.

Complementing is Ccache, with special patches from Peter0x44 to improve Windows support. If you switch between branches often then using ccache will speed up your (re-)builds. I also set up the conventional Ccache lib/ccache/ directory. It’s a directory in w64devkit that when added your $PATH transparently backs all your builds with Ccache. On a typical Linux distribution this would be /usr/lib/ccache/. You can enabled it in your w64devkit.ini using path type, too:

path type = minimal+ccache

In my case Ccache hasn’t been useful as I hoped when I added it. I’ve adopted Git worktrees workflow instead, so each branch sits still with its own build tree(s).

To aid COM programming, the kit now includes widl and uuidgen, the open source alternative to Microsoft MIDL, for generating Interface Definition Language (IDL) files. uuidgen is a minimalist rewrite by yours truly of the Microsoft tool.

G. Berthiaume wrote a new tool, make2compdb, that extracts JSON Compilation Database files, compile_commands.json, from Make builds. It operates as a unix filter:

$ make -Bwn | make2compdb >compile_commands.json

I wrote a recycle tool that sends files and folders to the recycle bin, which for the latter is usually faster than deleting them with rm -rf. The trash tool on other systems. I rarely delete files anymore, instead using this command to send them off to sit in the recycle bin for a couple weeks until Storage Sense automatically deletes them.

A few months ago I announced the addition of quilt, an old school patch management tool. The original doesn’t support Windows, so this is a full rewrite in C++. Now that every platform has Quilt, w64devkit’s patches are now managed with Quilt.

NSIS, an installer creator, is now included, so you can build your own application installers. Just the command line tool, makensis. It’s also in a “multilib” configuration, and x64 w64devkit can produce 32-bit and 64-bit installers. (My motivation to add NSIS was my alternative alt-tab switcher, which requires an installer to install properly.)

Zstandard zstd/unzstd is now included because source tarballs are often in .tar.zst format. It’s a great compression format, and ought to be your default (rather than gzip) when you need compression. busybox-w32 tar knows to use zstd for .tar.zst.

Runtimes

Unique to w64devkit, C11 threads are now part of the Mingw-w64 runtime. A new implementation by yours truly, independent of winpthreads, smaller than winpthreads, and requires no special compiler or linker flags. (If you’re careful, you can even use it in CRT-free programs.) The catch is that it requires Windows 7 or later because I built it on SRW locks; XP would have required substantial complexity.

Writing an implementation taught me that C11 threads are underspecified poorly designed — in case it wasn’t already obvious by the presence of recursive locks! This C standard addition did not get nearly the attention is needed and should have been cut (par for the course). My implementation excludes recursive locks. Trying to create one always fails. It also has a weakened thrd_current because the semantics don’t map onto the Windows threading model.

std::terminate actually terminates (traps) instead of calling exit, so it traps in GDB for inspection. This includes uncaught exceptions. It also no longer prints a half-baked stack trace, shedding ~100k of dead weight from most C++ programs. I did this in two steps a year apart, the second just recently. I misplaced the trap in the first change, preventing it from shedding as much weight as it could.

Future directions

Years ago I disabled Link-Time Optimization (LTO) due to bugs in GCC and Binutils. Just having it available in the toolchain triggered LTO bugs. These problems reduced my faith that LTO could produce correct programs, so I disabled it.

However, I’m toying with the idea of not only re-enabling LTO, but even distributing “FatLTO” runtimes, particularly for C++ and Fortran. That is, runtime object files will contain both native code (at -O2 per above) and LTO bytecode. If you don’t request LTO, you get the pre-compiled native code just like today, at no link-time cost. If you enable LTO, it essentially rebuilds the runtime itself at link time. Very computationally expensive, but you get full control, and you can even coerce it back to the old -Os (or even -Oz) if you prefer that.

As far as I can tell, nobody actually distributes a toolchain with FatLTO runtimes, so w64devkit might be the first. I know I’m the first to cross compile FatLTO objects with GCC — remember that everything in w64devkit is cross-compiled — because it’s never worked correctly in a GCC release. My first attempt produced a toolchain that didn’t work.

I’m feeling more confident this time around because fixing LTO bugs is now very easy: Strap a frontier AI into a good coding harness, show it what’s broken, give it all the sources (source.tar), and ~15 minutes later I have a patch. We live in a sci-fi world. I started by fixing known-to-me LTO bugs, enabled FatLTO, then worked through newly-discovered LTO bugs. I’ve been dogfooding it, and nothing new has popped up in awhile, but it will take some time for me to feel confident. I’m less confident in my ability to discover Fortran runtime bugs. Upstream GCC rejects these fixes out of hand, so through no choice of my own w64devkit will have a unique edge in this space.

While some of the new features require at least Windows 7, I’m still committed to supporting Windows XP as a baseline for the x86 release. I keep an old XP laptop on hand, on which I test and enjoy w64devkit. You just have to make do with some older tools, but that’s what you were already doing anyway.

The signed releases on GitHub embed a compressed copy of the original 16-bit game, so that single EXE is ready to play out-of-the-box with no further setup or downloads. I’m distributing 32-bit builds (but requires SSE2) because there’s no advantage to 64-bit here, and these builds work (almost) everywhere except 16-bit Windows. 32-bit Windows could run the original 16-bit game, but the VM-encapsulated version is better behaved. It doesn’t dump a Stars.ini under C:\WINDOWS, it interacts properly with the task bar, and copy protection is neutralized via the OS bridge.

If you ever been curious about Stars!, now’s the time to try it. The game has a thorough, built-in tutorial, but also check out the manual, wiki, strategy guide, and AutoHost (play-by-email service). The game predates the modern search engine concept, otherwise they might have chosen a better name. I suggest using “stars 4x” in your searches.

If you want to build from source and hack on the VM yourself, the best tool for the job is w64devkit, of course, because it comes with everything you’ll need. Plus the game itself: stars.exe from stars27jrc3.zip.

Implementation details

The emulator itself requires x86 or x86-64 because it does not implement x87 (80-bit floating point) in software, but instead runs these operation directly on the host’s x87 hardware. This is simple, fast, and precise. The project validates the emulation as a whole with a differential fuzzer against the host. The fuzzer randomly generates a 16-bit instruction, emulates it, then runs it with JIT on the host and compares the results.

Handles on Windows are pointer-sized, and so the Win16-to-Win32 bridge maps 16-bit handles to host handles. It marshals between different struct layouts when translating these calls, services the DOS interrupts the game requires, an copies data in and out of guest memory. It’s rather like running a Wasm instance, which of course makes sense in retrospect.

The Win32 bridge is also monitorable and manipulatable using the Model Context Protocol (MCP). AI agents can “see” the UI “DOM” as it’s built, and can drive it by injecting synthetic events into the event pump, all without going through the usual desktop control. The MCP can also read and write guest memory. Opus 5 played a complete game through MCP — which is quite fun to watch — requesting my assistance at just two points when it got stuck in the UI. A foundation for a new Stars!Bench?

Targeting old 16-bit computers, the authors couldn’t afford to build a sloppy, wasteful UI, and so by modern standards the game UI is remarkably fast and responsive. They don’t make ‘em like they used to. Computing the next turn, or “turn generation,” is the computational bottleneck, and so that’s where I focused my optimization efforts. The emulator can trace executed instructions, so I gathered traces of turn generation, then had Fable 5.1 identify and reverse engineer the hottest common routines (e.g. the game’s L’Ecuyer MCG PRNG) and basic blocks. Each was re-written in C and mapped into the instruction decoder as new 80286 instructions. On load the emulator identifies these routines and patches them with the new instruction. This resulted in a nearly ~2x speedup of turn generation. By exploiting local conditions, emulating a particular known program, I get JIT performance without JIT complexity.

The original game doesn’t use buffered I/O, and instead issues many small reads and writes. Passing these small reads/writes straight to Win32 made I/O take ~5% turn generation time, probably worse today than it was back then. Plus it’s just rude. The emulator buffers the game’s I/O calls, further speeding up turn generation.

The game has some sound effects in the “battle VCR” and the final version of the game shipped with Microsoft’s WaveMix.dll. Rather than load and link this DLL, the emulator implements the DLL’s interfaces natively, and these routines are dynamically linked into the 16-bit process. You will not need this DLL with the emulator, nor is it embedded in releases.

The game also has art assets embedded uncompressed in the original EXE, forming the bulk of its ~3MB. Stars!VM uses a custom LZ-based compression algorithm tailored to compressing the original game. It’s compressed when embedded in releases. So the 32-bit version of the game is half the size of the original, at ~1.5MB.

The original game requires a serial code, serving as its copy protection. A code is 8 alpha-numeric characters that must pass two checks. Failing the first is loud, but failing the second will sabotage your game with penalties. You’ll know because it will announce that your people suspect you are a usurper. Most codes you’ll find online are such “usurper” codes. Play-by-email (PBEM) saves embed a hardware signature derived from C and D drive configuration. People playing on different machines using the same serial code recieve the usurper penalty.

I bought a serial code back in the day, but it hasn’t been possible to purchase one a for at least decade now. So the emulator injects a fixed serial code on first run (disable with --prompt-serial), and you won’t need to worry about it. The VM also produces a fixed hardware signature (same as any other emulator), so it looks like everyone running Stars!VM is sharing the a machine, meaning no penalty for key reuse. I cracked the serial code checks anyway, allowing me discover interesting ones. My favorites: CLONEMUM, CROSSNUT, EGGSWAIN, GHOSTKIN, GONKSHOW, GRIPMIME, SEEKCAPS, SIFTBOLD, SIRBUOYS, SLIMFAZE, SLOTMOPS, SPAWNELK, SUNBLOND, WANTNEAR, and WRONGPOX. These look like some of my passwords.

Endless possibilities

I’m quite pleased and excited with the results, especially for a weekend project. It’s breathed life back into the game for me, not only having a better experience running it, but also that I can trivially bend the game to my will in the ways I dreamed about. A few hooks in the right places should open the game to easy modding, but I’m more engineer than modder.