For those unfamiliar with xxd, examples are in order. Its default hexdump output looks like this:

$ echo hello world | xxd | tee dump
00000000: 6865 6c6c 6f20 776f 726c 640a            hello world.

Octets display in pairs with an ASCII text listing on the right. All configurable. I can run this in reverse (-r), recovering the original input:

$ xxd -r dump
hello world

The tool reads the offset before the colon, the hexadecimal octets, and ignores the text column. By editing dump with a text editor, I can change the raw octets of the original input. From this point of view, the hexdump is actually a program of two alternating instructions: seek and write. xxd seeks to the offset, writes the octets, then repeats. It also doesn’t truncate the output file, so a hexdump can express binary patches as a seek/write program.

$ echo hello world >hello
$ echo 6: 65766572796f6e650a | xxd -r - hello
$ cat hello
hello everyone

That seeks to offset 0x6, then writes the 9 octets. The xxd parser is flexible, and I did not need to follow the default format. It figured out the format on its own, and rexxd further improves on this. We can use it to create large files out of thin air, too:

$ echo 3fffffff: 00 | xxd -r - >1G

This command creates an all-zero, 1GiB file, 1G, by seeking to just before 1GiB then writing a zero. I used >1G so that the shell would truncate the file before starting xxd — in case it was larger or contained non-zeros.

This is a “smart seek” of course, and its not literally seeking on every line. The tool tracks its file position and only seeks when necessary. If seeking fails, it simulates the seek using a write if possible. When would it not be possible? Lines need not be in order, of course, and so it may need to seek backwards. Lines can also overlap in contents. If it weren’t for buffering — or if rexxd had a unified buffer cache — then by using the same file for input and output an “xxd program” could write new instructions for itself and accidentally become Turing-complete.

The other common mode, -i, looks like this:

$ echo hello world >hello
$ xxd -i hello hello.c

Which produces this hello.c:

unsigned char hello[] = {
  0x68, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x77, 0x6f, 0x72, 0x6c, 0x64, 0x0a
};
unsigned int hello_len = 12;

Note how it converted the file name into variable names. Characters disallowed in variable names become underscores _. When reading from standard input, xxd only emits the octets. Unless the new-ish -n name option is given, in which case that becomes the variable name. This remains popular because, #embed notwithstanding, as of this writing all major toolchains remain stubborn about embedding data on their own.

The case for replacement

The idea of replacing it began with backporting the -n name option to Vim 9.0 xxd. The feature did not appear in a release until a year ago, 28 years after -i, despite its obviousness. I’ve also felt that xxd is slower than it could be, and a momentary examination reveals it’s buggier than it ought to be. As expected, a few seconds of fuzz testing xxd -r reveals bugs, and it doesn’t even require writing a single line of code:

$ afl-gcc -fsanitize=address,undefined xxd.c
$ mkdir inputs
$ echo >inputs/sample
$ afl-fuzz -i inputs/ -o fuzzout/ ./a.out -r

The Windows port is lacking in the usual ways, unable to handle Unicode paths. The new Vim 9.1 xxd -R color feature broke the Windows port, and if w64devkit included Vim 9.1 then I’d need to patch out the new bugs. As demonstrated above, at least it’s trivial to compile! It’s a single source file, xxd.c, and requires no configuration. I love that.

The more I looked, the more problems I found. It’s not doing anything terribly complex, so I expected it wouldn’t be difficult to rewrite it with a better foundation. So I did. Ignoring tests and documentation, my rewrite is about twice as long. In exchange, it’s substantially faster:

$ dd if=/dev/urandom of=bigfile bs=1M count=64

$ time orig-xxd bigfile dump
real    0m 4.40s
user    0m 2.89s
sys     0m 1.46s

$ time rexxd bigfile dump
real    0m 0.31s
user    0m 0.07s
sys     0m 0.21s

Same in reverse:

$ time orig-xxd -r dump nul
real    0m 5.81s
user    0m 5.67s
sys     0m 0.07s

$ time rexxd -r dump nul
real    0m 0.33s
user    0m 0.23s
sys     0m 0.09s

Or embedding data with rexxd:

$ time orig-xxd -i bigfile bigfile.c
real    0m 10.32s
user    0m 9.85s
sys     0m 0.37s

$ time rexxd -i bigfile bigfile.c
real    0m 0.40s
user    0m 0.07s
sys     0m 0.34s

I wanted to keep it portable and simple, so that’s without fancy SIMD processing. Just SWAR parsing, branch avoidance, no division on hot paths, and sound architecture. I also optimized for the typical case at the cost of the atypical case. It’s a little unfair to compare it to a program probably first written on a 16-bit machine, but there was time for it to pick up these techniques over the decades, too.

Unicode support works well:

$ cat π
3.14159265358979323846264338327950288419716939937510582097494
$ rexxd -i π π.c

Producing this source with Unicode variables:

unsigned char π[] = {
  0x33, 0x2e, 0x31, 0x34, 0x31, 0x35, 0x39, 0x32, 0x36, 0x35, 0x33, 0x35,
  // ...
  0x34, 0x0a
};
unsigned int π_len = 62;

Whereas the original xxd on Windows has the usual CRT problems:

$ orig-xxd -i π
orig-xxd: p: No such file or directory

It also struggles with 64-bit offsets, particularly on 32-bit hosts and LLP64 hosts like Windows. In contrast, I designed rexxd to robustly process file offsets as 64-bit on all hosts. Its tests operate on a virtual file system with virtual files at those sizes, so those paths really have been tested, too.

The original xxd only uses static allocation, which places small range limits on the configuration:

$ orig-xxd -c 1000
orig-xxd: invalid number of columns (max. 256).

In rexxd everything is arena allocated of course, and options are limited only by the available memory, so the above, and more, would work. The arena helps make the SWAR tricks possible, too, providing a fast runway to load more data at a time.

While reverse engineering the original, I documented bugs I discovered and noted them with a BUG: comment if you wanted to see more. I’m not aiming for bug compatibility, so these are not present in rexxd.

Platform layer

The xxd man page suggests using strace to examine the execution of -r reverse. That is, to monitor the seeks and writes of a binary patch in order to debug it. That’s so insightful that I decided to build that as a new -x option (think sh -x). That is, rexxd has a built-in strace on all platforms! The trace is expressed in terms of unix system calls, even on Windows:

$ printf '00:41 \n02:42 \n04:43' | rexxd -x -r - data.bin
open("data.bin", O_CREAT|O_WRONLY, 0666) = 1
read(0, ..., 4096) = 19
write(1, "A", 1) = 1
lseek(1, 2, SEEK_SET) = 2
read(0, ..., 4096) = 0
write(1, "B", 1) = 1
lseek(1, 4, SEEK_SET) = 4
write(1, "C", 1) = 1
exit(0) = ?

Is this doing some kind of self-ptrace debugger voodoo? Nope. Like u-config, it has a platform layer, and it simply logs the platform layer calls — except for the trace printout itself of course. While the intention is to debug binary patches, it was also quite insightful in examining rexxd itself. It helped me spot that rexxd flushed more often than strictly necessary.

To port rexxd to any system, define Plt as needed, implement these five plt_ functions, then call xxd. The five functions mostly have the expected unix-like semantics:

typedef struct Plt Plt;
b32  plt_open(Plt *, i32 fd, u8 *path, b32 trunc, Arena *);
i64  plt_seek(Plt *, i32 fd, i64 off, i32 whence);
i32  plt_read(Plt *, u8 *buf, i32 len);
b32  plt_write(Plt *, i32 fd, u8 *buf, i32 len);
void plt_exit(Plt *, i32);
i32  xxd(i32 argc, u8 **argv, Plt *, byte *heap, iz heapsize);

If the platform wants these functions to be “virtual” then it can put function pointers in the Plt struct. Otherwise it stores anything it might need in Plt. Global variables are never necessary. The application layer doesn’t use the standard library except (indirectly) memset and memcpy, and it allocates everything it uses from the provided heap parameter.

plt_open is a little unusual in that it picks the file descriptor: 0 to replace standard input, or 1 to replace standard output. All platforms currently use a virtual file descriptor table, and these do not map onto the real process file descriptors. But they could! Calls are straced in the application layer, so they log virtual file descriptors as seen by rexxd. The arena parameter offers scratch space for the Windows platform layer to convert paths from narrow to wide for CreateFileW, so it can handle long path names with ease.

plt_read doesn’t accept a file descriptor because there’s only one from which to read, 0. plt_write on the other hand allows writing to standard error, 2.

plt_exit doesn’t return, of course. In tests it longjmps back to the top level, as though returning from xxd with a status. This lets me skip allocation null pointer checks, with OOM unwinding safely back to the top level. Since rexxd allocates everything from the arena, it’s all automatically deallocated, so it’s a clean exit.

On Windows, plt_seek calls SetFilePointerEx. I learned the hard way that the behavior of calling it on a non-file is undefined, not an error, so at least one GetFileType call is mandatory. I also learned that Windows will successfully seek all the way to INT64_MAX. If the file system doesn’t support that offset, it’s a write failure later. For correct operation, rexxd must take care not to overflow its own internal file position tracking near these offsets with Windows allowing seeks to operate at the edge until the first flush. Tests run on a virtual file system thanks to the platform layer, and some tests permit huge seeks and simulate impossibly enormous files in order to probe behavior at the extremes.

This is in contrast to Linux, where seeks beyond the underlying file system’s supported file size is a seek error. For example, on ext4 with the default configuration:

$ echo ffffffff000: 00 | rexxd -x -r - somefile
open("somefile", O_CREAT|O_WRONLY, 0666) = 1
read(0, ..., 4096) = 16
lseek(1, 17592186040320, SEEK_SET) = 17592186040320
read(0, ..., 4096) = 0
write(1, "\0", 1) = -1
exit(3) = ?

We can see the seek succeeded then the write failed because it went one byte beyond the file system limit. While seeking one byte further will cause the seek to fail (22 EINVAL), and rexxd falls back on write until it fills the storage and runs out of space:

$ echo ffffffff001: 00 | rexxd -x -r - somefile
open("somefile", O_CREAT|O_WRONLY, 0666) = 1
read(0, ..., 4096) = 16
lseek(1, 17592186040321, SEEK_SET) = -1
write(1, "\0\0\0\0\0\0...\0\0\0\0\0\0", 4096) = 4096
write(1, "\0\0\0\0\0\0...\0\0\0\0\0\0", 4096) = 4096
...

Mostly for fun, I wrote a libc-free platform layer using raw Linux system calls, and it maps almost perfectly onto the kernel interface:

struct Plt { int fds[3]; };

b32 plt_open(Plt *plt, i32 fd, u8 *path, b32 trunc, Arena *)
{
    i32 mode = fd ? O_CREAT|O_WRONLY : 0;
    mode |= trunc ? O_TRUNC : 0;
    plt->fds[fd] = (i32)syscall3(SYS_open, (uz)path, mode, 0666);
    return plt->fds[fd] >= 0;
}

i64 plt_seek(Plt *plt, i32 fd, i64 off, i32 whence)
{
    return syscall3(SYS_lseek, plt->fds[fd], off, whence);
}

i32 plt_read(Plt *plt, u8 *buf, i32 len)
{
    return (i32)syscall3(SYS_read, plt->fds[0], (uz)buf, len);
}

b32 plt_write(Plt *plt, i32 fd, u8 *buf, i32 len)
{
    return len == syscall3(SYS_write, plt->fds[fd], (uz)buf, len);
}

void plt_exit(Plt *, i32 r)
{
    syscall3(SYS_exit, r, 0, 0);
}

On Windows I use the artisanal function prototypes of which I’ve grown so fond. It’s also my first time using w64devkit’s -lmemory in a serious application. I’m using -lchkstk in the “xxd as a DLL” platform layer, too, but that one’s just a toy. In that one I use alloca to allocate an arena, which is a rather novel combination, and the large stack frame requires a stack probe. Otherwise none of rexxd requires stack probes.

w64devkit’s new xxd.exe is delightfully tidy as viewed by peports:

$ du -h xxd.exe
28.0K   xxd.exe
$ peports xxd.exe
KERNEL32.dll
        0       CreateFileW
        0       ExitProcess
        0       GetCommandLineW
        0       GetFileType
        0       GetStdHandle
        0       MultiByteToWideChar
        0       ReadFile
        0       SetFilePointerEx
        0       VirtualAlloc
        0       WideCharToMultiByte
        0       WriteFile
SHELL32.dll
        0       CommandLineToArgvW

Other notes

Buffered output and buffered input is custom tailored for rexxd. When parsing line-oriented input, like -r, it attempts to parse from of a view of the input buffer, no copying. The view is the usual string representation:

typedef struct {
    u8 *data;
    iz  len;
} Str;

Does it fail if the line is longer than the buffer? If it straddles reads, does that hurt efficiency? The answer to both is “no” due to the spillover arena. Input is the buffered input struct, and here’s the interface to get the next line:

Str nextline(Input *, Arena *);

If the line isn’t entirely contained in the input buffer, the complete line is concatenated in the arena. So it comfortably handles huge lines while no-copy optimizing for typical short, non-straddling lines. With a per-iteration arena, any arena-backed line is automatically freed at the end of the iteration, so it’s all transparent:

    for (;;) {
        Arena scratch = perm;
        Str line = nextline(b, &scratch);
        // ... line may point into an Input or scratch ...
    }

If the line doesn’t fit in the arena, it triggers OOM handling. That is, it calls plt_exit and something platform-appropriate happens without returning. Beats the pants off old getline!

I came up with a maxof macro that evaluates the maximum of any integral type, signed or unsigned. It appears in overflow checks and more, I really like how it turned out. For example:

    if (pos > maxof(i64) - off) {
        // overflow
    }
    pos += off;

Or:

i32 trunc32(iz n)
{
    return n>maxof(i32) ? maxof(i32) : (i32)n;
}

Now that I have -lmemory and generally solved string function issues for myself, I leaned into __builtin_memset and __builtin_memcpy for this project. Despite restrict, it’s surprisingly difficult to get compilers to optimize loops into semantically equivalent string function calls. An explicit built-in solves that. It also produces faster debug builds, which is what I run while I work. At -O0, rexxd is about half the speed of a release build.

Other than -x, I don’t plan on inventing new features. I’d like to maintain compatibility with the xxd found everywhere else, and I don’t expect adoption beyond w64devkit. Overall the project took about twice as long as I anticipated — two weekends instead of one — but it turned out better than I expected and I’m very pleased with the results.

LoadLibraryA is a similar situation, and potentially applies the code page to a longer portion of the module path. LoadLibraryW is unaffected, at least for the directly-named module, because it’s Unicode all the way through.

For my contrived demonstration I came up with two names that to English-reading eyes appears as two words with extraneous markings:

• õral.dll: CP-1252="C3 B5 …"
• õral.dll: CP-1252="F5 …"; UTF-8="C3 B5 …"

Both end with ral.dll. I’ve included the CP-1252 encoding for the differing prefixes, and the UTF-8 encoding for the second. I’m using CP-1252 because it’s the most common system code page in the world, especially the Western hemisphere. Due to case insensitivity, the actual DLL may be named ãµral.dll — i.e. to match the second library case — but the module name must be encoded as uppercase when building the import library. Alternatively the second could be Õral.dll, particularly because I won’t use it when constructing an import library.

The plan is to store the octets C3 B5 … in the import table. A process using CP-1252 decodes it to õral.dll. In the UTF-8 code page it decodes to õral.dll. For testing we can use an application manifest to control the code page for a particular PE image — a lot easier than changing the system code page. Otherwise, this trick could dynamically change the behavior of a program in response to the system code page without actually inspecting the active code page.

The libraries will have a single function get, which returns a string indicating which library was loaded:

#define X(s) #s
#define S(s) X(s)
__declspec(dllexport) char *get(void) { return S(V); }

Constructing the import library can be tricky because you must consider how the toolchain, editors, and shells decode and encode text, which may involve the build system’s code page. It’s shockingly difficult to script! Binutils dlltool cannot process these names and cannot be used at all. With bleeding edge w64devkit I could reliably construct the DLLs and import library like so, even in a script (Windows 10 and later only):

$ gcc -shared -DV=UTF-8 -o Õral.dll  detect.c
$ gcc -shared -DV=ANSI  -o õral.dll detect.c -Wl,--out-implib=detect.lib

That produces two DLLs and one import library, detect.lib, with the desired module name octets. A straightforward MSVC cl invocation also works so long as it’s not from a batch file. It will quite correctly warn about the strange name situation, which I like. My test program, main.c:

#include <stdio.h>

char *get(void);
int main(void) { puts(get()); }

I link detect.lib when I build it:

$ cc -o main.exe main.c detect.lib

I designed peports to print non-ASCII octets unambiguously (\xXX), and it’s the only tool I know that does so:

$ peports main.exe | tail -n 2
\xc3\xb5ral.dll
        1       get

The module name has the C3 B5 … prefix octets. When I run it under my system code page, CP-1252:

$ ./main
ANSI

If I add a UTF-8 manifest, even just a “side-by-side” manifest, it loads the other library despite an identical import table:

$ cc -o main.exe main.c detect.lib libwinsane.o
$ ./main
UTF-8

Again, without the manifest, if I switched my system code page to UTF-8 then UTF-8 would still be the result.

I can’t think of much practical use for this trick outside of malware. In a real program it would be simpler to inspect code page, and there’s no benefit to avoiding such a check if it’s needed. Malware could use it to trick inspection tools and scanners that decode module names differently than the dynamic linker. Such tools often incorrectly assume UTF-8, which is what motivated this article.

If you’d like to see/try a complete, working demonstration before diving into the details: demo.cpp. We’re going to build this from the ground up, so let’s establish a few primitive integer definitions:

using b32 = signed;
using i32 = signed;
using uz  = decltype(0uz);

Think of uz as like uintptr_t. This implementation will support both 32-bit and 64-bit targets, and we’ll need it as the basis for locks and condition variables:

enum Lock : uz;
enum Cond : uz;

Opaque enums provide additional type safety: They have the properties of an integer, including trivial destruction, but are distinct types which compilers forbid mixing with other integers. We can’t, say, accidentally cross condition variable and lock parameters — my main concern. Aside from zero-initialization, we do not actually care about the values of these variables, so enumerators are unnecessary. (Caveat: GDB cannot display opaque enums, which is slightly irritating.)

The documentation doesn’t explicitly mention zero initialization, but the official *_INIT constants are defined as zero. That locks in zero at the ABI level, so we can count on it.

All the functions we’ll need are exported by kernel32.dll. Locks have two variations on lock/unlock: “exclusive” (write) and “shared” (read). There are also “try” versions, but I won’t be using them.

#define W32(r, p) extern "C" __declspec(dllimport) r __stdcall p noexcept
W32(void, AcquireSRWLockExclusive(Lock *));
W32(void, AcquireSRWLockShared(Lock *));
W32(void, ReleaseSRWLockExclusive(Lock *));
W32(void, ReleaseSRWLockShared(Lock *));

Declaring Win32 functions in C++ is a mouthful, and everything must be written in just the right order, but it’s mostly tucked away in a macro. Usually there’s a stack discipline to these locks, so an RAII scoped guard is in order:

struct Guard {
    Lock *l;
    Guard(Lock *l) : l{l} { AcquireSRWLockExclusive(l); }
    ~Guard()              { ReleaseSRWLockExclusive(l); }
};

struct RGuard {
    Lock *l;
    RGuard(Lock *l) : l{l} { AcquireSRWLockShared(l); }
    ~RGuard()              { ReleaseSRWLockShared(l); }
};

Dead simple. (What about rule of three? Instead of working around this language design flaw, reach into the distant future where it’s been fixed: -Werror=deprecated-copy-dtor.) Usage might look like:

struct Example {
    Lock lock = {};
    i32  value;
};

i32 incr(Example *e)
{
    Guard g(&e->lock);
    return ++e->value;
}

Note the = {} to guarantee the lock is always ready for use. It gets more interesting with condition variables in the mix. That’s three more functions:

W32(b32,  SleepConditionVariableSRW(Cond *, Lock *, i32, b32));
W32(void, WakeAllConditionVariable(Cond *));
W32(void, WakeConditionVariable(Cond *));

The last parameter on SleepConditionVariableSRW indicates if the lock was acquired shared. Why do locks have distinct acquire and release functions while condition variables use a flag for the same purpose? Beats me. I’ll unfold it into two functions, selected by type, with a default infinite wait:

b32 wait(Cond *c, Guard *g, i32 ms = -1)
{
    return SleepConditionVariableSRW(c, g->l, ms, 0);
}

b32 wait(Cond *c, RGuard *g, i32 ms = -1)
{
    return SleepConditionVariableSRW(c, g->l, ms, 1);
}

Usage might look like:

for (RGuard g(&lock); remaining;) {
    wait(&done, &g);
}

The other side is nothing more than a rename (but could also be accomplished through linking):

void signal(Cond *c)
{
    WakeConditionVariable(c);
}

void broadcast(Cond *c)
{
    WakeAllConditionVariable(c);
}

And a couple examples of its usage:

if (Guard g(&lock); !--remaining) {
    signal(&done);
}

// Or:

Guard g(&lock);
ready = true;
broadcast(&init);
while (remaining) {
    wait(&done, &g);
}

A satisfying, powerful synchronization interface with hardly any code!

In PE executable images, an import isn’t just a symbol, but a tuple of DLL name and symbol. For human display, a tuple is typically formatted with an exclamation point delimiter, as in msvcrt.dll!malloc, though sometimes without the .dll suffix. You’ve likely seen this in stack traces. Because it’s a tuple and not just a symbol, it’s possible to refer to, and import, the same symbol from different DLLs. Contrast that with ELF, which has a list of shared objects, and a separate list of symbols, with the dynamic linker pairing them up at load time. That permits cool tricks like LD_PRELOAD, but for the same reason loading is less predictable.

Windows comes with several CRTs, and various libraries and applications use one or another (or none) depending on how they were built. As C standard library implementations they export mostly the same symbols, malloc, printf, etc. With imports as tuples, it’s not so unusual for an application to load multiple CRTs at once. Typically coexistence is transitive. That is, a module does not directly access both CRTs but depends on modules that use different CRTs. One module calls, say, msvcrt.dll!malloc, and another module calls ucrtbase.dll!malloc. With DLL-qualified symbols, this is sound so long as modules don’t cross the streams, e.g. an allocation in one module must not be freed in the other. Libraries in this ecosystem must avoid exposing their CRT through their interfaces, such as expecting the library’s caller to free() objects: The caller might not have access to the right free!

Contrast again with the unix ecosystem generally, where a process can only load one libc and everyone is expected to share. Libraries commonly expect callers to free() their objects (e.g. libreadline, xcb), blending their interface with libc.

Suppose you’re in such a situation where, due to unix-oriented libraries, your application must use functions from two different CRTs at once. One might have been compiled with Mingw-w64 and linked with MSVCRT, and the other compiled with MSVC and linked with UCRT. We need to call malloc and free in each, but they have the same name. What a pickle!

There’s an obvious, and probably most common, solution: run-time dynamic linking. Use load-time linking on one CRT, and LoadLibrary on the other CRT with GetProcAddress to obtain function pointers. However, it’s possible to do this entirely with load-time linking!

A malloc by any other name would allocate as well

Think about it a moment and you might wonder: If the names are the same, how can I pick which I’m calling? The tuple representation won’t work because ! cannot appear in an identifier, which is, after all, why it was chosen. The trick is that we’re going to rename one of them! To demonstrate, I’ll use my Windows development kit, w64devkit, a Mingw-w64 distribution that links MSVCRT. I’m going to use UCRT as the second CRT to access ucrtbase.dll!malloc.

I can choose whatever valid identifier I’d like, so I’m going to pick ucrt_malloc. This will require a declaration:

__declspec(dllimport) void *ucrt_malloc(size_t);

If I stop here and try to use it, of course it won’t work:

ld: undefined reference to `__imp_ucrt_malloc'

The linker hasn’t yet been informed of the change in management. For that we’ll need an import library. I’ll define one using a .def file, which I’ll name ucrtbase.def:

LIBRARY ucrtbase.dll
EXPORTS
ucrt_malloc == malloc

The last line says that this library has the symbol ucrt_malloc, but that it should be imported as malloc. This line is the lynchpin to the whole scheme. Note: The double equals is important, as a single equals sign means something different. Next, use dlltool to build the import library:

$ dlltool -d ucrtbase.def -l ucrtbase.lib

The equivalent MSVC tool is lib, but as far as I know it cannot quite do this sort of renaming. However, MSVC link will work just fine with this dlltool-created import library. The name ucrtbase.lib, while obvious, is irrelevant. It’s that LIBRARY line that ties it to the DLL. My test source file looks like this:

#include <stdlib.h>

__declspec(dllimport) void *ucrt_malloc(size_t);

int main(void)
{
    void *msvcrt[] = {malloc(1), malloc(1), malloc(1)};
    void *ucrt[] = {ucrt_malloc(1), ucrt_malloc(1), ucrt_malloc(1)};
    return 0;
}

It compiles successfully:

$ cc -g3 -o main.exe main.c ucrtbase.lib

I can see the two malloc imports with objdump:

$ objdump -p main.exe
...
DLL Name: msvcrt.dll
...
844a	 1021  malloc
...
DLL Name: ucrtbase.dll
847e	    1  malloc

It loads and runs successfully, too:

$ gdb main.exe
Reading symbols from main.exe...
(gdb) break 9
Breakpoint 1 at 0x1400013cd: file main.c, line 9.
(gdb) run
Thread 1 hit Breakpoint 1, main () at main.c:9
9           return 0;
(gdb) p msvcrt
$1 = {0xd06a30, 0xd06a70, 0xd06ab0}
(gdb) p ucrt
$2 = {0x6e9490, 0x6eb7c0, 0x6eb800}

The pointer addresses confirm that these are two, distinct allocators. Perhaps you’re wondering what happens if I cross the streams?

int main(void)
{
    free(ucrt_malloc(1));
}

The MSVCRT allocator justifiably panics over the bad pointer:

$ cc -g3 -o chaos.exe chaos.c ucrtbase.lib
$ gdb -ex run chaos.exe
Starting program: chaos.exe
warning: HEAP[chaos.exe]:
warning: Invalid address specified to RtlFreeHeap
Thread 1 received signal SIGTRAP, Trace/breakpoint trap.
0x00007ffc42c369af in ntdll!RtlRegisterSecureMemoryCacheCallback ()
(gdb)

While you’re probably not supposed to meddle with ucrtbase.dll like this, the general principle of export renames is reasonable. I don’t expect I’ll ever need to do it, but I like that I have the option.

Win32 has two interfaces for interacting with environment variables:

1. GetEnvironmentVariable and SetEnvironmentVariable
2. GetEnvironmentStrings and FreeEnvironmentStrings

The first, which I’ll call get/set, is the easy interface, with Windows doing all the searching and sorting on your behalf. It’s also the only supported interface through which a process can manipulate its own variables. It has no function for enumerating variables.

The second, which I’ll call get/free, allocates a copy of the environment block. Calls to get/set does not modify existing copies. Similarly, manipulating this block has no effect on the environment as viewed through get/set. In other words, it’s read only. We can enumerate our environment variables by walking the environment block. As I will discuss below, enumeration is it’s only consistently useful purpose!

Technically it’s possible to access the actual environment block through undocumented fields in the PEB. It’s the same content as returned by get/free except that it’s not a copy. It cannot be accessed safely, so I’m ignoring this route.

The environment block format is a null-terminated block of null-terminated strings:

keyA=a\0keyBB=bb\0keyCCC=ccc\0\0

Each string begins with a character other than = and contains at least one =. In my tests this rule was strictly enforced by Windows, and I could not construct an environment block that broke this rule. This list is usually, but not always, sorted. It may contain repeated variables, but they’re always assigned the same value, which is also strictly enforced by Windows.

The get/free interface has no “set” function, and a process cannot set its own environment block to a custom buffer. (Update: Stefan Kanthak points out SetEnvironmentStringsW. I missed it because it was only officially documented a few months before this article was written.) There is one interface where a process gets to provide a raw environment block: CreateProcess. That is, a parent can construct one for its children.

    wchar_t env[] = L"HOME=C:\\Users\\me\0PATH=C:\\bin;C:\\Windows\0";
    CreateProcessW(L"example.exe", ..., env, ...);

Windows imposes some rules upon this environment block:

• If an element begins with = or does not contain =, CreateProcess fails.

• Repeated variables are modified to match the first instance. If you’re potentially overriding using a duplicate, put the override first.

• Some cases of bad formatting become memory access violations.

As usual for Win32, there are no rules against ill-formed UTF-16, and I could always pass such “UTF-16” through into the child environment block. Keep that in mind even when using the get/set interface.

The SetEnvironmentVariable documentation gives a maximum variable size:

The maximum size of a user-defined environment variable is 32,767 characters. There is no technical limitation on the size of the environment block.

At least on more recent versions of Windows, my experiments proved exactly the opposite. There is no limit on a user-defined environment variables, but environment blocks are limited to 2GiB, for both 32-bit and 64-bit processes. I could even create such huge environments in large address aware 32-bit processes, though the interfaces are prone to error due to allocations problems.

There’s one special case where CreateProcess is illogical, and it’s certainly a case of confusion within its implementation. An environment block is not allowed to be empty. An empty environment is represented as a block containing one empty (zero length) element. That is, two null terminators in a row. It’s the one case where an environment block may contain an element without a =. The logical empty environment block would be just one null terminator, to terminate the block itself, because it contains no variables. You can safely pretend that’s the case when parsing an environment block, as this special case is superfluous.

However, CreateProcess partially enforces this silly, unnecessary special case! If an environment block begins with a null terminator, the next character must be in a mapped memory region because it will read this character. If it’s not mapped, the result is a memory access violation. Its actual value doesn’t matter, and CreateProcess will treat it as though it was another null terminator. Surely someone at Microsoft would have noticed by now that this behavior makes no sense, but I guess it’s kept for backwards compatibility?

The CreateProcess documentation says that “the system uses a sorted environment” but this made no difference in my tests. The word “must” appears in this sentence, but it’s unclear if it applies to sorting, or even outside the special case being discussed. GetEnvironmentVariable works fine on an unsorted environment block. SetEnvironmentVariable maintains sorting, but given an unsorted block it goes somewhere in the middle, probably wherever a bisection happens to land. Perhaps look-ups in sorted blocks are faster, but environment blocks are so small — a maximum of 32K characters (Update: only true for ANSI) — that, in practice, it really does not matter.

Suppose you’re meticulous and want to sort your environment block before spawning a process. How do you go about it? There’s the rub: The official documentation is incomplete! The Changing Environment Variables page says:

All strings in the environment block must be sorted alphabetically by name. The sort is case-insensitive, Unicode order, without regard to locale.

What do they mean by “case-insensitive” sort? Does “Unicode order” mean case folding? A reasonable guess, but no, that’s not how get/set works. Besides, how does “Unicode order” apply to ill-formed UTF-16? Worse, get/set sorting is certainly not “Unicode order” even outside of case-insensitivity! For example, U+1F31E (SUN WITH FACE) sorts ahead of U+FF01 (FULLWIDTH EXCLAMATION MARK) because the former encodes in UTF-16 as U+D83C U+DF1E. Maybe it’s case-insensitive only in ASCII? Nope, π (U+03C0) and Π (U+03A0) are considered identical. Windows uses some kind of case-insensitive, but not case-folded, undocumented early 1990s UCS-2 sorting logic for environment variables.

Update: John Doty suspects the RtlCompareUnicodeString function for sorting. It lines up perfectly with get/set for all possible inputs.

Without better guidance, the only reliable way to “correctly” sort an environment block is to build it with get/set, then retrieve the result with get/free. The algorithm looks like:

1. Get a copy of the environment with GetEnvironmentStrings.
2. Walk the environment and call SetEnvironmentVariable on each name with a null pointer as the value. This clears out the environment.
3. Call SetEnvironmentVariable for each variable in the new environment.
4. Get a sorted copy of the new environment with GetEnvironmentStrings.

Unfortunately that’s all global state, so you can only construct one new environment block at a time.

If you know all your variable names ahead of time, then none of this is a problem. Determine what Windows thinks the order should be, then use that in your program when constructing the environment block. It’s the general case where this is a challenge, such as a language runtime designed to operate on arbitrary environment variables with behavior congruent to the rest of the system.

There are similar issues with looking up variables in an environment block. How does case-insensitivity work? Sorting is “without regard to locale” but what about when comparing variable names? The documentation doesn’t say. When enumerating variables using get/free, you might read what get/set considers to be duplicates, though at least values will always agree with get/set, i.e. they’re aliases of one variables. Windows maintains that invariant in my tests. The above algorithm would also delete these duplicates.

For example, if someone passed you a “dirty” environment with duplicates, or that was unsorted, this would clean it up in a way that allows get/free to be traversed in order without duplicates.

    wchar_t *env = GetEnvironmentStringsW();

    // Clear out the environment
    for (wchar_t *var = env; *var;) {
        size_t len = wcslen(var);
        size_t split = wcscspn(var, L"=");
        var[split] = 0;
        SetEnvironmentVariableW(var, 0);
        var[split] = '=';
        var += len + 1;
    }

    // Restore the original variables
    for (wchar_t *var = env; *var;) {
        size_t len = wcslen(var);
        size_t split = wcscspn(var, L"=");
        var[split] = 0;
        SetEnvironmentVariableW(var, var+split+1);
        var += len + 1;
    }

    FreeEnvironmentStringsW(env);

On the second pass, SetEnvironmentVariableW will gobble up all the duplicates.

As a final note, the CreateProcess page had said this up until February 2023 about the environment block parameter:

If this parameter is NULL and the environment block of the parent process contains Unicode characters, you must also ensure that dwCreationFlags includes CREATE_UNICODE_ENVIRONMENT.

That seems to indicate it’s virtually always wrong to call CreateProcess without that flag — that is, Windows will trash the child’s environment unless this flag is passed — which is a bonkers default. Fortunately this appears to be wrong, which is probably why the documentation was finally corrected (after several decades). Omitting this flag was fine under all my tests, and I was unable to produce surprising behavior on any system.

In summary:

• Prefer get/set for all operations except enumeration
• Environment blocks are not necessarily sorted
• Repeat variables are forced to the value of the first instance
• Variables may contain ill-formed UTF-16
• Empty environment blocks have a superfluous special case
• Entries cannot begin with =
• Entries must contain at least one =
• Sort order is ambiguous, so you cannot reliably do it yourself
• Case-insensitivity of names is ambiguous, so rely on get/set
• CREATE_UNICODE_ENVIRONMENT necessary only for non-null environment

Update September 2024: Correction from Kasper Brandt regarding variables beginning with =. I misunderstood how it was parsed and came to the wrong conclusion.

I no longer call these “freestanding” programs since that term is, at best, inaccurate. In fact, we will be actively avoiding GCC features associated with that label. Instead I call these CRT-free programs, where CRT stands for the C runtime the Windows-oriented term for libc. This term communicates both intent and scope.

Entry point

You should already know that main is not the program’s entry point, but a C application’s entry point. The CRT provides the entry point, where it initializes the CRT, including parsing command line options, then calls the application’s main. The real entry point doesn’t have a name. It’s just the address of the function to be called by the loader without arguments.

You might naively assume you could continue using the name main and tell the linker to use it as the entry point. You would be wrong. Avoid the name main! It has a special meaning in C gets special treatment. Using it without a conventional CRT will confuse your tools an may cause build issues.

While you can use almost any other name you like, the conventional names are mainCRTStartup (console subsystem) and WinMainCRTStartup (windows subsystem). It’s easy to remember: Append CRTStartup to the name you’d use in a normal CRT-linking application. I strongly recommend using these names because it reduces friction. Your tools are already familiar with them, so you won’t need to do anything special.

int mainCRTStartup(void);     // console subsystem
int WinMainCRTStartup(void);  // windows subsystem

The MSVC linker documentation says the entry point uses the __stdcall calling convention. Ignore this and do not use __stdcall for your entry point! Since entry points may take no arguments, there is no practical difference from the __cdecl calling convention, so it matters little. Rather, the goal is to avoid __stdcall function decorations. In particular, the GNU linker --entry option does not understand them, nor can it find decorated entry points on its own. If you use __stdcall, then the 32-bit GNU linker will silently (!) choose the beginning of your .text section as the entry point. (This bug was fixed in Binutils 2.42, released January 2024. __stdcall entry points now link correctly.)

If you’re using C++, then of course you will also need to use extern "C" so that it’s not name-mangled. Otherwise the results are similarly bad.

If using -fwhole-program, you will need to mark your entry point as externally visible for GCC so that it knows its an entry point. While linkers are familiar with conventional entry point names, GCC the compiler is not. Normally you do not need to worry about this.

__attribute((externally_visible))  // for -fwhole-program
int mainCRTStartup(void)
{
    return 0;
}

The entry point returns int. If there are no other threads then the process will exit with the returned value as its exit status. In practice this is only useful for console programs. Windows subsystem programs have threads started automatically, without warning, and it’s almost certain your main thread is not the last thread. You probably want to use ExitProcess or even TerminateProcess instead of returning. The latter exits more abruptly and can avoid issues with certain subsystems, like DirectSound, not shutting down gracefully: It doesn’t even let them try.

int WinMainCRTStartup(void)
{
    // ...
    TerminateProcess(GetCurrentProcess(), 0);
}

Compilation

Starting with the GNU toolchain, you have two ways to get into “CRT-free mode”: -nostartfiles and -nostdlib. The former is more dummy-proof, and it’s what I use in build documentation. The latter can be a more complicated, but when it succeeds you get guarantees about the result. I use it in build scripts I intend to run myself, which I want to fail if they don’t do exactly what I expect. To illustrate, consider this trivial program:

#include <windows.h>

int mainCRTStartup(void)
{
    ExitProcess(0);
}

This program uses ExitProcess from kernel32.dll. Compiling is easy:

$ cc -nostartfiles example.c

The -nostartfiles prevents it from linking the CRT entry point, but it still implicitly passes other “standard” linker flags, including libraries -lmingw32 and -lkernel32. Programs can use kernel32.dll functions without explicitly linking that DLL. But, hey, isn’t -lmingw32 the CRT, the thing we’re avoiding? It is, but it wasn’t actually linked because the program didn’t reference it.

$ objdump -p a.exe | grep -Fi .dll
        DLL Name: KERNEL32.dll

However, -nostdlib does not pass any of these libraries, so you need to do so explicitly.

$ cc -nostdlib example.c -lkernel32

The MSVC toolchain behaves a little like -nostartfiles, not linking a CRT unless you need it, semi-automatically. However, you’ll need to list kernel32.dll and tell it which subsystem you’re using.

$ cl example.c /link /subsystem:console kernel32.lib

However, MSVC has a handy little feature to list these arguments in the source file.

#ifdef _MSC_VER
  #pragma comment(linker, "/subsystem:console")
  #pragma comment(lib, "kernel32.lib")
#endif

This information must go somewhere, and I prefer the source file rather than a build script. Then anyone can point MSVC at the source without worrying about options.

$ cl example.c

I try to make all my Windows programs so simply built.

Stack probes

On Windows, it’s expected that stacks will commit dynamically. That is, the stack is merely reserved address space, and it’s only committed when the stack actually grows into it. This made sense 30 years ago as a memory saving technique, but today it no longer makes sense. However, programs are still built to use this mechanism.

To function properly, programs must touch each stack page for the first time in order. Normally that’s not an issue, but if your stack frame exceeds the page size, there’s a chance it might step over a page. When a function has a large stack frame, GCC inserts a call to a “stack probe” in libgcc that touches its pages in the prologue. It’s not unlike stack clash protection.

For example, if I have a 4kiB local variable:

int mainCRTStartup(void)
{
    char buf[1<<12] = {0};
    return 0;
}

When I compile with -nostdlib:

$ cc -nostdlib example.c
ld: ... undefined reference to `___chkstk_ms'

It’s trying to link the CRT stack probe. You can disable this behavior with -mno-stack-arg-probe.

$ cc -mno-stack-arg-probe -nostdlib example.c

Or you can just link -lgcc to provide a definition:

$ cc -nostdlib example.c -lgcc

Had you used -nostartfiles, you wouldn’t have noticed because it passes -lgcc automatically. It’s “dummy-proof” because this sort of issue goes away before it comes up, though for the same reason it’s harder to tell exactly what went into a program.

If you disable the probe altogether — my preference — you’ve only solved the linker problem, but the underlying stack commit problem remains and your program may crash. You can solve that by telling the linker to ask the loader to commit a larger stack up front rather than grow it at run time. Say, 2MiB:

$ cc -mno-stack-arg-probe -Xlinker --stack=0x200000,0x200000 example.c

Of course, I wish that this was simply the default behavior because it’s far more sensible! A much better option is to avoid large stack frames in the first place. Allocate locals larger than, say, 1KiB in a scratch arena instead of on the stack.

MSVC doesn’t have libgcc of course, but it still generates stack probes both for growing the stack and for security checks. The latter requires kernel32.dll, so if I compile the same program with MSVC, I get a bunch of linker failures:

$ cl example.c /link /subsystem:console
... unresolved external symbol __imp_RtlCaptureContext ...
... and 7 more ...

Using /Gs1000000000 turns off the stack probes, /GS- turns off the checks, /stack commits a larger stack:

$ cl /GS- /Gs1000000000 example.c /link
     /subsystem:console /stack:0x200000,200000

Though, as before, better to avoid large stack frames in the first place.

Built-in functions… ugh

The three major C and C++ compilers — GCC, MSVC, Clang — share a common, evil weakness: “built-in” functions. No matter what, they each assume you will supply definitions for standard string functions at link time, particularly memset and memcpy. They do this no matter how many “seriously now, do not use standard C functions” options you pass. When you don’t link a CRT, you may need to define them yourself.

With GCC there’s a catch: it will transform your memset definition — that is, in a function named memset — into a call to itself. After all, it looks an awful lot like memset! This typically manifests as an infinite loop. Use -fno-builtin to prevent GCC from mis-compiling built-in functions.

Even with -fno-builtin, both GCC and Clang will continue inserting calls to built-in functions elsewhere. For example, making an especially large local variable (and using volatile to prevent it from being optimized out):

int mainCRTStartup(void)
{
    volatile char buf[1<<14] = {0};
    return 0;
}

As of this writing, the latest GCC and Clang will generate a memset call despite -fno-builtin:

$ cc -mno-stack-arg-probe -fno-builtin -nostdlib example.c
ld: ... undefined reference to `memset' ...

To be absolutely pure, you will need to address this in just about any non-trivial program. On the other hand, -nostartfiles will grab a definition from msvcrt.dll for you:

$ cc -nostartfiles example.c
$ objdump -p a.exe | grep -Fi .dll
        DLL Name: msvcrt.dll

To be clear, this is a completely legitimate and pragmatic route! You get the benefits of both worlds: the CRT is still out of the way, but there’s also no hassle from misbehaving compilers. If this sounds like a good deal, then do it! (For on-lookers feeling smug: there is no such easy, general solution for this problem on Linux.)

When you write your own definitions, I suggest putting each definition in its own section so that they can be discarded via -Wl,--gc-sections when unused:

__attribute((section(".text.memset")))
void *memset(void *d, int c, size_t n)
{
    // ...
}

So far, for all three compilers, I’ve only needed to provide definitions for memset and memcpy.

Stack alignment on 32-bit x86

GCC expects a 16-byte aligned stack and generates code accordingly. Such is dictated by the x64 ABI, so that’s a given on 64-bit Windows. However, the x86 ABIs only guarantee 4-byte alignment. If no care is taken to deal with it, there will likely be unaligned loads. Some may not be valid (e.g. SIMD) leading to a crash. UBSan disapproves, too. Fortunately there’s a function attribute for this:

__attribute((force_align_arg_pointer))
int mainCRTStartup(void)
{
    // ...
}

GCC will now align the stack in this function’s prologue. Adjustment is only necessary at entry points, as GCC will maintain alignment through its own frames. This includes all entry points, not just the program entry point, particularly thread start functions. Rule of thumb for i686 GCC: If WINAPI or __stdcall appears in a definition, the stack likely requires alignment.

__attribute((force_align_arg_pointer))
DWORD WINAPI mythread(void *arg)
{
    // ...
}

It’s harmless to use this attribute on x64. The prologue will just be a smidge larger. If you’re worried about it, use #ifdef __i686__ to limit it to 32-bit builds.

Putting it all together

If I’ve written a graphical application with WinMainCRTStartup, used large stack frames, marked my entry point as externally visible, plan to support 32-bit builds, and defined a couple of needed string functions, my optimal entry point may look something like:

#ifdef __GNUC__
__attribute((externally_visible))
#endif
#ifdef __i686__
__attribute((force_align_arg_pointer))
#endif
int WinMainCRTStartup(void)
{
    // ...
}

Then my “optimize all the things” release build may look something like:

$ cc -O3 -fno-builtin -Wl,--gc-sections -s -nostdlib -mwindows
     -fno-asynchronous-unwind-tables -o app.exe app.c -lkernel32

Or with MSVC:

$ cl /O2 /GS- app.c /link kernel32.lib /subsystem:windows

Or if I’m taking it easy maybe just:

$ cc -O3 -fno-builtin -s -nostartfiles -mwindows -o app.exe app.c

Or with MSVC (linker flags in source):

$ cl /O2 app.c

SDL has grown on me over the past year. I didn’t understand its value until viewing it in the right lens: as a complete platform and runtime replacing the host’s runtime, possibly including libc. Ideally an SDL application links exclusively against SDL and otherwise not directly against host libraries, though in practice it’s somewhat porous. With care — particularly in avoiding mistakes covered in this article — that ideal is quite achievable for C applications that fit within SDL’s feature set.

SDL applications are always interesting one way or another, so I like to dig in when I come across them. The items in this article are mistakes I’ve either made myself or observed across many such passion projects in the wild.

Mistake 1: Not using sdl2-config

This shell script comes with SDL2 and smooths over differences between platforms, even when cross compiling. It informs your compiler where to find and how to link SDL2. The script even works on Windows if you have a unix shell, such as via w64devkit. Use it as a command substitution at the end of the build command, particularly when using --libs. A one-shot or unity build (my preference) looks like so:

$ cc app.c $(sdl2-config --cflags --libs)

Or under separate compilation:

$ cc -c app.c $(sdl2-config --cflags)
$ cc app.o $(sdl2-config --libs)

Alternatively, static link by replacing --libs with --static-libs, though this is discouraged by the SDL project. When dynamically linked, users can, and do, trivially substitute a different SDL2 binary, such as one patched for their system. In my experience, static linking works reliably on Windows but poorly on Linux.

Alternatively, use the general purpose pkg-config. Don’t forget eval!

$ eval cc app.c $(pkg-config sdl2 --cflags --libs)

I wrote a pkg-config for Windows specifically for this case.

Caveats:

• Some circumstances require special treatment, and sdl2-config may be too blunt a tool. That’s fine, but generally prefer sdl2-config as the default approach.

• sdl2-config does not support extensions such as SDL2_image, so you will need to use pkg-config. Personally I don’t think they’re worth the trouble when there’s stb, or QOI instead of PNG.

• There’s an alternative build option using CMake, without any use of sdl2-config, but I won’t discuss it here.

Mistake 2: Including SDL2/SDL.h

A lot of examples, including tutorials linked from the official SDL website, have SDL2/ in their include paths. That’s because they’re making mistake 1, not using sdl2-config, and are instead relying on Linux distributions having installed SDL2 in a place coincidentally accessible through that include path.

This is annoying when SDL2 not installed there, or if I don’t want it using the system’s SDL2. Worse, it can result in subtly broken builds as it mixes and matches different SDL installations. The correct SDL2 include is the following:

#include "SDL.h"

Note the quotes, which helps prevent picking up an arbitrary system header by accident. When carefully and narrowly targeting SDL-the-platform, this will be the only “system” include anywhere in your application.

Mistake 3: Not surrendering main

A conventional SDL application has a main function defined in its source, but despite the name, this is distinct from C main. To smooth over platform differences, SDL may rename the application’s main to SDL_main and substitute its own C main. Because of this, main must have the conventional argc/argv prototype and must return a value. (As a special case, C permits main to implicitly return 0, so it’s an easy mistake to make.)

With this in mind, the bare minimum SDL2 application:

#include "SDL.h"

int main(int argc, char **argv)
{
    return 0;
}

Caveat: Like with sdl2-config, some special circumstances require control over the application entry point — see SDL_MAIN_HANDLED and SDL_SetMainReady — but that should be reserved until there’s a need.

One such special case is avoiding linking a CRT on Windows. In principle it’s this simple:

#include "SDL.h"

int WinMainCRTStartup(void)
{
    SDL_SetMainReady();
    // ...
    return 0;
}

Then it’s the usual compiler and linker flags:

$ cc -nostdlib -o app.exe app.c $(sdl2-config --cflags --libs)

This will create a tiny .exe that doesn’t link any system DLL, just SDL2.dll. Quite platform agnostic indeed!

$ objdump -p app.exe | grep -Fi .dll
        DLL Name: SDL2.dll

Alas, as of this writing, this does not work reliably. SDL2’s accelerated renderers on Windows do not clean up properly in SDL_QuitSubSystem nor SDL_Quit, so the process cannot exit without calling ExitProcess in kernel32.dll (or similar). This is still an open experiment.

Mistake 4: Using the SDL wiki for API documentation

The SDL wiki is not authoritative documentation, merely a convenient web-linkable — and downloadable (see “offline html”) — information source. However, anyone who’s spent time on it can tell you it’s incomplete. The authoritative API documentation is the SDL headers, which fortunately are already on hand for building SDL applications. The SDL maintainers themselves use the headers, not the wiki.

If, like me, you’re using ctags, this is actually good news! With a bit of configuration, you can jump to any bit of SDL documentation at any time in your editor, treating the SDL headers like a hyperlinked wiki built into your editor. Just like building, sdl2-config can tell ctags where find those headers:

$ ctags -a -R --kinds-c=dept $(sdl2-config --prefix)/include/SDL2

I’m using -a (--append) to append to the tags file I’ve already generated for my own program, -R (--recurse) to automatically find all the headers, and --kinds-c=dept capture exactly the kinds of symbols I care about — #define, enum, prototypes, typedef — no more no less.

In Vim I CTRL-] over any SDL symbol to jump to its documentation, and then I can use it again within its documentation comment to jump further still to any symbols it mentions, then finally use the jump or tag stack to return. As long as I have t in 'complete' ('cpt'), which is the default, I can also “tab”-complete any SDL symbol using the tags table. There are a few rough edges here and there, but overall it’s a solid editing paradigm.

By the way, with sdl2-config in your $PATH, all the above works out of the box in w64devkit! That’s where I’ve mostly been working with SDL.

Mistake 5: Using stdio streams

A common bit of code in real SDL programs and virtually every tutorial:

if (SDL_Init(...)) {
    fprintf(stderr, "SDL_Init(): %s\n", SDL_GetError());
    return 1;
}

This is not ideal:

• fprintf is not part of the SDL platform. This is going behind SDL’s back, reaching around the abstraction to a different platform. Strictly speaking, this API may not even be available to an SDL application.

• SDL applications are graphical, so stderr is likely disconnected from anything useful. Few would ever see this message.

Fortunately SDL provides two alternatives:

• SDL_Log: like C printf, but SDL will strive to connect it to somewhere useful. If the application was launched from a terminal or console, SDL will find it and hook it up to the logger. On Windows, if there’s a debugger attached, SDL will use OutputDebugString to send logs to the debugger.

• SDL_ShowSimpleMessageBox: using any means possible, attempt to display a message to the user. Like SDL_Log, it’s safe to use before/without initializing SDL subsystems.

If you’re paranoid, you could even use both:

if (SDL_Init(...)) {
    SDL_ShowSimpleMessageBox(
        SDL_MESSAGEBOX_ERROR, "SDL_Init()", SDL_GetError(), 0
    );
    SDL_Log("SDL_Init(): %s", SDL_GetError());
    return 1;
}

Though note that SDL_ShowSimpleMessageBox can fail, which will set a new, different error message for SDL_Log!

There’s a similar story again with fopen and loading assets. SDL has an I/O API, SDL_RWops. It’s probably better than the host’s C equivalent, particularly with regards to paths. If you’re not already embedding your assets, use the SDL API instead.

Mistake 6: Using SDL_RENDERER_ACCELERATED

This flag — and its surrounding bit set, SDL_RendererFlags — are a subtle design flaw in the SDL2 API. Its existence is misleading, causing to widespread misuse. It does not help that the documentation, both header and wiki, is incomplete and unclear. The SDL_CreateRenderer function accepts a bit set as its third argument, and it serves two simultaneous purposes:

• Indicates mandatory properties of the renderer. Examples: “must use accelerated rendering,” “must use software rendering,” “must support vertical synchronization (vsync).” Drivers without the chosen properties are skipped.

• If SDL_RENDERER_PRESENTVSYNC is set, also enables vsync in the created render.

The common mistake is thinking that this bit indicates preference: “prefer an accelerated renderer if possible”. But it really means “accelerated renderer or bust.”

Given a zero for renderer flags, SDL will first attempt to create an accelerated renderer. Failing that, it will then attempt to create a software renderer. A software renderer fallback is exactly the behavior you want! After all, this fallback is one of the primary features of the SDL renderer API. This is so straightforward there are no caveats.

Mistake 7: Not accounting for vsync

For a game, you probably ought to enable vsync in your renderer. The hint: You’re using SDL_PollEvent in your main event loop. Otherwise you will waste lots of resources rendering thousands of frames per second. If my laptop fan spins up running your SDL application, it’s probably because you didn’t do this. The following should be the most conventional SDL renderer configuration:

r = SDL_CreateRenderer(window, -1, SDL_RENDERER_PRESENTVSYNC);

The software renderer supports vsync, so it will not be excluded from the driver search when vsync is requested.

That’s only for SDL renderers. If you’re using OpenGL, set a non-zero SDL_GL_SetSwapInterval so that SDL_GL_SwapWindow synchronizes. For the other rendering APIs, consult their documentation. (I can only speak to SDL and OpenGL from experience.)

Caveat: Beware accidentally relying on vsync for timing in your game. You don’t want your game’s physics to depend on the host’s display speed. Even the pros make this mistake from time to time.

However, if you’re not making a game – perhaps instead an IMGUI application without active animations — there’s a good chance you don’t need or want vsync. The hint: You’re using SDL_WaitEvent in your main event loop.

In summary, graphical SDL applications fall into one of two cases:

• SDL_PollEvent with vsync
• SDL_WaitEvent without vsync

Mistake 8: Using assert.h instead of SDL_assert

Alright, this one isn’t so common, but I’d like to highlight it. The SDL_assert macro is fantastic, easily beating assert.h which doesn’t even break in the right place. It uses SDL to present a user interface to the assertion, with support for retrying and ignoring. It also works great under debuggers, breaking exactly as it should. I have nothing but praise for it, so don’t pass up the chance to use it when you can.

While I’m at it: during developing and testing, always always always run your application under a debugger. Don’t close the debugger, just launch through it again after rebuilding. Also, enable UBSan and ASan when available for the extra assertions.

SDL wishlist

For months I had wondered why SDL provides no memory allocation API. I’m fine if it doesn’t have a general purpose allocator since I just want to grab a chunk of host memory for an arena. However, SDL does have allocations functions — SDL_malloc, etc. I didn’t know about them until I stopped making mistake 4.

It was the same story again with math functions: I’d like not to stray from SDL as a platform, but what if I need transcendental functions? I could whip up crude implementations myself, but I’d prefer not. SDL has those too: SDL_sin, etc. Caveat: The math.h functions are built-ins, and compilers use that information to better optimize programs, e.g. cool stuff like -mrecip, or SIMD vectorization. That cannot be done with SDL’s equivalents.

I’m surprised SDL has no random number generator considering how important it is to games. Since I prefer to handle this myself, I don’t mind that so much, but it does leave a lot of toy programs out there calling C rand. I would like SDL if provided a single, good seed early during startup. There isn’t even a wall clock function for the classic srand(time(0)) seeding event! My solution has been to mix event timestamps into the random state:

static Uint32 rand32(Uint64 *);

Uint64 rng = 0;
for (SDL_Event e; SDL_PollEvent(&e);) {
    rng ^= e.common.timestamp;
    rand32(&rng);  // stir
    switch (e.type) { /* ... */ }
}

As I learn more in the future, I may come back and add to this list. At the very least I expect to use SDL increasingly in my own projects.