A window procedure has this prototype:

LRESULT Wndproc(
  HWND hWnd,
  UINT Msg,
  WPARAM wParam,
  LPARAM lParam,
);

To create a window we must first register a class with RegisterClass, which accepts a set of properties describing a window class, including a pointer to one of these functions.

    MyState *state = ...;

    RegisterClassA(&(WNDCLASSA){
        // ...
        .lpfnWndProc   = my_wndproc,
        .lpszClassName = "my_class",
        // ...
    });

    HWND hwnd = CreateWindowExA("my_class", ..., state);

The thread drives a message pump with events from the operating system, dispatching them to this procedure, which then manipulates the program state in response:

    for (MSG msg; GetMessageW(&msg, 0, 0, 0);) {
        TranslateMessage(&msg);
        DispatchMessageW(&msg);  // calls the window procedure
    }

All four WNDPROC parameters are determined by Win32. There is no context pointer argument. So how does this procedure access the program state? We generally have two options:

1. Global variables. Yucky but easy. Frequently seen in tutorials.
2. A GWLP_USERDATA pointer attached to the window.

The second option takes some setup. Win32 passes the last CreateWindowEx argument to the window procedure when the window created, via WM_CREATE. The procedure attaches the pointer to its window as GWLP_USERDATA. This pointer is passed indirectly, through a CREATESTRUCT. So ultimately it looks like this:

    case WM_CREATE:
        CREATESTRUCT *cs = (CREATESTRUCT *)lParam;
        void *arg = (struct state *)cs->lpCreateParams;
        SetWindowLongPtr(hwnd, GWLP_USERDATA, (LONG_PTR)arg);
        // ...

In future messages we can retrieve it with GetWindowLongPtr. Every time I go through this I wish there was a better way. What if there was a fifth window procedure parameter though which we could pass a context?

typedef LRESULT Wndproc5(HWND, UINT, WPARAM, LPARAM, void *);

We’ll build just this as a trampoline. The x64 calling convention passes the first four arguments in registers, and the rest are pushed on the stack, including this new parameter. Our trampoline cannot just stuff the extra parameter in the register, but will actually have to build a stack frame. Slightly more complicated, but barely so.

Allocating executable memory

In previous articles, and in the programs where I’ve applied techniques like this, I’ve allocated executable memory with VirtualAlloc (or mmap elsewhere). This introduces a small challenge for solving the problem generally: Allocations may be arbitrarily far from our code and data, out of reach of relative addressing. If they’re further than 2G apart, we need to encode absolute addresses, and in the simple case would just assume they’re always too far apart.

These days I’ve more experience with executable formats, and allocation, and I immediately see a better solution: Request a block of writable, executable memory from the loader, then allocate our trampolines from it. Other than being executable, this memory isn’t special, and allocation works the usual way, using functions unaware it’s executable. By allocating through the loader, this memory will be part of our loaded image, guaranteed to be close to our other code and data, allowing our JIT compiler to assume a small code model.

There are a number of ways to do this, and here’s one way to do it with GNU-styled toolchains targeting COFF:

        .section .exebuf,"bwx"
        .globl exebuf
exebuf:	.space 1<<21

This assembly program defines a new section named .exebuf containing 2M of writable ("w"), executable ("x") memory, allocated at run time just like .bss ("b"). We’ll treat this like an arena out of which we can allocate all trampolines we’ll probably ever need. With careful use of .pushsection this could be basic inline assembly, but I’ve left it as a separate source. On the C side I retrieve this like so:

typedef struct {
    char *beg;
    char *end;
} Arena;

Arena get_exebuf()
{
    extern char exebuf[1<<21];
    Arena r = {exebuf, exebuf+sizeof(exebuf)};
    return r;
}

Unfortunately I have to repeat myself on the size. There are different ways to deal with this, but this is simple enough for now. I would have loved to define the array in C with the GCC section attribute, but as is usually the case with this attribute, it’s not up to the task, lacking the ability to set section flags. Besides, by not relying on the attribute, any C compiler could compile this source, and we only need a GNU-style toolchain to create the tiny COFF object containing exebuf.

While we’re at it, a reminder of some other basic definitions we’ll need:

#define S(s)            (Str){s, sizeof(s)-1}
#define new(a, n, t)    (t *)alloc(a, n, sizeof(t), _Alignof(t))

typedef struct {
    char     *data;
    ptrdiff_t len;
} Str;

Str clone(Arena *a, Str s)
{
    Str r = s;
    r.data = new(a, r.len, char);
    memcpy(r.data, s.data, (size_t)r.len);
    return r;
}

Which have been discussed at length in previous articles.

Trampoline compiler

From here the plan is to create a function that accepts a Wndproc5 and a context pointer to bind, and returns a classic WNDPROC:

WNDPROC make_wndproc(Arena *, Wndproc5, void *arg);

Our window procedure now gets a fifth argument with the program state:

LRESULT my_wndproc(HWND, UINT, WPARAM, LPARAM, void *arg)
{
    MyState *state = arg;
    // ...
}

When registering the class we wrap it in a trampoline compatible with RegisterClass:

    RegisterClassA(&(WNDCLASSA){
        // ...
        .lpfnWndProc   = make_wndproc(a, my_wndproc, state),
        .lpszClassName = "my_class",
        // ...
    });

All windows using this class will readily have access to this state object through their fifth parameter. It turns out setting up exebuf was the more complicated part, and make_wndproc is quite simple!

WNDPROC make_wndproc(Arena *a, Wndproc5 proc, void *arg)
{
    Str thunk = S(
        "\x48\x83\xec\x28"      // sub   $40, %rsp
        "\x48\xb8........"      // movq  $arg, %rax
        "\x48\x89\x44\x24\x20"  // mov   %rax, 32(%rsp)
        "\xe8...."              // call  proc
        "\x48\x83\xc4\x28"      // add   $40, %rsp
        "\xc3"                  // ret
    );
    Str r   = clone(a, thunk);
    int rel = (int)((uintptr_t)proc - (uintptr_t)(r.data + 24));
    memcpy(r.data+ 6, &arg, sizeof(arg));
    memcpy(r.data+20, &rel, sizeof(rel));
    return (WNDPROC)r.data;
}

The assembly allocates a new stack frame, with callee shadow space, and with room for the new argument, which also happens to re-align the stack. It stores the new argument for the Wndproc5 just above the shadow space. Then calls into the Wndproc5 without touching other parameters. There are two “patches” to fill out, which I’ve initially filled with dots: the context pointer itself, and a 32-bit signed relative address for the call. It’s going to be very near the callee. The only thing I don’t like about this function is that I’ve manually worked out the patch offsets.

It’s probably not useful, but it’s easy to update the context pointer at any time if hold onto the trampoline pointer:

void set_wndproc_arg(WNDPROC p, void *arg)
{
    memcpy((char *)p+6, &arg, sizeof(arg));
}

So, for instance:

    MyState *state[2] = ...;  // multiple states
    WNDPROC proc = make_wndproc(a, my_wndproc, state[0]);
    // ...
    set_wndproc_arg(proc, state[1]);  // switch states

Though I expect the most common case is just creating multiple procedures:

    WNDPROC procs[] = {
        make_wndproc(a, my_wndproc, state[0]),
        make_wndproc(a, my_wndproc, state[1]),
    };

To my slight surprise these trampolines still work with an active Control Flow Guard system policy. Trampolines do not have stack unwind entries, and I thought Windows might refuse to pass control to them.

Here’s a complete, runnable example if you’d like to try it yourself: main.c and exebuf.s

Better cases

This is more work than going through GWLP_USERDATA, and real programs have a small, fixed number of window procedures — typically one — so this isn’t the best example, but I wanted to illustrate with a real interface. Again, perhaps the best real use is a library with a weak custom allocator interface:

typedef struct {
    void *(*malloc)(size_t);   // no context pointer!
    void  (*free)(void *);     // "
} Allocator;

void *arena_malloc(size_t, Arena *);

// ...

    Allocator perm_allocator = {
        .malloc = make_trampoline(exearena, arena_malloc, perm);
        .free   = noop_free,
    };
    Allocator scratch_allocator = {
        .malloc = make_trampoline(exearena, arena_malloc, scratch);
        .free   = noop_free,
    };

Something to keep in my back pocket for the future.

If you’d like to skip ahead, here’s the full source up front. Tested with w64devkit, MSVC cl, and clang-cl: scratch/regex-wrap

Interface design

The C interface I came up with, regex.h:

#pragma once
#include <stddef.h>

#define S(s) (str){s, sizeof(s)-1}

typedef struct {
    char     *data;
    ptrdiff_t len;
} str;

typedef struct {
    char *beg;
    char *end;
} arena;

typedef struct regex regex;

typedef struct {
    str      *data;
    ptrdiff_t len;
} strlist;

regex  *regex_new(str, arena *);
strlist regex_match(regex *, str, arena *);

Longtime readers will find it familiar: my favorite non-owning, counted strings form in place of null-terminated strings — similar to C++ std::string_view — and arena allocation. Yes, such fundamental types wouldn’t “belong” to a regex library like this, but imagine they’re standardized by the project or whatever. Also, this is purely a C header, not a C/C++ polyglot, and will not be used by the C++ portion.

In particular note the lack of “free” functions. The regex engine allocates everything in the arena, including all temporary working memory used while compiling, matching, etc. So in a sense, it could be called a non-allocating library. This requires a bit of C++ abuse: I will not call some C++ regex destructors. It shouldn’t matter because they only redundantly manage memory in the arena. (If regex objects are holding file handles or something else unnecessary then its implementation so poor as to not be worth using, and we should just use a better regex library.)

Now’s a good time to mention a caveat: In order to pull this off the regex library lives in its own Dynamic-Link Library with its own copy of the C++ standard library, i.e. statically linked. My demo is Windows-only, but this concept theoretically extends to shared objects on Linux. Since it’s a C interface that doesn’t expose standard library objects, the DLL can be used by programs compiled with different toolchains. Though that wouldn’t apply to my inciting hypothetical.

Example usage:

regex  *re = regex_new(S("(\\w+)"), perm);
str     s  = S("Hello, world! This is a test.");
strlist m  = regex_match(re, s, perm);
for (ptrdiff_t i = 0; i < m.len; i++) {
    printf("%2td = %.*s\n", i, (int)m.data[i].len, m.data[i].data);
}

This program prints:

 0 = Hello
 1 = world
 2 = This
 3 = is
 4 = a
 5 = test

If matching lots of source strings, scope the arena to the loop and then the results, and any regex working memory, are automatically freed in O(1) at the end of each iteration:

for (ptrdiff_t i = 0; i < ninputs; i++) {
    arena   scratch = *perm;
    strlist matches = regex_match(re, inputs[i], &scratch);
    // ... consume matches ...
}

C++ implementation

On the C++ side the first thing I do is replace new and delete, which is how I force it to allocate from the arena. This replaces new/delete for globally, but recall that the regex library has its own, private C++ implementation. Replacements apply only to itself even if there’s other C++ present in the process. If this is the only C++ in the process then it doesn’t require such careful isolation.

I can’t tell std::regex about the arena — it calls operator new the usual way, without extra arguments — so I have to smuggle it in through a thread-local variable:

static thread_local arena *perm;

If I’m sure the library is only used by a single thread then I can omit thread_local, but it’s useful here to demonstrate and measure. Using it in my operator replacements:

void *operator new(size_t size, std::align_val_t align)
{
    arena    *a     = perm;
    ptrdiff_t ssize = size;
    ptrdiff_t pad   = (uintptr_t)a->end & ((int)align - 1);
    if (ssize < 0 || ssize > a->end - a->beg - pad) {
        throw std::bad_alloc{};
    }
    return a->end -= size + pad;
}

void *operator new(size_t size)
{
    return operator new(
        size,
        std::align_val_t(__STDCPP_DEFAULT_NEW_ALIGNMENT__)
    );
}

Starting in C++17, replacing the global allocator requires definitions for both plain new/delete and aligned new/delete. The many other variants, including arrays, call these four and so may be skipped. Allocating over-aligned objects isn’t a special case for arenas, so I implemented plain new by calling aligned new. I’d prefer to allocate through a template so that I can “see” the type, but that’s not an option in this case.

After converting to signed sizes because they’re simpler, it’s the usual from-the-end allocation. I prefer -fno-exceptions but std::regex is inherently exceptional — and I mean that in at least two bad ways — so they’re required. The good news is this library gracefully and reliably handles out-of-memory errors. (The arena makes this trivial to test, so try it for yourself!)

I added a little extra flair replacing delete:

void operator delete(void *) noexcept {}
void operator delete(void *, std::align_val_t) noexcept {}

void operator delete(void *p, size_t size) noexcept
{
    arena *a = perm;
    if (a->end == (char *)p) {
        a->end += size;
    }
}

The two mandatory replacements are no-ops because that’s simply how arenas work. We don’t free individual objects, but many at once. It’s completely optional, but I also replaced sized delete for little other reason than sized deallocation is cool. C++ destructs in reverse order, so this is likely to work out. At least with GCC libstdc++, it freed about a third of the workspace memory before returning to C. I’d rather it didn’t try to free anything at all, but since it’s going to call delete anyway I can get some use out of it.

Interesting side note: In a rough benchmark these replacements made MSVC std::regex matching four times faster! I expected a small speedup, but not that. In the typical case it appears to be wasting most of its time on allocation. On the other hand, libstdc++ std::regex is overall quite a bit slower than MSVC, and my replacements had no performance effect. It’s spending its time elsewhere, and the small gains are lost interacting with the thread-local.

Finally the meat:

extern "C" std::regex *regex_new(str re, arena *a)
{
    perm = a;
    try {
        return new std::regex(re.data, re.data+re.len);
    } catch (...) {
        return {};
    }
}

It sets the thread-local to the arena, then constructs with “iterators” at each end of the input. All exceptions are caught and turned into a null return. Depending on need, we may want to indicate why it failed — out of memory, invalid regex, etc. — by returning an error value of some sort. An exercise for the reader.

The matcher is a little more complicated:

extern "C" strlist regex_match(std::regex *re, str s, arena *a)
{
    perm = a;
    try {
        std::cregex_iterator it(s.data, s.data+s.len, *re);
        std::cregex_iterator end;

        strlist r = {};
        r.len  = std::distance(it, end);
        r.data = new str[r.len]();
        for (ptrdiff_t i = 0; it != end; it++, i++) {
            r.data[i].data = s.data + it->position();
            r.data[i].len  = it->length();
        }
        return r;

    } catch (...) {
        return {};
    }
}

I create a char * “cregex” iterator, again giving it each end of the input. I hope it’s not just making a copy (MSVC std::regex does grumble grumble). The result is allocated out of the arena. As before, exceptions convert to a null return. Callers can distinguish errors because no-match results have a non-null pointer. The iterator, being a local variable, is destroyed before returning, uselessly calling delete. I could avoid this by allocating it with new, but in practice it doesn’t matter.

You might have noticed the lack of declspec(dllexport). DEF files are great, and I’ve come to appreciate and prefer them. GCC and MSVC accept them as another input on the command line, and the source need not be aware exports. My regex.def:

LIBRARY regex
EXPORTS
regex_new
regex_match

In w64devkit, the command to build the DLL:

$ g++ -shared -std=c++17 -o regex.dll regex.cpp regex.def

The MSVC command almost maps 1:1 to the GCC command:

$ cl /LD /std:c++17 /EHsc regex.cpp regex.def

In either case only the C interface is exported (via peports):

$ peports -e regex.dll
EXPORTS
        1       regex_match
        2       regex_new

Reasons against

Though this library is conveniently on hand, and my minimalist C wrapper interface is nicer than a typical C regex library interface, and even hides some std::regex problems, trade-offs must be considered:

• No Unicode support, particularly UTF-8
• std::regex implementations are universally poor and slow
• libstdc++ std::regex is especially slow to compile
• Isolating in a DLL (if needed) is inconvenient
• DLL is 200K (MSVC) to 700K (GCC) or so

Depending on what I’m doing, some of these may have me looking elsewhere.

To give a sense of the numbers, here’s empty.c, which does nothing but include windows.h.

#include <windows.h>

With the current Mingw-w64 headers, that’s ~82kLOC (non-blank):

$ gcc -E empty.c | grep -vc '^$'
82041

With w64devkit this takes my system ~450ms to compile with GCC:

$ time gcc -c empty.c
real    0m 0.45s
user    0m 0.00s
sys     0m 0.00s

Compiling an actually empty source file takes ~10ms, so it really is spending practically all that time processing headers. MSVC is a faster compiler, and this extends to processing an even larger windows.h that crosses over 100kLOC (VS2022). It clocks in at 120ms on the same system:

$ cl /nologo /E empty.c | grep -vc '^$'
empty.c
100944
$ time cl /nologo /c empty.c
empty.c
real    0m 0.12s
user    0m 0.09s
sys     0m 0.01s

That’s just low enough to be tolerable, but I’d like the situation with GCC to be better. Defining WIN32_LEAN_AND_MEAN reduces the number of included headers, which has a significant effect:

$ gcc -E -DWIN32_LEAN_AND_MEAN empty.c | grep -vc '^$'
55025
$ time gcc -c -DWIN32_LEAN_AND_MEAN empty.c
real    0m 0.30s
user    0m 0.00s
sys     0m 0.00s

$ cl /nologo /E /DWIN32_LEAN_AND_MEAN empty.c | grep -vc '^$'
empty.c
41436
$ time cl /nologo /c /DWIN32_LEAN_AND_MEAN empty.c
empty.c
real    0m 0.07s
user    0m 0.01s
sys     0m 0.01s

Precompiled headers

The official solution is precompiled headers. Put all the system header includes, or similar, into a dedicated header, then compile that header into a special format. For example, headers.h:

#define WIN32_LEAN_AND_MEAN
#include <windows.h>

Then main.c includes windows.h through this header:

#include "headers.h"

int mainCRTStartup(void)
{
    return 0;
}

If I ask GCC to compile headers.h:

$ gcc headers.h

It produces headers.h.gch. When a source includes headers.h, GCC first searches for an appropriate .gch. Not only must the name match, but so must all the definitions at the moment of inclusion: headers.h should always be the first included header, otherwise it may not work. Now when I compile main.c:

$ time gcc -c main.c
real    0m 0.04s
user    0m 0.00s
sys     0m 0.00s

Much better! MSVC has a conventional name for this header recognizable to every Visual Studio user: stdafx.h. It works a bit differently, and I’ve never used it myself, but I trust it has similar results.

Precompiled headers requires some extra steps that vary by toolchain. Can we do better? That depends on your definition of “better!”

Artisan, handcrafted prototypes

As mentioned, systems software tends to need only a few declarations: open, read, write, stat, etc. What if I wrote these out manually? A bit tedious, but it doesn’t require special precompiled header handling. It also creates some new possibilities. To illustrate, a CRT-free “hello world” program:

#include <windows.h>

int mainCRTStartup(void)
{
    HANDLE stdout = GetStdHandle(STD_OUTPUT_HANDLE);
    char message[] = "Hello, world!\n";
    DWORD len;
    return !WriteFile(stdout, message, sizeof(message)-1, &len, 0);
}

This takes my system half a second to compile — quite long to produce just 26 assembly instructions:

$ time cc -nostartfiles -o hello.exe hello.c
real    0m 0.50s
user    0m 0.00s
sys     0m 0.00s
$ ./hello.exe
Hello, world!

The program requires prototypes only for GetStdHandle and WriteFile, a definition for STD_OUTPUT_HANDLE, and some typedefs. Starting with the easy stuff, the definition and types look like this:

#define STD_OUTPUT_HANDLE ((DWORD)-11)

typedef int BOOL;
typedef void *HANDLE;
typedef unsigned long DWORD;

By the way, here’s a cheat code for quickly finding preprocessor definitions, faster than looking them up elsewhere:

$ echo '#include <windows.h>' | gcc -E -dM - | grep 'STD_\w*_HANDLE'
#define STD_INPUT_HANDLE ((DWORD)-10)
#define STD_ERROR_HANDLE ((DWORD)-12)
#define STD_OUTPUT_HANDLE ((DWORD)-11)

Did you catch the pattern? It’s -10 - fd, where fd is the conventional unix file descriptor number: a kind of mnemonic.

Prototypes are a little trickier, especially if you care about 32-bit. The Windows API uses the “stdcall” calling convention, which is distinct from the “cdecl” calling convention on x86, though the same on x64. Of course, you must already be aware of this merely using the API, as your own callbacks must usually be stdcall themselves. Further, API functions are DLL imports and should be declared as such. Putting it together, here’s GetStdHandle:

__declspec(dllimport)
HANDLE __stdcall GetStdHandle(DWORD);

This works with both Mingw-w64 and MSVC. MSVC requires __stdcall between the return type and function name, so don’t get clever about it. If you only care about GCC then you can declare both at once using attributes:

HANDLE GetStdHandle(DWORD)
    __attribute__((dllimport,stdcall));

I like to hide all this behind a macro, with a “table” of all my imports listed just below:

#define W32(r) __declspec(dllimport) r __stdcall
W32(HANDLE) GetStdHandle(DWORD);
W32(BOOL)   WriteFile(HANDLE, const void *, DWORD, DWORD *, void *);

In WriteFile you may have noticed I’m taking shortcuts. The “official” definition uses an ugly pointer typedef, LPCVOID, instead of pointer syntax, but I skipped that type definition. I also replaced the last argument, an OVERLAPPED pointer, with a generic pointer. I only need to pass null. I can keep sanding it down to something more ergonomic:

W32(int)    WriteFile(void *, void *, int, int *, void *);

That’s how I typically write these prototypes. I dropped the const because it doesn’t help me. I used signed sizes because I like them better and it’s what I’m usually holding at the call site. But doesn’t changing the signedness potentially break compatibility? It makes no difference to any practical ABI: It’s passed the same way. In general, signedness is a matter for operators, and only some of them — mainly comparisons (<, >, etc.) and division. It’s a similar story for pointers starting with the 32-bit era, so I can choose whatever pointer types are convenient.

In general, I can do anything I want so long as I know my compiler will produce an appropriate function call. These are not standard functions, like printf or memcpy, which are implemented in part by the compiler itself, but foreign functions. It’s no different than teaching an FFI how to make a call. This is also, in essence, how OpenGL and Vulkan work, with applications defining the API for themselves.

Considering all this, my new hello world:

#define W32(r) __declspec(dllimport) r __stdcall
W32(void *) GetStdHandle(int);
W32(int)    WriteFile(void *, void *, int, int *, void *);

int mainCRTStartup(void)
{
    void *stdout = GetStdHandle(-10 - 1);
    char message[] = "Hello, world!\n";
    int len;
    return !WriteFile(stdout, message, sizeof(message)-1, &len, 0);
}

You know, there’s a kind of beauty to a program that requires no external definitions. It builds quickly and produces a binary bit-for-bit identical to the original:

$ time cc -nostartfiles -o hello.exe main.c
real    0m 0.04s
user    0m 0.00s
sys     0m 0.00s

$ time cl /nologo hello.c /link /subsystem:console kernel32.lib
hello.c
real    0m 0.03s
user    0m 0.00s
sys     0m 0.00s

I’ve also been using this to patch over API rough edges. For example, WSARecvFrom takes WSAOVERLAPPED, but GetQueuedCompletionStatus takes OVERLAPPED. These types are explicitly compatible, and only defined separately for annoying technical reasons. I must use the same overlapped object with both APIs at once, meaning I would normally need ugly pointer casts on my Winsock calls, or vice versa with I/O completion ports. But because I’m writing all these definitions myself, I can define a common overlapped structure for both!

Perhaps you’re worried that this would be too fragile. Well, as a legacy software aficionado, I enjoy building and running my programs on old platforms. So far these programs still work properly going back 30 years to Windows NT 3.5 and Visual C++ 4.2. When I do hit a snag, it’s always been a bug (now long fixed) in the old operating system, not in my programs or these prototypes. So, in effect, this technique has worked well for the past 30 years!

Writing out these definitions is a bit of a chore, but after paying that price I’ve been quite happy with the results. I will likely continue doing it in the future, at least for non-graphical applications.

In case you’re not familiar with it, a typical WaitGroup use case in Go:

var wg sync.WaitGroup
for _, task := range tasks {
    wg.Add(1)
    go func(t Task) {
        // ... do task ...
        wg.Done()
    }(task)
}
wg.Wait()

I zero-initialize the WaitGroup, the main goroutine increments the counter before starting each task goroutine, each goroutine decrements the counter when done, and the main goroutine waits until the counter reaches zero. My goal is to build the same mechanism in C:

void workfunc(task t, int *wg)
{
    // ... do task ...
    waitgroup_done(wg);
}

int main(void)
{
    // ...
    int wg = 0;
    for (int i = 0; i < ntasks; i++) {
        waitgroup_add(&wg, 1);
        go(workfunc, tasks[i], &wg);
    }
    waitgroup_wait(&wg);
    // ...
}

When it’s done, the WaitGroup is back to zero, and no cleanup is required.

I’m going to take it a little further than that: Since its meaning and contents are explicit, you may initialize a WaitGroup to any non-negative task count! In other words, waitgroup_add is optional if the total number of tasks is known up front.

    int wg = ntasks;
    for (int i = 0; i < ntasks; i++) {
        go(workfunc, tasks[i], &wg);
    }
    waitgroup_wait(&wg);

A sneak peek at the full source: waitgroup.c

The four elements (of synchronization)

To build this WaitGroup, we’re going to need four primitives from the host platform, each operating on an int. The first two are atomic operations, and the second two interact with the system scheduler. To port the WaitGroup to a platform you need only implement these four functions, typically as one-liners.

static int  load(int *);           // atomic load
static int  addfetch(int *, int);  // atomic add-then-fetch
static void wait(int *, int);      // wait on change at address
static void wake(int *);           // wake all waiters by address

The first two should be self-explanatory. The wait function waits for the pointed-at integer to change its value, and the second argument is its expected current value. The scheduler will double-check the integer before putting the thread to sleep in case it changes at the last moment — in other words, an atomic check-then-maybe-sleep. The wake function is the other half. After changing the integer, a thread uses it to wake all threads waiting for the pointed-at integer to change. Together, this mechanism is known as a futex.

I’m going to simplify the WaitGroup semantics a bit in order to make my implementation even simpler. Go’s WaitGroup allows adding negatives, and the Add method essentially does double-duty. My version forbids adding negatives. That means the “add” operation is just an atomic increment:

void waitgroup_add(int *wg, int delta)
{
    addfetch(wg, delta);
}

Since it cannot bring the counter to zero, there’s nothing else to do. The “done” operation can decrement to zero:

void waitgroup_done(int *wg)
{
    if (!addfetch(wg, -1)) {
        wake(wg);
    }
}

If the atomic decrement brought the count to zero, we finished the last task, so we need to wake the waiters. We don’t know if anyone is actually waiting, but that’s fine. Some futex use cases will avoid making the relatively expensive system call if nobody’s waiting — i.e. don’t waste time on a system call for each unlock of an uncontended mutex — but in the typical WaitGroup case we expect a waiter when the count finally goes to zero. That’s the common case.

The most complicated of the three is waiting:

void waitgroup_wait(int *wg)
{
    for (;;) {
        int c = load(wg);
        if (!c) {
            break;
        }
        wait(wg, c);
    }
}

First check if the count is already zero and return if it is. Otherwise use the futex to wait for it to change. Unfortunately that’s not exactly the semantics we want, which would be to wait for a certain target. This doesn’t break the wait, but it’s a potential source of inefficiency. If a thread finishes a task between our load and wait, we don’t go to sleep, and instead try again. However, in practice, I ran thousands of threads through this thing concurrently and I couldn’t observe such a “miss.” As far as I can tell, it’s so rare it doesn’t matter.

If this was a concern, the WaitGroup could instead be a pair of integers: the counter and a “latch” that is either 0 or 1. Waiters wait on the latch, and the latch is modified (atomically) when the counter transitions to or from zero. That gives waiters a stable value on which to wait, proxying the counter. However, since this doesn’t seem to matter in practice, I prefer the elegance and simplicity of the single-integer WaitGroup.

Four elements: Linux

With the WaitGroup done at a high level, we now need the per-platform parts. Both GCC and Clang support GNU-style atomics, so I’ll just assume these are available on Linux without worrying about the compiler. The first two functions wrap these built-ins:

static int load(int *p)
{
    return __atomic_load_n(p, __ATOMIC_SEQ_CST);
}

static int addfetch(int *p, int addend)
{
    return __atomic_add_fetch(p, addend, __ATOMIC_SEQ_CST);
}

For wait and wake we need the futex(2) system call. In an attempt to discourage its direct use, glibc doesn’t wrap this system call in a function, so we must make the system call ourselves.

static void wait(int *p, int current)
{
    syscall(SYS_futex, p, FUTEX_WAIT, current, 0, 0, 0);
}

static void wake(int *p)
{
    syscall(SYS_futex, p, FUTEX_WAKE, INT_MAX, 0, 0, 0);
}

The INT_MAX means “wake as many as possible.” The other common value is 1 for waking a single waiter. Also, these system calls can’t meaningfully fail, so there’s no need to check the return value. If wait wakes up early (e.g. EINTR), it’s going to check the counter again anyway. In fact, if your kernel is more than 20 years old, predating futexes, and returns ENOSYS (“Function not implemented”), it will still work correctly, though it will be incredibly inefficient.

Four elements: Windows

Windows didn’t support futexes until Windows 8 in 2012, and were still supporting Windows without it into 2020, so they’re still relatively “new” for this platform. Nonetheless, they’re now mature enough that we can count on them being available.

I’d like to support both GCC-ish (via Mingw-w64) and MSVC-ish compilers. Mingw-w64 provides a compatible intrin.h, so I can stick to MSVC-style atomics and cover both at once. On the other hand, MSVC doesn’t define atomics for int (or even int32_t), strictly long, so I have to sneak in a little cast. (Recall: sizeof(long) == sizeof(int) on every version of Windows supporting futexes.) The other option is to typedef the WaitGroup so that it’s int on Linux (for the futex) and long on Windows (for atomics).

static int load(int *p)
{
    return _InterlockedOr((long *)p, 0);
}

static int addfetch(int *p, int addend)
{
    return addend + _InterlockedExchangeAdd((long *)p, addend);
}

The official, sanctioned futex functions are WaitOnAddress and WakeByAddressAll. They used to be in kernel32.dll, but as of this writing they live in API-MS-Win-Core-Synch-l1-2-0.dll, linked via -lsynchronization. Gross. Since I can’t stomach this, I instead call the low-level RTL functions where it’s actually implemented: RtlWaitOnAddress and RtlWakeAddressAll. These live in the nice neighborhood of ntdll.dll. They’re undocumented as far as I can tell, but thankfully Wine comes to the rescue, providing both documentation and several different implementations. Reading through it is educational, and hints at ways to construct futexes on systems lacking them.

These functions aren’t declared in any headers, so I have to do it myself. On the plus side, so far I haven’t paid the substantial compile-time costs of including windows.h, and so I can continue avoiding it. These functions are listed in the ntdll.dll import library, so I don’t need to invent the import library entries.

__declspec(dllimport)
long __stdcall RtlWaitOnAddress(void *, void *, size_t, void *);
__declspec(dllimport)
long __stdcall RtlWakeAddressAll(void *);

Rather conveniently, the semantics perfectly line up with Linux futexes!

static void wait(int *p, int current)
{
    RtlWaitOnAddress(p, &current, sizeof(*p), 0);
}

static void wake(int *p)
{
    RtlWakeAddressAll(p);
}

Like with Linux, there’s no meaningful failure, so the return values don’t matter.

That’s the whole implementation. Considering just a single platform, a flexible, lightweight, and easy-to-use synchronization facility in ~50 lines of relatively simple code is a pretty good deal if you ask me!

Unix-like systems pass the argv array directly from parent to child. On Linux it’s literally copied onto the child’s stack just above the stack pointer on entry. The runtime just bumps the stack pointer address a few bytes and calls it argv. Here’s a minimalist x86-64 Linux runtime in just 6 instructions (22 bytes):

_start: mov   edi, [rsp]     ; argc
        lea   rsi, [rsp+8]   ; argv
        call  main
        mov   edi, eax
        mov   eax, 60        ; SYS_exit
        syscall

It’s 5 instructions (20 bytes) on ARM64:

_start: ldr  w0, [sp]        ; argc
        add  x1, sp, 8       ; argv
        bl   main
        mov  w8, 93          ; SYS_exit
        svc  0

On Windows, argv is passed in serialized form as a string. That’s how MS-DOS did it (via the Program Segment Prefix), because that’s how CP/M did it. It made more sense when processes were mostly launched directly by humans: The string was literally typed by a human operator, and somebody has to parse it after all. Today, processes are nearly always launched by other programs, but despite this, must still serialize the argument array into a string as though a human had typed it out.

Windows itself provides an operating system routine for parsing command line strings: CommandLineToArgvW. Fetch the command line string with GetCommandLineW, pass it to this function, and you have your argc and argv. Plus maybe LocalFree to clean up. It’s only available in “wide” form, so if you want to work in UTF-8 you’ll also need WideCharToMultiByte. It’s around 20 lines of C rather than 6 lines of assembly, but it’s not too bad.

My GetCommandLineW

GetCommandLineW returns a pointer into static storage, which is why it doesn’t need to be freed. More specifically, it comes from the Process Environment Block. This got me thinking: Could I locate this address myself without the API call? First I needed to find the PEB. After some research I found a PEB pointer in the Thread Information Block, itself found via the gs register (x64, fs on x86), an old 386 segment register. Buried in the PEB is a UNICODE_STRING, with the command line string address. I worked out all the offsets for both x86 and x64, and the whole thing is just three instructions:

wchar_t *cmdline_fetch(void)
{
    void *cmd = 0;
    #if __amd64
    __asm ("mov %%gs:(0x60), %0\n"
           "mov 0x20(%0), %0\n"
           "mov 0x78(%0), %0\n"
           : "=r"(cmd));
    #elif __i386
    __asm ("mov %%fs:(0x30), %0\n"
           "mov 0x10(%0), %0\n"
           "mov 0x44(%0), %0\n"
           : "=r"(cmd));
    #endif
    return cmd;
}

From Windows XP through Windows 11, this returns exactly the same address as GetCommandLineW. There’s little reason to do it this way other than to annoy Raymond Chen, but it’s still neat and maybe has some super niche use. Technically some of these offsets are undocumented and/or subject to change, except Microsoft’s own static link CRT also hardcodes all these offsets. It’s easy to find: disassemble any statically linked program, look for the gs register, and you’ll find it using these offsets, too.

If you look carefully at the UNICODE_STRING you’ll see the length is given by a USHORT in units of bytes, despite being a 16-bit wchar_t string. This is the source of Windows’ maximum command line length of 32,767 characters (including terminator).

GetCommandLineW is from kernel32.dll, but CommandLineToArgvW is a bit more off the beaten path in shell32.dll. If you wanted to avoid linking to shell32.dll for important reasons, you’d need to do the command line parsing yourself. Many runtimes, including Microsoft’s own CRTs, don’t call CommandLineToArgvW and instead do their own parsing. It’s messier than I expected, and when I started digging into it I wasn’t expecting it to involve a few days of research.

The GetCommandLineW has a rough explanation: split arguments on whitespace (not defined), quoting is involved, and there’s something about counting backslashes, but only if they stop on a quote. It’s not quite enough to implement your own, and if you test against it, it’s quickly apparent that this documentation is at best incomplete. It links to a deprecated page about parsing C++ command line arguments with a few more details. Unfortunately the algorithm described on this page is not the algorithm used by GetCommandLineW, nor is it used by any runtime I could find. It even varies between Microsoft’s own CRTs. There is no canonical command line parsing result, not even a de facto standard.

I eventually came across David Deley’s How Command Line Parameters Are Parsed, which is the closest there is to an authoritative document on the matter (also). Unfortunately it focuses on runtimes rather than CommandLineToArgvW, and so some of those details aren’t captured. In particular, the first argument (i.e. argv[0]) follows entirely different rules, which really confused me for while. The Wine documentation was helpful particularly for CommandLineToArgvW. As far as I can tell, they’ve re-implemented it perfectly, matching it bug-for-bug as they do.

My CommandLineToArgvW

Before finding any of this, I started building my own implementation, which I now believe matches CommandLineToArgvW. These other documents helped me figure out what I was missing. In my usual fashion, it’s a little state machine: cmdline.c. The interface:

int cmdline_to_argv8(const wchar_t *cmdline, char **argv);

Unlike the others, mine encodes straight into WTF-8, a superset of UTF-8 that can round-trip ill-formed UTF-16. The WTF-8 part is negative lines of code: invisible since it involves not reacting to ill-formed input. If you use the new-ish UTF-8 manifest Win32 feature then your program cannot handle command line strings with ill-formed UTF-16, a problem solved by WTF-8.

As documented, that argv must be a particular size — a pointer-aligned, 224kB (x64) or 160kB (x86) buffer — which covers the absolute worst case. That’s not too bad when the command line is limited to 32,766 UTF-16 characters. The worst case argument is a single long sequence of 3-byte UTF-8. 4-byte UTF-8 requires 2 UTF-16 code points, so there would only be half as many. The worst case argc is 16,383 (plus one more argv slot for the null pointer terminator), which is one argument for each pair of command line characters. The second half (roughly) of the argv is actually used as a char buffer for the arguments, so it’s all a single, fixed allocation. There is no error case since it cannot fail.

int mainCRTStartup(void)
{
    static char *argv[CMDLINE_ARGV_MAX];
    int argc = cmdline_to_argv8(cmdline_fetch(), argv);
    return main(argc, argv);
}

Also: Note the FUZZ option in my source. It has been pretty thoroughly fuzz tested. It didn’t find anything, but it does make me more confident in the result.

I also peeked at some language runtimes to see how others handle it. Just as expected, Mingw-w64 has the behavior of an old (pre-2008) Microsoft CRT. Also expected, CPython implicitly does whatever the underlying C runtime does, so its exact command line behavior depends on which version of Visual Studio was used to build the Python binary. OpenJDK pragmatically calls CommandLineToArgvW. Go (gc) does its own parsing, with behavior mixed between CommandLineToArgvW and some of Microsoft’s CRTs, but not quite matching either.

Building a command line string

I’ve always been boggled as to why there’s no complementary inverse to CommandLineToArgvW. When spawning processes with arbitrary arguments, everyone is left to implement the inverse of this under-specified and non-trivial command line format to serialize an argv. Hopefully the receiver parses it compatibly! There’s no falling back on a system routine to help out. This has lead to a lot of repeated effort: it’s not limited to high level runtimes, but almost any extensible application (itself a kind of runtime). Fortunately serializing is not quite as complex as parsing since many of the edge cases simply don’t come up if done in a straightforward way.

Naturally, I also wrote my own implementation (same source):

int cmdline_from_argv8(wchar_t *cmdline, int len, char **argv);

Like before, it accepts a WTF-8 argv, meaning it can correctly pass through ill-formed UTF-16 arguments. It returns the actual command line length. Since this one can fail when argv is too large, it returns zero for an error.

char *argv[] = {"python.exe", "-c", code, 0};
wchar_t cmd[CMDLINE_CMD_MAX];
if (!cmdline_from_argv8(cmd, CMDLINE_CMD_MAX, argv)) {
    return "argv too large";
}
if (!CreateProcessW(0, cmd, /*...*/)) {
    return "CreateProcessW failed";
}

How do others handle this?

• The aged Emacs implementation is written in C rather than Lisp, steeped in history with vestigial wrong turns. Emacs still only calls the “narrow” CreateProcessA despite having every affordance to do otherwise, and uses the wrong encoding at that. A personal source of headaches.

• CPython uses Python rather than C via subprocess.list2cmdline. While undocumented, it’s accessible on any platform and easy to test against various inputs. Try it out!

• Go (gc) is just as delightfully boring I’d expect.

• OpenJDK optimistically optimizes for command line strings under 80 bytes, and like Emacs, displays the weathering of long use.

I don’t plan to write a language implementation anytime soon, where this might be needed, but it’s nice to know I’ve already solved this problem for myself!

There are multiple C implementations, so how could they all be bad, even the early ones? Microsoft’s C runtime has defined how the standard library should work on the platform, and everyone else followed along for the sake of compatibility. I’m excluding Cygwin and its major fork, MSYS2, despite not inheriting any of these flaws. They change so much that they’re effectively whole new platforms, not truly “native” to Windows.

In practice, C++ standard libraries are implemented on top of a C standard library, which is why C++ shares the same problems. CPython dodges these issues: Though written in C, on Windows it bypasses the broken C standard library and directly calls the proprietary interfaces. Other language implementations, such “gc” Go, simply aren’t built on C at all, and instead do things correctly in the first place — the behaviors the C runtimes should have had all along.

If you’re just working on one large project, bypassing the C runtime isn’t such a big deal, and you’re likely already doing so to access important platform functionality. You don’t really even need a C runtime. However, if you write many small programs, as I do, writing the same special Windows support for each one ends up being most of the work, and honestly makes properly supporting Windows not worth the trouble. I end up just accepting the broken defaults most of the time.

Before diving into the details, if you’re looking for a quick-and-easy solution for the Mingw-w64 toolchain, including w64devkit, which magically makes your C and C++ console programs behave well on Windows, I’ve put together a “library” named libwinsane. It solves all problems discussed in this article, except for one. No source changes required, simply link it into your program.

What exactly is broken?

The Windows API comes in two flavors: narrow with an “A” (“ANSI”) suffix, and wide (Unicode, UTF-16) with a “W” suffix. The former is the legacy API, where an active code page maps 256 bytes onto (up to) 256 specific characters. On typical machines configured for European languages, this means code page 1252. Roughly speaking, Windows internally uses UTF-16, and calls through the narrow interface use the active code page to translate the narrow strings to wide strings. The result is that calls through the narrow API have limited access to the system.

The UTF-8 encoding was invented in 1992 and standardized by January 1993. UTF-8 was adopted by the unix world over the following years due to its backwards-compatibility with its existing interfaces. Programs could read and write Unicode data, access Unicode paths, pass Unicode arguments, and get and set Unicode environment variables without needing to change anything. Today UTF-8 has become the dominant text encoding format in the world, in large part due to the world wide web.

In July 1993, Microsoft introduced the wide Windows API with the release of Windows NT 3.1, placing all their bets on UCS-2 (later UTF-16) rather than UTF-8. This turned out to be a mistake, since UTF-16 is inferior to UTF-8 in practically every way, though admittedly some problems weren’t so obvious at the time.

The major problem: The C and C++ standard libraries only hook up to the narrow Windows interfaces. The standard library, and therefore typical portable software on Windows, cannot handle anything but ASCII. The effective result is that these programs:

• Cannot accept non-ASCII arguments
• Cannot get/set non-ASCII environment variables
• Cannot access non-ASCII paths
• Cannot read and write non-ASCII on a console

Doing any of these requires calling proprietary functions, treating Windows as a special target. It’s part of what makes correctly porting software to Windows so painful.

The sensible solution would have been for the C runtime to speak UTF-8 and connect to the wide API. Alternatively, the narrow API could have been changed over to UTF-8, phasing out the old code page concept. In theory this is what the UTF-8 “code page” is about, though it doesn’t always work. There would have been compatibility problems with abruptly making such a change, but until very recently, this wasn’t even an option. Why couldn’t there be a switch I could flip to get sane behavior that works like every other platform?

How to mostly fix Unicode support

In 2019, Microsoft introduced a feature to allow programs to request UTF-8 as their active code page on start, along with supporting UTF-8 on more narrow API functions. This is like the magic switch I wanted, except that it involves embedding some ugly XML into your binary in a particular way. At least it’s now an option.

For Mingw-w64, that means writing a resource file like so:

#include <winuser.h>
CREATEPROCESS_MANIFEST_RESOURCE_ID RT_MANIFEST "utf8.xml"

Compiling it with windres:

$ windres -o manifest.o manifest.rc

Then linking that into your program. Amazingly it mostly works! Programs can access Unicode arguments, Unicode environment variables, and Unicode paths, including with fopen, just as it’s worked on other platforms for decades. Since the active code page is set at load time, it happens before argv is constructed (from GetCommandLineA), which is why that works out.

Alternatively you could create a “side-by-side assembly” placing that XML in a file with the same name as your EXE but with .manifest suffix (after the .exe suffix), then placing that next to your EXE. Just be mindful that there’s a “side-by-side” cache (WinSxS), and so it might not immediately pick up your changes.

What doesn’t work is console input and output since the console is external to the process, and so isn’t covered by the process’s active code page. It must be configured separately using a proprietary call:

SetConsoleOutputCP(CP_UTF8);

Annoying, but at least it’s not that painful. This only covers output, though, meaning programs can only print UTF-8. Unfortunately UTF-8 input still doesn’t work, and setting the input code page doesn’t do anything despite reporting success:

SetConsoleCP(CP_UTF8);  // doesn't work

If you care about reading interactive Unicode input, you’re stuck bypassing the C runtime since it’s still broken.

Text stream translation

Another long-standing issue is that C and C++ on Windows has distinct “text” and “binary” streams, which it inherited from DOS. Mainly this means automatic newline conversion between CRLF and LF. The C standard explicitly allows for this, though unix-like platforms have never actually distinguished between text and binary streams.

The standard also specifies that standard input, output, and error are all open as text streams, and there’s no portable method to change the stream mode to binary — a serious deficiency with the standard. On unix-likes this doesn’t matter, but on Windows it means programs can’t read or write binary data on standard streams without calling a non-standard function. It also means reading and writing standard streams is slow, frequently a bottleneck unless I route around it.

Personally, I like writing binary data to standard output, including video, and sometimes binary filters that also read binary input. I do it so often that in probably half my C programs I have this snippet in main just so they work correctly on Windows:

    #ifdef _WIN32
    int _setmode(int, int);
    _setmode(0, 0x8000);
    _setmode(1, 0x8000);
    #endif

That incantation sets standard input and output in the C runtime to binary mode without the need to include a header, making it compact, simple, and self-contained.

This built-in newline translation, along with the Windows standard text editor, Notepad, lagging decades behind, meant that many other programs, including Git, grew their own, annoying, newline conversion misfeatures that cause other problems.

libwinsane

I introduced libwinsane at the beginning of the article, which fixes all this simply by being linked into a program. It includes the magic XML manifest .rsrc section, configures the console for UTF-8 output, and sets standard streams to binary before main (via a GCC constructor). I called it a “library”, but it’s actually a single object file. It can’t be a static library since it must be linked into the program despite not actually being referenced by the program.

So normally this program:

#include <stdio.h>
#include <string.h>

int main(int argc, char **argv)
{
    char *arg = argv[argc-1];
    size_t len = strlen(arg);
    printf("%zu %s\n", len, arg);
}

Compiled and run:

C:\>cc -o example example.c
C:\>example π
1 p

As usual, the Unicode argument is silently mangled into one byte. Linked with libwinsane, it just works like everywhere else:

C:\>gcc -o example example.c libwinsane.o
C:\>example π
2 π

If you’re maintaining a substantial program, you probably want to copy and integrate the necessary parts of libwinsane into your project and build, rather than always link against this loose object file. This is more for convenience and for succinctly capturing the concept. You may even want to enable ANSI escape processing in your version.

Update December 2024: Pavel Galkin demonstrates how libwinsane.o changes the console state, which affects all processes associated with the terminal. This is mostly unavoidable, and it’s one reason I’ve since concluded that UTF-8 manifests are a poor solution. Better to solve the problem using a platform layer.

My small development distribution for Windows, w64devkit, is my own little way of pushing back against this trend where it affects me most. Following in the footsteps of projects like Handmade Hero and Making a Video Game from Scratch, this is my guide to no-nonsense software development using my development kit. It’s an overview of the tooling and development workflow, and I’ve tried not to assume too much knowledge of the reader. Being a guide rather than manual, it is incomplete on its own, and I link to substantial external resources to fill in the gaps. The guide is capped with a small game I wrote entirely using my development kit, serving as a demonstration of what sorts of things are not only possible, but quite reasonably attainable.

Game repository: https://github.com/skeeto/asteroids-demo
Guide to source: Understanding Asteroids

Initial setup

Of course you cannot use the development kit if you don’t have it yet. Go to the releases section and download the latest release. It will be a .zip file named w64devkit-x.y.z.zip where x.y.z is the version.

You will need to unzip the development kit before using it. Windows has built-in support for .zip files, so you can either right-click to access “Extract All…” or navigate into it as a folder then drag-and-drop the w64devkit directory somewhere outside the .zip file. It doesn’t care where it’s unzipped (aka it’s “portable”), so put it where ever is convenient: your desktop, user profile directory, a thumb drive, etc. You can move it later if you change your mind just so long as you’re not actively running it. If you decide you don’t need it anymore then delete it.

Entering the development environment

There is a w64devkit.exe in the unzipped w64devkit directory. This is the easiest way to enter the development environment, and will not require system configuration changes. This program puts the kit’s programs in the PATH environment variable then runs a Bourne shell — the standard unix shell. Aside from the text editor, this is the primary interface for developing software. In time you may even extend this environment with your own tools.

If you want an additional “terminal” window, run w64devkit.exe again. If you use it a lot, you may want to create a shortcut and even pin it to your task bar.

Whether on Windows or unix-like systems, when you type a command into the system shell it uses the PATH environment variable to locate the actual program to run for that command. In practice, the PATH variable is a concatenation of multiple directories, and the shell searches these directories in order. On unix-like systems, PATH elements are separated by colons. However, Windows uses colons to delimit drive letters, so its PATH elements are separated by semicolons.

# Prepending to PATH on unix
PATH="$HOME/bin:$PATH"

# Prepending to PATH on Windows (w64devkit)
PATH="$HOME/bin;$PATH"

For more advanced users: Rather than use w64devkit.exe, you could “Edit environment variables for your account” and manually add w64devkit’s bin directory to your PATH, making the tools generally available everywhere on your system. If you’ve gone this route, you can start a Bourne shell at any time with sh -l. (The -l option requests a login shell.)

Also borrowed from the unix world is the concept of a home directory, specified by the HOME environment variable. By default this will be your user profile directory, typically C:/Users/$USER. Login shells always start in the home directory. This directory is often indicated by tilde (~), and many programs automatically expand a leading tilde to the home directory.

Shell basics

The shell is a command interpreter. It’s named such because it was originally a shell around the operating system kernel — the user interface to the kernel. Your system’s graphical interface — Windows Explorer, or Explorer.exe — is really just a kind of shell, too. That shell is oriented around the mouse and graphics. This is fine for some tasks, but a keyboard-oriented command shell is far better suited for development tasks. It’s more efficient, but more importantly its features are composable: Complex operations and processes can be constructed from simple, easy-to-understand tools. Embrace it!

In the shell you can navigate between directories with cd, make directories with mkdir, remove files with rm, regular expression text searches with grep, etc. Run busybox to see a listing of the available standard commands. Unfortunately there are no manual pages, but you can access basic usage information for any command with busybox CMD --help.

Windows’ standard command shell is cmd.exe. Unfortunately this shell is terrible and exists mostly for legacy compatibility. The intended replacement is PowerShell for users who regularly use a shell. However, PowerShell is fundamentally broken, does virtually everything incorrectly, and manages to be even worse than cmd.exe. Besides, sticking to POSIX shell conventions significantly improves build portability, and unix tool knowledge is transferable to basically every other operating system.

Unix’s standard shell was the Bourne shell, sh. The shells in use today are Bourne shell clones with a superset of its features. The most popular interactive shells are Bash and Zsh. On Linux, dash (Debian Almquist shell) has become popular for non-interactive use (scripting). The shell included with w64devkit is the BusyBox fork of the Almquist shell (ash), closely related to dash. The Almquist shell has almost no non-interactive features beyond the standard Bourne shell, and so as far as scripts are concerned can be regarded as a plain Bourne shell clone. That’s why I typically refer to it by the name sh.

However, BusyBox’s Almquist shell has interactive features much like Bash, and Bash users should be quite comfortable. It’s not just tab-completion but a slew of Emacs-like keybindings:

• Ctrl-r: search backwards in history
• Ctrl-s: search forwards in history
• Ctrl-p: previous command (Up)
• Ctrl-n: next command (Down)
• Ctrl-a: cursor to the beginning of line (Home)
• Ctrl-e: cursor to the end of line (End)
• Alt-b: cursor back one word
• Alt-f: cursor forward one word
• Ctrl-l: clear the screen
• Alt-d: delete word after the cursor
• Ctrl-w: delete the word before the cursor
• Ctrl-k: delete to the end of the line
• Ctrl-u: delete to the beginning of the line
• Ctrl-f: cursor forward one character (Right)
• Ctrl-b: cursor backward one character (Left)
• Ctrl-d: delete character under the cursor (Delete)
• Ctrl-h: delete character before the cursor (Backspace)

Take special note of Ctrl-r, which is the most important and powerful shortcut of the bunch. Frequent use is a good habit. Don’t mash the up arrow to search through the command history.

Special note for Cygwin and MSYS2 users: the shell is aware of Windows paths and does not present a virtual unix file system scheme. This has important consequences for scripting, both good and bad. The shell even supports backslash as a directory separator, though you should of course prefer forward slashes.

Shell customization

Login shells (-l) evaluate the contents of ~/.profile on startup. This is your chance to customize the shell configuration, such as setting environment variables or defining aliases and functions. For instance, if you wanted the prompt to show the working directory in green you’d set PS1 in your ~/.profile:

PS1="$(printf '\x1b[33;1m\\w\x1b[0m$ ')"

If you find yourself using the same command sequences or set of options again and again, you might consider putting those commands into a script, and then installing that script somewhere on your PATH so that you can run it as a new command. First make a directory to hold your scripts, say in ~/bin:

mkdir ~/bin

In ~/.profile prepend it to your PATH:

PATH="$HOME/bin;$PATH"

If you don’t want to start a fresh shell to try it out, then load the new configuration in your current shell:

source ~/.profile

Suppose you keep getting the tar switches mixed up and you’d like to just have an untar command that does the right thing. Create a file named untar or untar.sh in ~/bin with these contents:

#!/bin/sh
set -e
tar -xaf "$@"

Now a command like untar something.tar.gz will extract the archive contents.

To learn more about Bourne shell scripting, the POSIX shell command language specification is a good reference. All of the features listed in that document are available to your shell scripts.

Text editing

The development kit includes the powerful and popular text editor Vim. It takes effort to learn, but is well worth the investment. It’s packed with features, but since you only need a small number of them on a regular basis it’s not as daunting as it might appear. Using Vim effectively, you will write and edit text so much more quickly than before. That includes not just code, but prose: READMEs, documentation, etc.

(The catch: Non-modal editing will forever feel frustratingly inefficient. That’s not because you will become unpracticed at it, or even have trouble code switching between input styles, but because you’ll now be aware how bad it is. Ignorance is bliss.)

Vim includes its own tutorial for absolute beginners which you can access with the vimtutor command. It will run in the console window and guide you through the basics in about half an hour. Do not be afraid to return to the tutorial at any time since this is the stuff you need to know by heart.

When it comes time to actually use Vim to write code, you can continue writing code via the terminal interface (vim), or you can run the graphical interface (gvim). The latter is recommended since it has some nice quality-of-life features, but it’s not strictly necessary. When starting the GUI, put an ampersand (&) on the command so that it runs in the background. For instance this brings up the editor with two files open but leaves the shell running in the foreground so you can continue using it while you edit:

gvim main.c Makefile &

Vim’s defaults are good but imperfect. Before getting started with actually editing code you should establish at least the following minimal configuration in ~/_vimrc. (To understand these better, use :help to jump the built-in documentation.)

set hidden encoding=utf-8 shellslash
filetype plugin indent on
syntax on

The graphical interface defaults to a white background. Many people prefer “dark mode” when editing code, so inverting this is simply a matter of choosing a dark color scheme. Vim comes with a handful of color schemes, around half of which have dark backgrounds. Use :colorscheme to change it, and put it in your ~/_vimrc to persist it.

colorscheme slate

The default graphical interface includes a menu bar and tool bar. There are better ways to accomplish all these operations, none of which require touching the mouse, so consider removing all that junk:

set guioptions=ac

Finally, since the development kit is oriented around C and C++, here’s my own entire Vim configuration for C which makes it obey my own style:

set cinoptions+=t0,l1,:0 cinkeys-=0#

Once you’re comfortable with the basics, the best next step is to read Practical Vim: Edit Text at the Speed of Thought by Drew Neil. It’s an opinionated guide to Vim that instills good habits. If you want something cost-free to whet your appetite, check out Seven habits of effective text editing.

Writing an application

We’ve established a shell and text editor. Next is the development workflow for writing an actual application. Ultimately you will invoke a compiler from within Vim, which will parse compiler messages and take you directly to the parts of your source code that need attention. Before we get that far, let’s start with the basics.

The classic example is the “hello world” program, which we’ll suppose is in a file called hello.c:

#include <stdio.h>

int main(void)
{
    puts("Hello, world!");
}

While this development kit provides a version of the GNU compiler, gcc, this guide mostly speaks of it in terms of the generic unix C compiler name, cc. Unix-like systems install cc as an alias for the system’s default C compiler, and w64devkit is no exception.

cc -o hello.exe hello.c

This command creates hello.exe from hello.c. Since this is not (yet?) on your PATH, you must invoke it via a path name (i.e. the command must include a slash), since otherwise the shell will search for it via the PATH variable. Typically this means putting ./ in front of the program name, meaning “run the program in the current directory”. As a convenience you do not need to include the .exe extension:

./hello

Unlike the untar shell script from before, this hello.exe is entirely independent of w64devkit. You can share it with anyone running Windows and they’ll be able to execute it. There’s a little bit of runtime embedded in the executable, but the bulk of the runtime is in the operating system itself. I want to highlight this point because most programming languages don’t work like this, or at least doing so is unnatural with lots of compromises. The users of your software do not need to install a runtime or other supporting software. They just run the executable you give them!

That executable is probably pretty small, less than 50kB — basically a miracle by today’s standards. Sure, it’s hardly doing anything right now, but you can add a whole lot more functionality without that executable getting much bigger. In fact, it’s entirely unoptimized right now and could be even smaller. Passing the -Os flag tells the compiler to optimize for size and -s flag tells the linker to strip out unneeded information.

cc -Os -s -o hello.exe hello.c

That cuts the program down to around a third of its previous size. If necessary you can still do even better than this, but that’s outside the scope of this guide.

So far the program could still be valid enough to compile but contain obvious mistakes. The compiler can warn about many of these mistakes, and so it’s always worth enabling these warnings. This requires two flags: -Wall (“all” warnings) and -Wextra (extra warnings).

cc -Wall -Wextra -o hello.exe hello.c

When you’re working on a program, you often don’t want optimization enabled since it makes it more difficult to debug. However, some warnings aren’t fired unless optimization is enabled. Fortunately there’s an optimization level to resolve this, -Og (optimize for debugging). Combine this with -g3 to embed debug information in the program. This will be handy later.

cc -Wall -Wextra -Og -g3 -o hello.exe hello.c

These are the compiler flags you typically want to enable while developing your software. When you distribute it, you’d use either -Os -s (optimize for size) or -O3 -s (optimize for speed).

Makefiles

I mentioned running the compiler from Vim. This isn’t done directly but via special build script called a Makefile. You invoke the make program from Vim, which invokes the compiler as above. The simplest Makefile would look like this, in a file literally named Makefile:

hello.exe: hello.c
    cc -Wall -Wextra -Og -g3 -o hello.exe hello.c

This tells make that the file named hello.exe is derived from another file called hello.c, and the tab-indented line is the recipe for doing so. Running the make command will run the compiler command if and only if hello.c is newer than hello.exe.

To run make from Vim, use the :make command inside Vim. It will not only run make but also capture its output in an internal buffer called the quickfix list. If there is any warning or error, Vim will jump to it. Use :cn (next) and :cp (prev) to move between issues and correct them, or :cc to re-display the current issue. When you’re done fixing the issues, run :make again to start the cycle over.

Try that now by changing the printed message and recompiling from within Vim. Intentionally create an error (bad syntax, too many arguments, etc.) and see what happens.

Makefiles are a powerful and conventional way to build C and C++ software. Since the development kit includes the standard set of unix utilities, it’s very easy to write portable Makefiles that work across a variety a operating systems and environments. Your software isn’t necessarily tied to Windows just because you’re using a Windows-based development environment. If you want to learn how Makefiles work and how to use them effectively, read A Tutorial on Portable Makefiles. From here on I’ll assume you’ve read that tutorial.

Ultimately I’d probably write my “hello world” Makefile like so:

.POSIX:
CC      = cc
CFLAGS  = -Wall -Wextra -Og -g3
LDFLAGS =
LDLIBS  =
EXE     = .exe

hello$(EXE): hello.c
    $(CC) $(CFLAGS) $(LDFLAGS) -o $@ hello.c $(LDLIBS)

When building a release, optimize for size or speed:

make CFLAGS=-Os LDFLAGS=-s

This is very much a Windows-first style of Makefile, but still allows it to be comfortably used on other systems. On Linux this make invocation strips away the .exe extension:

make EXE=

For a Windows-second Makefile, remove the line with EXE = .exe. This allows EXE to come from the environment. So, for instance, I already define the EXE environment variable in my w64devkit ~/.profile:

export EXE=.exe

On Linux running make does the right thing, as does running make on Windows. No special configuration required.

If my software is truly limited to Windows, I’m likely still interested in supporting cross-compilation. A common convention for GNU toolchains is a CROSS Makefile macro. For example:

.POSIX:
CROSS   =
CC      = $(CROSS)gcc
CFLAGS  = -Wall -Wextra -Og -g3
LDFLAGS =
LDLIBS  =

hello.exe: hello.c
    $(CC) $(CFLAGS) $(LDFLAGS) -o $@ hello.c $(LDLIBS)

On Windows I just run make, but on Linux I’d set CROSS appropriately.

make CROSS=x86_64-w64-mingw32-

Navigating

What happens if you’re working on a larger program and you need to jump to the definition of a function, macro, or variable? It would be tedious to use grep all the time to find definitions. The development kit includes a solid implementation of ctags for building a tags database lists the locations for various kinds of definitions, and Vim knows how to read this database. Most often you’ll want to run it recursively like so:

ctags -R

You can of course do this from Vim, too: :!ctags -R

With the cursor over an identifier, press CTRL-] to jump to a definition for that name. Use :tn and :tp to move between different definitions (e.g. when the name is overloaded). Or if you have a tag in mind rather than a name listed in the buffer, use the :tag command to jump by name. Vim maintains a tag stack and jump list for going back and forth, like the backward and forward buttons in a browser.

Debugging

I had mentioned that the -g3 option embeds extra information in the executable. This is for debuggers, and the development kit includes the GNU Debugger, gdb, to help you debug your programs. To use it, invoke GDB on your executable:

gdb hello.exe

From here you can set breakpoints and such, then run the program with start or run, then step through it line by line. See Beej’s Quick Guide to GDB for a guide. During development, always run your program through GDB, and never exit GDB. See also: Assertions should be more debugger-oriented.

Learning C and C++

So far this guide hasn’t actually assumed any C knowledge. One of the best ways to learn C is by reading the highly-regarded The C Programming Language and doing the exercises. Alternatively, cost-free options are Beej’s Guide to C Programming and Modern C (more advanced). You can use the development kit to go through any of these.

I’ve focused on C, but everything above also applies to C++. To learn C++ A Tour of C++ is a safe bet.

Demonstration

To illustrate how much you can do with nothing beyond than this 76MB development kit, here’s a taste in the form of a weekend project: an Asteroids Clone for Windows. That’s the game in the video at the top of this guide.

The development kit doesn’t include Git so you’d need to install it separately in order to clone the repository, but you could at least skip that and download a .zip snapshot of the source. It has no third-party dependencies yet it includes hardware-accelerated graphics, real-time sound mixing, and gamepad input. Building a larger and more complex game is much less about tooling and more about time and skill. That’s what I mean about w64devkit being (almost) everything you need.

As a computer engineer, my job is to use computers to solve important problems. Ideally my solutions will be efficient, and typically that means making the best use of the resources at hand. Quite often these resources are machines running Windows and, despite my misgivings about the platform, there is much to be gained by properly and effectively leveraging it.

Sometimes targeting Windows while working from another platform is sufficient, but other times I must work on the platform itself. There are various options available for C development, and I’ve finally formalized my own development kit: w64devkit.

For most users, the value is in the 78MiB .zip available in the “Releases” on GitHub. This (relatively) small package includes a state-of-the-art C and C++ compiler (latest GCC), a powerful text editor, debugger, a complete x86 assembler, and miniature unix environment. It’s “portable” in that there’s no installation. Just unzip it and start using it in place. With w64devkit, it literally takes a few seconds on any Windows to get up and running with a fully-featured, fully-equipped, first-class development environment.

The development kit is cross-compiled entirely from source using Docker, though Docker is not needed to actually use it. The repository is just a Dockerfile and some documentation. The only build dependency is Docker itself. It’s also easy to customize it for your own personal use, or to audit and build your own if, for whatever reason, you didn’t trust my distribution. This is in stark contrast to Windows builds of most open source software where the build process is typically undocumented, under-documented, obtuse, or very complicated.

From script to Docker

Publishing this is not necessarily a commitment to always keep w64devkit up to date, but this Dockerfile is derived from (and replaces) a shell script I’ve been using continuously for over two years now. In this period, every time GCC has made a release, I’ve built myself a new development kit, so I’m already in the habit.

I’ve been using Docker on and off for about 18 months now. It’s an oddball in that it’s something I learned on the job rather than my own time. I formed an early impression that still basically holds: The main purpose of Docker is to contain and isolate misbehaved software to improve its reliability. Well-behaved, well-designed software benefits little from containers.

My unusual application of Docker here is no exception. Most software builds are needlessly complicated and fragile, especially Autoconf-based builds. Ironically, the worst configure scripts I’ve dealt with come from GNU projects. They waste time on superfluous checks (“Does your compiler define size_t?”) then produce a build that doesn’t work anyway because you’re doing something slightly unusual. Worst of all, despite my best efforts, the build will be contaminated by the state of the system doing the build.

My original build script was fragile by extension. It would work on one system, but not another due to some subtle environment change — a slightly different system header that reveals a build system bug (example in GCC), or the system doesn’t have a file at a certain hard-coded absolute path that shouldn’t be hard-coded. Converting my script to a Dockerfile locks these problems in place and makes builds much more reliable and repeatable. The misbehavior is contained and isolated by Docker.

Unfortunately it’s not completely contained. In each case I use make’s -j option to parallelize the build since otherwise it would take hours. Some of the builds have subtle race conditions, and some bad luck in timing can cause a build to fail. Docker is good about picking up where it left off, so it’s just a matter of trying again.

In one case a build failed because Bison and flex were not installed even though they’re not normally needed. Some dependency isn’t expressed correctly, and unlucky ordering leads to an unused .y file having the wrong timestamp. Ugh. I’ve had this happen a lot more in Docker than out, probably because file system operations are slow inside Docker and it creates greater timing variance.

Other tools

The README explains some of my decisions, but I’ll summarize a few here:

• Git. Important and useful, so I’d love to have it. But it has a weird installation (many .zip-unfriendly symlinks) tightly-coupled with msys2, and its build system does not support cross-compilation. I’d love to see a clean, straightforward rewrite of Git in a single, appropriate implementation language. Imagine installing the latest Git with go get git-scm.com/git. (Update: libgit2 is working on it!)

• Bash. It’s a much nicer interactive shell than BusyBox-w32 ash. But the build system doesn’t support cross-compilation, and I’m not sure it supports Windows without some sort of compatibility layer anyway.

• Emacs. Another powerful editor. But the build system doesn’t support cross-compilation. It’s also way too big.

• Go. Tempting to toss it in, but Go already does this all correctly and effectively. It simply doesn’t require a specialized distribution. It’s trivial to manage a complete Go toolchain with nothing but Go itself on any system. People may say its language design comes from the 1970s, but the tooling is decades ahead of everyone else.

Alternatives

For a long, long time Cygwin filled this role for me. However, I never liked its bulky nature, the complete opposite of portable. Cygwin processes always felt second-class on Windows, particularly in that it has its own view of the file system compared to other Windows processes. They could never fully cooperate. I also don’t like that there’s no toolchain for cross-compiling with Cygwin as a target — e.g. compile Cygwin binaries from Linux. Finally it’s been essentially obsoleted by WSL which matches or surpasses it on every front.

There’s msys and msys2, which are a bit lighter. However, I’m still in an isolated, second-class environment with weird path translation issues. These tools do have important uses, and it’s the only way to compile most open source software natively on Windows. For those builds that don’t support cross-compilation, it’s the only path for producing Windows builds. It’s just not what I’m looking for when developing my own software.

Update: llvm-mingw is an eerily similar project using Docker the same way, but instead builds LLVM.

Using Docker for other builds

I also converted my GnuPG build script to a Dockerfile. Of course I don’t plan to actually use GnuPG on Windows. I just need it for passphrase2pgp, which I test against GnuPG. This tests the Windows build.

In the future I may extend this idea to a few other tools I don’t intend to include with w64devkit. If you have something in mind, you could use my Dockerfiles as a kind of starter template.

Suppose you’re writing a command line program that prompts the user for a password or passphrase, and Windows is one of the supported platforms (even very old versions). This program uses UTF-8 for its string representation, as it should, and so ideally it receives the password from the user encoded as UTF-8. On most platforms this is, for the most part, automatic. However, on Windows finding the correct answer to this problem is a maze where all the signs lead towards dead ends. I recently navigated this maze and found the way out.

I knew it was possible because my passphrase2pgp tool has been using the golang.org/x/crypto/ssh/terminal package, which gets it very nearly perfect. Though they were still fixing subtle bugs as recently as 6 months ago.

The first step is to ignore just everything you find online, because it’s either wrong or it’s solving a slightly different problem. I’ll discuss the dead ends later and focus on the solution first. Ultimately I want to implement this on Windows:

// Display prompt then read zero-terminated, UTF-8 password.
// Return password length with terminator, or zero on error.
int read_password(char *buf, int len, const char *prompt);

I chose int for the length rather than size_t because it’s a password and should not even approach INT_MAX.

The correct way

For the impatient: complete, working, ready-to-use example

On a unix-like system, the program would:

1. open(2) the special /dev/tty file for reading and writing
2. write(2) the prompt
3. tcgetattr(3) and tcsetattr(3) to disable ECHO
4. read(2) a line of input
5. Restore the old terminal attributes with tcsetattr(3)
6. close(2) the file

A great advantage of this approach is that it doesn’t depend on standard input and standard output. Either or both can be redirected elsewhere, and this function still interacts with the user’s terminal. The Windows version will have the same advantage.

Despite some tempting shortcuts that don’t work, the steps on Windows are basically the same but with different names. There are a couple subtleties and extra steps. I’ll be ignoring errors in my code snippets below, but the complete example has full error handling.

Create console handles

Instead of /dev/tty, the program opens two files: CONIN$ and CONOUT$ using CreateFileA(). Note: The “A” stands for ANSI, as opposed to “W” for wide (Unicode). This refers to the encoding of the file name, not to how the file contents are encoded. CONIN$ is opened for both reading and writing because write permissions are needed to change the console’s mode.

HANDLE hi = CreateFileA(
    "CONIN$",
    GENERIC_READ | GENERIC_WRITE,
    0,
    0,
    OPEN_EXISTING,
    0,
    0
);
HANDLE ho = CreateFileA(
    "CONOUT$",
    GENERIC_WRITE,
    0,
    0,
    OPEN_EXISTING,
    0,
    0
);

Print the prompt

To write the prompt, call WriteConsoleA() on the output handle. On its own, this assumes the prompt is plain ASCII (i.e. "password: "), not UTF-8 (i.e. "contraseña: "):

WriteConsoleA(ho, prompt, strlen(prompt), 0, 0);

If the prompt may contain UTF-8 data, perhaps because it displays a username or isn’t in English, you have two options:

• Convert the prompt to UTF-16 and call WriteConsoleW() instead.
• Use SetConsoleOutputCP() with CP_UTF8 (65001). This is a global (to the console) setting and should be restored when done.

Disable echo

Next use GetConsoleMode() and SetConsoleMode() to disable echo. The console usually has ENABLE_PROCESSED_INPUT already set, which tells the console to handle CTRL-C and such, but I set it explicitly just in case. I also set ENABLE_LINE_INPUT so that the user can use backspace and so that the entire line is delivered at once.

DWORD orig = 0;
GetConsoleMode(hi, &orig);

DWORD mode = orig;
mode |= ENABLE_PROCESSED_INPUT;
mode &= ~ENABLE_ECHO_INPUT;
SetConsoleMode(hi, mode);

There are reports that ENABLE_LINE_INPUT limits reads to 254 bytes, but I was unable to reproduce it. My full example can read huge passwords without trouble.

The old mode is saved in orig so that it can be restored later.

Read the password

Here’s where you have to pay the piper. As of the date of this article, the Windows API offers no method for reading UTF-8 input from the console. Give up on that hope now. If you use the “ANSI” functions to read input under any configuration, they will to the usual Windows thing of silently mangling your input.

So you must use the UTF-16 API, ReadConsoleW(), and then encode it yourself. Fortunately Win32 provides a UTF-8 encoder, WideCharToMultiByte(), which will even handle surrogate pairs for all those people who like putting PILE OF POO (U+1F4A9) in their passwords:

SIZE_T wbuf_len = (len - 1 + 2)*sizeof(*wbuf);
WCHAR *wbuf = HeapAlloc(GetProcessHeap(), 0, wbuf_len);
DWORD nread;
ReadConsoleW(hi, wbuf, len - 1 + 2, &nread, 0);
wbuf[nread-2] = 0;  // truncate "\r\n"
int r = WideCharToMultiByte(CP_UTF8, 0, wbuf, -1, buf, len, 0, 0);
SecureZeroMemory(wbuf, wbuf_len);
HeapFree(GetProcessHeap(), 0, wbuf);

I use SecureZeroMemory() to erase the UTF-16 version of the password before freeing the buffer. The + 2 in the allocation is for the CRLF line ending that will later be chopped off. The error handling version checks that the input did indeed end with CRLF. Otherwise it was truncated (too long).

Clean up

Finally print a newline since the user-typed one wasn’t echoed, restore the old console mode, close the console handles, and return the final encoded length:

WriteConsoleA(ho, "\n", 1, 0, 0);
SetConsoleMode(hi, orig);
CloseHandle(ho);
CloseHandle(hi);
return r;

The error checking version doesn’t check for errors from any of these functions since either they cannot fail, or there’s nothing reasonable to do in the event of an error.

Dead ends

If you look around the Win32 API you might notice SetConsoleCP(). A reasonable person might think that setting the “code page” to UTF-8 (CP_UTF8) might configure the console to encode input in UTF-8. The good news is Windows will no longer mangle your input as before. The bad news is that it will be mangled differently.

You might think you can use the CRT function _setmode() with _O_U8TEXT on the FILE * connected to the console. This does nothing useful. (The only use for _setmode() is with _O_BINARY, to disable braindead character translation on standard input and output.) The best you’ll be able to do with the CRT is the same sort of wide character read using non-standard functions, followed by conversion to UTF-8.

CredUICmdLinePromptForCredentials() promises to be both a mouthful of a function name, and a prepacked solution to this problem. It only delivers on the first. This function seems to have broken some time ago and nobody at Microsoft noticed — probably because nobody has ever used this function. I couldn’t find a working example, nor a use in any real application. When I tried to use it, I got a nonsense error code it never worked. There’s a GUI version of this function that does work, and it’s a viable alternative for certain situations, though not mine.

At my most desperate, I hoped ENABLE_VIRTUAL_TERMINAL_PROCESSING would be a magical switch. On Windows 10 it magically enables some ANSI escape sequences. The documentation in no way suggests it would work, and I confirmed by experimentation that it does not. Pity.

I spent a lot of time searching down these dead ends until finally settling with ReadConsoleW() above. I hoped it would be more automatic, but I’m glad I have at least some solution figured out.

The Windows API — a.k.a. Win32 — is notorious for being clunky, ugly, and lacking good taste. Microsoft has done a pretty commendable job with backwards compatibility, but the trade-off is that the API is filled to the brim with historical cruft. Every hasty, poor design over the decades is carried forward forever, and, in many cases, even built upon, which essentially doubles down on past mistakes. POSIX certainly has its own ugly corners, but those are the exceptions. In the Windows API, elegance is the exception.

That’s why, when I recently revisited the Fibers API, I was pleasantly surprised. It’s one of the exceptions — much cleaner than the optional, deprecated, and now obsolete POSIX equivalent. It’s not quite an apples-to-apples comparison since the POSIX version is slightly more powerful, and more complicated as a result. I’ll cover the difference in this article.

For the last part of this article, I’ll walk through an async/await framework build on top of fibers. The framework allows coroutines in C programs to await on arbitrary kernel objects.

Fiber Async/await Demo

Fibers

Windows fibers are really just stackful, symmetric coroutines. From a different point of view, they’re cooperatively scheduled threads, which is the source of the analogous name, fibers. They’re symmetric because all fibers are equal, and no fiber is the “main” fiber. If any fiber returns from its start routine, the program exits. (Older versions of Wine will crash when this happens, but it was recently fixed.) It’s equivalent to the process’ main thread returning from main(). The initial fiber is free to create a second fiber, yield to it, then the second fiber destroys the first.

For now I’m going to focus on the core set of fiber functions. There are some additional capabilities I’m going to ignore, including support for fiber local storage. The important functions are just these five:

void *CreateFiber(size_t stack_size, void (*proc)(void *), void *arg);
void  SwitchToFiber(void *fiber);
bool  ConvertFiberToThread(void);
void *ConvertThreadToFiber(void *arg);
void  DeleteFiber(void *fiber);

To emphasize its simplicity, I’ve shown them here with more standard prototypes than seen in their formal documentation. That documentation uses the clunky Windows API typedefs still burdened with its 16-bit heritage — e.g. LPVOID being a “long pointer” from the segmented memory of the 8086:

• CreateFiber
• SwitchToFiber
• ConvertFiberToThread
• ConvertThreadToFiber
• DeleteFiber

Fibers are represented using opaque, void pointers. Maybe that’s a little too simple since it’s easy to misuse in C, but I like it. The return values for CreateFiber() and ConvertThreadToFiber() are void pointers since these both create fibers.

The fiber start routine returns nothing and takes a void “user pointer”. That’s nearly what I’d expect, except that it would probably make more sense for a fiber to return int, which is more in line with main / WinMain / mainCRTStartup / WinMainCRTStartup. As I said, when any fiber returns from its start routine, it’s like returning from the main function, so it should probably have returned an integer.

A fiber may delete itself, which is the same as exiting the thread. However, a fiber cannot yield (e.g. SwitchToFiber()) to itself. That’s undefined behavior.

#include <stdio.h>
#include <stdlib.h>
#include <windows.h>

void
coup(void *king)
{
    puts("Long live the king!");
    DeleteFiber(king);
    ConvertFiberToThread(); /* seize the main thread */
    /* ... */
}

int
main(void)
{
    void *king = ConvertThreadToFiber(0);
    void *pretender = CreateFiber(0, coup, king);
    SwitchToFiber(pretender);
    abort(); /* unreachable */
}

Only fibers can yield to fibers, but when the program starts up, there are no fibers. At least one thread must first convert itself into a fiber using ConvertThreadToFiber(), which returns the fiber object that represents itself. It takes one argument analogous to the last argument of CreateFiber(), except that there’s no start routine to accept it. The process is reversed with ConvertFiberToThread().

Fibers don’t belong to any particular thread and can be scheduled on any thread if properly synchronized. Obviously one should never yield to the same fiber in two different threads at the same time.

Contrast with POSIX

The equivalent POSIX systems was context switching. It’s also stackful and symmetric, but it has just three important functions: getcontext(3), makecontext(3), and swapcontext.

int  getcontext(ucontext_t *ucp);
void makecontext(ucontext_t *ucp, void (*func)(), int argc, ...);
int  swapcontext(ucontext_t *oucp, const ucontext_t *ucp);

These are roughly equivalent to GetCurrentFiber(), CreateFiber(), and SwitchToFiber(). There is no need for ConvertFiberToThread() since threads can context switch without preparation. There’s also no DeleteFiber() because the resources are managed by the program itself. That’s where POSIX contexts are a little bit more powerful.

The first argument to CreateFiber() is the desired stack size, with zero indicating the default stack size. The stack is allocated and freed by the operating system. The downside is that the caller doesn’t have a choice in managing the lifetime of this stack and how it’s allocated. If you’re frequently creating and destroying coroutines, those stacks are constantly being allocated and freed.

In makecontext(3), the caller allocates and supplies the stack. Freeing that stack is equivalent to destroying the context. A program that frequently creates and destroys contexts can maintain a stack pool or otherwise more efficiently manage their allocation. This makes it more powerful, but it also makes it a little more complicated. It would be hard to remember how to do all this without a careful reading of the documentation:

/* Create a context */
ucontext_t ctx;
ctx.uc_stack.ss_sp = malloc(SIGSTKSZ);
ctx.uc_stack.ss_size = SIGSTKSZ;
ctx.uc_link = 0;
getcontext(&ctx);
makecontext(&ctx, proc, 0);

/* Destroy a context */
free(ctx.uc_stack.ss_sp);

Note how makecontext(3) is variadic (...), passing its arguments on to the start routine of the context. This seems like it might be better than a user pointer. Unfortunately it’s not, since those arguments are strictly limited to integers.

Ultimately I like the fiber API better. The first time I tried it out, I could guess my way through it without looking closely at the documentation.

Async / await with fibers

Why was I looking at the Fiber API? I’ve known about coroutines for years but I didn’t understand how they could be useful. Sure, the function can yield, but what other coroutine should it yield to? It wasn’t until I was recently bit by the async/await bug that I finally saw a “killer feature” that justified their use. Generators come pretty close, though.

Windows fibers are a coroutine primitive suitable for async/await in C programs, where it can also be useful. To prove that it’s possible, I built async/await on top of fibers in 95 lines of code.

The alternatives are to use a third-party coroutine library or to do it myself with some assembly programming. However, having it built into the operating system is quite convenient! It’s unfortunate that it’s limited to Windows. Ironically, though, everything I wrote for this article, including the async/await demonstration, was originally written on Linux using Mingw-w64 and tested using Wine. Only after I was done did I even try it on Windows.

Before diving into how it works, there’s a general concept about the Windows API that must be understood: All kernel objects can be in either a signaled or unsignaled state. The API provides functions that block on a kernel object until it is signaled. The two important ones are WaitForSingleObject() and WaitForMultipleObjects(). The latter behaves very much like poll(2) in POSIX.

Usually the signal is tied to some useful event, like a process or thread exiting, the completion of an I/O operation (i.e. asynchronous overlapped I/O), a semaphore being incremented, etc. It’s a generic way to wait for some event. However, instead of blocking the thread, wouldn’t it be nice to await on the kernel object? In my aio library for Emacs, the fundamental “wait” object was a promise. For this API it’s a kernel object handle.

So, the await function will take a kernel object, register it with the scheduler, then yield to the scheduler. The scheduler — which is a global variable, so there’s only one scheduler per process — looks like this:

struct {
    void *main_fiber;
    HANDLE handles[MAXIMUM_WAIT_OBJECTS];
    void *fibers[MAXIMUM_WAIT_OBJECTS];
    void *dead_fiber;
    int count;
} async_loop;

While fibers are symmetric, coroutines in my async/await implementation are not. One fiber is the scheduler, main_fiber, and the other fibers always yield to it.

There is an array of kernel object handles, handles, and an array of fibers. The elements in these arrays are paired with each other, but it’s convenient to store them separately, as I’ll show soon. fibers[0] is waiting on handles[0], and so on.

The array is a fixed size, MAXIMUM_WAIT_OBJECTS (64), because there’s a hard limit on the number of fibers that can wait at once. This pathetically small limitation is an unfortunate, hard-coded restriction of the Windows API. It kills most practical uses of my little library. Fortunately there’s no limit on the number of handles we might want to wait on, just the number of co-existing fibers.

When a fiber is about to return from its start routine, it yields one last time and registers itself on the dead_fiber member. The scheduler will delete this fiber as soon as it’s given control. Fibers never truly return since that would terminate the program.

With this, the await function, async_await(), is pretty simple. It registers the handle with the scheduler, then yields to the scheduler fiber.

void
async_await(HANDLE h)
{
    async_loop.handles[async_loop.count] = h;
    async_loop.fibers[async_loop.count] = GetCurrentFiber();
    async_loop.count++;
    SwitchToFiber(async_loop.main_fiber);
}

Caveat: The scheduler destroys this handle with CloseHandle() after it signals, so don’t try to reuse it. This made my demonstration simpler, but it might be better to not do this.

A fiber can exit at any time. Such an exit is inserted implicitly before a fiber actually returns:

void
async_exit(void)
{
    async_loop.dead_fiber = GetCurrentFiber();
    SwitchToFiber(async_loop.main_fiber);
}

The start routine given to async_start() is actually wrapped in the real start routine. This is how async_exit() is injected:

struct fiber_wrapper {
    void (*func)(void *);
    void *arg;
};

static void
fiber_wrapper(void *arg)
{
    struct fiber_wrapper *fw = arg;
    fw->func(fw->arg);
    async_exit();
}

int
async_start(void (*func)(void *), void *arg)
{
    if (async_loop.count == MAXIMUM_WAIT_OBJECTS) {
        return 0;
    } else {
        struct fiber_wrapper fw = {func, arg};
        SwitchToFiber(CreateFiber(0, fiber_wrapper, &fw));
        return 1;
    }
}

The library provides a single awaitable function, async_sleep(). It creates a “waitable timer” object, starts the countdown, and returns it. (Notice how SetWaitableTimer() is a typically-ugly Win32 function with excessive parameters.)

HANDLE
async_sleep(double seconds)
{
    HANDLE promise = CreateWaitableTimer(0, 0, 0);
    LARGE_INTEGER t;
    t.QuadPart = (long long)(seconds * -10000000.0);
    SetWaitableTimer(promise, &t, 0, 0, 0, 0);
    return promise;
}

A more realistic example would be overlapped I/O. For example, you’d open a file (CreateFile()) in overlapped mode, then when you, say, read from that file (ReadFile()) you create an event object (CreateEvent()), populate an overlapped I/O structure with the event, offset, and length, then finally await on the event object. The fiber will be resumed when the operation is complete.

Side note: Unfortunately overlapped I/O doesn’t work correctly for files, and many operations can’t be done asynchronously, like opening files. When it comes to files, you’re better off using dedicated threads as libuv does instead of overlapped I/O. You can still await on these operations. You’d just await on the signal from the thread doing synchronous I/O, not from overlapped I/O.

The most complex part is the scheduler, and it’s really not complex at all:

void
async_run(void)
{
    while (async_loop.count) {
        /* Wait for next event */
        DWORD nhandles = async_loop.count;
        HANDLE *handles = async_loop.handles;
        DWORD r = WaitForMultipleObjects(nhandles, handles, 0, INFINITE);

        /* Remove event and fiber from waiting array */
        void *fiber = async_loop.fibers[r];
        CloseHandle(async_loop.handles[r]);
        async_loop.handles[r] = async_loop.handles[nhandles - 1];
        async_loop.fibers[r] = async_loop.fibers[nhandles - 1];
        async_loop.count--;

        /* Run the fiber */
        SwitchToFiber(fiber);

        /* Destroy the fiber if it exited */
        if (async_loop.dead_fiber) {
            DeleteFiber(async_loop.dead_fiber);
            async_loop.dead_fiber = 0;
        }
    }
}

This is why the handles are in their own array. The array can be passed directly to WaitForMultipleObjects(). The return value indicates which handle was signaled. The handle is closed, the entry removed from the scheduler, and then the fiber is resumed.

That WaitForMultipleObjects() is what limits the number of fibers. It’s not possible to wait on more than 64 handles at once! This is hard-coded into the API. How? A return value of 64 is an error code, and changing this would break the API. Remember what I said about being locked into bad design decisions of the past?

To be fair, WaitForMultipleObjects() was a doomed API anyway, just like select(2) and poll(2) in POSIX. It scales very poorly since the entire array of objects being waited on must be traversed on each call. That’s terribly inefficient when waiting on large numbers of objects. This sort of problem is solved by interfaces like kqueue (BSD), epoll (Linux), and IOCP (Windows). Unfortunately IOCP doesn’t really fit this particular problem well — awaiting on kernel objects — so I couldn’t use it.

When the awaiting fiber count is zero and the scheduler has control, all fibers must have completed and there’s nothing left to do. However, the caller can schedule more fibers and then restart the scheduler if desired.

That’s all there is to it. Have a look at demo.c to see how the API looks in some trivial examples. On Linux you can see it in action with make check. On Windows, you just need to compile it, then run it like a normal program. If there was a better function than WaitForMultipleObjects() in the Windows API, I would have considered turning this demonstration into a real library.

I thought it would be interesting to revisit this software, to reevaluate it from a far more experienced perspective. Keep in mind that C++ wasn’t even standardized yet, and the most recent C standard was from 1989. Given this, what was it like to be a professional software developer using a Borland toolchain on Windows 20 years ago? Was it miserable, made bearable only by ignorance of how much better the tooling could be? Or maybe it actually wasn’t so bad, and these tools are better than I expect?

Ultimately my conclusion is that it’s a little bit of both. There are some significant capability gaps compared to today, but the core toolchain itself is actually quite reasonable, especially for the mid 1990s.

The setup

Before getting into the evaluation, let’s discuss how I got it all up and running. While it’s technically possible to run Windows 95 on a modern x86-64 machine thanks to the architecture’s extreme backwards compatibility, it’s more compatible, simpler, and safer to virtualize it. Most importantly, I can emulate older hardware that will have better driver support.

Despite that early start in Windows all those years ago, I’m primarily a Linux user. The premier virtualization solution on Linux these days is KVM, a kernel module that turns Linux into a hypervisor and makes efficient use of hardware virtualization extensions. Unfortunately pre-XP Windows doesn’t work well on KVM, so instead I’m using QEmu (with KVM disabled), a hardware emulator closely associated with KVM. Since it doesn’t take advantage of hardware virtualization extensions, it will be slower. This is fine since my goal is to emulate slow, 20+ year old hardware anyway.

There’s very little practical difference between Windows 95 and Windows 98. Since Windows 98 runs a lot smoother virtualized, I decided to go with that instead. This will be perfectly sufficient for my toolchain evaluation.

Software

To get started, I’ll need an installer for Windows 98. I thought this would be difficult to find, but there’s a copy available on the Internet Archive. I don’t know how “legitimate” this is, but it works. Since it’s running in a virtual machine without network access, I also don’t really care if this copy is somehow infected with malware.

Internet Archive: Windows 98 Second Edition

Also on the Internet Archive is a complete copy of Borland C++ 5.02, with the same caveats of legitimacy. It works, which is good enough for my purposes.

Internet Archive: Borland C++ 5.02

Thank you Internet Archive!

Hardware

I’ve got my software, now to set up the virtualized hardware. First I create a drive image:

$ qemu-image create -fqcow2 win98.img 8G

I gave it 8GB, which is actually a bit overkill. Giving Windows 98 a virtual hard drive with modern sizes would probably break the installer. This sort of issue is a common theme among old software, where there may be complaints about negative available disk space due to signed integer overflow.

I decided to give the machine 256MB of memory (-m 256). This is also a little excessive, but I wanted to be sure memory didn’t limit Borland’s capabilities. This amount of memory is close to the upper bound, and going much beyond will likely cause problems with Windows 98.

For the CPU I settled on a Pentium (-cpu pentium). My original goal was to go a little simpler with a 486 (-cpu 486), but the Windows 98 installer kept crashing when I tried this.

I experimented with different configurations for the network card, but I couldn’t get anything to work. So I’ve disabled networking (-net none). The only reason I’d want this is that it would be easier to move files in and out of the virtual machine.

Finally, here’s how I ran QEmu. The last two lines are only needed when installing.

$ qemu-system-x86_64 \
    -localtime \
    -cpu pentium \
    -no-acpi \
    -no-hpet \
    -m 256 \
    -hda win98.img \
    -soundhw sb16 \
    -vga cirrus \
    -net none \
    -cdrom "Windows 98 Second Edition.iso" \
    -boot d

Installation

Installation is just a matter of following the instructions. You’ll need that product key listed on the Internet Archive site.

That copy of Borland is just a big .zip file. This presents two problems.

1. Without network access, I’ll need to figure out how to get this inside the virtual machine.

2. This version of Windows doesn’t come with software to unzip this file. I’d need to find and install an unzip tool first.

Fortunately I can kill two birds with one stone by converting that .zip archive into a .iso and mounting it in the virtual machine.

unzip "BORLAND C++.zip"
genisoimage -R -J -o borland.iso "BORLAND C++"

Then in the QEmu console (C-A-2) I attach it:

change ide1-cd0 borland.iso

This little trick of generating .iso files and mounting them is how I will be moving all the other files into the virtual machine.

Borland C++

The first thing I did was play around with with Borland IDE. This is what I would have been using 20 years ago.

Despite being Borland C++, I’m personally most interested in its ANSI C compiler. As I already pointed out, this software pre-dates C++’s standardization, and a lot has changed over the past two decades. On the other hand, C hasn’t really changed all that much. The 1999 update to the C standard (e.g. “C99”) was big and important, but otherwise little has changed. The biggest drawback is the lack of “declare anywhere” variables, including in for-loop initializers. Otherwise it’s the same as writing C today.

To test drive the IDE, I made a couple of test projects, built and ran them with different options, and poked around with the debugger. The debugger is actually pretty decent, especially for the 1990s. It can be operated via the IDE or standalone, so I could use it without firing up the IDE and making a project.

The toolchain includes an assembler, and I can inspect the compiler’s assembly output. To nobody’s surprise this is Intel-flavored assembly, which is very welcome. Imagining myself as a software developer in the mid 1990s, this means I can see exactly what the compiler’s doing as well as write some of the performance sensitive parts in assembly if necessary.

The built-in editor is the worst part of the IDE, which is unfortunate since it really spoils the whole experience. It’s easy to jump between warnings and errors, it has incremental search, and it has good syntax highlighting. But these are the only positive things I can say about it. If I had to work with this editor full-time, I’d spend my days pretty irritated.

Switch to command line tools

Like with the debugger, the Borland people did a good job modularizing their development tools. As part of the installation process, all of the Borland command line tools are added to the system PATH (reminder: this is a single-user system). This includes compiler, linker, assembler, debugger, and even an incomplete implementation of make.

With this, I can essentially pretend the IDE doesn’t exist and replace that crummy editor with something better: Vim.

The last version of Vim to support MS-DOS and Windows 95/98 is Vim 7.3, released in 2010. I download those binaries, trim a few things from my .vimrc, and smuggle it all into my virtual machine via a virtual CD. I’ve now got a powerful text editor in Windows 98 and my situation has drastically improved.

Since I hardly use features added since Vim 7.3, this feels right at home to me. I can invoke the build from Vim, and it can populate the quickfix list from Borland’s output, so I could actually be fairly productive in these circumstances! I’m honestly really impressed with how well this all works together.

At this point I only have two significant annoyances:

1. Borland’s command line tools belong to that category of irritating programs that print their version banner on every invocation. There’s not even a command line switch to turn this off. All this noise is quickly tiresome. The Visual Studio toolchain does the same thing by default, though it can be turned off (-nologo). I dislike that some GNU tools also commit this sin, but at least GNU limits this to interactive programs.

2. The Windows/DOS command shell and console is even worse than it is today. I didn’t think that was possible. This is back when it was still genuinely DOS and not just pretending to be (e.g. in NT). The worst part by far is the lack of command history. There’s no using the up-arrow to get previous commands. There’s no tab completion. Forward slash is not a substitute for backslash in paths. If I wanted to improve my productivity, replacing this console and shell would be the first priority.

Update: In an email, Aristotle Pagaltzis informed me that Windows 98 comes with DOSKEY.COM, which provides command history for COMMAND.EXE. Alternatively there’s Enhanced DOSKEY.com, an open source, alternative implementation that also provides tab completion for commands and filesnames. This makes the console a lot more usable (and, honestly, in some ways better than the modern defaults).

Building Enchive with Borland

Last year I wrote a backup encryption tool called Enchive, and I still use it regularly. One of my design goals was high portability since it may be needed to decrypt something important in the distant future. It should be as bit-rot-proof as possible. In software, the best way to future-proof is to past-proof.

If I had a time machine that could send source code back in time, and I sent Enchive to a competant developer 20 years ago, would they be able to compile it and run it? If the answer is yes, then that means Enchive already has 20 years of future-proofing built into it.

To accomplish this, Enchive is 3,300 lines of strict ANSI C, 1989-style, with no dependencies other than the C standard library and a handful of operating system functions — e.g. functionality not in the C standard library. In practice, any ANSI C compiler targeting either POSIX, or Windows 95 or later, should be able to compile it.

My Windows 98 virtual machine includes an ANSI C compiler, and can be used to simulate this time machine. I generated an “amalgamation” build (make amalgamation) — essentially a concatenation of all the source files — and sent this into the virtual machine. Before Borland was able to compile it, I needed to make three small changes.

First, Enchive includes stdint.h to get fixed-width integers needed for the encryption routines. This header comes from C99, and C89 has no equivalent. I anticipated this problem from the beginning and made it easy for the person performing the build to correct it. This header is included exactly once, in config.h, and this is placed at the top of the amalgamation build. The include only needs to be replaced with a handful of manual typedefs. For Borland that looks like this:

typedef unsigned char    uint8_t;
typedef unsigned short   uint16_t;
typedef unsigned long    uint32_t;
typedef unsigned __int64 uint64_t;

typedef long             int32_t;
typedef __int64          int64_t;

#define INT8_C(n)   (n)
#define INT16_C(n)  (n)
#define INT32_C(n)  (n##U)

Second, in more recent versions of Windows, GetFileAttributes() can return the value INVALID_FILE_ATTRIBUTES. Checking for an error that cannot happen is harmless, but this value isn’t defined in Borland’s SDK. I only had to eliminate that check.

Third, the CryptGenRandom() interface isn’t defined in Borland’s SDK. This is used by Enchive to generate keys. MSDN reports this function wasn’t available until Windows XP, but it’s definitely there in Windows 98, exported by ADVAPI32.dll. I’m able to call it, though it always reports an error. Perhaps it’s been disabled in this version due to cryptographic export restrictions?

Regardless of what’s wrong, I ripped this out and replaced it with a fatal error. This version of Enchive can’t generate new keys — unless derived from a passphrase — nor encrypt files, including the use of a protection key to encrypt the secret key. However, it can decrypt files, which is the important part that needs to be future-proofed.

With this three changes — which took me about 10 minutes to sort out — Enchive builds and runs, and it correctly decrypts files I encrypted on Linux. So Enchive has at least 20 years of past-proofing! The screenshot at the top of this article shows it running successfully in an MS-DOS console window.

What’s wrong? What’s missing?

I mentioned that there were some gaps. The most obvious is the lack of the standard POSIX utilities, especially a decent shell. I don’t know if any had been ported to Windows in the mid 1990s. But that could be solved one way or another without too much trouble, even if it meant doing some of that myself.

No, the biggest capability I’d miss, and which wouldn’t be easily obtained, is Git, or a least a decent source control system. I really don’t want to work without proper source control. Git’s support for Windows is second tier, and the port to modern Windows is already a bit of a hack. Getting it to run in Windows 98 would probably be a challenge, especially if I had to compile it with Borland.

The other major issue is the lack of stability. In this experiment, I’ve been seeing this screen a lot:

I remember Windows crashing a lot back in those days, and it certainly had a bad reputation for being unstable, but this is far worse than I remembered. While the hardware emulator may be somewhat at fault here, keep in mind that I never installed third party drivers. Most of these crashes are Windows’ fault. I found I can reliably bring the whole system down with a single GetProcAddress() call on a system DLL. The only way I can imagine this instability was so tolerated back then was general ignorance that computing could be so much better.

I was tempted to write this article in Vim on Windows 98, but all this crashing made me too nervous. I didn’t want some stupid filesystem corruption to wipe out my work. Too risky.

A better alternative

If I was stuck working in Windows 98 — or was at least targeting it as a platform — but had access to a modern tooling ecosystem, could I do better than Borland? Yes! Programs built by Mingw-w64 can be run even as far back as Windows 95.

Now, there’s a catch. I thought it would be this simple:

$ i686-w64-mingw32-gcc -Os hello.c

But when I brought the resulting binary into the virtual machine it crashed when ran it: illegal instruction. Turns out it contained a conditional move (cmov) which is an instruction not available until the Pentium Pro (686). The “pentium” emulation is just a 586.

I tried to disable cmov by picking the specific architecture:

$ i686-w64-mingw32-gcc -march=pentium -Os hello.c

This still didn’t work because the statically-linked part of the CRT contained the cmov. I’d have to recompile that as well.

I could have switched the QEmu options to “upgrade” to a Pentium Pro, but remember that my goal was really the 486. Fortunately this was easy to fix: compile my own Mingw-w64 cross-compiler. I’ve done this a number of times before, so I knew it wouldn’t be difficult.

I could go step by step, but it’s all fairly well documented in the Mingw-64 “howto-build” document. I used GCC 7.3 (the latest version), and for the target I picked “i486-w64-mingw32”. When it was done I could compile binaries on Linux to run in my Windows 98 virtual machine:

$ i486-w64-mingw32-gcc -Os hello.c

This should enable quite a bit of modern software to run inside my virtual machine if I so wanted. I didn’t actually try this (yet?), but, to take this concept all the way, I could use this cross-compiler to cross-compile Mingw-w64 itself to run inside the virtual machine, directly replacing Borland C++.

And the only thing I’d miss about Borland is its debugger.

With some up-front attention to detail, this is actually not terribly difficult. Unix-like systems are probably the least diverse and least buggy they’ve ever been. Writing portable code is really just a matter of coding to the standards and ignoring extensions unless absolutely necessary. Knowing what’s standard and what’s extension is the tricky part, but I’ll explain how to find this information.

You might be tempted to reach for an overly complicated solution such as GNU Autoconf. Sure, it creates a configure script with the familiar, conventional interface. This has real value. But do you really need to run a single-threaded gauntlet of hundreds of feature/bug tests for things that sometimes worked incorrectly in some weird unix variant back in the 1990s? On a machine with many cores (parallel build, -j), this may very well be the slowest part of the whole build process.

For example, the configure script for Emacs checks that the compiler supplies stdlib.h, string.h, and getenv — things that were standardized nearly 30 years ago. It also checks for a slew of POSIX functions that have been standard since 2001.

There’s a much easier solution: Document that the application requires, say, C99 and POSIX.1-2001. It’s the responsibility of the person building the application to supply these implementations, so there’s no reason to waste time testing for it.

How to code to the standards

Suppose there’s some function you want to use, but you’re not sure if it’s standard or an extension. Or maybe you don’t know what standard it comes from. Luckily the man pages document this stuff very well, especially on Linux. Check the friendly “CONFORMING TO” section. For example, look at getenv(3). Here’s what that section has to say:

CONFORMING TO
    getenv(): SVr4, POSIX.1-2001, 4.3BSD, C89, C99.

    secure_getenv() is a GNU extension.

This says this function comes from the original C standard. It’s always available on anything that claims to be a C implementation. The man page also documents secure_getenv(), which is a GNU extension: to be avoided in anything intended to be portable.

What about sleep(3)?

CONFORMING TO
    POSIX.1-2001.

This function isn’t part of standard C, but it’s available on any system claiming to implement POSIX.1-2001 (the POSIX standard from 2001). If the program needs to run on an operating system not implementing this POSIX standard (i.e. Windows), you’ll need to call an alternative function, probably inside a different #if .. #endif branch. More on this in a moment.

If you’re coding to POSIX, you must define the _POSIX_C_SOURCE feature test macro to the standard you intend to use prior to any system header includes:

A POSIX-conforming application should ensure that the feature test macro _POSIX_C_SOURCE is defined before inclusion of any header.

For example, to properly access POSIX.1-2001 functions in your application, define _POSIX_C_SOURCE to 200112L. With this defined, it’s safe to assume access to all of C and everything from that standard of POSIX. You can do this at the top of your sources, but I personally like the tidiness of a global config.h that gets included before everything.

How to create a portable build

So you’ve written clean, portable C to the standards. How do you build this application? The natural choice is make. It’s available everywhere and it’s part of POSIX.

Again, the tricky part is teasing apart the standard from the extension. I’m a long-time sinner in this regard, having far too often written Makefiles that depend on GNU Make extensions. This is a real pain when building programs on systems without the GNU utilities. I’ve been making amends (and finding some bugs as a result).

No implementation makes the division clear in its documentation, and especially don’t bother looking at the GNU Make manual. Your best resource is the standard itself. If you’re already familiar with make, coding to the standard is largely a matter of unlearning the various extensions you know.

Outside of some hacks, this means you don’t get conditionals (if, else, etc.). With some practice, both with sticking to portable code and writing portable Makefiles, you’ll find that you don’t really need them. Following the macro conventions will cover most situations. For example:

• CC: the C compiler program
• CFLAGS: flags to pass to the C compiler
• LDFLAGS: flags to pass to the linker (via the C compiler)
• LDLIBS: libraries to pass to the linker

You don’t need to do anything weird with the assignments. The user invoking make can override them easily. For example, here’s part of a Makefile:

CC     = c99
CFLAGS = -Wall -Wextra -Os

But the user wants to use clang, and their system needs to explicitly link -lsocket (e.g. Solaris). The user can override the macro definitions on the command line:

$ make CC=clang LDLIBS=-lsocket

The same rules apply to the programs you invoke from the Makefile. Read the standards documents and ignore your system’s man pages as to avoid accidentally using an extension. It’s especially valuable to learn the Bourne shell language and avoid any accidental bashisms in your Makefiles and scripts. The dash shell is good for testing your scripts.

Makefiles conforming to the standard will, unfortunately, be more verbose than those taking advantage of a particular implementation. If you know how to code Bourne shell — which is not terribly difficult to learn — then you might even consider hand-writing a configure script to generate the Makefile (a la metaprogramming). This gives you a more flexible language with conditionals, and, being generated, redundancy in the Makefile no longer matters.

As someone who frequently dabbles with BSD systems, my life has gotten a lot easier since learning to write portable Makefiles and scripts.

But what about Windows

It’s the elephant in the room and I’ve avoided talking about it so far. If you want to build with Visual Studio’s command line tools — something I do on occasion — build portability goes out the window. Visual Studio has nmake.exe, which nearly conforms to POSIX make. However, without the standard unix utilities and with the completely foreign compiler interface for cl.exe, there’s absolutely no hope of writing a Makefile portable to this situation.

The nice alternative is MinGW(-w64) with MSYS or Cygwin supplying the unix utilities, though it has the problem of linking against msvcrt.dll. Another option is a separate Makefile dedicated to nmake.exe and the Visual Studio toolchain. Good luck defining a correctly working “clean” target with del.exe.

My preferred approach lately is an amalgamation build (as seen in Enchive): Carefully concatenate all the application’s sources into one giant source file. First concatenate all the headers in the right order, followed by all the C files. Use sed to remove and local includes. You can do this all on a unix system with the nice utilities, then point cl.exe at the amalgamation for the Visual Studio build. It’s not very useful for actual development (i.e. you don’t want to edit the amalgamation), but that’s what MinGW-w64 resolves.

What about all those POSIX functions? You’ll need to find Win32 replacements on MSDN. I prefer to do this is by abstracting those operating system calls. For example, compare POSIX sleep(3) and Win32 Sleep().

#if defined(_WIN32)
#include <windows.h>

void
my_sleep(int s)
{
    Sleep(s * 1000);  // TODO: handle overflow, maybe
}

#else /* __unix__ */
#include <unistd.h>

void
my_sleep(int s)
{
    sleep(s);  // TODO: fix signal interruption
}
#endif

Then the rest of the program calls my_sleep(). There’s another example in the OpenMP article with pwrite(2) and WriteFile(). This demonstrates that supporting a bunch of different unix-like systems is really easy, but introducing Windows portability adds a disproportionate amount of complexity.

Caveat: paths and filenames

There’s one major complication with filenames for applications portable to Windows. In the unix world, filenames are null-terminated bytestrings. Typically these are Unicode strings encoded as UTF-8, but it’s not necessarily so. The kernel just sees bytestrings. A bytestring doesn’t necessarily have a formal Unicode representation, which can be a problem for languages that want filenames to be Unicode strings (also).

On Windows, filenames are somewhere between UCS-2 and UTF-16, but end up being neither. They’re really null-terminated unsigned 16-bit integer arrays. It’s almost UTF-16 except that Windows allows unpaired surrogates. This means Windows filenames also don’t have a formal Unicode representation, but in a completely different way than unix. Some heroic efforts have gone into working around this issue.

As a result, it’s highly non-trivial to correctly support all possible filenames on both systems in the same program, especially when they’re passed as command line arguments.

Summary

The key points are:

1. Document the standards your application requires and strictly stick to them.
2. Ignore the vendor documentation if it doesn’t clearly delineate extensions.

This was all a discussion of non-GUI applications, and I didn’t really touch on libraries. Many libraries are simple to access in the build (just add it to LDLIBS), but some libraries — GUIs in particular — are particularly complicated to manage portably and will require a more complex solution (pkg-config, CMake, Autoconf, etc.).

I’ve been using tools like this going back 20 years, but I never tried to write one myself until now. There are many memory cheat tools to pick from these days, the most prominent being Cheat Engine. These tools use the platform’s debugging API, so of course any good debugger could do the same thing, though a debugger won’t be specialized appropriately (e.g. locating the particular address and locking its value).

My motivation was bypassing an in-app purchase in a single player Windows game. I wanted to convince the game I had made the purchase when, in fact, I hadn’t. Once I had it working successfully, I ported MemDig to Linux since I thought it would be interesting to compare. I’ll start with Windows for this article.

Windows

Only three Win32 functions are needed, and you could almost guess at how it works.

• OpenProcess()
• ReadProcessMemory()
• WriteProcessMemory()

It’s very straightforward and, for this purpose, is probably the simplest API for any platform (see update).

As you probably guessed, you first need to open the process, given its process ID (integer). You’ll need to select the desired access bit a bit set. To read memory, you need the PROCESS_VM_READ and PROCESS_QUERY_INFORMATION rights. To write memory, you need the PROCESS_VM_WRITE and PROCESS_VM_OPERATION rights. Alternatively you could just ask for all rights with PROCESS_ALL_ACCESS, but I prefer to be precise.

DWORD access = PROCESS_VM_READ |
               PROCESS_QUERY_INFORMATION |
               PROCESS_VM_WRITE |
               PROCESS_VM_OPERATION;
HANDLE proc = OpenProcess(access, FALSE, pid);

And then to read or write:

void *addr; // target process address
SIZE_T written;
ReadProcessMemory(proc, addr, &value, sizeof(value), &written);
// or
WriteProcessMemory(proc, addr, &value, sizeof(value), &written);

Don’t forget to check the return value and verify written. Finally, don’t forget to close it when you’re done.

CloseHandle(proc);

That’s all there is to it. For the full cheat tool you’d need to find the mapped regions of memory, via VirtualQueryEx. It’s not as simple, but I’ll leave that for another article.

Linux

Unfortunately there’s no standard, cross-platform debugging API for unix-like systems. Most have a ptrace() system call, though each works a little differently. Note that ptrace() is not part of POSIX, but appeared in System V Release 4 (SVr4) and BSD, then copied elsewhere. The following will all be specific to Linux, though the procedure is similar on other unix-likes.

In typical Linux fashion, if it involves other processes, you use the standard file API on the /proc filesystem. Each process has a directory under /proc named as its process ID. In this directory is a virtual file called “mem”, which is a file view of that process’ entire address space, including unmapped regions.

char file[64];
sprintf(file, "/proc/%ld/mem", (long)pid);
int fd = open(file, O_RDWR);

The catch is that while you can open this file, you can’t actually read or write on that file without attaching to the process as a debugger. You’ll just get EIO errors. To attach, use ptrace() with PTRACE_ATTACH. This asynchronously delivers a SIGSTOP signal to the target, which has to be waited on with waitpid().

You could select the target address with lseek(), but it’s cleaner and more efficient just to do it all in one system call with pread() and pwrite(). I’ve left out the error checking, but the return value of each function should be checked:

ptrace(PTRACE_ATTACH, pid, 0, 0);
waitpid(pid, NULL, 0);

off_t addr = ...; // target process address
pread(fd, &value, sizeof(value), addr);
// or
pwrite(fd, &value, sizeof(value), addr);

ptrace(PTRACE_DETACH, pid, 0, 0);

The process will (and must) be stopped during this procedure, so do your reads/writes quickly and get out. The kernel will deliver the writes to the other process’ virtual memory.

Like before, don’t forget to close.

close(fd);

To find the mapped regions in the real cheat tool, you would read and parse the virtual text file /proc/pid/maps. I don’t know if I’d call this stringly-typed method elegant — the kernel converts the data into string form and the caller immediately converts it right back — but that’s the official API.

Update: Konstantin Khlebnikov has pointed out the process_vm_readv() and process_vm_writev() system calls, available since Linux 3.2 (January 2012) and glibc 2.15 (March 2012). These system calls do not require ptrace(), nor does the remote process need to be stopped. They’re equivalent to ReadProcessMemory() and WriteProcessMemory(), except there’s no requirement to first “open” the process.

I primarily work on and develop for unix-like operating systems — Linux in particular. However, when it comes to desktop applications, most potential users are on Windows. Rather than develop on Windows, which I’d rather avoid, I’ll continue developing, testing, and debugging on Linux while keeping portability in mind. Unfortunately every option I’ve found for building Windows C programs has some significant limitations. These limitations advise my approach to portability and restrict the C language features used by the program for all platforms.

As of this writing I’ve identified four different practical ways to build C applications for Windows. This information will definitely become further and further out of date as this article ages, so if you’re visiting from the future take a moment to look at the date. Except for LLVM shaking things up recently, development tooling on unix-like systems has had the same basic form for the past 15 years (i.e. dominated by GCC). While Visual C++ has been around for more than two decades, the tooling on Windows has seen more churn by comparison.

Before I get into the specifics, let me point out a glaring problem common to all four: Unicode arguments and filenames. Microsoft jumped the gun and adopted UTF-16 early. UTF-16 is a kludge, a worst of all worlds, being a variable length encoding (surrogate pairs), backwards incompatible (unlike UTF-8), and having byte-order issues (BOM). Most Win32 functions that accept strings generally come in two flavors, ANSI and UTF-16. The standard, portable C library functions wrap the ANSI-flavored functions. This means portable C programs can’t interact with Unicode filenames. (Update 2021: Now they can.) They must call the non-portable, Windows-specific versions. This includes main itself, which is only handed ANSI-truncated arguments.

Compare this to unix-like systems, which generally adopted UTF-8, but rather as a convention than as a hard rule. The operating system doesn’t know or care about Unicode. Program arguments and filenames are just zero-terminated bytestrings. Implicitly decoding these as UTF-8 would be a mistake anyway. What happens when the encoding isn’t valid?

This doesn’t have to be a problem on Windows. A Windows standard C library could connect to Windows’ Unicode-flavored functions and encode to/from UTF-8 as needed, allowing portable programs to maintain the bytestring illusion. It’s only that none of the existing standard C libraries do it this way.

Mingw-w64

Of course my first natural choice is MinGW, specifically the Mingw-w64 fork. It’s GCC ported to Windows. You can continue relying on GCC-specific features when you need them. It’s got all the core language features up through C11, plus the common extensions. It’s probably packaged by your Linux distribution of choice, making it trivial to cross-compile programs and libraries from Linux — and with Wine you can even execute them on x86. Like regular GCC, it outputs GDB-friendly DWARF debugging information, so you can debug applications with GDB.

If I’m using Mingw-w64 on Windows, I prefer to do so from inside Cygwin. Since it provides a complete POSIX environment, it maximizes portability for the whole tool chain. This isn’t strictly required.

However, it has one big flaw. Unlike unix-like systems, Windows doesn’t supply a system standard C library. That’s the compiler’s job. But Mingw-w64 doesn’t have one. Instead it links against msvcrt.dll, which isn’t officially supported by Microsoft. It just happens to exist on modern Windows installations. Since it’s not supported, it’s way out of date and doesn’t support much of C99. A lot of these problems are patched over by the compiler, but if you’re relying on Mingw-w64, you still have to stick to some C89 library features, such as limiting yourself to the C89 printf specifiers.

Update: Mārtiņš Možeiko has pointed out __USE_MINGW_ANSI_STDIO, an undocumented feature that fixes the printf family. I now use this by default in all of my Mingw-w64 builds. It fixes most of the formatted output issues, except that it’s incompatible with the format function attribute. (Update 2021: Mingw-w64 now does the right thing out of the box.)

Another problem is that position-independent code generation is broken, and so ASLR is not an option. This means binaries produced by Mingw-w64 are less secure than they should be. There are also a number of subtle code generation bugs that might arise if you’re doing something unusual. (Update 2021: Mingw-w64 makes PIE mandatory.)

Visual C++

The behemoth usually considered in this situation is Visual Studio and the Visual C++ build tools. I strongly prefer open source development tools, and Visual Studio obviously the least open source option, but at least it’s cost-free these days. Now, I have absolutely no interest in Visual Studio, but fortunately the Visual C++ compiler and associated build tools can be used standalone, supporting both C and C++.

Included is a “vcvars” batch file — vcvars64.bat for x64. Execute that batch file in a cmd.exe console and the Visual C++ command line build tools will be made available in that console and in any programs executed from it (your editor). It includes the compiler (cl.exe), linker (link.exe), assembler (ml64.exe), disassembler (dumpbin.exe), and more. It also includes a mostly POSIX-complete make called nmake.exe. All these tools are noisy and print a copyright banner on every invocation, so get used to passing -nologo every time, which suppresses some of it.

When I said behemoth, I meant it. In my experience it literally takes hours (unattended) to install Visual Studio 2015. The good news is you don’t actually need it all anymore. The build tools are available standalone. While it’s still a larger and slower installation process than it really should be, it’s is much more reasonable to install. It’s good enough that I’d even say I’m comfortable relying on it for Windows builds. (Update: The build tools are unfortunately no longer standalone.)

That being said, it’s not without its flaws. Microsoft has never announced any plans to support C99. They only care about C++, with C as a second class citizen. Since C++11 incorporated most of C99 and Microsoft supports C++11, Visual Studio 2015 supports most of C99. The only things missing as far as I can tell are variable length arrays (VLAs), complex numbers, and C99’s array parameter declarators, since none of these were adopted by C++. Some C99 features are considered extensions (as they would be for C89), so you’ll also get warnings about them, which can be disabled.

The command line interface (option flags, intermediates, etc.) isn’t quite reconcilable with the unix-like ecosystem (i.e. GCC, Clang), so you’ll need separate Makefiles, or you’ll need to use a build system that generates Visual C++ Makefiles.

Debugging is a major problem. (Update 2022: It’s actually quite good once you know how to do it.) Visual C++ outputs separate .pdb program database files, which aren’t usable from GDB. Visual Studio has a built-in debugger, though it’s not included in the standalone Visual C++ build tools. I’m still searching for a decent debugging solution for this scenario. I tried WinDbg, but I can’t stand it. (Update 2022: RemedyBG is amazing.)

In general the output code performance is on par with GCC and Clang, so you’re not really gaining or losing performance with Visual C++.

Clang

Unsurprisingly, Clang has been ported to Windows. It’s like Mingw-w64 in that you get the same features and interface across platforms.

Unlike Mingw-w64, it doesn’t link against msvcrt.dll. Instead it relies directly on the official Windows SDK. You’ll basically need to install the Visual C++ build tools as if were going to build with Visual C++. This means no practical cross-platform builds and you’re still relying on the proprietary Microsoft toolchain. In the past you even had to use Microsoft’s linker, but LLVM now provides its own.

It generates GDB-friendly DWARF debug information (in addition to CodeView) so in theory you can debug with GDB again. I haven’t given this a thorough evaluation yet.

Pelles C

Finally there’s Pelles C. It’s cost-free but not open source. It’s a reasonable, small install that includes a full IDE with an integrated debugger and command line tools. It has its own C library and Win32 SDK with the most complete C11 support around. It also supports OpenMP 3.1. All in all it’s pretty nice and is something I wouldn’t be afraid to rely upon for Windows builds.

Like Visual C++, it has a couple of “povars” batch files to set up the right environment, which includes a C compiler, linker, assembler, etc. The compiler interface mostly mimics cl.exe, though there are far fewer code generation options. The make program, pomake.exe, mimics nmake.exe, but is even less POSIX-complete. The compiler’s output code performance is also noticeably poorer than GCC, Clang, and Visual C++. It’s definitely a less mature compiler.

It outputs CodeView debugging information, so GDB is of no use. The best solution is to simply use the compiler built into the IDE, which can be invoked directly from the command line. You don’t normally need to code from within the IDE just to use the debugger.

Like Visual C++, it’s Windows only, so cross-compilation isn’t really in the picture.

If performance isn’t of high importance, and you don’t require specific code generation options, then Pelles C is a nice choice for Windows builds.

Other Options

I’m sure there are a few other options out there, and I’d like to hear about them so I can try them out. I focused on these since they’re all cost free and easy to download. If I have to register or pay, then it’s not going to beat these options.

The Native API is a low-level API, a foundation for the implementation of the Windows API and various components that don’t use the Windows API (drivers, etc.). It includes a runtime library (RTL) suitable for replacing important parts of the C standard library, unavailable to freestanding programs. Very useful for a minimal program.

Unfortunately, using the Native API is a bit of a minefield. Not all of the documented Native API functions are actually exported by ntdll.dll, making them inaccessible both for linking and GetProcAddress(). Some are exported, but not documented as such. Others are documented as exported but are not documented when (which release of Windows). If a particular function wasn’t exported until Windows 8, I don’t want to use when supporting Windows 7.

This is further complicated by the Microsoft Windows SDK, where many of these functions are just macros that alias C runtime functions. Naturally, MinGW closely follows suit. For example, in both cases, here is how the Native API function RtlCopyMemory is “declared.”

#define RtlCopyMemory(dest,src,n) memcpy((dest),(src),(n))

This is certainly not useful for freestanding programs, though it has a significant benefit for hosted programs: The C compiler knows the semantics of memcpy() and can properly optimize around it. Any C compiler worth its salt will replace a small or aligned, fixed-sized memcpy() or memmove() with the equivalent inlined code. For example:

    char buffer0[16];
    char buffer1[16];
    // ...
    memcpy(buffer0, buffer1, 16);
    // ...

On x86_64 (GCC 4.9.3, -Os), this memmove() call is replaced with two instructions. This isn’t possible when calling an opaque function in a non-standard dynamic library. The side effects could be anything.

    movaps  xmm0, [rsp + 48]
    movaps  [rsp + 32], xmm0

These Native API macro aliases are what have allowed certain Wine issues to slip by unnoticed for years. Very few user space applications actually call Native API functions, even when addressed directly by name in the source. The development suite is pulling a bait and switch.

Like last time I danced at the edge of the compiler, this has caused headaches in my recent experimentation with freestanding executables. The MinGW headers assume that the programs including them will link against a C runtime. Dirty hack warning: To work around it, I have to undo the definition in the MinGW headers and make my own. For example, to use the real RtlMoveMemory():

#include <windows.h>

#undef RtlMoveMemory
__declspec(dllimport)
void RtlMoveMemory(void *, const void *, size_t);

Anywhere where I might have previously used memmove() I can instead use RtlMoveMemory(). Or I could trivially supply my own wrapper:

void *
memmove(void *d, const void *s, size_t n)
{
    RtlMoveMemory(d, s, n);
    return d;
}

As of this writing, the same approach is not reliable with RtlCopyMemory(), the cousin to memcpy(). As far as I can tell, it was only exported starting in Windows 7 SP1 and Wine 1.7.46 (June 2015). Use RtlMoveMemory() instead. The overlap-handling overhead is negligible compared to the function call overhead anyway.

As a side note: one reason besides minimalism for not implementing your own memmove() is that it can’t be implemented efficiently in a conforming C program. According to the language specification, your implementation of memmove() would not be permitted to compare its pointer arguments with <, >, <=, or >=. That would lead to undefined behavior when pointing to unrelated objects (ISO/IEC 9899:2011 §6.5.8¶5). The simplest legal approach is to allocate a temporary buffer, copy the source buffer into it, then copy it into the destination buffer. However, buffer allocation may fail — i.e. NULL return from malloc() — introducing a failure case to memmove(), which isn’t supposed to fail.

Update July 2016: Alex Elsayed pointed out a solution to the memmove() problem in the comments. In short: iterate over the buffers bytewise (char *) using equality (==) tests to check for an overlap. In theory, a compiler could optimize away the loop and make it efficient.

I keep mentioning Wine because I’ve been careful to ensure my applications run correctly with it. So far it’s worked perfectly with both Windows API and Native API functions. Thanks to the hard work behind the Wine project, despite being written sharply against the Windows API, these tiny programs remain relatively portable (x86 and ARM). It’s a good fit for graphical applications (games), but I would never write a command line application like this. The command line has always been a second class citizen on Windows.

Now that I’ve got these Native API issues sorted out, I’ve significantly expanded the capabilities of my tiny, freestanding programs without adding anything to their size. Functions like RtlUnicodeToUTF8N() and RtlUTF8ToUnicodeN() will surely be handy.