Normally we’d extend Python in order to access an external interface that Python cannot access on its own. Wasm runs in a sandbox with no access to the outside world whatsoever, so it obviously isn’t useful for that case. Extensions may also grant Python more speed, which is one of Wasm’s main selling points. We can also use Wasm to access embeddable capabilities written in a different programming language which do not require external access.

For preferred non-WASI Wasm runtime is Volodymyr Shymanskyy’s wasm3. It’s plain old C and very friendly to embedding in the same was as, say, SQLite. Performance is middling, though a C program running on wasm3 is still quite a bit faster than an equivalent Python program. It has Python bindings, pywasm3, but it’s distributed only in source code form. That is, the host machine must have a C toolchain in order to use pywasm3, which defeats my purposes here. If there’s a C toolchain, I might as well just use that instead of going through Wasm.

For the use cases in this article, the best option is wasmtime-py. The distribution includes binaries for Windows, macOS, and Linux on x86-64 and ARM64, which covers nearly all Python installations. Hosts require nothing more than a Python interpreter, no native toolchains. It’s almost as good as having Wasm built into Python itself. In my tests it’s 3x–10x faster than wasm3, so for my first use case the situation is even better. The catch is that it currently weighs ~18MiB (installed), and in the future will likely rival the Python interpreter itself. The API also breaks on a monthly basis, so you’re signing up for the upgrade treadmill lest your own program perishes to bitrot after a couple of years. This article is about version 40.

Usage examples and gotchas

The official examples don’t do anything non-trivial or interesting, and so to figure things out I had to study the documentation, which does not offer many hints. Basic setup looks like this:

import functools
import wasmtime

store    = wasmtime.Store()
module   = wasmtime.Module.from_file(store.engine, "example.wasm")
instance = wasmtime.Instance(store, module, ())
exports  = instance.exports(store)

memory = exports["memory"].get_buffer_ptr(store)
func1  = functools.partial(exports["func1"], store)
func2  = functools.partial(exports["func2"], store)
func3  = functools.partial(exports["func3"], store)

A store is an allocation region from which we allocate all Wasm objects. It is not possible to free individual objects except to discard the whole store. Quite sensible, honestly. What’s not sensible is how often I have to repeat myself, passing the store back into every object in order to use it. These objects are associated with exactly one store and cannot be used with different stores. Use the wrong store and it panics: It’s already keeping track internally! I do not understand why the interface works this way. So to make things simpler, I use functools.partial to bind the store parameter and so get the interface I expect.

The get_buffer_ptr object is a buffer protocol object, and if you’re moving anything other than bytes that’s probably what you want to use to access memory. The usual caveats apply for this object: If you change the memory size you probably want to grab a fresh buffer object. For bytes (e.g. buffers and strings) I prefer the read and write methods.

Because multi-value is still in an experimental state in the Wasm ecosystem, you will likely not pass structs with Wasm. Anything more complicated than scalars will require pointers and copying data in and out of Wasm linear memory. This involves the usual trap that catches nearly everyone: Wasm interfaces make no distinction between pointers and integers, and Wasm runtimes interpret generally interpret all integers as signed. What that means is your pointers are signed unless you take action. Addresses start at 0, so this is bad, bad news.

malloc = functools.partial(exports["func1"], store)

hello = b"hello"
pointer = malloc(len(hello))
assert pointer
memory = exports["memory"].write(store, hello, pointer)  # WRONG!

To make matters worse, wasmtime-py adds its own footgun: The read and write methods adopt the questionable Python convention of negative indices acting from the end. If malloc returns a pointer in the upper half of memory, the negative pointer will pass the bounds check inside write because negative is valid, then quietly store to the wrong address! Doh!

I wondered how common this error, so I searched online. I could find only one non-trivial wasmtime-py use in the wild, in a sandboxed PDF reader. It falls into the negative pointer trap as I expected. Not only that, it’s a buffer overflow into Python’s memory space:

            buf_ptr = malloc(store, len(pdf_data))
            mem_data = memory.data_ptr(store)

            for i, byte in enumerate(pdf_data):
                mem_data[buf_ptr + i] = byte

The data_ptr method returns a non-bounds-checked raw ctypes pointer, so this is actually a double mistake. First, it shouldn’t trust pointers coming out of Wasm if it cares at all about sandboxing. The second is the potential negative pointer, which in this case would write outside of the Wasm memory and in Python’s memory, hopefully seg-faulting.

What’s one to do? Every pointer coming out of Wasm must be truncated with a mask:

pointer = malloc(...) & 0xffffffff   # correct for wasm32!

This interprets the result as unsigned. 64-bit Wasm needs a 64-bit mask, though in practice you will never get a valid negative pointer from 64-bit Wasm. This rule applies to JavaScript as well, where the idiom is:

let pointer = malloc(...) >>> 0

Wasm runtimes cannot help — they lack the necessary information — and this is perhaps a fundamental flaw in Wasm’s design. Once you know about it you see this mistake happening everywhere.

Now that you have a proper address, you can apply it to a buffer protocol view of memory. If you’re using NumPy there are various ways to interact with this memory by wrapping it in NumPy types, though only if you’re on a little endian host. (If you’re on a big endian machine, just give up on running Wasm anyway.) The first use case I have in mind typically involves copying plain Python values in and out. The struct package is quite handy here:

vec2   = malloc(...) & 0xffffffff
memory = exports["memory"].get_buffer_ptr(store)
struct.pack_into("<ii", memory, vec2, x, y)

It fills a similar role to JavaScript DataView. If you’re copying lots of numbers, with CPython it’s faster to construct a custom format string rather than use a loop:

nums: list[int] = ...
struct.pack_into(f"<{len(nums)}i", memory, buf, *nums)

To copy structures back out, use struct.unpack_from. If you’re moving strings, you’ll need to .encode() and .decode() to convert to and from bytes, which are well-suited to read and write.

In practice with real Wasm programs you’re going to be interacting with the “guest” allocator from the outside, to request memory into which you copy inputs for a function. In my examples I’ve used malloc because it requires no elaboration, but as usual a bump allocator solves this so much better, especially because it doesn’t require stuffing a whole general purpose allocator inside the Wasm program. Have one global arena — no other threads will sharing that Wasm instance — rapid fire a bunch of allocations as needed without any concern for memory management in the “host”, call the function, which might allocate a result from that arena, then reset the arena to clean up. In essence a stack for passing values in and out.

WebAssembly as faster Python

Suppose we noticed a computational hot spot in our Python program in a pure Python function (e.g. not calling out to an extension). Optimizing this function would be wise. Based on my experiments if I re-implement that function in C, compile it to Wasm, then run that bit of Wasm in place of the original function, I can expect around a 10x speed-up. In general C is more like 100x faster than Python, and the overhead of interfacing with Wasm — copying stuff in and out, etc. — can be high, but not so high as to not be profitable. This improves further if I can change the interface, e.g. require callers to use the buffer protocol.

Thanks to wasmtime-py, I could introduce this change without fussing with cross-compilers to build distribution binaries, nor require a toolchain on the target, just a hefty Python package. Might be worth it.

My main experimental benchmark is a variation on my solution to the “Two Sum” problem, which I originally wrote for JavaScript, then extended to pywasm3 and later wasmtime-py. It’s simple, just interesting enough, and representative of the sort of Wasm drop-in I have in mind. It has the same interface, but implements it with Wasm.

# Original Pythonic interface
def twosum(nums: list[int], target: int) -> tuple[int, int] | None:
    ...

# Stateful Wasm interface
class TwoSumWasm():
    def __init__(self):
        store    = wasmtime.Store()
        module   = wasmtime.Module.from_file(store.engine, ...)
        instance = wasmtime.Instance(store, module, ())
        ...

    def twosum(self, nums, target):
        # ... use wasm instance ...

There’s some state to it with the Wasm instance in tow. If you hide that by making it global you’ll need to synchronize your threads around it. In a multi-threaded program perhaps these would be lazily-constructed thread locals. I haven’t had to solve this yet.

However, the weakness of the wasmtime “store” really shows: Notice how compilation and instantiation are bound together in one store? I cannot compile once and then create disposable instances on the fly, e.g. as required for each run of a WASI program. Every instance permanently extends the compilation store. In practice we must wastefully re-compile the Wasm program for each disposable instance. Despite appearances, compilation and instantiation are not actually distinct steps, as they are in JavaScript’s Wasm API. wasmtime.Instance accepts a store as its first argument, suggesting use of a different store for instantiation. That would solve this problem, but as of this writing it must be the same store used to compile the module. This is a fatal flaw for certain real use cases, particularly WASI.

Update: Wolfgang Meier points out the serialize and deserialize methods, which detaches a compiled module from its store, allowing for independent instantations. I tried it, and it’s a practical workaround. Overhead is low; no validation when deserializing. My benchmark now does it for future reference, as I expect it to be my typical use case.

WebAssembly as embedded capabilities

Loup Vaillant’s Monocypher is a wonderful cryptography library. Lean, efficient, and embedding-friendly, so much so it’s distributed in amalgamated form. It requires no libc or runtime, so we can compile it straight to Wasm with almost any Clang toolchain:

$ clang --target=wasm32 -nostdlib -O2 -Wl,--no-entry -Wl,--export-all
        -o monocypher.wasm monocypher.c

It’s not “Wasm-aware” so I need --export-all to expose the interface. This is swell because, as single translation unit, anything with external linkage is the interface. Though remember what I said about interacting with the guest allocator? This has no allocator, nor should it. It’s not so usable in this form because we’d need to manage memory from the outside. Do-able, but it’s easy to improve by adding a couple more functions, sticking to a single translation unit:

#include "monocypher.c"

extern char  __heap_base[];
static char *heap_used;
static char *heap_high;

void *bump_alloc(ptrdiff_t size)
{
    // ...
}

void bump_reset()
{
    ptrdiff_t len = heap_used - __heap_base;
    __builtin_memset(__heap_base, 0, len);  // wipe keys, etc.
    heap_used = __heap_base;
}

I’ve discussed __heap_base before, which is part of the ABI. We’ll push keys, inputs, etc. onto this “stack”, run our cryptography routine, copy out the result, then reset the bump allocator, which wipes out all sensitive data. Often memset is insufficient — typically it’s zero-then-free, and compilers see the lifetime about to end — but no lifetime ends here, and stores to this “heap” memory externally observable as far as the abstract machine can tell. (Otherwise we couldn’t reliably copy out our results!)

There’s a lot to this API, but I’m only going to look at the AEAD interface. We “lock” up some data in an encrypted box, write any unencrypted label we’d like on the outside. Then later we can unlock the box, which will only open for us if neither the contents of the box nor the label were tampered with. That’s some solid API design:

void crypto_aead_lock(uint8_t       *cipher_text,
                      uint8_t        mac  [16],
                      const uint8_t  key  [32],
                      const uint8_t  nonce[24],
                      const uint8_t *ad,         size_t ad_size,
                      const uint8_t *plain_text, size_t text_size);
int crypto_aead_unlock(uint8_t       *plain_text,
                       const uint8_t  mac  [16],
                       const uint8_t  key  [32],
                       const uint8_t  nonce[24],
                       const uint8_t *ad,          size_t ad_size,
                       const uint8_t *cipher_text, size_t text_size);

By compiling to Wasm we can access this functionality from Python almost like it was pure Python, and interact with other systems using Monocypher.

Since Monocypher does not interact with the outside world on its own, it relies on callers to use their system’s CSPRNG to create those nonces and keys, which we’ll do using the secrets built-in package:

class Monocypher:
    def __init__(self):
        ...
        self._read   = functools.partial(memory.read, store)
        self._write  = functools.partial(memory.write, store)
        self.__alloc = functools.partial(exports["bump_alloc"], store)
        self._reset  = functools.partial(exports["bump_reset"], store)
        self._lock   = functools.partial(exports["crypto_aead_lock"], store)
        self._unlock = functools.partial(exports["crypto_aead_unlock"], store)
        self._csprng = secrets.SystemRandom()

    def _alloc(self, n):
        return self.__alloc(n) & 0xffffffff

    def generate_key(self):
        return self._csprng.randbytes(32)

    def generate_nonce(self):
        return self._csprng.randbytes(24)

    ...

With a solid foundation, all that follows comes easily. A finally guarantees secrets are always removed from Wasm memory, and the rest is just about copying bytes around:

    def aead_lock(self, text, key, ad = b""):
        assert len(key) == 32
        try:
            macptr   = self._alloc(16)
            keyptr   = self._alloc(32)
            nonceptr = self._alloc(24)
            adptr    = self._alloc(len(ad))
            textptr  = self._alloc(len(text))

            self._write(key, keyptr)
            nonce = self.generate_nonce()
            self._write(nonce, nonceptr)
            self._write(ad,    adptr)
            self._write(text,  textptr)

            self._lock(
                textptr,
                macptr,
                keyptr,
                nonceptr,
                adptr, len(ad),
                textptr, len(text),
            )
            return (
                self._read(macptr, macptr+16),
                nonce,
                self._read(textptr, textptr+len(text)),
            )
        finally:
            self._reset()

And aead_unlock is basically the same in reverse, but throws if the box fails to unlock, perhaps due to tampering:

    def aead_unlock(self, text, mac, key, nonce, ad = b""):
        assert len(mac) == 16
        assert len(key) == 32
        assert len(nonce) == 24
        try:
            macptr   = self._alloc(16)
            keyptr   = self._alloc(32)
            nonceptr = self._alloc(24)
            adptr    = self._alloc(len(ad))
            textptr  = self._alloc(len(text))

            self._write(mac, macptr)
            self._write(key, keyptr)
            self._write(nonce, nonceptr)
            self._write(ad, adptr)
            self._write(text, textptr)

            if self._unlock(
                textptr,
                macptr,
                keyptr,
                nonceptr,
                adptr, len(ad),
                textptr, len(text),
            ):
                raise ValueError("AEAD mismatch")
            return self._read(textptr, textptr+len(text))
        finally:
            self._reset()

Usage:

mc = Monocypher()
key = mc.generate_key()
message = "Hello, world!"
mac, nonce, encrypted = mc.aead_lock(message.encode(), key)

Transmit mac, nonce, and encrypted to the other party (or your future self), who already has the key:

decrypted = mc.aead_unlock(encrypted, mac, key, nonce)

Find the complete source in my scratch repository.

While I have a few reservations about wasmtime-py, it fascinates me how well this all works. It’s been my hammer in search of a nail for some time now.

Linked list basics

For the sake of an interesting example, I’m will demonstrate with the same concept as last time I talked about data structures: a collection of key/value strings, in the form of an environment variables. This time in linked list form:

typedef struct {
    char     *data;
    ptrdiff_t len;
} Str;

uint64_t hash64(Str);
bool     equals(Str, Str);

typedef struct Env Env;
struct Env {
    Env *next;
    Str  key;
    Str  value;
};

It will be sourced from some string, formatted like the env program:

    Str input = S(
        "EDITOR=vim\n"
        "HOME=/home/user\n"
        "PATH=/bin:/usr/bin\n"
        "SHELL=/bin/bash\n"
        "TERM=xterm-256color\n"
        "USER=user\n"
        "SHELL=/bin/sh\n"   // <- repeated entry
    );

And all the parser heavy lifting will be done by our ever-handy cut function:

typedef struct {
    Str tail;
    Str head;
} Cut;

Cut cut(Str, char);

The simplest way to build up a linked list is like a stack, pushing objects into the front. Zero-initialized head pointer, point the new node at it, then make that node the new head element:

Env *parse_reversed(Str s, Arena *a)
{
    Env *head = 0;  // 1
    for (Cut line = {s}; line.tail.len;) {
        line = cut(line.tail, '\n');
        Cut  pair  = cut(line.head, '=');
        Env *env   = new(a, 1, Env);
        env->key   = pair.head;
        env->value = pair.tail;
        env->next  = head;  // 2
        head = env;  // 3
    }
    return head;
}

That’s it, a complete linked list implementation in three lines of code. No big deal. Because of the bump allocator, nodes are packed in order in memory, so the usual cache objections for linked lists do not apply. LIFO semantics mean the linked list is in reverse order from the source order. If we’re doing a linear scan through the linked list, the last entry in the source wins, which may be what you wanted:

Str lookup_linear(Env *env, Str key)
{
    for (Env *var = env; var; var = var->next) {
        if (equals(key, var->key)) {
            return var->value;
        }
    }
    return (Str){};
}

    // ...
    Env *env  = parse_reversed(input, &scratch);
    Str value = lookup_linear(env, S("SHELL"));  // <- "/bin/sh"

It’s just one more line of code to maintain the original order, using a very simple double-pointer technique:

Env *parse_ordered(Str s, Arena *a)
{
    Env  *head = 0;  // 1
    Env **tail = &head;  // 2
    for (Cut line = {s}; line.tail.len;) {
        // ...
        *tail = env;  // 3
        tail = &env->next;  // 4
    }
    return head;
}

No branches necessary, nor dummy nodes. A pointer to the last pointer in the list works even for empty lists. The tail pointer is unneeded once the list is complete. This form has queue behavior.

Faster look-up with a tree

If you’re doing many look-ups, or if the list is long, those linear scans to find items in the list are not ideal. We can introduce an intrusive hash map, in the form of a hash trie, by adding two more pointers to the linked list:

typedef struct Env Env;
struct Env {
    Env *next;
    Env *child[2];  // <- hash map linkage
    Str  key;
    Str  value;
};

I’ve found it’s simplest to construct a node into the hash map, then link it onto the list tail. That constructor looks like this:

Env *new_env(Arena *a, Env **env, Str key, Str value)
{
    for (uint64_t h = hash64(key); *env; h <<= 1) {
        env = &(*env)->child[h>>63];
    }
    *env = new(a, 1, Env);
    (*env)->key = key;
    (*env)->value = value;
    return *env;
}

Then we swap that into the head/tail version in place of the original new macro call:

Env *parse_mapped(Str s, Arena *a)
{
    Env  *head = 0;
    Env **tail = &head;
    for (Cut line = {s}; line.tail.len;) {
        // ...
        Env *env = new_env(a, &head, pair.head, pair.tail);
        *tail = env;
        tail = &env->next;
    }
    return head;
}

This is now a linked list and a hash map at the same time, built-up piece by piece without any resizing. We still have the original linked list, but we can now search it in log time. The look-up function resembles the constructor:

Str lookup_logn(Env *env, Str key)
{
    for (uint64_t h = hash64(key); env; h <<= 1) {
        if (equals(key, env->key)) {
            return env->value;
        }
        env = env->child[h>>63];
    }
    return (Str){};
}

Because of the FIFO semantics, it finds the first match in the source:

    Env *env   = parse_mapped(input, &scratch);
    Str  value = lookup_logn(env, S("SHELL"));  // <- /bin/bash

The other matches are also in the tree, and we can find those as well by continuing traversal. That is, it’s already a multi-map. This particular interface can’t pick up where it left off, but we can build one that does using an iterator/cursor:

typedef struct {
    uint64_t hash;
    Str      key;
    Env     *env;
} EnvIter;

EnvIter new_enviter(Env *env, Str key)
{
    return (EnvIter){hash64(key), key, env};
}

Str enviter_next(EnvIter *it)
{
    while (it->env) {
        Env *cur = it->env;
        it->env = it->env->child[it->hash>>63];
        it->hash <<= 1;
        if (equals(it->key, cur->key)) {
            return cur->value;
        }
    }
    return (Str){};
}

Update: Thanks to Daniel Kareh for a correction.

Then we can use a loop to visit every match in source order:

    Env *env = parse_mapped(input, &scratch);
    for (EnvIter it = new_enviter(env, S("SHELL"));;) {
        Str value = enviter_next(&it);
        if (!value.data) break;
        // ...
    }

Faster look-up with an index table

If the list is static once constructed, or if look-ups happen much more frequently than the list grows, we can find list items even faster by constructing an index table over the list: an MSI hash table. This table avoids redundancy by sharing structure with the list. Because it’s a flat table, if we keep adding to the list then eventually we’ll need to reconstruct a larger table when it becomes overloaded.

The table itself has a very simple structure, just an array and its size, expressed as a power-of-two exponent:

typedef struct {
    Env **slots;
    int   exp;
} EnvTable;

We do not need the child nodes, and so linked list nodes are untouched. That is, it’s not intrusive. In fact, we can build any arbitrary number of tables over a list, perhaps indexing different properties for different sorts of queries. The idea is that we build the list first, then create the table:

EnvTable new_table(Arena *a, Env *env)
{
    // Compute list length
    ptrdiff_t len = 0;
    for (Env *var = env; var; var = var->next) {
        len++;
    }

    // Then compute an appropriate table size
    EnvTable table = {};
    table.exp = 3;
    ptrdiff_t one = 1;
    for (; (one<<table.exp) - (one<<(table.exp-3)) < len; table.exp++) {}
    table.slots = new(a, one<<table.exp, Env *);

    // Then insert linked list items into the table
    for (Env *var = env; var; var = var->next) {
        uint64_t hash = hash64(var->key);
        size_t   mask = ((size_t)1 << table.exp) - 1;
        size_t   step = (size_t)(hash >> (64 - table.exp)) | 1;
        for (size_t i = (size_t)hash;;) {
            i = (i + step) & mask;
            if (!table.slots[i]) {
                table.slots[i] = var;
                break;
            }
        }
    }

    return table;
}

Note how only searches for an empty slot, not for a matching entry. That’s because this too is a multi-map, also with elements in insertion order. Look-ups are constant time:

Str lookup_constant(EnvTable table, Str key)
{
    uint64_t hash = hash64(key);
    size_t   mask = ((size_t)1 << table.exp) - 1;
    size_t   step = (size_t)(hash >> (64 - table.exp)) | 1;
    for (size_t i = (size_t)hash;;) {
        i = (i + step) & mask;
        if (!table.slots[i]) {
            return (Str){};
        } else if (equals(table.slots[i]->key, key)) {
            return table.slots[i]->value;
        }
    }
}

It finds the earliest match in the list, meaning an index over the “reverse” list will find the last entry in the source. The indexed-over property is the input to hash64 and equals. By using a different input to these functions we could build another table on, say, value length if that’s a property on which we needed to find elements efficiently. Again, for multi-map iteration we need some kind of iterator or cursor:

typedef struct {
    EnvTable table;
    Str      key;
    size_t   step;
    size_t   i;
} TableIter;

TableIter new_tableiter(EnvTable table, Str key)
{
    uint64_t hash = hash64(key);
    size_t   step = (size_t)(hash >> (64 - table.exp)) | 1;
    size_t   idx  = (size_t)hash;
    return (TableIter){table, key, step, idx};
}

Str table_next(TableIter *it)
{
    size_t mask  = ((size_t)1 << it->table.exp) - 1;
    Env  **slots = it->table.slots;
    for (;;) {
        it->i = (it->i + it->step) & mask;
        if (!slots[it->i]) {
            return (Str){};
        } else if (equals(slots[it->i]->key, it->key)) {
            return slots[it->i]->value;
        }
    }
}

Its usage looks just like the other multi-map:

    Env *env = parse_ordered(input, &scratch);
    EnvTable table = new_table(&scratch, env);
    for (TableIter it = new_tableiter(table, S("SHELL"));;) {
        Str value = table_next(&it);
        if (!value.data) break;
        // ...
    }

With these techniques at hand, I can start with linked lists when they are convenient, and later add needed features without fundamentally changing the underlying data structure. None of this requires runtime support, and so it fits comfortably on embedded systems, tiny WebAssembly programs, etc. All the above code is available ready to run: list.c.

I continue to title this “speculations” because, unlike arenas in C, I have not (yet?) put these C++ techniques into practice in real software. I haven’t refined them through use. Even ignoring its standard library as I do here, C++ is an enormously complex programming language — far more so than C — and I’m less confident that I’m not breaking a rule by accident. I only want to break rules with intention!

As a reminder here’s where we left things off:

struct Arena {
    char *beg;
    char *end;
};

template<typename T>
T *raw_alloc(Arena *a, ptrdiff_t count = 1)
{
    ptrdiff_t size = sizeof(T);
    ptrdiff_t pad  = -(uintptr_t)a->beg & (alignof(T) - 1);
    if (count >= (a->end - a->beg - pad)/size) {
        throw std::bad_alloc{};  // OOM policy
    }
    void *r = a->beg + pad;
    a->beg += pad + count*size;
    return new(r) T[count]{};
}

I used throw when out of memory mainly to emphasize that this works, but you’re free to pick whatever is appropriate for your program. Remember, that’s the entire allocator, including implicit deallocation, sufficient to fulfill the allocation needs for most programs, though they must be designed for it. Also note that it’s now raw_alloc, as we’ll be writing a new, enhanced alloc that builds upon this one.

Also a reminder on usage, I’ll draw on an old example, updated for C++:

wchar_t   *towidechar(Str, Arena *);   // convert to UTF-16
Str        slurpfile(wchar_t *path);   // read an entire file
Slice<Str> split(Str, char, Arena *);  // split on delimiter

Slice<Str> getlines(Str path, Arena *perm, Arena scratch)
{
    // Use scratch for path conversion, auto-free on return
    wchar_t *wpath = towidechar(path, &scratch);

    // Use perm for file contents, which are returned
    Str buf = slurpfile(wpath, perm);

    // Use perm for the slice, pointing into buf
    return split(buf, '\n', perm);
}

Changes to scratch do not persist after getlines returns, so objects allocated from that arena are automatically freed on return. So far this doesn’t rely on C++ RAII features, just simple value semantics. It works well because all the objects in question have trivial destructors. But suppose there’s a resource to manage:

struct TcpSocket {
    int socket = ::socket(AF_INET, SOCK_STREAM, 0);
    TcpSocket() = default;
    TcpSocket(TcpSocket &) = delete;
    void operator=(TcpSocket &) = delete;
    // TODO: move ctor/operator
    ~TcpSocket() { if (socket >= 0) close(socket); }
    operator int() { return socket; }
};

If we allocate a TcpSocket in an arena, including as a member of another object, the destructor will never run unless we call it manually. To deal with this we’ll need to keep track of objects requiring destruction, which we’ll do with a linked list of destructors, forming a LIFO stack:

struct Dtor {
    Dtor     *next;
    void     *objects;
    ptrdiff_t count;
    void     (*dtor)(void *objects, ptrdiff_t count);
};

Each Dtor points to a homogeneous array, a count (typically one), and a pointer to a function that knows how to destroy these objects. The linked list itself is heterogeneous, with dynamic type. The function pointer is like a kind of type tag. The dtor functions will be generated using a template function:

template<class T>
void destroy(void *ptr, ptrdiff_t count)
{
    T *objects = (T *)ptr;
    for (ptrdiff_t i = count-1; i >= 0; i--) {
        objects[i].~T();
    }
}

Notice it destroys end-to-beginning, in reverse order that these objects would be instantiated by placement new[]. It’s essentially a placement delete[]. An arena initializes with an empty list of Dtors as a new member:

struct Arena {
    char *beg;
    char *end;
    Dtor *dtors = 0;

    // ...

};

There are two different ways to construct an arena: over a block of raw memory (unowned), or from an existing arena to borrow a scratch arena over its free space. So that’s two constructors:

struct Arena {
    // ...

    Arena(char *mem, ptrdiff_t len) : beg{mem}, end{mem+len} {}
    Arena(Arena &a) : beg{a.beg}, end{a.end} {}

    // ...
};

Finally a destructor that pops the Dtor linked list until empty, which runs the destructors in reverse order when the arena is destroyed:

struct Arena {
    // ...

    void operator=(Arena &) = delete;  // rule of three

    ~Arena()
    {
        while (dtors) {
            Dtor *dead = dtors;
            dtors = dead->next;
            dead->dtor(dead->objects, dead->count);
        }
    }
};

(Note: This should probably use a local variable instead of manipulating the dtors member directly. Updates to dtors are potentially visible to destructors, inhibiting optimization.) The new, enhanced alloc building upon raw_alloc:

template<typename T>
T *alloc(Arena *a, ptrdiff_t count = 1)
{
    if (__has_trivial_destructor(T) || !count) {
        return raw_alloc<T>(a, count);
    }

    Dtor *dtor    = raw_alloc<Dtor>(a);  // allocate first
    T    *r       = raw_alloc<T>(a, count);
    dtor->next    = a->dtors;
    dtor->objects = r;
    dtor->count   = count;
    dtor->dtor    = destroy<T>;

    a->dtors = dtor;
    return r;
}

I’m using the non-standard __has_trivial_destructor built-in supported by all major C++ implementations, meaning we still don’t need the C++ standard library, but std::is_trivially_destructible is the usual tool here. LLVM is pushing __is_trivially_destructible instead, but it’s not supported by GCC until GCC 16.

Since it’s so simple to do it, if the count is zero then it doesn’t care about non-trivial destruction, as there’s nothing to destroy. Things get more interesting for a non-zero number of non-trivially destructible objects. First allocate a Dtor, important because failing to allocate it second would cause a leak (no Dtor entry in place). Then allocate the array, attach it to the Dtor, attach the Dtor to the arena, registering the objects for cleanup.

If a constructor throws, placement new[] will automatically destroy objects that have been created so far — i.e. the real placement delete[] — before returning, so that case was already covered at the start.

With a little more cleverness we could omit the objects pointer and discover the array using pointer arithmetic off the Dtor object itself. That’s tricky (consider alignment), and generally unnecessary, so I didn’t worry about it. With arenas, allocator overhead is already well below that of conventional allocation, so slack is plentiful. Chances are we will also never need an array of non-trivially destructible objects, and so we could probably omit count, then write a single-object allocator that forwards constructor arguments (e.g. a handles to the resource to be managed). That involves no new concepts, and I leave it as an exercise for the reader.

With that in place, we could now allocate an array of TcpSockets:

void example(Arena scratch)
{
    TcpSocket *sockets = alloc<TcpSocket>(&scratch, 100);
    // ...
}

These sockets will all be closed when example exits via their singular Dtor entry on scratch. When calling this example with an arena:

void caller(Arena *perm)
{
    example(*perm);  // creates a scratch arena
    // ...
}

This invokes the copy constructor, creating a scratch arena with an empty dtors list to be passed into example. Objects existing in *perm will not be destroyed by example because dtors isn’t passed in. If we had passed a pointer to an arena, the Arena constructor isn’t invoked, so the callee uses the caller’s arena, pushing its Dtors onto the callee’s list.

In other words, the interface hasn’t changed! That’s the most exciting part for me. This by-copy, by-pointer interfacing has really grown on me the past two years.

Patrice Roy’s new book, C++ Memory Management, has made me more conscious of object lifetimes. C++ is stricter than C about lifetimes, and common, textbook memory management that’s sound in C is less so in C++ — more than I realized. The book also presents a form of arena allocation so watered down as to enjoy none of the benefits. (Despite its precision otherwise, the second half is also littered with integer overflows lacking the appropriate checks, and near the end has some pointer overflows invalidating the check.) However, I’m grateful for the new insights, and it’s made me revisit my own C++ arena allocation. In this new light I see I got it subtly wrong myself!

Surprising to most C++ programmers, but not language lawyers, idiomatic C memory allocation was ill-formed in C++ until recently:

int *newint(int v)
{
    int *r = (int *)malloc(sizeof(*r));
    if (r) {
        *r = v;  // <-- undefined behavior before C++20
    }
    return r;
}

This program allocates memory for an object but never starts a lifetime. Assignment without a lifetime is invalid. Pointer casts are that much more suspicious in C++, and due to lifetime semantics, in many cases indicate incorrect code. (To be clear, I’m not arguing in favor of these semantics, but reasoning about the facts on the ground.) C++20 carved out special exceptions for malloc and friends, but addressing this kind of thing in general is the purpose of the brand new start_lifetime_as (and similar), the slightly older construct_at, or a classic placement new. They all start lifetimes. The last looks like:

int *newint(int v)
{
    void *r = malloc(sizeof(int));
    if (r) {
        return new(r) int{v};
    }
    return nullptr;
}

That’s no good as a C/C++ polyglot, though per the differing old semantics that was impossible anyway without macros. Which is basically cheating. An important detail: The corrected version has no casts, and it returns the result of new. That’s important because only the pointer returned by new is imbued as a pointer to the new lifetime, not r. There are no side effects affecting the provenance of r, which still points to raw memory as far as the language is concerned.

With that in mind let’s revisit my arena from last time, which does not necessarily benefit from the recent changes, not being one of the special case C standard library functions:

struct Arena {
    char *beg;
    char *end;
};

template<typename T>
T *alloc(Arena *a, ptrdiff_t count = 1)
{
    ptrdiff_t size = sizeof(T);
    ptrdiff_t pad  = -(uintptr_t)a->beg & (alignof(T) - 1);
    assert(count < (a->end - a->beg - pad)/size);  // OOM policy
    T *r = (T *)(a->beg + pad);
    a->beg += pad + count*size;
    for (ptrdiff_t i = 0; i < count; i++) {
        new((void *)&r[i]) T{};
    }
    return r;
}

Hey, look, placement new! I did that to produce a nicer interface, but I lucked out also starting lifetimes appropriately. Except it returns the wrong pointer. This allocator discards the pointer blessed with the new lifetime. Both pointers have the same address but different provenance. That matters. But I’m calling new many times, so how do I fix this? Array new, duh.

template<typename T>
T *alloc(Arena *a, ptrdiff_t count = 1)
{
    ptrdiff_t size = sizeof(T);
    ptrdiff_t pad  = -(uintptr_t)a->beg & (alignof(T) - 1);
    assert(count < (a->end - a->beg - pad)/size);  // OOM policy
    void *r = a->beg + pad;
    a->beg += pad + count*size;
    return new(r) T[count]{};
}

Wow… that’s actually much better anyway. No explicit casts, no loop. Why didn’t I think of this in the first place? The catch is I can’t forward constructor arguments, emplace-style — the part that gave me the trouble with perfect forwarding — but that’s for the best. Forwarding more than once was unsound, made more obvious by new[].

Caveat: This only works starting in C++20, and strictly with operator new[](size_t, void *). Any other placement new[] may require array overhead — e.g. it prepends an array size so that delete[] can run non-trivial destructors — which is unknowable and therefore impossible to provide or align correctly. Overhead for placement new[] is nonsense, of course, but as of this writing, all three major C++ compilers do it and essentially have broken custom placement new[].

Since I’m thinking about lifetimes, what about the other end? My arena does not call destructors, by design, and starts new lifetimes on top of objects that are technically still alive. Is that undefined behavior? As far as I can tell this is allowed, even for non-trivial destructors, with the caveat that it might leak resources. In this case the resource is memory managed by the arena, so that’s fine of course.

So addressing pointer provenance also produced a nicer definition. What a great result from reading that book! While researching, I noticed Jonathan Müller, who personally gave me great advice and feedback on my previous article, talked about lifetimes just a couple weeks later. I recommend both.

The above description is probably confusing on its own, and an example is worth a thousand words, so here’s a listing naming 7 fields:

timestamp,point.x,point.y,foo.bar.z,point.z,foo.bar.y,foo.bar.x

Where point is a substructure, as is foo and bar, but note they’re interleaved in the record. So if a record contains these values:

{1758158348, 1.23, 4.56, -100, 7.89, -200, -300}

The JSON representation would look like:

{
  "timestamp": 1758158348,
  "point": {
    "x": 1.23,
    "y": 4.56,
    "z": 7.89
  },
  "foo": {
    "bar": {
      "z": -100,
      "y": -200,
      "x": -300
    }
  }
}

Notice point.z moved up and foo.bar.z down, so that substructures are contiguous in this representation as required for JSON. Sorting the field names lexicographically would group them together as a simple solution. However, as an additional constraint I want to retain the original field order as much as possible. For example, timestamp is first in both the original and JSON representations, but sorting would put it last. If all substructures are already contiguous, nothing should change.

Solution with string interning

My solution is to intern the segment strings, assigning each a unique, monotonic integral token in the order they’re observed. In my program, zero is reserved as a special “root” token, and so the first string has the value 1. The concrete values aren’t important, only that they’re assigned monotonically.

The trick is that a string is always interned in the “namespace” of a previous token. That is, we’re building a (token, string) -> token map. For our segments that namespace is the token for the parent structure, and the top-level fields are interned in the reserved “root” namespace. When applied to the example, we get the token sequences:

timestamp  -> 1
point.x    -> 2 3
point.y    -> 2 4
foo.bar.z  -> 5 6 7
point.z    -> 2 8
foo.bar.y  -> 5 6 9
foo.bar.x  -> 5 6 10

And our map looks like:

{0, "timestamp"} -> 1
{0, "point"}     -> 2
{2, "x"}         -> 3
{2, "y"}         -> 4
{0, "foo"}       -> 5
{5, "bar"}       -> 6
{6, "z"}         -> 7
{2, "z"}         -> 8
{6, "y"}         -> 9
{6, "x"}         -> 10

Notice how "x" is assigned 3 and 10 due to different namespaces. That’s important because otherwise the fields of foo.bar would sort in the same order as point. Namespace gives these fields unique identities.

Once we have the token representation, sort lexicographically by token. That pulls point.z up to its siblings.

timestamp  -> 1
point.x    -> 2 3
point.y    -> 2 4
point.z    -> 2 8
foo.bar.z  -> 5 6 7
foo.bar.y  -> 5 6 9
foo.bar.x  -> 5 6 10

Now we have the “output” order with minimal re-ordering. If substructures were already contiguous, nothing changes. Assuming a reasonable map, this is O(n log n), primarily due to sorting.

Alternatives

Before I thought of namespaces, my initial idea was to intern the whole prefix of a segment. The sequence of look-ups would be:

"timestamp"    -> 1  -> {1}
"point"        -> 2
"point.x"      -> 3  -> {2, 3}
"point"        -> 2
"point.y"      -> 4  -> {2, 4}
"foo"          -> 5
"foo.bar"      -> 6
"foo.bar.z"    -> 7  -> {5, 6, 7}
"point"        -> 2
"point.z"      -> 8  -> {2, 8}
"foo"          -> 5
"foo.bar"      -> 6
"foo.bar.y"    -> 9  -> {5, 6, 9}
"foo"          -> 5
"foo.bar"      -> 6
"foo.bar.x"    -> 10 -> {5, 6, 10}

Ultimately it produces the same tokens, and this is a more straightforward string -> string map. The prefixes are acting as namespaces. However, I wrote it this way as a kind of visual proof: Notice the right triangle shape formed by the strings for each field. From the area we can see that processing prefixes as strings is O(n^2) quadratic time on the number of segments! In my real problem the inputs were never large enough for this to matter, but I hate leaving behind avoidable quadratic algorithms. Using a token as a namespace flattens the prefix to a constant size.

Another option is a different map for each namespace. So for foo.bar.z lookup the "foo" map (string -> map) in the root (string -> map), then within that lookup the "bar" table (string -> token) (since this is the penultimate segment), then intern "z" within that to get its token. That wouldn’t have quadratic time complexity, but it seems quite a bit more complicated than a single, flat (token, string) -> token map.

Implementation in C

Because the standard library has little useful for us, I am building on previously-established definitions, so refer to that article for basic definitions like Str. To start off, tokens will be a size-typed integer so we never need to worry about overflowing the token counter. We’d run out of memory first:

typedef ptrdiff Token;

We’re building a (token, string) -> token) map, so we’ll need a hash function for such keys:

uint64_t hash(Token t, Str s)
{
    uint64_t r = (uint64_t)t << 8;
    for (ptrdiff i = 0; i < s.len; i++) {
        r ^= s.data[i];
        r *= 1111111111111111111u;
    }
    return r;
}

The map itself is a forever-useful hash trie.

typedef struct Map Map;
struct Map {
    Map  *child[4];
    Token namespace;
    Str   segment;
    Token token;
};

Token *upsert(Map **m, Token namespace, Str segment, Arena *a)
{
    for (uint64_t h = hash(ns, segment); *m; h <<= 2) {
        if (namespace==(*m)->namespace && equals(segment, (*m)->segment)) {
            return &(*m)->token;
        }
        m = &(*m)->child[h>>62];
    }
    *m = new(a, 1, Map);
    (*m)->namespace = namespace;
    (*m)->segment = segment;
    return &(*m)->token;  // caller will assign
}

We’ll use this map to convert a string naming a field into a sequence of tokens, so we’ll need a slice. Fields also have an offset within the record and a type, which we’ll track via its original ordering, which I’ll do with an index field (e.g. into the original header). Also track the original name.

typedef struct {
    Str          name;
    ptrdiff_t    index;
    Slice(Token) tokens;
} Field;

To sort fields we’ll need a comparator:

ptrdiff_t field_compare(Field a, Field b)
{
    ptrdiff_t len = min(a.tokens.len, b.tokens.len);
    for (ptrdiff_t i = 0; i < len; i++) {
        Token d = a.tokens.data[i] - b.tokens.data[i];
        if (d) {
            return d;
        }
    }
    return a.tokens.len - b.tokens.len;
}

Because field names are unique, each token sequence is unique, and so we need not use index in the comparator.

Finally down to business: cut up the list and build the token sequences with the established push macro. The sort function isn’t interesting, and could be as simple as libc qsort with the above comparator (and adapter), so I’m only listing the prototype.

void field_sort(Slice(Field), Arena scratch);

Slice(Field) parse_fields(Str fieldlist, Arena *a)
{
    Slice(Field) fields  = {};
    Map         *strtab  = 0;
    ptrdiff_t    ntokens = 0;

    for (Cut c = {.tail=fieldlist, .ok=true}; c.ok;) {
        c = cut(c.tail, ',');
        Field field = {};
        field.name  = c.head;
        field.index = fields.len;

        Token prev = 0;
        for (Cut f = {.tail=field.name, .ok=true}; f.ok;) {
            f = cut(f.tail, '.');
            Token *token = upsert(&strtab, prev, f.head, a);
            if (!*token) {
                *token = ++ntokens;
            }
            *push(a, &field.tokens) = *token;
            prev = *token;
        }

        *push(a, &fields) = field;
    }

    field_sort(fields, *a);
    return fields;
}

Usage here suggests Cut::ok should be inverted to Cut::done so that it better zero-initializes. Something I’ll need to consider. Because it’s all allocated from an arena, no need for destructors or anything like that, so this is the complete implementation. Back to the example:

    Str fieldlist = S(
        "timestamp,"
        "point.x,"
        "point.y,"
        "foo.bar.z,"
        "point.z,"
        "foo.bar.y,"
        "foo.bar.x"
    );
    Slice(Field) fields = parse_fields(fieldlist, &scratch);
    for (ptrdiff_t i = 0; i < fields.len; i++) {
        Str name = fields.data[i].name;
        fwrite(name.data, 1, name.len, stdout);
        putchar('\n');
    }

This program will print the proper output field order. In a real program we’d hold onto the string table, define an inverse lookup to translate tokens back into strings, and use it when in producing output. I do just that in my exploratory program, rec2json.c, written a little differently than presented above. It uses the sorted tokens to compile a simple bytecode program that, when run against a record, produces its JSON representation. It compiles the example to:

OPEN          # print '{'
KEY     1     # print token 1 as a key, i.e. "timestamp:"
READ    0     # print double at record offset 0
COMMA         # print ','
KEY     2     # print token 2 as a key, i.e. "point:"
OPEN
KEY     3
READ    8     # print double at record offset 8
COMMA
KEY     4
READ    16
COMMA
KEY     8
READ    32
CLOSE         # print '}'
COMMA
KEY     5
OPEN
KEY     6
OPEN
KEY     7
READ    24
COMMA
KEY     9
READ    40
COMMA
KEY     10
READ    48
CLOSE
CLOSE
CLOSE

Seeing it written out, I notice more room for improvement. An optimization pass could coalesce instructions so that, for instance, OPEN then KEY concatenate to a single string at compile time so that it only needs one instruction. This program could be 15 instructions instead of 31. In my real case I didn’t need anything quite this sophisticated, but it was fun to explore.