The original program uses ARM64 assembly. I’m a lot more comfortable with x86-64 assembly, plus that’s the hardware I have readily on hand. So the assembly language will be different, but all the concepts apply to both these architectures. Almost none of these OpenBSD system interfaces are formally documented (or stable for that matter), and I had to dig around the OpenBSD source tree to figure it out (along with a helpful jart nudge). So don’t be afraid to get your hands dirty.

There are lots of subtle problems in the original demo, so let’s go through the program piece by piece, starting with the entry point:

void
start()
{
        w("hello\n", 6);
        x();
}

This function is registered as the entry point in the ELF image, so it has no caller. That means no return address on the stack, so the stack is not aligned for a function.(Correction: The stack alignment issue is true for x86, but not ARM, so the original demo is fine.) In toy programs that goes unnoticed, but compilers generate code assuming the stack is aligned. In a real application this is likely to crash deep on the first SIMD register spill.

We could fix this with a force_align_arg_pointer attribute, at least for architectures that support it, but I prefer to write the entry point in assembly. Especially so we can access the command line arguments and environment variables, which is necessary in a real application. That happens to work the same as it does on Linux, so here’s my old, familiar entry point:

asm (
    "        .globl _start\n"
    "_start: mov   %rsp, %rdi\n"
    "        call  start\n"
);

Per the ABI, the first argument passes through rdi, so I pass a copy of the stack pointer, rsp, as it appeared on entry. Entry point arguments argc, argv, and envp are all pushed on the stack at rsp, so the first real function can retrieve it all from just the stack pointer. The original demo won’t use it, though. Using call to pass control pushes a return address, which will never be used, and aligns the stack for the first real function. I name it _start because that’s what the linker expects and so things will go a little smoother, so it’s rather convenient that the original didn’t use this name.

Next up, the “write” function:

int
w(void *what, size_t len) {
        __asm(
"       mov x2, x1;"
"       mov x1, x0;"
"       mov w0, #1;"
"       mov x8, #4;"
"       svc #0;"
        );
        return 0;
}

There are two serious problems with this assembly block. First, the function arguments are not necessarily in those registers by the time control reaches the basic assembly block. The function prologue could move them around. Even more so if this function was inlined. This is exactly the problem extended inline assembly is intended to solve. Second, it clobbers a number of registers. Compilers assume this does not happen when generating their own code. This sort of assembly falls apart the moment it comes into contact with a non-zero optimization level.

Solving this is just a matter of using inline assembly properly:

long w(void *what, long len)
{
    char err;
    long rax = 4;  // SYS_write
    asm volatile (
        "syscall"
        : "+a"(rax), "+d"(len), "=@ccc"(err)
        : "D"(1), "S"(what)
        : "rcx", "r11", "memory"
    );
    return err ? -rax : rax;
}

I’ve enhanced it a bit, returning a Linux-style negative errno on error. In the BSD ecosystem, syscall errors are indicated using the carry flag, which here is output into err via =@ccc. When set, the return value is an errno. Further, the OpenBSD kernel uses both rax and rdx for return values, so I’ve also listed rdx as an input+output despite not consuming the result. Despite all these changes, this function is not yet complete! We’ll get back to it later.

The “exit” function, x, is just fine:

void
x() {
        __asm(
"       mov x8, #1;"
"       svc #0;"
        );
}

It doesn’t set an exit status, so it passes garbage instead, but otherwise this works. No inputs, plus clobbers and outputs don’t matter when control never returns. In a real application I might write it:

__attribute((noreturn))
void x(int status)
{
    asm volatile ("syscall" :: "a"(1), "D"(status));
    __builtin_unreachable();
}

This function will need a little additional work later, too.

The ident section is basically fine as-is:

__asm(" .section \".note.openbsd.ident\", \"a\"\n"
"       .p2align 2\n"
"       .long   8\n"
"       .long   4\n"
"       .long   1\n"
"       .ascii \"OpenBSD\\0\"\n"
"       .long   0\n"
"       .previous\n");

The compiler assumes the current section remains the same at the end of the assembly block, which here is accomplished with .previous. Though it clobbers the assembler’s remembered “other” section and so may interfere with surrounding code using .previous. Better to use .pushsection and .popsection for good stack discipline. There are many such examples in the OpenBSD source tree.

asm (
    ".pushsection .note.openbsd.ident, \"a\"\n"
    ".long  8, 4, 1, 0x6e65704f, 0x00445342, 0\n"
    ".popsection\n"
);

Now the trickiest part, the pinsyscall table:

struct whats {
        unsigned int offset;
        unsigned int sysno;
} happening[] __attribute__((section(".openbsd.syscalls"))) = {
        { 0x104f4, 4 },
        { 0x10530, 1 },
};

Those offsets — offsets from the beginning of the ELF image — were entered manually, and it kind of ruins the whole demo. We don’t have a good way to get at those offsets from C, or any high level language. However, we can solve that by tweaking the inline assembly with some labels:

__attribute((noinline))
long w(void *what, long len)
{
    // ...
    asm volatile (
        "_w: syscall"
        // ...
    );
    // ...
}

__attribute((noinline,noreturn))
void x(int status)
{
    asm volatile (
        "_x: syscall"
        // ...
    );
    // ...
}

Very importantly I’ve added noinline to prevent these functions from being inlined into additional copies of the syscall instruction, which of course won’t be registered. This also prevents duplicate labels causing assembler errors. Once we have the labels, we can use them in an assembly block listing the allowed syscall instructions:

asm (
    ".pushsection .openbsd.syscalls\n"
    ".long  _x, 1\n"
    ".long  _w, 4\n"
    ".popsection\n"
);

That lets the linker solve the offsets problem, which is its main job after all. With these changes the demo works reliably, even under high optimization levels. I suggest these flags:

$ cc -static -nostdlib -no-pie -o where where.c

Disabling PIE with -no-pie is necessary in real applications or else strings won’t work. You can apply more flags to strip it down further, but these are the flags generally necessary to compile these sorts of programs on at least OpenBSD 7.6.

So, how do I know this stuff works in general? Because I ported my ultra portable pkg-config clone, u-config, to use raw OpenBSD syscalls: openbsd_main.c. Everything still works at high optimization levels.

$ cc -static -nostartfiles -no-pie -o pkg-config openbsd_main.c libmemory.a
$ ./pkg-config --cflags --libs libcurl
-I/usr/local/include -L/usr/local/lib -lcurl

Because the new syscall wrappers behave just like Linux system calls, it leverages the linux_noarch.c platform, and the whole port is ~70 lines of code. A few more flags (-fno-stack-protector, -Oz, -s, etc.), and it squeezes into a slim 21.6K static binary.

Despite making no libc calls, it’s not possible stop compilers from fabricating (hallucinating?) string function calls, so the build above depends on external definitions. In the command above, libmemory.a comes from libmemory.c found in w64devkit. Alternatively, and on topic, you could link the OpenBSD libc string functions by omitting libmemory.a from the build.

$ cc -static -nostartfiles -no-pie -o pkg-config openbsd_main.c

Though it pulls in a lot of bloat (~8x size increase), and teasing out the necessary objects isn’t trivial.

Years ago, OpenBSD gained two new security system calls, pledge(2) (originally tame(2)) and unveil. In both, an application surrenders capabilities at run-time. The idea is to perform initialization like usual, then drop capabilities before handling untrusted input, limiting unwanted side effects. This feature is applicable even where type safety isn’t an issue, such as Python, where a program might still get tricked into accessing sensitive files or making network connections when it shouldn’t. So how can a Python program access these system calls?

As discussed previously, it’s quite easy to access C APIs from Python through its ctypes package, and this is no exception. In this article I show how to do it. Here’s the full source if you want to dive in: openbsd.py.

I’ve chosen these extra constraints:

• As extra safety features, unnecessary for correctness, attempts to call these functions on systems where they don’t exist will silently do nothing, as though they succeeded. They’re provided as a best effort.

• Systems other than OpenBSD may support these functions, now or in the future, and it would be nice to automatically make use of them when available. This means no checking for OpenBSD specifically but instead feature sniffing for their presence.

• The interfaces should be Pythonic as though they were implemented in Python itself. Raise exceptions for errors, and accept strings since they’re more convenient than bytes.

For reference, here are the function prototypes:

int pledge(const char *promises, const char *execpromises);
int unveil(const char *path, const char *permissions);

The string-oriented interface of pledge will make this a whole lot easier to implement.

Finding the functions

The first step is to grab functions through ctypes. Like a lot of Python documentation, this area is frustratingly imprecise and under-documented. I want to grab a handle to the already-linked libc and search for either function. However, getting that handle is a little different on each platform, and in the process I saw four different exceptions, only one of which is documented.

I came up with passing None to ctypes.CDLL, which ultimately just passes NULL to dlopen(3). That’s really all I wanted. Currently on Windows this is a TypeError. Once the handle is in hand, try to access the pledge attribute, which will fail with AttributeError if it doesn’t exist. In the event of any exception, just assume the behavior isn’t available. If found, I also define the function prototype for ctypes.

_pledge = None
try:
    _pledge = ctypes.CDLL(None, use_errno=True).pledge
    _pledge.restype = ctypes.c_int
    _pledge.argtypes = ctypes.c_char_p, ctypes.c_char_p
except Exception:
    _pledge = None

Catching a broad Exception isn’t great, but it’s the best we can do since the documentation is incomplete. From this block I’ve seen TypeError, AttributeError, FileNotFoundError, and OSError. I wouldn’t be surprised if there are more possibilities, and I don’t want to risk missing them.

Note that I’m catching Exception rather than using a bare except. My code will not catch KeyboardInterrupt nor SystemExit. This is deliberate, and I never want to catch these.

The same story for unveil:

_unveil = None
try:
    _unveil = ctypes.CDLL(None, use_errno=True).unveil
    _unveil.restype = ctypes.c_int
    _unveil.argtypes = ctypes.c_char_p, ctypes.c_char_p
except Exception:
    _unveil = None

Pythonic wrappers

The next and final step is to wrap the low-level call in an interface that hides their C and ctypes nature.

Python strings must be encoded to bytes before they can be passed to C functions. Rather than make the caller worry about this, we’ll let them pass friendly strings and have the wrapper do the conversion. Either may also be NULL, so None is allowed.

def pledge(promises: Optional[str], execpromises: Optional[str]):
    if not _pledge:
        return  # unimplemented

    r = _pledge(None if promises is None else promises.encode(),
                None if execpromises is None else execpromises.encode())
    if r == -1:
        errno = ctypes.get_errno()
        raise OSError(errno, os.strerror(errno))

As usual, a return of -1 means there was an error, in which case we fetch errno and raise the appropriate OSError.

unveil works a little differently since the first argument is a path. Python functions that accept paths, such as open, generally accept either strings or bytes. On unix-like systems, paths are fundamentally bytestrings and not necessarily Unicode, so it’s necessary to accept bytes. Since strings are nearly always more convenient, they take both. The unveil wrapper here will do the same. If it’s a string, encode it, otherwise pass it straight through.

def unveil(path: Union[str, bytes, None], permissions: Optional[str]):
    if not _unveil:
        return  # unimplemented

    r = _unveil(path.encode() if isinstance(path, str) else path,
                None if permissions is None else permissions.encode())
    if r == -1:
        errno = ctypes.get_errno()
        raise OSError(errno, os.strerror(errno))

That’s it!

Trying it out

Let’s start with unveil. Initially a process has access to the whole file system with the usual restrictions. On the first call to unveil it’s immediately restricted to some subset of the tree. Each call reveals a little more until a final NULL which locks it in place for the rest of the process’s existence.

Suppose a program has been tricked into accessing your shell history, perhaps by mishandling a path:

def hackme():
    try:
        with open(pathlib.Path.home() / ".bash_history"):
            print("You've been hacked!")
    except FileNotFoundError:
        print("Blocked by unveil.")

hackme()

If you’re a Bash user, this prints:

You've been hacked!

Using our new feature to restrict the program’s access first:

# restrict access to static program data
unveil("/usr/share", "r")
unveil(None, None)

hackme()

On OpenBSD this now prints:

Blocked by unveil.

Working just as it should!

With pledge we declare what abilities we’d like to keep by supplying a list of promises, pledging to use only those abilities afterward. A common case is the stdio promise which allows reading and writing of open files, but not opening files. A program might open its log file, then drop the ability to open files while retaining the ability to write to its log.

An invalid or unknown promise is an error. Does that work?

>>> pledge("doesntexist", None)
OSError: [Errno 22] Invalid argument

So far so good. How about the functionality itself?

pledge("stdio", None)
hackme()

The program is instantly killed when making the disallowed system call:

Abort trap (core dumped)

If you want something a little softer, include the error promise:

pledge("stdio error", None)
hackme()

Instead it’s an exception, which will be a lot easier to debug when it comes to Python, so you probably always want to use it.

OSError: [Errno 78] Function not implemented

The core dump isn’t going to be much help to a Python program, so you probably always want to use this promise. In general you need to be extra careful about pledge in complex runtimes like Python’s which may reasonably need to do many arbitrary, undocumented things at any time.

Unfortunately I was unable to determine the origin of $RANDOM. Nobody was doing a good job tracking source code changes before the mid-1990s, so that history appears to be lost. Bash was first released in 1989, but the earliest version I could find was 1.14.7, released in 1996. KornShell was first released in 1983, but the earliest source I could find was from 1993. In both cases $RANDOM already existed. My guess is that it first appeared in one of these two shells, probably KornShell.

Update: Quentin Barnes has informed me that his 1986 copy of KornShell (a.k.a. ksh86) implements $RANDOM. This predates Bash and makes it likely that this feature originated in KornShell.

Bash

Of all the shells I’m going to discuss, Bash has the most interesting history. It never made use use of srand(3) / rand(3) and instead uses its own generator — which is generally what I prefer. Prior to Bash 4.0, it used the crummy linear congruential generator (LCG) found in the C89 standard:

static unsigned long rseed = 1;

static int
brand ()
{
  rseed = rseed * 1103515245 + 12345;
  return ((unsigned int)((rseed >> 16) & 32767));
}

For some reason it was naïvely decided that $RANDOM should never produce the same value twice in a row. The caller of brand() filters the output and discards repeats before returning to the shell script. This actually reduces the quality of the generator further since it increases correlation between separate outputs.

When the shell starts up, rseed is seeded from the PID and the current time in seconds. These values are literally summed and used as the seed.

/* Note: not the literal code, but equivalent. */
rseed = getpid() + time(0);

Subshells, which fork and initally share an rseed, are given similar treatment:

rseed = rseed + getpid() + time(0);

Notice there’s no hashing or mixing of these values, so there’s no avalanche effect. That would have prevented shells that start around the same time from having related initial random sequences.

With Bash 4.0, released in 2009, the algorithm was changed to a Park–Miller multiplicative LCG from 1988:

static int
brand ()
{
  long h, l;

  /* can't seed with 0. */
  if (rseed == 0)
    rseed = 123459876;
  h = rseed / 127773;
  l = rseed % 127773;
  rseed = 16807 * l - 2836 * h;
  return ((unsigned int)(rseed & 32767));
}

There’s actually a subtle mistake in this implementation compared to the generator described in the paper. This function will generate different numbers than the paper, and it will generate different numbers on different hosts! More on that later.

This algorithm is a much better choice than the previous LCG. There were many more options available in 2009 compared to 1989, but, honestly, this generator is pretty reasonable for this application. Bash is so slow that you’re never practically going to generate enough numbers for the small state to matter. Since the Park–Miller algorithm is older than Bash, they could have used this in the first place.

I considered submitting a patch to switch to something more modern. However, given Bash’s constraints, it’s harder said than done. Portability to weird systems is still a concern, and I expect they’d reject a patch that started making use of long long in the PRNG. They still support pre-ANSI C compilers that don’t have 64-bit arithmetic.

However, what still really could be improved is seeding. In Bash 4.x here’s what it looks like:

static void
seedrand ()
{
  struct timeval tv;

  gettimeofday (&tv, NULL);
  sbrand (tv.tv_sec ^ tv.tv_usec ^ getpid ());
}

Seeding is both better and worse. It’s better that it’s seeded from a higher resolution clock (milliseconds), so two shells started close in time have more variation. However, it’s “mixed” with XOR, which, in this case, is worse than addition.

For example, imagine two Bash shells started one millsecond apart. Both tv_usec and getpid() are incremented by one. Those increments are likely to cancel each other out by an XOR, and they end up with the same seed.

Instead, each of those quantities should be hashed before mixing. Here’s a rough example using my triple32() hash (adapted to glorious GNU-style pre-ANSI C):

static unsigned long
hash32 (x)
     unsigned long x;
{
  x ^= x >> 17;
  x *= 0xed5ad4bbUL;
  x &= 0xffffffffUL;
  x ^= x >> 11;
  x *= 0xac4c1b51UL;
  x &= 0xffffffffUL;
  x ^= x >> 15;
  x *= 0x31848babUL;
  x &= 0xffffffffUL;
  x ^= x >> 14;
  return x;
}

static void
seedrand ()
{
  struct timeval tv;

  gettimeofday (&tv, NULL);
  sbrand (hash32 (tv.tv_sec) ^
          hash32 (hash32 (tv.tv_usec) ^ getpid ()));
}

I had said there’s there’s a mistake in the Bash implementation of Park–Miller. Take a closer look at the types and the assignment to rseed:

  /* The variables */
  long h, l;
  unsigned long rseed;

  /* The assignment */
  rseed = 16807 * l - 2836 * h;

The result of the substraction can be negative, and that negative value is converted to unsigned long. The C standard says ULONG_MAX + 1 is added to make the value positive. ULONG_MAX varies by platform — typicially long is either 32 bits or 64 bits — so the results also vary. Here’s how the paper defined it:

  long test;

  test = 16807 * l - 2836 * h;
  if (test > 0)
    rseed = test;
  else
    rseed = test + 2147483647;

As far as I can tell, this mistake doesn’t hurt the quality of the generator.

$ 32/bash -c 'RANDOM=127773; echo $RANDOM $RANDOM'
29932 13634

$ 64/bash -c 'RANDOM=127773; echo $RANDOM $RANDOM'
29932 29115

Zsh

In contrast to Bash, Zsh is the most straightforward: defer to rand(3). Its $RANDOM can return the same value twice in a row, assuming that rand(3) does.

zlong
randomgetfn(UNUSED(Param pm))
{
    return rand() & 0x7fff;
}

void
randomsetfn(UNUSED(Param pm), zlong v)
{
    srand((unsigned int)v);
}

A cool feature is that means you could override it if you wanted with a custom generator.

int
rand(void)
{
    return 4; // chosen by fair dice roll.
              // guaranteed to be random.
}

Usage:

$ gcc -shared -fPIC -o rand.so rand.c
$ LD_PRELOAD=./rand.so zsh -c 'echo $RANDOM $RANDOM $RANDOM'
4 4 4

This trick also applies to the rest of the shells below.

KornShell (ksh)

KornShell originated in 1983, but it was finally released under an open source license in 2005. There’s a clone of KornShell called Public Domain Korn Shell (pdksh) that’s been forked a dozen different ways, but I’ll get to that next.

KornShell defers to rand(3), but it does some additional naïve filtering on the output. When the shell starts up, it generates 10 values from rand(). If any of them are larger than 32,767 then it will shift right by three all generated numbers.

#define RANDMASK 0x7fff

    for (n = 0; n < 10; n++) {
        // Don't use lower bits when rand() generates large numbers.
        if (rand() > RANDMASK) {
            rand_shift = 3;
            break;
        }
    }

Why not just look at RAND_MAX? I guess they didn’t think of it.

Update: Quentin Barnes pointed out that RAND_MAX didn’t exist until POSIX standardization in 1988. The constant first appeared in Unix in 1990. This KornShell code either predates the standard or needed to work on systems that predate the standard.

Like Bash, repeated values are not allowed. I suspect one shell got this idea from the other.

    do {
        cur = (rand() >> rand_shift) & RANDMASK;
    } while (cur == last);

Who came up with this strange idea first?

OpenBSD’s Public Domain Korn Shell (pdksh)

I picked the OpenBSD variant of pdksh since it’s the only pdksh fork I ever touch in practice, and its $RANDOM is the most interesting of the pdksh forks — at least since 2014.

Like Zsh, pdksh simply defers to rand(3). However, OpenBSD’s rand(3) is infamously and proudly non-standard. By default it returns non-deterministic, cryptographic-quality results seeded from system entropy (via the misnamed arc4random(3)), à la /dev/urandom. Its $RANDOM inherits this behavior.

    setint(vp, (int64_t) (rand() & 0x7fff));

However, if a value is assigned to $RANDOM in order to seed it, it reverts to its old pre-2014 deterministic generation via srand_deterministic(3).

    srand_deterministic((unsigned int)intval(vp));

OpenBSD’s deterministic rand(3) is the crummy LCG from the C89 standard, just like Bash 3.x. So if you assign to $RANDOM, you’ll get nearly the same results as Bash 3.x and earlier — the only difference being that it can repeat numbers.

That’s a slick upgrade to the old interface without breaking anything, making it my favorite version $RANDOM for any shell.

For some time Elfeed was experiencing a strange, spurious failure. Every so often users were seeing an error (spoiler warning) when updating feeds: “error in process sentinel: Search failed.” If you use Elfeed, you might have even seen this yourself. From the surface it appeared that curl, tasked with the responsibility for downloading feed data, was producing incomplete output despite reporting a successful run. Since the run was successful, Elfeed assumed certain data was in curl’s output buffer, but, since it wasn’t, it failed hard.

Unfortunately this issue was not reproducible. Manually running curl outside of Emacs never revealed any issues. Asking Elfeed to retry fetching the feeds would work fine. The issue would only randomly rear its head when Elfeed was fetching many feeds in parallel, under stress. By the time the error was discovered, the curl process had exited and vital debugging information was lost. Considering that this was likely to be a bug in Emacs itself, there really wasn’t a reliable way to capture the necessary debugging information from within Emacs Lisp. And, indeed, this later proved to be the case.

A quick-and-dirty work around is to use condition-case to catch and swallow the error. When the bizarre issue shows up, rather than fail badly in front of the user, Elfeed could attempt to swallow the error — assuming it can be reliably detected — and treat the fetch as simply a failure. That didn’t sit comfortably with me. Elfeed had done its due diligence checking for errors already. Someone was lying to Elfeed, and I intended to catch them with their pants on fire. Someday.

I’d just need to witness the bug on one of my own machines. Elfeed is part of my daily routine, so surely I’d have to experience this issue myself someday. My plan was, should that day come, to run a modified Elfeed, instrumented to capture extra data. I would have also routinely run Emacs under GDB so that I could inspect the failure more deeply.

For now I just had to wait to hunt that zebra.

Bryan Cantrill, DTrace, and FreeBSD

Over the holidays I re-discovered Bryan Cantrill, a systems software engineer who worked for Sun between 1996 and 2010, and is most well known for DTrace. My first exposure to him was in a BSD Now interview in 2015. I had re-watched that interview and decided there was a lot more I had to learn from him. He’s become a personal hero to me. So I scoured the internet for more of his writing and talks. Besides what I’ve already linked in this article, here are a couple more great presentations:

• Oral Tradition in Software Engineering
• Fork Yeah! The Rise and Development of illumos

You can also find some of his writing scattered around the DTrace blog.

Some interesting operating system technology came out of Sun during its final 15 or so years — most notably DTrace and ZFS — and Bryan speaks about it passionately. Almost as a matter of luck, most of it survived the Oracle acquisition thanks to Sun releasing it as open source in just the nick of time. Otherwise it would have been lost forever. The scattered ex-Sun employees, still passionate about their prior work at Sun, along with some of their old customers have since picked up the pieces and kept going as a community under the name illumos. It’s like an open source flotilla.

Naturally I wanted to get my hands on this stuff to try it out for myself. Is it really as good as they say? Normally I stick to Linux, but it (generally) doesn’t have these Sun technologies. The main reason is license incompatibility. Sun released its code under the CDDL, which is incompatible with the GPL. Ubuntu does infamously include ZFS, but other distributions are unwilling to take that risk. Porting DTrace is a serious undertaking since it’s got its fingers throughout the kernel, which also makes the licensing issues even more complicated.

(Update Feburary 2018: DTrace has been released under the GPLv2, allowing it to be legally integrated with Linux.)

Linux has a reputation for Not Invented Here (NIH) syndrome, and these licensing issues certainly contribute to that. Rather than adopt ZFS and DTrace, they’ve been reinvented from scratch: btrfs instead of ZFS, and a slew of partial options instead of DTrace. Normally I’m most interested in system call tracing, and my go to is strace, though it certainly has its limitations — including this situation of debugging curl under Emacs. Another famous example of NIH is Linux’s epoll(2), which is a broken version of BSD kqueue(2).

So, if I want to try these for myself, I’ll need to install a different operating system. I’ve dabbled with OmniOS, an OS built on illumos, in virtual machines, using it as an alien environment to test some of my software (e.g. enchive). OmniOS has a philosophy called Keep Your Software To Yourself (KYSTY), which is really just code for “we don’t do packaging.” Honestly, you can’t blame them since they’re a tiny community. The best solution to this is probably pkgsrc, which is essentially a universal packaging system. Otherwise you’re on your own.

There’s also openindiana, which is a more friendly desktop-oriented illumos distribution. Still, the short of it is that you’re very much on your own when things don’t work. The situation is like running Linux a couple decades ago, when it was still difficult to do.

If you’re interested in trying DTrace, the easiest option these days is probably FreeBSD. It’s got a big, active community, thorough documentation, and a huge selection of packages. Its license (the BSD license, duh) is compatible with the CDDL, so both ZFS and DTrace have been ported to FreeBSD.

What is DTrace?

I’ve done all this talking but haven’t yet described what DTrace really is. I won’t pretend to write my own tutorial, but I’ll provide enough information to follow along. DTrace is a tracing framework for debugging production systems in real time, both for the kernel and for applications. The “production systems” part means it’s stable and safe — using DTrace won’t put your system at risk of crashing or damaging data. The “real time” part means it has little impact on performance. You can use DTrace on live, active systems with little impact. Both of these core design principles are vital for troubleshooting those really tricky bugs that only show up in production.

There are DTrace probes scattered all throughout the system: on system calls, scheduler events, networking events, process events, signals, virtual memory events, etc. Using a specialized language called D (unrelated to the general purpose programming language D), you can dynamically add behavior at these instrumentation points. Generally the behavior is to capture information, but it can also manipulate the event being traced.

Each probe is fully identified by a 4-tuple delimited by colons: provider, module, function, and probe name. An empty element denotes a sort of wildcard. For example, syscall::open:entry is a probe at the beginning (i.e. “entry”) of open(2). syscall:::entry matches all system call entry probes.

Unlike strace on Linux which monitors a specific process, DTrace applies to the entire system when active. To run curl under strace from Emacs, I’d have to modify Emacs’ behavior to do so. With DTrace I can instrument every curl process without making a single change to Emacs, and with negligible impact to Emacs. That’s a big deal.

So, when it comes to this Elfeed issue, FreeBSD is much better poised for debugging the problem. All I have to do is catch it in the act. However, it’s been months since that bug report and I’m not really making this connection yet. I’m just hoping I eventually find an interesting problem where I can apply DTrace.

FreeBSD on a Raspberry Pi 2

So I’ve settled in FreeBSD as the playground for these technologies, I just have to decide where. I could always run it in a virtual machine, but it’s always more interesting to try things out on real hardware. FreeBSD supports the Raspberry Pi 2 as a Tier 2 system, and I had a Raspberry Pi 2 sitting around collecting dust, so I put it to use.

I wrote the image to an SD card, and for a few days I stretched my legs on this new system. I cloned a couple dozen of my own git repositories, ran the builds and the tests, and just got a feel for things. I tried out the ports system for the first time, mainly to discover that the low-powered Raspberry Pi 2 takes days to build some of the packages I want to try.

I mostly program in Vim these days, so it’s some days before I even set up Emacs. Eventually I do build Emacs, clone my configuration, fire it up, and give Elfeed a spin.

And that’s when the “search failed” bug strikes! Not just once, but dozens of times. Perfect! This low-powered platform is the jackpot for this particular bug, triggering it left and right. Given that I’ve got DTrace at my disposal, it’s the perfect place to debug this. Something is lying to Elfeed and DTrace will play the judge.

Before I dive in I see three possibilities:

1. curl is reporting success but truncating its output.
2. Emacs is quietly truncating curl’s output.
3. Emacs is misinterpreting curl’s exit status.

With Dtrace I can observe what every curl process writes to Emacs, and I can also double check curl’s exit status. I come up with the following (newbie) DTrace script:

syscall::write:entry
/execname == "curl"/
{
    printf("%d WRITE %d \"%s\"\n",
           pid, arg2, stringof(copyin(arg1, arg2)));
}

syscall::exit:entry
/execname == "curl"/
{
    printf("%d EXIT  %d\n", pid, arg0);
}

The /execname == "curl"/ is a predicate that (obviously) causes the behavior to only fire for curl processes. The first probe has DTrace print a line for every write(2) from curl. arg0, arg1, and arg2 correspond to the arguments of write(2): fd, buf, count. It logs the process ID (pid) of the write, the length of the write, and the actual contents written. Remember that these curl processes are run in parallel by Emacs, so the pid allows me to associate the separate writes and the exit status.

The second probe prints the pid and the exit status (the first argument to exit(2)).

I also want to compare this to exactly what is delivered to Elfeed when curl exits, so I modify the process sentinel — the callback that handles a subprocess exiting — to call write-file before any action is taken. I can compare these buffer dumps to the logs produced by DTrace.

There are two important findings.

First, when the “search failed” bug occurs, the buffer was completely empty (95% of the time) or truncated at the end of the HTTP headers (5% of the time), right at the blank line. DTrace indicates that curl did its job to the full, so it’s Emacs who’s the liar. It’s not delivering all of curl’s data to Elfeed. That’s pretty annoying.

Second, curl was line-buffered. Each line was a separate, independent write(2). I was certainly not expecting this. Normally the C library only does line buffering when the output is a terminal. That’s because it’s guessing a user may be watching, expecting the output to arrive a line at a time.

Here’s a sample of what it looked like in the log:

88188 WRITE 32 "Server: Apache/2.4.18 (Ubuntu)
"
88188 WRITE 46 "Location: https://blog.plover.com/index.atom
"
88188 WRITE 21 "Content-Length: 299
"
88188 WRITE 45 "Content-Type: text/html; charset=iso-8859-1
"
88188 WRITE 2 "
"

Why would curl think Emacs is a terminal?

Oh. That’s right. This is the same problem I ran into four years ago when writing EmacSQL. By default Emacs connects to subprocesses through a pseudo-terminal (pty). I called this a mistake in Emacs back then, and I still stand by that claim. The pty causes weird, annoying problems for little benefit:

• Interpreting control characters. Hope you weren’t transferring binary data!
• Subprocesses will generally get line buffered. This makes them slower, though in some situations it might be desirable.
• Stdout and stderr get mixed together. (Optional since Emacs 25.)
• New! There’s a bug somewhere in Emacs that causes truncation when ptys are used heavily in parallel.

Just from eyeballing the DTrace log I knew what to do: dump the pty and switch to a pipe. This is controlled with the process-connection-type variable, and fixing it is a one-liner.

Not only did this completely resolve the truncation issue, Elfeed is noticeably faster at fetching feeds on all machines. It’s no longer receiving mountains of XML one line at a time, like sucking pudding through a straw. It’s now quite zippy even on my Raspberry Pi 2, which had never been the case before (without the “search failed” bug). Even if you were never affected by this bug, you will benefit from the fix.

I haven’t officially reported this as an Emacs bug yet because reproducibility is still an issue. It needs something better than “fire off a bunch of HTTP requests across the internet in parallel from a Raspberry Pi.”

The fix reminds me of that old boilermaker story about charging a lot of money just to swing a hammer. Once the problem arose, DTrace quickly helped to identify the place to hit Emacs with the hammer.

Finally, a big thanks to alphapapa for originally taking the time to report this bug months ago.