The usual way to work around the lack of operating system support for a particular asynchronous operation is to dedicate threads to waiting on those operations. By using a thread pool, we can even avoid the overhead of spawning threads when we need them. Plus asyncio is designed to play nicely with thread pools anyway.
Before we get started, we’ll need some way to test that it’s working. We
need a slow file system. One thought is to use ptrace to intercept the
relevant system calls, though this isn’t quite so simple. The
other threads need to continue running while the thread waiting on
open(2) is paused, but ptrace pauses the whole process. Fortunately
there’s a simpler solution anyway: LD_PRELOAD.
Setting the LD_PRELOAD environment variable to the name of a shared
object will cause the loader to load this shared object ahead of
everything else, allowing that shared object to override other
libraries. I’m on x86-64 Linux (Debian), and so I’m looking to override
open64(2) in glibc. Here’s my open64.c:
#define _GNU_SOURCE
#include <dlfcn.h>
#include <string.h>
#include <unistd.h>
int
open64(const char *path, int flags, int mode)
{
if (!strncmp(path, "/tmp/", 5)) {
sleep(3);
}
int (*f)(const char *, int, int) = dlsym(RTLD_NEXT, "open64");
return f(path, flags, mode);
}
Now Python must go through my C function when it opens files. If the
file resides where under /tmp/, opening the file will be delayed by 3
seconds. Since I still want to actually open a file, I use dlsym() to
access the real open64() in glibc. I build it like so:
$ cc -shared -fPIC -o open64.so open64.c -ldl
And to test that it works with Python, let’s time how long it takes to
open /tmp/x:
$ touch /tmp/x
$ time LD_PRELOAD=./open64.so python3 -c 'open("/tmp/x")'
real 0m3.021s
user 0m0.014s
sys 0m0.005s
Perfect! (Note: It’s a little strange putting time before setting the
environment variable, but that’s because I’m using Bash and it time is
special since this is the shell’s version of the command.)
Python’s standard open() is most commonly used as a context manager
so that the file is automatically closed no matter what happens.
with open('output.txt', 'w') as out:
print('hello world', file=out)
I’d like my asynchronous open to follow this pattern using async
with. It’s like with, but the context manager is acquired and
released asynchronously. I’ll call my version aopen():
async with aopen('output.txt', 'w') as out:
...
So aopen() will need to return an asynchronous context manager, an
object with methods __aenter__ and __aexit__ that both return
awaitables. Usually this is by virtue of these methods being
coroutine functions, but a normal function that directly returns
an awaitable also works, which is what I’ll be doing for __aenter__.
class _AsyncOpen():
def __init__(self, args, kwargs):
...
def __aenter__(self):
...
async def __aexit__(self, exc_type, exc, tb):
...
Ultimately we have to call open(). The arguments for open() will be
given to the constructor to be used later. This will make more sense
when you see the definition for aopen().
def __init__(self, args, kwargs):
self._args = args
self._kwargs = kwargs
When it’s time to actually open the file, Python will call __aenter__.
We can’t call open() directly since that will block, so we’ll use a
thread pool to wait on it. Rather than create a thread pool, we’ll use
the one that comes with the current event loop. The run_in_executor()
method runs a function in a thread pool — where None means use the
default pool — returning an asyncio future representing the future
result, in this case the opened file object.
def __aenter__(self):
def thread_open():
return open(*self._args, **self._kwargs)
loop = asyncio.get_event_loop()
self._future = loop.run_in_executor(None, thread_open)
return self._future
Since this __aenter__ is not a coroutine function, it returns the
future directly as its awaitable result. The caller will await it.
The default thread pool is limited to one thread per core, which I suppose is the most obvious choice, though not ideal here. That’s fine for CPU-bound operations but not for I/O-bound operations. In a real program we may want to use a larger thread pool.
Closing a file may block, so we’ll do that in a thread pool as well. First pull the file object from the future, then close it in the thread pool, waiting until the file has actually closed:
async def __aexit__(self, exc_type, exc, tb):
file = await self._future
def thread_close():
file.close()
loop = asyncio.get_event_loop()
await loop.run_in_executor(None, thread_close)
The open and close are paired in this context manager, but it may be
concurrent with an arbitrary number of other _AsyncOpen context
managers. There will be some upper limit to the number of open files, so
we need to be careful not to use too many of these things
concurrently, something which easily happens when using unbounded
queues. Lacking back pressure, all it takes is for tasks to be
opening files slightly faster than they close them.
With all the hard work done, the definition for aopen() is trivial:
def aopen(*args, **kwargs):
return _AsyncOpen(args, kwargs)
That’s it! Let’s try it out with the LD_PRELOAD test.
First define a “heartbeat” task that will tell us the asyncio loop is still chugging away while we wait on opening the file.
async def heartbeat():
while True:
await asyncio.sleep(0.5)
print('HEARTBEAT')
Here’s a test function for aopen() that asynchronously opens a file
under /tmp/ named by an integer, (synchronously) writes that integer
to the file, then asynchronously closes it.
async def write(i):
async with aopen(f'/tmp/{i}', 'w') as out:
print(i, file=out)
The main() function creates the heartbeat task and opens 4 files
concurrently though the intercepted file opening routine:
async def main():
beat = asyncio.create_task(heartbeat())
tasks = [asyncio.create_task(write(i)) for i in range(4)]
await asyncio.gather(*tasks)
beat.cancel()
asyncio.run(main())
The result:
$ LD_PRELOAD=./open64.so python3 aopen.py
HEARTBEAT
HEARTBEAT
HEARTBEAT
HEARTBEAT
HEARTBEAT
HEARTBEAT
$ cat /tmp/{1,2,3,4}
1
2
3
4
As expected, 6 heartbeats corresponding to 3 seconds that all 4 tasks
spent concurrently waiting on the intercepted open(). Here’s the full
source if you want to try it our for yourself:
https://gist.github.com/skeeto/89af673a0a0d24de32ad19ee505c8dbd
Only opening and closing the file is asynchronous. Read and writes are
unchanged, still fully synchronous and blocking, so this is only a half
solution. A full solution is not nearly as simple because asyncio is
async/await. Asynchronous reads and writes would require all new APIs
with different coloring. You’d need an aprint() to complement
print(), and so on, each returning an awaitable to be awaited.
This is one of the unfortunate downsides of async/await. I strongly prefer conventional, preemptive concurrency, but we don’t always have that luxury.
]]>A common situation in asyncio Python programs is asynchronous initialization. Some resource must be initialized exactly once before it can be used, but the initialization itself is asynchronous — such as an asyncpg database. Let’s talk about a couple of solutions.
The naive “solution” would be to track the initialization state in a variable:
initialized = False
async def one_time_setup():
"Do not call more than once!"
...
async def maybe_initialize():
global initialized
if not initialized:
await one_time_setup()
initialized = True
The reasoning for initialized is the expectation of calling the
function more than once. However, if it might be called from concurrent
tasks there’s a race condition. If the second caller arrives while the
first is awaiting one_time_setup(), the function will be called a
second time.
Switching the order of the call and the assignment won’t help:
async def maybe_initialize():
global initialized
if not initialized:
initialized = True
await one_time_setup()
Since asyncio is cooperative, the first caller doesn’t give up control
until to other tasks until the await, meaning one_time_setup() will
never be called twice. However, the second caller may return before
one_time_setup() has completed. What we want is for one_time_setup()
to be called exactly once, but for no caller to return until it has
returned.
My first thought was to use a mutex lock. This will protect the
variable and prevent followup callers from progressing too soon. Tasks
arriving while one_time_setup() is still running will block on the
lock.
initialized = False
initialized_lock = asyncio.Lock()
async def maybe_initialize():
global initialized
async with initialized_lock:
if not initialized:
await one_time_setup()
initialized = True
Unfortunately this has a serious downside: asyncio locks are
associated with the loop where they were created. Since the
lock variable is global, maybe_initialize() can only be called from
the same loop that loaded the module. asyncio.run() creates a new loop
so it’s incompatible.
# create a loop: always an error
asyncio.run(maybe_initialize())
# reuse the loop: maybe an error
loop = asyncio.get_event_loop()
loop.run_until_complete((maybe_initialize()))
(IMHO, it was a mistake for the asyncio API to include explicit loop objects. It’s a low-level concept that unavoidably leaks through most high-level abstractions.)
A workaround is to create the lock lazily. Thank goodness creating a lock isn’t itself asynchronous!
initialized = False
initialized_lock = None
async def maybe_initialize():
global initialized, initialized_lock
if not initialized_lock:
initialized_lock = asyncio.Lock()
async with initialized_lock:
if not initialized:
await one_time_setup()
initialized = True
This is better, but maybe_initialize() can still only ever be called
from a single loop.
asyncio.run(maybe_initialize()) # ok
asyncio.run(maybe_initialize()) # error!
The pthreads API provides pthread_once to solve this problem.
C++11 has similarly has std::call_once. We can build something
similar using a future-like object.
future = None
async def maybe_initialize():
global future
if not future:
future = asyncio.create_task(one_time_setup())
await future
Awaiting a coroutine more than once is an error, but tasks are future-like objects and can be awaited more than once. At least on CPython, they can also be awaited in other loops! So not only is this simpler, it also solves the loop problem!
asyncio.run(maybe_initialize()) # ok
asyncio.run(maybe_initialize()) # still ok
This can be tidied up nicely in a @once decorator:
def once(func):
future = None
async def once_wrapper(*args, **kwargs):
nonlocal future
if not future:
future = asyncio.create_task(func(*args, **kwargs))
return await future
return once_wrapper
No more need for maybe_initialize(), just decorate the original
one_time_setup():
@once
async def one_time_setup():
...
pdb showed this wasn’t the case. Instead, the
program’s author had made a couple of fundamental mistakes using
asyncio. Let’s discuss them using small examples.
Setting the stage: There’s a heartbeat coroutine that “beats” once per second. A real program would send out a packet as the heartbeat, but here it just prints how late it was scheduled.
async def heartbeat():
while True:
start = time.time()
await asyncio.sleep(1)
delay = time.time() - start - 1
print(f'heartbeat delay = {delay:.3f}s')
Running this with asyncio.run(heartbeat()):
heartbeat delay = 0.001s
heartbeat delay = 0.001s
heartbeat delay = 0.001s
It’s consistently 1ms late, but good enough, especially considering
what’s to come. A program that only sends a heartbeat is pretty
useless, so a real program will be busy working on other things
concurrently. In this example, we have little 10ms payloads of work to
do, which are represented by this process() function:
JOB_DURATION = 0.01 # 10ms
async def process():
time.sleep(JOB_DURATION) # simulate CPU time
That’s a synchronous sleep because it’s standing in for actual CPU work. Maybe it’s parsing JSON in a loop or crunching numbers in NumPy. Use your imagination. During this 10ms no other coroutines can be scheduled because this is, after all, still just a single-threaded program.
JOB_COUNT = 200
async def main():
asyncio.create_task(heartbeat())
await asyncio.sleep(2.5)
print('begin processing')
count = JOB_COUNT
for _ in range(JOB_COUNT):
asyncio.create_task(process())
await asyncio.sleep(5)
This program starts the heartbeat coroutine in a task. A coroutine doesn’t make progress unless someone is waiting on it, and that something can be a task. So it will continue along independently without prodding.
The arbitrary 2.5 second sleep simulates waiting, say, for a network request. In the output we’ll see the heartbeat tick a couple of times, then it will create and process 200 jobs concurrently. In a real program we’d have some way to collect the results, but we can ignore that part for now. They’re only 10ms, so the effect on the heartbeat should be pretty small right?
heartbeat delay = 0.001s
heartbeat delay = 0.001s
begin processing
heartbeat delay = 1.534s
heartbeat delay = 0.001s
heartbeat delay = 0.001s
The heartbeat was delayed for 1.5 seconds by a mere 200 tasks each doing only 10ms of work each. What happened?
Python calls the object that schedules tasks a loop, and this is no
coincidence. Everything to be scheduled gets put into a loop and is
scheduled round robin, one after another. The 200 tasks got scheduled
ahead of the heartbeat, and so it doesn’t get scheduled again until each
of those tasks either yields (await) or completes.
It really didn’t take much to significantly hamper the heartbeat, and, with a dumb bytecode compiler, 10ms may not be much work at all. The lesson here is to avoid spawning many tasks if latency is an important consideration.
My first idea at a solution: What if we used a semaphore to limit the number of “active” tasks at a time? Then perhaps the heartbeat wouldn’t have to compete with so many other tasks for time.
WORKER_COUNT = 4 # max "active" jobs at a time
async def main_with_semaphore():
asyncio.create_task(heartbeat())
await asyncio.sleep(2.5)
sem = asyncio.Semaphore(WORKER_COUNT)
async def process():
await sem.acquire()
time.sleep(JOB_DURATION)
sem.release()
print('begin processing')
for _ in range(JOB_COUNT):
asyncio.create_task(process())
await asyncio.sleep(5)
When the heartbeat sleep completes, about half the jobs will be complete and the other half blocked on the semaphore. So perhaps the heartbeat gets to skip ahead of all the blocked tasks since they’re not yet ready to run?
heartbeat delay = 0.001s
heartbeat delay = 0.001s
begin processing
heartbeat delay = 1.537s
heartbeat delay = 0.001s
heartbeat delay = 0.001s
It made no difference whatsoever because the tasks each “held their
place” in line in the loop! Even reducing WORKER_COUNT to 1 would have
no effect. As soon as a task completes, it frees the task waiting next
in line. The semaphore does practically nothing here.
Here’s what does work: a job queue. Create a queue to be populated with coroutines (not tasks), and have a small number of tasks run jobs from the queue. Since this is a real solution, I’ve made this example more complete.
async def main_with_queue():
asyncio.create_task(heartbeat())
await asyncio.sleep(2.5)
queue = asyncio.Queue(maxsize=1)
async def worker():
while True:
coro = await queue.get()
await coro # consider using try/except
queue.task_done()
workers = [asyncio.create_task(worker())
for _ in range(WORKER_COUNT)]
print('begin processing')
for _ in range(JOB_COUNT):
await queue.put(process())
await queue.join()
print('end processing')
for w in workers:
w.cancel()
await asyncio.sleep(2)
The task_done() and join() methods make it trivial synchronize on
full job completion. I also take the time to destroy the worker tasks.
It’s harmless to leave them blocked on the queue. They’ll be garbage
collected so it’s not a resource leak. However, CPython complains about
garbage collecting running tasks because it looks like a mistake — and
it usually is.
If you read carefully you might have noticed the queue’s maximum size is
set to 1: not much of a “queue”! Go developers will recognize this
as being (nearly) an unbuffered channel, the default and most common
kind of channel. So it’s more a synchronized rendezvous between producer
(put()) and consumer (get()). The producer waits at the queue with a
job until a task is free to come take it. A task waits at the queue
until a producer arrives with a job for it.
heartbeat delay = 0.001s
heartbeat delay = 0.001s
begin processing
heartbeat delay = 0.014s
heartbeat delay = 0.020s
end processing
heartbeat delay = 0.002s
heartbeat delay = 0.001s
The output shows that the impact to the heartbeat was modest — about the best we could hope for from async/await — and the heartbeat continued while jobs were running. The more concurrency — the more worker tasks running on the queue — the greater the latency.
Note: Increasing the WORKER_COUNT in this toy example won’t have an
impact on latency since the jobs aren’t actually concurrent. They start,
run, and complete before another worker task can draw from the queue.
Putting a couple awaits in process() allows for concurrency:
WORKER_COUNT = 200
async def process():
await asyncio.sleep(0.01)
time.sleep(JOB_DURATION)
await asyncio.sleep(0.01)
Since there are so many worker tasks, this is back to the initial problem:
heartbeat delay = 0.001s
heartbeat delay = 0.001s
begin processing
heartbeat delay = 1.655s
end processing
heartbeat delay = 0.001s
heartbeat delay = 0.001s
As WORKER_COUNT decreases, so does heartbeat latency.
Here’s another defect from the same program. Create an unbounded queue, a producer, and a consumer. The consumer prints the queue size so we can see what’s happening:
async def producer_consumer():
queue = asyncio.Queue()
done = asyncio.Condition()
async def producer():
for i in range(100_000):
await queue.put(i)
await queue.join()
async with done:
done.notify()
async def consumer():
while True:
await queue.get()
print(f'qsize = {queue.qsize()}')
queue.task_done()
asyncio.create_task(producer())
asyncio.create_task(consumer())
async with done:
await done.wait()
The output of this program begins:
qsize = 99999
qsize = 99998
qsize = 99997
qsize = 99996
...
So the entire queue is populated before the consumer does anything at
all: tons of latency for whatever is being consumed. Since the queue is
unbounded, the producer never needs to yield. You might be tempted to
use asyncio.sleep(0) in the producer to yield explicitly:
async def producer():
for i in range(100_000):
await queue.put(i)
await asyncio.sleep(0) # yield
await queue.join()
async with done:
done.notify()
This even seems to work! The output looks like this:
qsize = 0
qsize = 0
qsize = 0
qsize = 0
However, this is fragile and not a real solution. If the consumer yields just two times in its own loop, its nearly back to where we started:
async def consumer():
while True:
await queue.get()
print(f'qsize = {queue.qsize()}')
queue.task_done()
await asyncio.sleep(0)
await asyncio.sleep(0)
The output shows that the producer gradually creeps ahead of the consumer. On each consumer iteration, the producer iterates twice:
qsize = 0
qsize = 1
qsize = 2
qsize = 3
...
There’s a really simple solution to this: Never, ever use unbounded
queues. In fact every unbounded asyncio.Queue() is a bug.
It’s a serious API defect that asyncio allows unbounded queues to be
created at all. The default maxsize should have been actually zero
(unbuffered), not infinite. Because unbounded is the default, virtually
every example of asyncio.Queue — online, offline, and even the
official documentation — is broken in some way.
asyncio.Queue() is always wrong.asyncio.sleep(0) is nearly always used incorrectly.maxsize=1 job queue instead of spawning many identical tasks.Python linters should be updated to warn about 1 and 2 by default.
Update: A couple of people have pointed out an argument in the Trio
documentation for unbounded queues. This argument conflates two
different concepts: data structure queues and concurrent communication
infrastructure queues. To distinguish, the latter is often called a
channel. An unbounded queue (collections.deque) is necessary, but
and unbounded channel (asyncio.Queue) is always wrong. The Trio
documentation describes a web crawler, which is fundamentally a
breadth-first search (read: queue-oriented) of a graph. So this is a
plain old BFS queue, not a channel, which is why it’s reasonable for it
to be unbounded.
I’m a big fan of tarpits: a network service that intentionally inserts delays in its protocol, slowing down clients by forcing them to wait. This arrests the speed at which a bad actor can attack or probe the host system, and it ties up some of the attacker’s resources that might otherwise be spent attacking another host. When done well, a tarpit imposes more cost on the attacker than the defender.
The Internet is a very hostile place, and anyone who’s ever stood up an Internet-facing IPv4 host has witnessed the immediate and continuous attacks against their server. I’ve maintained such a server for nearly six years now, and more than 99% of my incoming traffic has ill intent. One part of my defenses has been tarpits in various forms. The latest addition is an SSH tarpit I wrote a couple of months ago:
This program opens a socket and pretends to be an SSH server. However, it actually just ties up SSH clients with false promises indefinitely — or at least until the client eventually gives up. After cloning the repository, here’s how you can try it out for yourself (default port 2222):
$ make
$ ./endlessh &
$ ssh -p2222 localhost
Your SSH client will hang there and wait for at least several days before finally giving up. Like a mammoth in the La Brea Tar Pits, it got itself stuck and can’t get itself out. As I write, my Internet-facing SSH tarpit currently has 27 clients trapped in it. A few of these have been connected for weeks. In one particular spike it had 1,378 clients trapped at once, lasting about 20 hours.
My Internet-facing Endlessh server listens on port 22, which is the standard SSH port. I long ago moved my real SSH server off to another port where it sees a whole lot less SSH traffic — essentially none. This makes the logs a whole lot more manageable. And (hopefully) Endlessh convinces attackers not to look around for an SSH server on another port.
How does it work? Endlessh exploits a little paragraph in RFC 4253, the SSH protocol specification. Immediately after the TCP connection is established, and before negotiating the cryptography, both ends send an identification string:
SSH-protoversion-softwareversion SP comments CR LF
The RFC also notes:
The server MAY send other lines of data before sending the version string.
There is no limit on the number of lines, just that these lines must not begin with “SSH-“ since that would be ambiguous with the identification string, and lines must not be longer than 255 characters including CRLF. So Endlessh sends and endless stream of randomly-generated “other lines of data” without ever intending to send a version string. By default it waits 10 seconds between each line. This slows down the protocol, but prevents it from actually timing out.
This means Endlessh need not know anything about cryptography or the vast majority of the SSH protocol. It’s dead simple.
Ideally the tarpit’s resource footprint should be as small as possible. It’s just a security tool, and the server does have an actual purpose that doesn’t include being a tarpit. It should tie up the attacker’s resources, not the server’s, and should generally be unnoticeable. (Take note all those who write the awful “security” products I have to tolerate at my day job.)
Even when many clients have been trapped, Endlessh spends more than 99.999% of its time waiting around, doing nothing. It wouldn’t even be accurate to call it I/O-bound. If anything, it’s timer-bound, waiting around before sending off the next line of data. The most precious resource to conserve is memory.
The most straightforward way to implement something like Endlessh is a
fork server: accept a connection, fork, and the child simply alternates
between sleep(3) and write(2):
for (;;) {
ssize_t r;
char line[256];
sleep(DELAY);
generate_line(line);
r = write(fd, line, strlen(line));
if (r == -1 && errno != EINTR) {
exit(0);
}
}
A process per connection is a lot of overhead when connections are expected to be up hours or even weeks at a time. An attacker who knows about this could exhaust the server’s resources with little effort by opening up lots of connections.
A better option is, instead of processes, to create a thread per connection. On Linux this is practically the same thing, but it’s still better. However, you still have to allocate a stack for the thread and the kernel will have to spend some resources managing the thread.
For Endlessh I went for an even more lightweight version: a
single-threaded poll(2) server, analogous to stackless green threads.
The overhead per connection is about as low as it gets.
Clients that are being delayed are not registered in poll(2). Their
only overhead is the socket object in the kernel, and another 78 bytes
to track them in Endlessh. Most of those bytes are used only for
accurate logging. Only those clients that are overdue for a new line
are registered for poll(2).
When clients are waiting, but no clients are overdue, poll(2) is
essentially used in place of sleep(3). Though since it still needs
to manage the accept server socket, it (almost) never actually waits
on nothing.
There’s an option to limit the total number of client connections so
that it doesn’t get out of hand. In this case it will stop polling the
accept socket until a client disconnects. I probably shouldn’t have
bothered with this option and instead relied on ulimit, a feature
already provided by the operating system.
I could have used epoll (Linux) or kqueue (BSD), which would be much
more efficient than poll(2). The problem with poll(2) is that it’s
constantly registering and unregistering Endlessh on each of the
overdue sockets each time around the main loop. This is by far the
most CPU-intensive part of Endlessh, and it’s all inflicted on the
kernel. Most of the time, even with thousands of clients trapped in
the tarpit, only a small number of them at polled at once, so I opted
for better portability instead.
One consequence of not polling connections that are waiting is that
disconnections aren’t noticed in a timely fashion. This makes the logs
less accurate than I like, but otherwise it’s pretty harmless.
Unforunately even if I wanted to fix this, the poll(2) interface
isn’t quite equipped for it anyway.
With a poll(2) server, the biggest overhead remaining is in the
kernel, where it allocates send and receive buffers for each client
and manages the proper TCP state. The next step to reducing this
overhead is Endlessh opening a raw socket and speaking TCP itself,
bypassing most of the operating system’s TCP/IP stack.
Much of the TCP connection state doesn’t matter to Endlessh and doesn’t need to be tracked. For example, it doesn’t care about any data sent by the client, so no receive buffer is needed, and any data that arrives could be dropped on the floor.
Even more, raw sockets would allow for some even nastier tarpit tricks. Despite the long delays between data lines, the kernel itself responds very quickly on the TCP layer and below. ACKs are sent back quickly and so on. An astute attacker could detect that the delay is artificial, imposed above the TCP layer by an application.
If Endlessh worked at the TCP layer, it could tarpit the TCP protocol itself. It could introduce artificial “noise” to the connection that requires packet retransmissions, delay ACKs, etc. It would look a lot more like network problems than a tarpit.
I haven’t taken Endlessh this far, nor do I plan to do so. At the moment attackers either have a hard timeout, so this wouldn’t matter, or they’re pretty dumb and Endlessh already works well enough.
Since writing Endless I’ve learned about Python’s asyncio, and
it’s actually a near perfect fit for this problem. I should have just
used it in the first place. The hard part is already implemented within
asyncio, and the problem isn’t CPU-bound, so being written in Python
doesn’t matter.
Here’s a simplified (no logging, no configuration, etc.) version of Endlessh implemented in about 20 lines of Python 3.7:
import asyncio
import random
async def handler(_reader, writer):
try:
while True:
await asyncio.sleep(10)
writer.write(b'%x\r\n' % random.randint(0, 2**32))
await writer.drain()
except ConnectionResetError:
pass
async def main():
server = await asyncio.start_server(handler, '0.0.0.0', 2222)
async with server:
await server.serve_forever()
asyncio.run(main())
Since Python coroutines are stackless, the per-connection memory overhead is comparable to the C version. So it seems asyncio is perfectly suited for writing tarpits! Here’s an HTTP tarpit to trip up attackers trying to exploit HTTP servers. It slowly sends a random, endless HTTP header:
import asyncio
import random
async def handler(_reader, writer):
writer.write(b'HTTP/1.1 200 OK\r\n')
try:
while True:
await asyncio.sleep(5)
header = random.randint(0, 2**32)
value = random.randint(0, 2**32)
writer.write(b'X-%x: %x\r\n' % (header, value))
await writer.drain()
except ConnectionResetError:
pass
async def main():
server = await asyncio.start_server(handler, '0.0.0.0', 8080)
async with server:
await server.serve_forever()
asyncio.run(main())
Try it out for yourself. Firefox and Chrome will spin on that server
for hours before giving up. I have yet to see curl actually timeout on
its own in the default settings (--max-time/-m does work
correctly, though).
Parting exercise for the reader: Using the examples above as a starting point, implement an SMTP tarpit using asyncio. Bonus points for using TLS connections and testing it against real spammers.
]]>In fact, both Python and JavaScript async functions are essentially just fancy generator functions with some specialized syntax and semantics. That is, they’re stackless coroutines. Both languages already had generators, so their generator-like async functions are a natural extension that — unlike stackful coroutines — do not require significant, new runtime plumbing.
Emacs officially got generators in 25.1 (September 2016), though, unlike Python and JavaScript, it didn’t require any additional support from the compiler or runtime. It’s implemented entirely using Lisp macros. In other words, it’s just another library, not a core language feature. In theory, the generator library could be easily backported to the first Emacs release to properly support lexical closures, Emacs 24.1 (June 2012).
For the same reason, stackless async/await coroutines can also be
implemented as a library. So that’s what I did, letting Emacs’ generator
library do most of the heavy lifting. The package is called aio:
It’s modeled more closely on JavaScript’s async functions than Python’s asyncio, with the core representation being promises rather than a coroutine objects. I just have an easier time reasoning about promises than coroutines.
I’m definitely not the first person to realize this was possible, and was beaten to the punch by two years. Wanting to avoid fragmentation, I set aside all formality in my first iteration on the idea, not even bothering with namespacing my identifiers. It was to be only an educational exercise. However, I got quite attached to my little toy. Once I got my head wrapped around the problem, everything just sort of clicked into place so nicely.
In this article I will show step-by-step one way to build async/await on top of generators, laying out one concept at a time and then building upon each. But first, some examples to illustrate the desired final result.
Ignoring all its problems for a moment, suppose you want to use
url-retrieve to fetch some content from a URL and return it. To keep
this simple, I’m going to omit error handling. Also assume that
lexical-binding is t for all examples. Besides, lexical scope
required by the generator library, and therefore also required by aio.
The most naive approach is to fetch the content synchronously:
(defun fetch-fortune-1 (url)
(let ((buffer (url-retrieve-synchronously url)))
(with-current-buffer buffer
(prog1 (buffer-string)
(kill-buffer)))))
The result is returned directly, and errors are communicated by an error signal (e.g. Emacs’ version of exceptions). This is convenient, but the function will block the main thread, locking up Emacs until the result has arrived. This is obviously very undesirable, so, in practice, everyone nearly always uses the asynchronous version:
(defun fetch-fortune-2 (url callback)
(url-retrieve url (lambda (_status)
(funcall callback (buffer-string)))))
The main thread no longer blocks, but it’s a whole lot less convenient. The result isn’t returned to the caller, and instead the caller supplies a callback function. The result, whether success or failure, will be delivered via callback, so the caller must split itself into two pieces: the part before the callback and the callback itself. Errors cannot be delivered using a error signal because of the inverted flow control.
The situation gets worse if, say, you need to fetch results from two different URLs. You either fetch results one at a time (inefficient), or you manage two different callbacks that could be invoked in any order, and therefore have to coordinate.
Wouldn’t it be nice for the function to work like the first example, but be asynchronous like the second example? Enter async/await:
(aio-defun fetch-fortune-3 (url)
(let ((buffer (aio-await (aio-url-retrieve url))))
(with-current-buffer buffer
(prog1 (buffer-string)
(kill-buffer)))))
A function defined with aio-defun is just like defun except that
it can use aio-await to pause and wait on any other function defined
with aio-defun — or, more specifically, any function that returns a
promise. Borrowing Python parlance: Returning a promise makes a
function awaitable. If there’s an error, it’s delivered as a error
signal from aio-url-retrieve, just like the first example. When
called, this function returns immediately with a promise object that
represents a future result. The caller might look like this:
(defcustom fortune-url ...)
(aio-defun display-fortune ()
(interactive)
(message "%s" (aio-await (fetch-fortune-3 fortune-url))))
How wonderfully clean that looks! And, yes, it even works with
interactive like that. I can M-x display-fortune and a fortune is
printed in the minibuffer as soon as the result arrives from the
server. In the meantime Emacs doesn’t block and I can continue my
work.
You can’t do anything you couldn’t already do before. It’s just a nicer way to organize the same callbacks: implicit rather than explicit.
The core object at play is the promise. Promises are already a
rather simple concept, but aio promises have been distilled to their
essence, as they’re only needed for this singular purpose. More on
this later.
As I said, a promise represents a future result. In practical terms, a promise is just an object to which one can subscribe with a callback. When the result is ready, the callbacks are invoked. Another way to put it is that promises reify the concept of callbacks. A callback is no longer just the idea of extra argument on a function. It’s a first-class thing that itself can be passed around as a value.
Promises have two slots: the final promise result and a list of
subscribers. A nil result means the result hasn’t been computed
yet. It’s so simple I’m not even bothering with cl-struct.
(defun aio-promise ()
"Create a new promise object."
(record 'aio-promise nil ()))
(defsubst aio-promise-p (object)
(and (eq 'aio-promise (type-of object))
(= 3 (length object))))
(defsubst aio-result (promise)
(aref promise 1))
To subscribe to a promise, use aio-listen:
(defun aio-listen (promise callback)
(let ((result (aio-result promise)))
(if result
(run-at-time 0 nil callback result)
(push callback (aref promise 2)))))
If the result isn’t ready yet, add the callback to the list of
subscribers. If the result is ready call the callback in the next
event loop turn using run-at-time. This is important because it
keeps all the asynchronous components isolated from one another. They
won’t see each others’ frames on the call stack, nor frames from
aio. This is so important that the Promises/A+ specification
is explicit about it.
The other half of the equation is resolving a promise, which is done
with aio-resolve. Unlike other promises, aio promises don’t care
whether the promise is being fulfilled (success) or rejected
(error). Instead a promise is resolved using a value function — or,
usually, a value closure. Subscribers receive this value function
and extract the value by invoking it with no arguments.
Why? This lets the promise’s resolver decide the semantics of the result. Instead of returning a value, this function can instead signal an error, propagating an error signal that terminated an async function. Because of this, the promise doesn’t need to know how it’s being resolved.
When a promise is resolved, subscribers are each scheduled in their own event loop turns in the same order that they subscribed. If a promise has already been resolved, nothing happens. (Thought: Perhaps this should be an error in order to catch API misuse?)
(defun aio-resolve (promise value-function)
(unless (aio-result promise)
(let ((callbacks (nreverse (aref promise 2))))
(setf (aref promise 1) value-function
(aref promise 2) ())
(dolist (callback callbacks)
(run-at-time 0 nil callback value-function)))))
If you’re not an async function, you might subscribe to a promise like so:
(aio-listen promise (lambda (v)
(message "%s" (funcall v))))
The simplest example of a non-async function that creates and delivers on a promise is a “sleep” function:
(defun aio-sleep (seconds &optional result)
(let ((promise (aio-promise))
(value-function (lambda () result)))
(prog1 promise
(run-at-time seconds nil
#'aio-resolve promise value-function))))
Similarly, here’s a “timeout” promise that delivers a special timeout error signal at a given time in the future.
(defun aio-timeout (seconds)
(let ((promise (aio-promise))
(value-function (lambda () (signal 'aio-timeout nil))))
(prog1 promise
(run-at-time seconds nil
#'aio-resolve promise value-function))))
That’s all there is to promises.
Before we get into pausing functions, lets deal with the slightly simpler matter of delivering their return values using a promise. What we need is a way to evaluate a “body” and capture its result in a promise. If the body exits due to a signal, we want to capture that as well.
Here’s a macro that does just this:
(defmacro aio-with-promise (promise &rest body)
`(aio-resolve ,promise
(condition-case error
(let ((result (progn ,@body)))
(lambda () result))
(error (lambda ()
(signal (car error) ; rethrow
(cdr error)))))))
The body result is captured in a closure and delivered to the promise. If there’s an error signal, it’s “rethrown” into subscribers by the promise’s value function.
This is where Emacs Lisp has a serious weak spot. There’s not really a concept of rethrowing a signal. Unlike a language with explicit exception objects that can capture a snapshot of the backtrace, the original backtrace is completely lost where the signal is caught. There’s no way to “reattach” it to the signal when it’s rethrown. This is unfortunate because it would greatly help debugging if you got to see the full backtrace on the other side of the promise.
So we have promises and we want to pause a function on a promise.
Generators have iter-yield for pausing an iterator’s execution. To
tackle this problem:
iter-next) with the promise’s result as the yield result.All the hard work is done in either side of the yield, so aio-await is
just a simple wrapper around iter-yield:
(defmacro aio-await (expr)
`(funcall (iter-yield ,expr)))
Remember, that funcall is here to extract the promise value from the
value function. If it signals an error, this propagates directly into
the iterator just as if it had been a direct call — minus an accurate
backtrace.
So aio-lambda / aio-defun needs to wrap the body in a generator
(iter-lamba), invoke it to produce a generator, then drive the
generator using callbacks. Here’s a simplified, unhygienic definition of
aio-lambda:
(defmacro aio-lambda (arglist &rest body)
`(lambda (&rest args)
(let ((promise (aio-promise))
(iter (apply (iter-lambda ,arglist
(aio-with-promise promise
,@body))
args)))
(prog1 promise
(aio--step iter promise nil)))))
The body is evaluated inside aio-with-promise with the result
delivered to the promise returned directly by the async function.
Before returning, the iterator is handed to aio--step, which drives
the iterator forward until it delivers its first promise. When the
iterator yields a promise, aio--step attaches a callback back to
itself on the promise as described above. Immediately driving the
iterator up to the first yielded promise “primes” it, which is
important for getting the ball rolling on any asynchronous operations.
If the iterator ever yields something other than a promise, it’s delivered right back into the iterator.
(defun aio--step (iter promise yield-result)
(condition-case _
(cl-loop for result = (iter-next iter yield-result)
then (iter-next iter (lambda () result))
until (aio-promise-p result)
finally (aio-listen result
(lambda (value)
(aio--step iter promise value))))
(iter-end-of-sequence)))
When the iterator is done, nothing more needs to happen since the iterator resolves its own return value promise.
The definition of aio-defun just uses aio-lambda with defalias.
There’s nothing to it.
That’s everything you need! Everything else in the package is merely
useful, awaitable functions like aio-sleep and aio-timeout.
Unfortunately url-retrieve doesn’t support timeouts. We can work
around this by composing two promises: a url-retrieve promise and
aio-timeout promise. First define a promise-returning function,
aio-select that takes a list of promises and returns (as another
promise) the first promise to resolve:
(defun aio-select (promises)
(let ((result (aio-promise)))
(prog1 result
(dolist (promise promises)
(aio-listen promise (lambda (_)
(aio-resolve
result
(lambda () promise))))))))
We give aio-select both our url-retrieve and timeout promises, and
it tells us which resolved first:
(aio-defun fetch-fortune-4 (url timeout)
(let* ((promises (list (aio-url-retrieve url)
(aio-timeout timeout)))
(fastest (aio-await (aio-select promises)))
(buffer (aio-await fastest)))
(with-current-buffer buffer
(prog1 (buffer-string)
(kill-buffer)))))
Cool! Note: This will not actually cancel the URL request, just move the async function forward earlier and prevent it from getting the result.
Despite aio being entirely about managing concurrent, asynchronous
operations, it has nothing at all to do with threads — as in Emacs 26’s
support for kernel threads. All async functions and promise callbacks
are expected to run only on the main thread. That’s not to say an
async function can’t await on a result from another thread. It just must
be done very carefully.
The package also includes two functions for realizing promises on processes, whether they be subprocesses or network sockets.
aio-process-filteraio-process-sentinelFor example, this function loops over each chunk of output (typically 4kB) from the process, as delivered to a filter function:
(aio-defun process-chunks (process)
(cl-loop for chunk = (aio-await (aio-process-filter process))
while chunk
do (... process chunk ...)))
Exercise for the reader: Write an awaitable function that returns a line
at at time rather than a chunk at a time. You can build it on top of
aio-process-filter.
I considered wrapping functions like start-process so that their aio
versions would return a promise representing some kind of result from
the process. However there are so many different ways to create and
configure processes that I would have ended up duplicating all the
process functions. Focusing on the filter and sentinel, and letting the
caller create and configure the process is much cleaner.
Unfortunately Emacs has no asynchronous API for writing output to a
process. Both process-send-string and process-send-region will block
if the pipe or socket is full. There is no callback, so you cannot await
on writing output. Maybe there’s a way to do it with a dedicated thread?
Another issue is that the process-send-* functions are
preemptible, made necessary because they block. The
aio-process-* functions leave a gap (i.e. between filter awaits)
where no filter or sentinel function is attached. It’s a consequence
of promises being single-fire. The gap is harmless so long as the
async function doesn’t await something else or get preempted. This
needs some more thought.
Update: These process functions no longer exist and have been
replaced by a small framework for building chains of promises. See
aio-make-callback.
The test suite for aio is a bit unusual. Emacs’ built-in test suite,
ERT, doesn’t support asynchronous tests. Furthermore, tests are
generally run in batch mode, where Emacs invokes a single function and
then exits rather than pump an event loop. Batch mode can only handle
asynchronous process I/O, not the async functions of aio. So it’s
not possible to run the tests in batch mode.
Instead I hacked together a really crude callback-based test suite. It
runs in non-batch mode and writes the test results into a buffer
(run with make check). Not ideal, but it works.
One of the tests is a sleep sort (with reasonable tolerances). It’s a
pretty neat demonstration of what you can do with aio:
(aio-defun sleep-sort (values)
(let ((promises (mapcar (lambda (v) (aio-sleep v v)) values)))
(cl-loop while promises
for next = (aio-await (aio-select promises))
do (setf promises (delq next promises))
collect (aio-await next))))
To see it in action (M-x sleep-sort-demo):
(aio-defun sleep-sort-demo ()
(interactive)
(let ((values '(0.1 0.4 1.1 0.2 0.8 0.6)))
(message "%S" (aio-await (sleep-sort values)))))
I’m quite happy with how this all came together. Once I had the concepts straight — particularly resolving to value functions — everything made sense and all the parts fit together well, and mostly by accident. That feels good.
]]>