MAAS has a couple of function decorators that are designed to help blocking code work with non-blocking code: synchronous and asynchronous.

The blocking, or synchronous, code in MAAS is primarily though not solely the realm of Django, which handles all web API calls and some web views. The biggest responsibility of Django these days is the ORM and database migrations.

Django doesn’t do non-blocking, or asynchronous, at all. For that MAAS uses Twisted. Twisted has many useful pieces that can be used in other projects, but most of the time the Twisted reactor is what you want.

If you don’t already know, the Twisted reactor is an event loop, a main loop, with some conveniences. In a typical application it is the main thread, in which case it will also handle a couple of signals, but it can run as Just Another Thread in your application.

Django often needs Twisted when making RPC calls to other components of MAAS, for example. Likewise, Twisted often needs to use the ORM when querying the database. The hand-off between the two needs to be simple to use and reliable in operation.

Crochet

Bridging the blocking and non-blocking worlds for MAAS is Crochet¹, a small library that makes it easy to embed a running Twisted reactor into a non-Twisted application. It sets up the reactor in a new thread and handles a bit of bookkeeping so you don’t have to. It also provides the decorators run_in_reactor and wait_for_reactor. The latter is a special case of the former so let’s consider only the former for now.

A function wrapped with run_in_reactor will find that it is always executed in the reactor’s thread. It has to be a good Twisted citizen, which means no blocking IO, but that’s okay because that’s the point.

The caller, on the other hand, is immediately returned a crochet.EventualResult instance. This object has one interesting method: wait():

• Calling wait() will block indefinitely, waiting for the result from the function body which is or will soon be run in the reactor’s thread.

• Calling wait(n) will block for up to n seconds before giving up and raising crochet.TimeoutError.

• When wait(n) times-out it does not cancel the execution of the function’s body in the reactor thread; wait() can be called again, with or without a time-out, or cancel() can be called to, well, cancel everything.

• Should it be useful, multiple threads can call wait() and each will get the result when it becomes available. Similarly, cancel() can be called from another thread.

• wait_for_reactor is exactly like run_in_reactor except that wait() is called on the EventualResult for you.

Limitations

Crochet is good, but we found that run_in_reactor and wait_for_reactor could not do everything we wanted:

• A decorated function would return an EventualResult to a caller already within the reactor thread.

• It’s not possible to provide a default time-out: run_in_reactor expects you to call wait(n); wait_for_reactor assumes you’re happy to wait forever.

@asynchronous

In response we created the asynchronous decorator. When called from a reactor thread it is transparent — it calls the decorated function and returns the result unaltered — but when called from another thread its behaviour is more nuanced.

Used au naturel, calls to the decorated function will return an EventualResult:

@asynchronous
def do_something_non_blocking(...):
    ...  # Runs in the reactor.
value = do_something_non_blocking().wait(...)

Decorate with a time-out and function calls will block for up to n seconds waiting for the result, after which TimeoutError will be raised:

@asynchronous(timeout=n)
def do_something_non_blocking(...):
    ...  # Runs in the reactor.
try:
    value = do_something_non_blocking()
except crochet.TimeoutError:
    ...  # Deal with it.

We use this when we want to convey expected time-outs for operations instead of expecting all callers to figure it out. When building a distributed application it’s hard to know what is a reasonable time to wait; it can help to make a choice and work with it until it becomes apparent that it needs to shift.

Finally, a time-out of FOREVER will cause calls to block indefinitely for the result:

@asynchronous(timeout=FOREVER)
def do_something_non_blocking(...):
    ...  # Runs in the reactor.
value = do_something_non_blocking()

Typically we use this when we expect the operation to always take a short amount of time, and the call would benefit from a clean calling convention, i.e. no .wait(...).

@synchronous

It is important to run blocking code outside of the reactor. Getting it wrong will block all non-blocking IO, so it pays to be both explicit and to prevent mistakes.

To these ends we have an @synchronous decorator. This does not call the function in a non-reactor thread, e.g. by deferToThread. Instead it raises AssertionError when called within the reactor thread, and that is all.

To a conscientious developer, however, this is a warning written in fire.

The intention is that liberal use of this decorator will cause unit tests to fail, so the engineer avoids committing bad code in the first place. It works: I cannot remember the last time I saw a related bug.

Use them

If you’re developing MAAS, use these decorators liberally. They add only a small overhead. If in doubt, use one.

If you want them and you’re not using MAAS, they’re in the MAAS source tree² and are licensed using the GNU Affero GPLv3 as is all of MAAS’s source.

… or build your own

These are simple decorators. You could implement them yourselves in a short time. Their value comes not from their lines of code but from being proved in MAAS. They didn’t come into being with exactly the behaviour I’ve described — for example @synchronous changed from a wrapper around deferToThread to an assertion. We’ve refined them over a couple of years and what we’ve now got is about right.

---