MAAS has a couple of function decorators that are
designed to help blocking code work with non-blocking code:
The blocking, or synchronous, code in MAAS is primarily though not solely the realm of Django, which handles all web API calls and some web views. The biggest responsibility of Django these days is the ORM and database migrations.
Django doesn’t do non-blocking, or asynchronous, at all. For that MAAS uses Twisted. Twisted has many useful pieces that can be used in other projects, but most of the time the Twisted reactor is what you want.
If you don’t already know, the Twisted reactor is an event loop, a main loop, with some conveniences. In a typical application it is the main thread, in which case it will also handle a couple of signals, but it can run as Just Another Thread in your application.
Django often needs Twisted when making RPC calls to other components of MAAS, for example. Likewise, Twisted often needs to use the ORM when querying the database. The hand-off between the two needs to be simple to use and reliable in operation.
Bridging the blocking and non-blocking worlds for MAAS is
Crochet1, a small library that makes it easy to embed a
running Twisted reactor into a non-Twisted application. It sets up the reactor
in a new thread and handles a bit of bookkeeping so you don’t have to. It also
provides the decorators
wait_for_reactor. The latter is a
special case of the former so let’s consider only the former for now.
A function wrapped with
run_in_reactor will find that it is always executed
in the reactor’s thread. It has to be a good Twisted citizen, which means no
blocking IO, but that’s okay because that’s the point.
The caller, on the other hand, is immediately returned a
crochet.EventualResult instance. This object has one interesting method:
wait()will block indefinitely, waiting for the result from the function body which is or will soon be run in the reactor’s thread.
wait(n)will block for up to
nseconds before giving up and raising
wait(n)times-out it does not cancel the execution of the function’s body in the reactor thread;
wait()can be called again, with or without a time-out, or
cancel()can be called to, well, cancel everything.
Should it be useful, multiple threads can call
wait()and each will get the result when it becomes available. Similarly,
cancel()can be called from another thread.
wait_for_reactoris exactly like
wait()is called on the
Crochet is good, but we found that
could not do everything we wanted:
A decorated function would return an
EventualResultto a caller already within the reactor thread.
It’s not possible to provide a default time-out:
run_in_reactorexpects you to call
wait_for_reactorassumes you’re happy to wait forever.
In response we created the
asynchronous decorator. When called from a
reactor thread it is transparent — it calls the decorated function and returns
the result unaltered — but when called from another thread its behaviour is
Used au naturel, calls to the decorated function will return an
@asynchronous def do_something_non_blocking(...): ... # Runs in the reactor. value = do_something_non_blocking().wait(...)
Decorate with a time-out and function calls will block for up to
waiting for the result, after which
TimeoutError will be raised:
@asynchronous(timeout=n) def do_something_non_blocking(...): ... # Runs in the reactor. try: value = do_something_non_blocking() except crochet.TimeoutError: ... # Deal with it.
We use this when we want to convey expected time-outs for operations instead of expecting all callers to figure it out. When building a distributed application it’s hard to know what is a reasonable time to wait; it can help to make a choice and work with it until it becomes apparent that it needs to shift.
Finally, a time-out of
FOREVER will cause calls to block indefinitely for
@asynchronous(timeout=FOREVER) def do_something_non_blocking(...): ... # Runs in the reactor. value = do_something_non_blocking()
Typically we use this when we expect the operation to always take a short
amount of time, and the call would benefit from a clean calling convention,
It is important to run blocking code outside of the reactor. Getting it wrong will block all non-blocking IO, so it pays to be both explicit and to prevent mistakes.
To these ends we have an
@synchronous decorator. This does not call the
function in a non-reactor thread, e.g. by
deferToThread. Instead it raises
AssertionError when called within the reactor thread, and that is all.
To a conscientious developer, however, this is a warning written in fire.
The intention is that liberal use of this decorator will cause unit tests to fail, so the engineer avoids committing bad code in the first place. It works: I cannot remember the last time I saw a related bug.
If you’re developing MAAS, use these decorators liberally. They add only a small overhead. If in doubt, use one.
… or build your own
These are simple decorators. You could implement them yourselves in a short
time. Their value comes not from their lines of code but from being proved in
MAAS. They didn’t come into being with exactly the behaviour I’ve described —
@synchronous changed from a wrapper around
deferToThread to an
assertion. We’ve refined them over a couple of years and what we’ve now got is
MAAS is using version 1.0.0 which is quite a long way behind Crochet’s current release.