In MAAS, to ensure that a function is run within its own database transaction, decorate it with @transactional:

from maasserver.utils.orm import transactional

@transactional
def do_something_databasey():
    ...

If a transaction is already in progress when do_something_databasey is called, it will instead be called within a savepoint.

That’s it.

Now for the why.

<cough>atomic<cough>

I thought you might ask about that.

On the face of it @transactional is the same as @django.db.transaction.atomic. Indeed, on the face of it, you would be right. In fact, transactional even uses Django’s atomic.

You might ponder some more and notice that atomic can be also used as a context manager:

from django.db import transaction

with transaction.atomic():
    ...  # Now in transaction or savepoint.

whereas transactional cannot. So atomic is better, right?

Transaction isolation

MAAS runs PostgreSQL with a higher-than-default isolation level, REPEATABLE READ. This gives us developers some reassurances that we can read from the database, make some decisions about what we’ve read, then mutate the database without the data having changed beneath our feet. It’s a bit like having a global lock on the database without the downsides of a global lock.

Aside: SERIALIZABLE

We originally chose to use the highest isolation level that PostgreSQL offers, SERIALIZABLE, but during QA we had “problems” which did not manifest with REPEATABLE READ. The latter offers slightly reduced isolation in that it permits phantom reads:

A transaction re-executes a query returning a set of rows that satisfy a search condition and finds that the set of rows satisfying the condition has changed due to another recently-committed transaction.

In practice this does not seem to harm us so we will stick with REPEATABLE READ for the foreseeable future.

How to “global lock without a global lock”

The database keeps track of the data read and written in a transaction. If another transaction’s reads and writes overlap and conflict, one of the transactions will be rejected with a serialization failure.

These failures are normal. After encountering one it’s typical to automatically retry the whole transaction, and this is what MAAS does. The hope is that MAAS will not often encounter these errors, meaning it gets the behaviour of a global lock but without the loss of concurrency.

If hot-spots arise where a lot of conflict occurs, explicit locking¹ can be used to force serialisation, or the application can be restructured to avoid the conflict.

The puzzle reveals itself

Django will let you set the isolation level for a database connection. Beyond that it doesn’t understand anything — it will dump its core on the floor when it sees a serialization failure — so the MAAS development team added this support.

The puzzle is twofold:

• Each HTTP request should run within its own transaction, and be retried when there’s a serialization failure.

• Database work happening outside of a request also needs its own transaction, with retries.

ATOMIC_REQUESTS is not even close

Django’s ATOMIC REQUESTS setting can be used to automatically wrap each view function with atomic. Unfortunately that’s only the view, not the whole request. Middleware — which can and does mutate the database — runs outside of that transaction.

We needed to go further upstream, so we subclassed Django’s django.core.handlers.wsgi.WSGIHandler into maasserver.utils.views.WebApplicationHandler. This wraps the whole request in a transaction.

But we still need to retry failed transactions.

Retrying serialization failures

PostgreSQL signals a serialization failure using error code 40001, which psycopg2 understands fine, but Django smooshes these failures into its own django.db.utils.OperationalError.

Fortunately Django’s developers have foreseen that this could be problematic to some, and set their exception’s __cause__ attribute (even in Python 2). Thus we are able to discover the PostgreSQL error code. This is encapsulated in MAAS’s is_serialization_failure function.

We now have all we need to detect when to retry a request.

To actually retry a request one more thing is critical: the request body — a file-like object — may have been read, so we need to rewind it. Twisted is MAAS’s WSGI container, and it stores the body in a temporary file so we can seek(0) to return it to a pristine state.

There are a few other nuances to get this to work smoothly. Django’s WSGIHandler is also a many-tentacled thing so the mechanics of MAAS’s WebApplicationHandler (reminder: the latter is a subclass of the former) are ugly and not generally useful elsewhere.

Hence transactional. It is the distillation of what we’ve so far learned, and can be used outside of a request.

Improving reliability

A serialization failure is a sign of contention, concurrent activity that’s touching the same data. Retrying such a transaction immediately makes it likely that there is still contention, so MAAS leaves a short delay between each retry.

MAAS also sets a limit of 10 on the number of times it will retry a transaction.

The retry_on_serialization_failure decorator encapsulates this and hides some extra smarts. The delays between each retry come from gen_retry_intervals which yields delays from an exponential series starting at 10ms, capped at 10s, with full jitter applied.

Full jitter is a fancy way of saying that each delay is multiplied by a random number between 0 and 1. The intended effect is to prevent two or more conflicting transactions from being retried at the same moment, decreasing the chance of repeated conflict.

The idea for this came from Exponential Backoff And Jitter on the AWS Architecture Blog. We have not yet reproduced that article’s analysis in MAAS, and I’ve not heard anything good or bad about MAAS’s behaviour since we implemented this, but I suspect we are silently benefiting from it in larger and busier MAAS installations².

@transactional and savepoint

It should be apparent now why transactional cannot behave as a context manager: it may need to call the transactional code multiple times. A decorator has a function it can call again and again, but a context manager has no means to re-execute the code block it surrounds.

MAAS does have a savepoint context manager because we never retry code within savepoints; it’s whole transactions or nothing.

Eating too much pizza

Suppose I make an EatPizza RPC call to a pizza-eating robot’s web API within a transaction. That transaction later fails and is retried n times by MAAS. The robot would rupture and short-circuit from the n + 1 pizzas in its belly.

MAAS has you covered here with post-commit hooks. They’re beyond the scope of this article, but transactional and savepoint are core to making them work.

For now the punchline is: both transactional and savepoint must be used instead of Django’s atomic.

Using this in other projects

If you want to use Django or just Django’s ORM with PostgreSQL (and an isolation level of REPEATABLE READ or SERIALIZABLE) you can find the code in MAAS’s source. All of MAAS’s code is licensed under the GNU Affero GPLv3.

---