In MAAS, to ensure that a function is run
within its own database transaction, decorate it with
from maasserver.utils.orm import transactional
If a transaction is already in progress when
called, it will instead be called within a savepoint.
Now for the why.
I thought you might ask about that.
On the face of it
@transactional is the same as
@django.db.transaction.atomic. Indeed, on the face of it, you would be
right. In fact,
transactional even uses Django’s
You might ponder some more and notice that
atomic can be also used as
a context manager:
from django.db import transaction
... # Now in transaction or savepoint.
transactional cannot. So
atomic is better, right?
MAAS runs PostgreSQL with a higher-than-default isolation level, REPEATABLE READ. This gives us developers some reassurances that we can read from the database, make some decisions about what we’ve read, then mutate the database without the data having changed beneath our feet. It’s a bit like having a global lock on the database without the downsides of a global lock.
We originally chose to use the highest isolation level that PostgreSQL offers, SERIALIZABLE, but during QA we had “problems” which did not manifest with REPEATABLE READ. The latter offers slightly reduced isolation in that it permits phantom reads:
A transaction re-executes a query returning a set of rows that satisfy a search condition and finds that the set of rows satisfying the condition has changed due to another recently-committed transaction.
In practice this does not seem to harm us so we will stick with REPEATABLE READ for the foreseeable future.
How to “global lock without a global lock”
The database keeps track of the data read and written in a transaction. If another transaction’s reads and writes overlap and conflict, one of the transactions will be rejected with a serialization failure.
These failures are normal. After encountering one it’s typical to automatically retry the whole transaction, and this is what MAAS does. The hope is that MAAS will not often encounter these errors, meaning it gets the behaviour of a global lock but without the loss of concurrency.
If hot-spots arise where a lot of conflict occurs, explicit locking1 can be used to force serialisation, or the application can be restructured to avoid the conflict.
The puzzle reveals itself
Django will let you set the isolation level for a database connection. Beyond that it doesn’t understand anything — it will dump its core on the floor when it sees a serialization failure — so the MAAS development team added this support.
The puzzle is twofold:
Each HTTP request should run within its own transaction, and be retried when there’s a serialization failure.
Database work happening outside of a request also needs its own transaction, with retries.
ATOMIC_REQUESTS is not even close
ATOMIC REQUESTS setting can be used to automatically wrap
each view function with
atomic. Unfortunately that’s only the view,
not the whole request. Middleware — which can and does mutate the
database — runs outside of that transaction.
We needed to go further upstream, so we subclassed Django’s
maasserver.utils.views.WebApplicationHandler. This wraps the whole
request in a transaction.
But we still need to retry failed transactions.
Retrying serialization failures
PostgreSQL signals a serialization failure using error code 40001,
psycopg2 understands fine, but
Django smooshes these failures into its own
Fortunately Django’s developers have foreseen that this could be
problematic to some, and set their exception’s
(even in Python 2). Thus we are able to discover the PostgreSQL error
code. This is encapsulated in MAAS’s
We now have all we need to detect when to retry a request.
To actually retry a request one more thing is critical: the request body
— a file-like object — may have been read, so we need to rewind
it. Twisted is MAAS’s WSGI container, and
it stores the body in a temporary file so we can
seek(0) to return it
to a pristine state.
There are a few other nuances to get this to work smoothly. Django’s
WSGIHandler is also a many-tentacled thing so the mechanics of MAAS’s
WebApplicationHandler (reminder: the latter is a subclass of the
former) are ugly and not generally useful elsewhere.
transactional. It is the distillation of what we’ve so far
learned, and can be used outside of a request.
A serialization failure is a sign of contention, concurrent activity that’s touching the same data. Retrying such a transaction immediately makes it likely that there is still contention, so MAAS leaves a short delay between each retry.
MAAS also sets a limit of 10 on the number of times it will retry a transaction.
retry_on_serialization_failure decorator encapsulates this and
hides some extra smarts. The delays between each retry come from
gen_retry_intervals which yields delays from an exponential series
starting at 10ms, capped at 10s, with full jitter applied.
Full jitter is a fancy way of saying that each delay is multiplied by a random number between 0 and 1. The intended effect is to prevent two or more conflicting transactions from being retried at the same moment, decreasing the chance of repeated conflict.
The idea for this came from Exponential Backoff And Jitter on the AWS Architecture Blog. We have not yet reproduced that article’s analysis in MAAS, and I’ve not heard anything good or bad about MAAS’s behaviour since we implemented this, but I suspect we are silently benefiting from it in larger and busier MAAS installations2.
It should be apparent now why
transactional cannot behave as a context
manager: it may need to call the transactional code multiple times. A
decorator has a function it can call again and again, but a context
manager has no means to re-execute the code block it surrounds.
MAAS does have a
savepoint context manager because we never retry code
within savepoints; it’s whole transactions or nothing.
Eating too much pizza
Suppose I make an
EatPizza RPC call to a pizza-eating robot’s web API
within a transaction. That transaction later fails and is retried n
times by MAAS. The robot would rupture and short-circuit from the n +
1 pizzas in its belly.
MAAS has you covered here with post-commit hooks. They’re beyond the
scope of this article, but
savepoint are core to
making them work.
For now the punchline is: both
be used instead of Django’s
Using this in other projects
If you want to use Django or just Django’s ORM with PostgreSQL (and an isolation level of REPEATABLE READ or SERIALIZABLE) you can find the code in MAAS’s source. All of MAAS’s code is licensed under the GNU Affero GPLv3.
For example, using PostgreSQL’s advisory locks.
It’s not a high priority but I would like to prove that jitter is helping. This would entail an experiment; it’s not something we can measure without altering the behaviour of MAAS. Less invasive metrics, like frequency of transaction retries, could be collected as a matter of course. These would help us tune MAAS. MAAS would not expose these numbers to end-users by default: our goal is that MAAS should work well without endless attention to meters, graphs, switches, and valves.