Porting MAAS to Python 3
MAAS, up to and including version 1.9, is a Python 2.7 application. We have wanted to move to Python 3 for a long time but it hasn’t been feasible. Recently the wind changed and we decided to port MAAS to Python 3.5 for MAAS’s 2.0 release.
The patch for the port ended up at 152564 lines, but it was actually a mostly
straightforward piece of work. We used 2to3 everywhere, with some
hand-holding, and only had to port a few bits entirely by hand.
It was not perfect. MAAS did not operate out-of-the-box afterwards. We honestly expected that; in fact, we chose that! We worked until the unit test suite passed under Python 3.5 and then landed it straight away because:
-
We did not want this large branch to be long-lived. The cost of maintaining something like that over time, especially when derived from a busy trunk branch, is horrendous. The risk of complete failure can go up rapidly if, say, project priorities shift and the branch languishes, even for a short while.
-
Perhaps more significantly, we decided it was good enough, and that there was a greater chance of success if we put it right in front of everyone in a kaizen-like stop the line approach. I had done most of the early porting work, then two others helped out in a week-long crunch to get it landable, but getting everyone in the team ironing out the last creases gave us the best chance of success.
So, it was not perfect, but it worked. We’ve fixed the issues that came up and have not looked back.
Python 3 is worth it
The MAAS team is learning its way around the shiny new stuff in Python 3 and increasingly taking advantage of it. It’s already clear that Python 3.5 is a better language than Python 2.7. The standard library is more consistent, better organised, and has several useful additions.
The split of byte strings and Unicode strings is The Big One though. The port
to Python 3 would have been worth it for that alone. Python 3.5 fortunately
makes it an easier split to live with because of the reintroduction of
%-formatting for byte strings.
Why had we not done this before?
MAAS’s biggest dependencies are Django and Twisted. Django, until somewhat recently, had no mature Python 3 port, but version 1.8 changed that. The story is somewhat similar for Twisted.
Prior to Python 3.3 it wasn’t considered possible to port Twisted to Python 3 without breaking compatibility with Python 2, so the porting effort is fairly new. A testament to Twisted’s modularity is that it can be ported bit by bit: Twisted in Python 3 is incomplete, but the parts that have been ported are rock solid.
Two modules were missing though: twisted.protocols.amp (which we use for RPC)
and twisted.web.wsgi (the WSGI container in which we host Django). With
guidance and plenty of discussion and reviews from the good people in the
Twisted development community, I ported both of those (Twisted tickets
6833 and 7993 for the curious). They’ll be in an
upcoming release of Twisted.
By the time both Django and Twisted were ready, almost all of MAAS’s other dependencies were ready too, and the work to port the remainder was small.
South to Django
Django posed one additional special problem for MAAS: the upgrade path.
MAAS has relied on South to migrate the PostgreSQL database as MAAS is upgraded. South is gone in Django 1.7, replaced by Django’s own migrations system. Handing over a database from South to Django is fine, as long as the database is in the state defined by the final South migration, mirrored by the initial state of Django’s native migrations.
However, we really wanted and needed MAAS to support upgrades from any supported 1.x release to any supported 2.x release, meaning we can’t guarantee that MAAS has applied all known South migrations before a version of MAAS based on Django >=1.7 has to run with it.
Another member of the MAAS team, Blake, built the mechanism to address this. Out of necessity it is a sausage factory — that is, you should not look inside if you like sausages (or your database) — but it’s a good one; I hope he’ll write about in more detail. Needless to say, it works, and your bare-metal clouds can be upgraded directly to MAAS in Ubuntu Xenial when it’s released.
All done
This port was an intense piece of work, but fun too, educational certainly, and it helps MAAS’s future. Nothing has caused us to regret our decision; the opposite in fact.
We’re running MAAS entirely on Python 3.5 in development, and it’ll be available in Ubuntu Xenial soon in the form of MAAS 1.10. This is essentially MAAS 1.9 running on Python 3.5. We’re already pushing towards a solid 2.0 release to coincide with the release of Xenial.