When Life Gives you Unicode

The epic journey of updating Django

In Lily we use Django as our backend framework. And as with all frameworks and external libraries, part of the work is to update to the newer versions in order to get the necessary security patches as well as new features that allow us to develop new features ourselves. Since Django is the core framework we use for our backend development, it’s important to keep it up to date so that we always have long-term support (LTS). Having LTS guarantees security updates.

We were running Django 1.8 which stops having long-term support in April 2018, so it was that time again – to update to a newer Django release. Especially since we wanted to update to Django 1.11 which is the next Django version with LTS, we knew this would be a big project and started it in Summer 2017. This proved to be a good decision since the whole update didn’t go quite as smoothly as we planned and took longer than estimated.

Since our other backenders were busy with building new features and I had some experience from the previous year from helping with our update to Django 1.8, it became my task to get us on Django 1.11. This blog post tells the epic journey of my Django update endeavour with all its highlights and pitfalls. Hopefully it gives some insight to our development process and also helps someone who is having a similar update project ahead of them.

Ok, so how do you exactly update a core framework in your project?

In addition to updating the actual used Django version and making the needed changes to our own code, a big part of this kind of update is listing all the external libraries and python packages we use and checking if they need to be updated for the new Django version as well.

Often big updates and other big features are developed in a separate branch and then merged as one big commit to the development branch. Although this was necessary for some parts of the project, I also learned when updating to Django 1.8 that having smaller tasks and commiting the code in smaller parts works better than releasing the entire project at once. The part where this was logical and possible to do was updating the external libraries and packages. I went through the libraries one by one and checked which version we are running and which version we need to have for Django 1.11. Then I updated to that version and tested it in a separate branch and finally deployed the changes to production.

This of course adds some overhead, but it proved to be a very good strategy. It made it easier and faster to discover issues caused by the individual libraries while ensuring us that the updated libraries work properly. All in all, it made the whole update process more manageable.

However, since we started the update process early, not all the libraries we use had support for Django 1.11 yet. This caused some extra work since I ended up checking every now and then if the support was now available. When it comes to determining if a particular package supports a certain Django version, it’s unfortunately not always as straightforward as one would hope for. This is because the quality of release notes, change logs, and readme files varies a lot between different projects. Sometimes projects mention the supported Django versions in their readme file, sometimes it’s only mentioned in the change log or release notes of a certain version, and sometimes nothing is mentioned at all. In this case I ended up checking the tox.ini file of the particular project to see which Django versions they test against.

Sometimes a project didn’t even test against Django 1.11 yet but reading between the lines made it safe to assume that most likely the package also works on Django 1.11. This seemed to be true particularly if the library supported Django 1.10. Figuring things like this felt like quite a treasure hunt and I sometimes felt myself more as a private investigator than a software developer.

When it comes to actually updating the Django version we use, that was also done gradually. Optimally, we would have first updated to 1.9 and released that to production, but unfortunately 1.9 was already not supported by Django anymore and many external packages we use had stopped supporting it. Thus I needed to update to 1.10 directly and release it as an intermediate step before going to the final target of Django 1.11.

A bumpy ride – Ditch the legacy

As often happens in a bigger project, not everything goes as smoothly as you could hope for. In this case I bumped into a few issues which took rather long to get around. First was getting our asynchronous https requests working with Django 1.10. We were using grequest for non-blocking http requests. However greguests caused issues after updating to django 1.10.

After some debugging, the reason proved to be the gevent package which grequests uses and specifically the fact that gevent does monkey patching for all requests. After many days of googling and trying out different possible fixes, this proved to be a very tricky issue to fix. Finally after a lot of research and after trying several possible solutions without luck, I decided to try something else and tried replacing greguests with an alternative library. After some research requests-futures proved to be a good alternative and switching to it was actually quite simple.

Another bigger issue we encountered was with email templates. Django 1.9 included major changes to the Django templating engine and as part of those, they had removed the django.template.base.add_to_builtins() function we used to restrict the available template tags we offer for users when they create their email templates. We do this for security reasons. My first attempt to fix the issue was to continue with the old solution and work around the changes in Django, even though from the very beginning switching away from the native Django templating to Jinja2 was identified as an alternative solution. After more than a week and several different approaches to get our templates working, my colleague reminded me about Jinja2 and I finally came to my senses and decided to try that instead. And as with the previous issue, this approach turned out to be the correct one and I got it mostly working rather quickly. There was one issue though…

Unicode can be a bitch

Once the Django 1.10 update was deployed, there was one issue with the new email template implementation which was initially missed and only detected in production. Saving new email templates failed if they contained any special characters. This was detected quickly and a solution was also found quickly and immediately deployed. And after that, Django 1.10 was live and fully working.

Since the changes between Django 1.10 and 1.11 are rather small, it was all smooth sailing from here; or so I hoped. Unfortunately unicode took its revenge. The morning after deploying the 1.11 update, we noticed that there were issues with email. It appeared that sending emails containing special characters failed since the update. We started investigating the issue. After a while came to the conclusion that it’s not a straightforward fix and I rolled back the update. There was a small scare that the failed messages were lost, but luckily we were able to recover and resend all the emails using our API. This was manual work though so it took me couple of hours to identify which emails suffered from the issue and then write and run a small script which re-sent all these emails. In the end we were able to recover quite fast and the only thing left was to find the fix to the unicode issue and redeploy the Django 1.11 update.

There were some unexpected issues when trying to find the fix which delayed the progress quite a bit. After determining where the issue seemed to originate from, I tried forcing the email content to be unicode before creating the email object. This was a quite obvious approach and proved to be a working fix. I realized later though that while trying the fix for the first time I accidentally had forgotten to clean my local environment properly. Thus, when building the docker, my environment was still using the old existing compiled code and docker containers, and therefore I thought that my fix didn’t work. Because of this, I continued trying to find different possible root causes and ways to fix the issue and bit by bit drifted away from the solution.

This cost me a few weeks of wasted work and a lot of frustration in researching unicode issues and possible ways to fix those and trying more and more exotic possible solutions. I eventually went back to the drawing board, and tried a similar fix to my original approach, this time after properly cleaning up all the old .pyc files, docker containers, and docker images. Suddenly it worked! Finally the last issue was fixed and it was time to redeploy the Django 1.11 update.

All’s well that ends well – lessons learned

And so, in the beginning of February 2018 we were finally running on Django 1.11, two months before the LTS ended for the version we were running previously. The project took a lot longer than I originally expected, so it’s a good thing I started it early. I didn’t work exclusively on this for the entire 6 months either. That would have been impossible since we needed to wait for several libraries to release their support for Django 1.11 and naturally  there were other tasks and projects which needed my attention.

With all the delays I experiences came some valuable learnings. One of these is to test more regularly with special characters in order to avoid possible unicode issues. The other valuable lesson is to not get stuck with one particular approach when trying to solve an issue, especially when it comes to external libraries. If you start having issues with one, then considering switching to a different library most likely is the correct solution. I’ll be sure to be faster in making these kinds of decisions in the future. The final lesson is always to make sure you clean up your environment properly before starting testing.  

Next update

The Django 1.11 we are now using in Lily is guaranteed to have Long-Term Support at least until April 2020. There is no hurry to update unless we need some new Django features for our development in the future. However, what makes the next Django update bigger is that Django 1.11 is the last Django version supporting Python 2.7. Since we are still using Python 2.7, the first big step which needs to happen within two years is to switch to Python 3. Thus I want to make a plan for the Python 3 conversion project and get started sooner rather than later. Luckily good conversion tools exist and we have gotten some valuable learnings from this update project so hopefully it will be a smoother ride. Wish us luck!