Python

The pain of Jinja2

There are many reasons to choose a generally better solution, but there is only one reason you don’t do that, which is it’s not suitable for your problem domain.
Who said that? Well, it’s me. LOL

The origination

In the past two weeks, we have been struggling with Jinja2 template. I have read the famous Two scoops of Django and Q&A on StackOverflow with regard to choosing a template system for Django web project, Jinja2 is praised here and there, so it seems we cannot be wrong to replace native template system called DTL(Django Template Language) with it.

When we first build the whole project, we use DTL. When the web project is open for test, it seems sort of slow when opening pages. After testing some environment changes, none of them can help. Considering the goodness of Jinja2, and we observe the template rendering time is long, so we decide to try Jinja2.

The substitution path

What have we done for this solution?

First of all, I changed the configuration to make DTL and Jinja2 coexist. Following completely the user guide from Jinja2 official website does not work, so I have to search around to make it work finally. Of course, at first I think the best solution is only use Jinja2 because it’s simple. But it turned out that would not work, because the 3rd-party module allauth(We use it for user management) brings lots of HTML pages and we cannot pay the cost to replace them all in Jinja2 syntax. So they have to coexist, and there are three ways to achieve it:

  1. Place DTL template pages and Jinj2 pages in different directories
  2. Use regex matching on URL pattern
  3. Use template page extensions

I tried the second method, it does not work fine. After comparing method 1 and method 3, I tried the third one cause it looks like an easy solution. BTW, since we have a common base template called base.html, by using this technology we have to copy it to two files: base.html and base.jinja2.html, and they have same content in different syntax.

After the configuration is done, then the real trouble surfaces. It is context processor. We use context processor to hold system menus and other static or relatively static content, but Jinja2 does not have that mechanism, and some guy even said if you have heavy logic in context processor, don’t use Jinja2. Anyhow, there’s always a solution, we refactor the function in context processor by moving it into back-end views. At this point, we don’t clearly recognize the difference. But in the end, we analyze and conclude this is the performance bottleneck for “our” Jinja2 solution. Is there any alternative solution for this problem? Yes, use global variables, but we don’t have time to try it again.

Back to our Jinja2 substitution path, after context processor is fixed, the last big thing is replace DTL syntax with Jinja2 syntax in HTML pages. Here first I want to write a progam, but the quickest way is to use regex replacement. The difficult point of regex replacement is writing correct regular expressions. We use an interesting open source tool called VerbalExpression(JSVerbalExpression) to achieve it. There are lots of examples in its test code. It’s fun and efficient.

Is it done? No. Last thing at last: we have some custom filters in DTL syntax, it’s not a simple substitution work, we have to change it by using Jinja2 syntax. After this is finished, the whole Jinj2 substitution work is done.

The result

We test the substitution result, and it is frustrating. We again studied the solution and the praising words about Jinja2. The big difference is when rendering strings in memory, Jinja2 is definitely faster, but for database queries, the answer is No. We feel the pain.

Anyway, we need to fix this.

The final solution

Since database queries are the main cause of performance bottleneck, we can cache static content in cache system such as Redis. We have used Redis before, so we take couple hours to set it up. Redis is simple and helpful, the result is positive. 2 weeks vs. couple hours, what a pain. So the conclusion is whatever they say about it, you need to measure it and make it work in your scenario, whilst you need to pay attention to the cost.

We finally decide not to use Jinja2, but stay with DTL. We keep the Jinja2 branch and we keep a close eye on both of them, maybe one day we still need it. And until then, the substitution experience can help a lot.

No pains, no gains. Have a good day!

The Way of Django

As a software architect, compared to software developer, I face more choices and challenges every day when constructing software systems. There’s no silver bullet in software development as told by Fred Brooks, but there’s always a better way to build software considering cost and productivity, and the last but the most important factor for modern software development: being happy programmers.