The first Django 3.1 release candidate came out earlier this week, and I’ve upgraded my blog to this (a one-line change I made through the GitHub web interface thanks to Heroku continuous deployment).
I also upgraded my Heroku PostgreSQL database to v11 so I could take advantage of the new search_type=”websearch” option for PostgreSQL full-text search added in Django 3.1—James Turk contributed that improvement and wrote about it on his blog.
This means my blog search engine now supports advanced operators, like “phrase searches” and -subtractions—try that out here: “signed cookies” -django
Huey as a minimal task queue for Django by J.V. Zammit ( https://twitter.com/jvzammit ) ¶
Can not schedule tasks under 60 seconds.
See https://github.com/coleifer/huey/blob/master/huey/consumer.py If periodic tasks are enabled, the scheduler will wake up every 60 seconds to enqueue any periodic tasks that should be run.
Are you considering adding a task queue to your Django project ? Then this article should be useful to you.
When I was younger task queue with Django project meant celery task queue.
Now that I’m older there are simpler alternatives.
The simplest I found was Huey . But the ideas presented here apply to evaluating all task queues for your Django project.
A colleague achieved significant gains in task execution time by moving off celery. To dramatiq.
This was on a Python 3 project I didn’t work directly on. By significant I mean ~50% throughput. In this case, “tasks” were about handling a simple message that wrote one row to the database, at most. With no real processing of that message.
At about the same time the above happened, I was listening to the DjangoChat podcast.
The below is a transcript from the “Caching” episode from November 2019. Transcript here. The podcast folks were discussing caches. And brought up the usual suspects; Memcached and Redis.
On mentioning Redis, Carlton Gibson calls out how easy it is to add a queue when you have Redis in place. And how much of an “overkill” celery is. Emphasis in the below quote is mine:
have Redis? Yeah. You want to use a queue. So let’s take a good queue package. So, you know, everyone always talks about celery, but celery is overkill for, you know, the majority of use cases. So what’s a good package? Well, there’s one called django-q, which I love and have fun with. That’s nice and simple. And that’s got a Redis back end. So you pip install, right or, you know, apt install Redis. And then you pip install django-q into your project, you know, a little bit of settings, magic, and you’re up and running [..]
I considered these packages to see if I could adopt a more lightweight alternative to celery. Before I continue, by “lightweight” I mean, “lightweight in terms of”:
package size and dependencies
code that I would need to rework to transition from celery to this task queue
works with Redis, a pre-existing component in my stack
The packages I compared considered:
huey (Recommended by Adam Johnson, Django blogger I follow).
I decided to move on with Huey. It was not a clear-cut decision.
My mindset was not about installing the best. It was about installing a minimal package. That removes my dependency on celery/django-celery. And allows me to continue taking that project’s codebase to Python 3.
Why not dramatiq? I did not go with dramatiq because of two reasons:
It required installation of another package. The “Advanced Python Scheduler”, APS, to allow scheduling of tasks.
django-dramatiq, while maintained by the author of dramatiq itself, is “yet another package”.
After my experience with django-celery any extra package scared me. I did not want to end up being unable to use a main library due to a smaller accompanying library not being maintained.
In comparison, for Huey I would only need to pip install huey. The Django integration part is part of the package. Docs here.
In addition, dramatiq is intended for rabbitmq. I see no need for advanced messaging such as that enabled by rabbitmq right now.
Why not django-q ? It is actively maintained. And targeted for Django. And offers a lot of features. But by the looks of it, it offered many features I was not going to be using. At the cost of being less lightweight than Huey.
The above does not mean dramatiq or django-q are not great packages. Far from it. I have them in mind in case the use case changes.