15 Apr 2011 17:28 - by pieterh

Previous: Loggly

Next: Salt

Dave Martorana writes:

Jason Allum, one of the owners of DmgCtrl ( and I happen to sit next to one and other in the same office space. Despite owning two separate companies, we both work heavily in Python and Django, and often either share code or try to impress each other with neat code tricks.

We were both looking for an asynchronous messaging framework to do some background processing for our apps. On the advice of a mutual friend, I implemented Celery. That required installing celery, kombu, django-celery, and django-kombu. While working just fine, when I showed it to Jason, he got it in his head that it was too heavy-handed a solution, with too many "moving parts" required just to get a simple task processed (and he was right).

Jason had heard of 0MQ a couple months ago, and found out there were Python bindings. He took a couple hours to build a prototype Django plugin using 0MQ and pyzmq, and django-ztask was born. It was extremely light-weight, used Django's built-in reloader support for reloading modified source files (something Celery did not do), and mimicked Celery's basic @task decorator support.

Over the next month, we built it out, adding in delayed processing, better decorator support, full logging, the ability to replay failed tasks, task persistence via the Django ORM, pre-processor calls, and so-on - all in less than 280 lines of code.

So we had almost all the power Celery had offered, plus some fun additions like the .after(…) call which uses 0MQ's ioloop.DelayedCallback to allow for future scheduling of tasks, with full Django integration in an incredibly light-weight package, all because 0MQ made it so easy to do.

So that's the story. 0MQ, Python language bindings, and ~280 lines of code, and we have a full message queue framework for Django - which could easily be ported to other frameworks. We love 0MQ!

Thanks for writing some awesome software!

Get the Django-ztask queue framework via github.

Comments: 1

Add a New Comment
or Sign in as Wikidot user
(will not be published)
- +