Stephen McDonald

For the past couple of years, this site has been powered by a static site generator called Jekyll. With Jekyll there isn’t a database, or even an admin area for managing content. Instead, content is stored in text files, which are edited locally on your computer, formatted in one of the lightweight markup languages supported by Jekyll, such as Markdown. Layout templates are used for the design, which can contain HTML and programming logic, similar to PHP code or Django templates, however this code is only executed once when the site is published, not on every page request. When the site is published, Jekyll will look at all of the Markdown content files, and use the layout templates to generate an entirely static site made up of HTML files.

For hacker types it’s the perfect blogging platform, as it addresses a number of complaints they typically have with traditional blogging tools like Wordpress or Mezzanine:

• Editing can be done via your favourite text editor, without dealing with the limitations of WYSIWYG editors
• All content can be version controlled through your favourite version control system, such as Git, Mercurial, or even SVN
• Programming logic can still be implemented in layout templates, and extended with Ruby plug-ins as required
• Site performance is blazing fast, since only static HTML files are served

Programming logic in Jekyll’s layout templates is provided by a templating language called Liquid. Those coming from Python who have used Django or Jinja templates, will feel right at home with Liquid, as it shares an almost identical syntax. As with Django templates, custom tags and filters can be written, in Ruby though, with an API that’s actually much more simple.

Jekyll powered blogs have no particular hosting requirements, beyond a web server that can serve up static files. Publishing changes to the site is simply a matter of running the Jekyll command locally to build the site, and pushing the generated files to your web server, perhaps via FTP. Now GitHub provides an even easier, more integrated approach than this. They provide a service called GitHub Pages, where you can nominate one of your code repositories for hosting a static website - simply push the changes for your static site to the repository via Git, and the site is up to date. GitHub Pages also integrates with Jekyll, so if your repository is set up as a Jekyll project, GitHub will automatically generate the static site that gets hosted.

In the Sandbox

Jekyll integration with GitHub Pages really comes close to a blogging nirvana for programmers, but it’s not all sunshine and roses. Understandably, the Jekyll instance on GitHub Pages is sandboxed, and you’re unable to extend Jekyll with your own custom Liquid tags and filters. This would require running arbitrary Ruby code, which would pose a security risk to GitHub. So you’re restricted to the built-in tags and filters provided by Liquid and Jekyll, which brings us to the point of this post. Since moving to Jekyll, I’ve had a few cases where I needed to extend things beyond what Jekyll provides. The adage constraints foster creativity has certainly rung true for me with these, and by bending the built-in Liquid tags and filters in strange and sometimes inefficient ways, I’ve been able to achieve what I’ve needed. Following are the details of some of these weird tricks I’ve come up with.

True Word Count

Jekyll adds a number_of_words tag to Liquid that can be used to display the number of words in an article. You’ll see I make use of it on this site, to generate the visual bars showing the size of each article on the homepage. Unfortunately this tag is particularly naive - it simply splits the contents up into chunks separated by spaces, and returns the number of items. The problem with this is that by the time the post makes its way into the template, it has already been converted to HTML, so all of the HTML tags and their attributes get included in the word count. Jekyll, aimed especially at programmers, also supports snippets of syntax highlighted code in articles. These have their own tag syntax, which would make extracting them out of article content prior to determining word count, seem particularly easy, so that the code in snippets is also omitted from the overall word count, however the number_of_words tag doesn’t do this either.

The Liquid code below is able to achieve a true word count for each article on the site. It isn’t perfect, but it works correctly for my site.

<ul>
{% for post in site.posts %}
    {% assign post_words = 0 %}
    {% assign lines = post.content|split:'pre>' %}
    {% for line in lines %}
        {% assign mod = forloop.index|modulo:2 %}
        {% assign line_words = line|strip_html|number_of_words|times:mod %}
        {% assign post_words = post_words|plus:line_words %}
    {% endfor %}
    <li><a href="{{ post.url }}">{{ post.title }}</a>({{ post_words }} words)</li>
{% endfor %}
</ul>

This code works off the assumption that the highlighted code snippets in articles are the only things that will generate HTML <pre> tags. It splits the entire article content on the string pre> which should match both opening and closing <pre> and </pre> tags. Note that it assigns the result of this split to a variable called lines. This is specifically necessary - the Liquid authors claimed that iterating through the results of the split filter isn’t possible, and they’re correct, it doesn’t work by default. Assigning it to a temporary variable is a trick I accidentally discovered though, that does in fact allow it to work.

We then loop through each of the lines, where we can assume every even line (2nd, 4th, etc) contains a block of code we want to omit from the overall word count. We then strip the HTML from the odd lines using Jekyll’s strip_html tag, to get the actual text content, and sum the result of the number_of_words tag on each of these. You’ll notice a strange bit here, where we multiply the modulo of 2 on the loop index, which will give us a value of 0 for even lines with code snippets, and 1 for lines with real words. The reason for this is that like Django templates, Liquid conditional tags like if don’t behave like regular programming languages - their conditional nature is only applicable to what’s rendered to the browser. Tags and filters within conditions that aren’t met are still executed, so we can’t simply wrap our word summing in {% if mod == 1 %}.

Frequency Tag Sort

Articles in Jekyll support traditional keyword tags as you’d expect. On the homepage of this site, I generate a list of all tags from all articles. The problem though, is that the tags are in arbitrary order when made available by Jekyll, making the list hard to digest in a meaningful way. We could sort them alphabetically, but I thought the best option would be to sort them by frequency, with the most commonly used tags appearing at the top of the list. Liquid provides the template tag sort for sorting data by a given property, but the tag structure provided by Jekyll is a hash, which as best as I can tell, isn’t supported by Liquid’s sort tag. I therefore came up with the following approach for sorting by frequency:

{% assign tags_max = 0 %}
{% for tag in site.tags %}
    {% if tag[1].size > tags_max %}
        {% assign tags_max = tag[1].size %}
    {% endif %}
{% endfor %}

<ul>
{% for i in (1..tags_max) reversed %}
    {% for tag in site.tags %}
        {% if tag[1].size == i %}
        <li>
            <a href="/tag/{{ tag[0] }}/">{{ tag[0] }}</a>
            ({{ tag[1].size }}){% unless forloop.last %}, {% endunless %}
        </li>
        {% endif %}
    {% endfor %}
{% endfor %}
</ul>

The above approach is quite ridiculous. It first iterates through each of the tags, to determine what the highest tag frequency is. Then it iterates from that highest frequency, down to 1, and within each iteration, loops again through all tags, displaying them if their frequency matches the current outer loop. This is insanely inefficient, but it gets the job done acceptably, considering again that this code is only run when the site is published, not on each request.

Conclusion

To be honest, none of the above is strictly necessary with Jekyll. I could easily achieve what I need by writing my own Liquid tags to do the job properly. This would even work with GitHub Pages, I’d just need to generate the static version of the site myself, and commit the generated HTML files to version control. But no. In the above cases, I saw the tasks as a challenge - a programming puzzle of sorts, and really enjoyed solving them in the end.

Mezzanine has come a long way over the last few years, now powering hundreds of rich, content-driven sites. For the most part these sites follow a similar pattern, one which Mezzanine is heavily geared towards: B2C sites comprised of heterogeneous, hierarchical content. In this regard Mezzanine has been a huge success, however in its entirety it’s capable of quite a lot more. Out of the box it provides a range of useful utilities, that aren’t particular geared towards a typical corporate site, but instead are aimed at a more social style of web application. These include things such as public user accounts with configurable profiles, threaded comment discussions, ratings, and much more - features that form the foundation of many of today’s most popular social web apps. This presents Mezzanine as a great foundation for building all types of social web applications, not just corporate, content-driven sites.

Interestingly, these extra utilities provided by Mezzanine form the core feature set of several very popular news aggregation sites, such as Hacker News and Reddit, and even their predecessors Slashdot and Digg. For those unfamiliar with these types of sites, they essentially provide crowd-sourced news, collectively controlled by contributing users of the site. Users create an account, submit links to interesting websites that they’ve discovered, and other registered users can then rate these links up or down - the end goal being that with enough positive votes, links are promoted to the front page of the site, gaining a ton of exposure. Users can then participate in threaded discussions which are provided for each website link, where comments themselves undergo the same crowd-sourced ranking, promoting the best comments to the top of the discussion.

To demonstrate these extra capabilities of Mezzanine, I set out to do something entirely lacking in innovation, by creating a direct Hacker News clone using Mezzanine. Creating a clone like this from scratch isn’t exactly a challenging task, and without doing anything remarkably different feature-wise, one that has little hope of gaining the type of traction that these sites enjoy. My intention here however, was to specifically explore how little work would actually be involved when leveraging Mezzanine as much as possible.

The end result is a project I’ve called Drum, and it turned out that by using Mezzanine as the foundation, each of the features required to implement a basic Hacker News clone required almost zero effort!

The remainder of this article assumes a basic understanding of Django, and will walk through what was involved in pulling each of these features together to create Drum.

Mezzanine is just Django: Links

The answer to many of the most frequently asked questions on the Mezzanine mailing list, often involves simply pointing people to the relevant section in Django’s documentation. This is one of the more commonly overlooked aspects of Mezzanine - it’s simply a standard Django project, and you’re free to implement your own models, views, templates, or any other component that Django provides.

With that in mind, we’ll get the boring stuff out of the way first, and put together some basic views and a model for the main content, namely the user contributed website links. Here’s our initial Link model:

from django.db import models
from mezzanine.core.models import Displayable, Ownable

class Link(Displayable, Ownable):
    link = models.URLField()

You’ll see we make good use of some Mezzanine building blocks here, even though we’re going it on our own. The abstract Displayable model is heavily used throughout Mezzanine. It provides all the common features needed for an actual web page, such as a title, a slug (the path portion of a URL), a published status and date, and meta data, which includes a description field we can expose to users when creating new links. The Ownable model provides the relationship between our links and the users who created them, so with all that in place, the only field we need to define initially is the actual link to the website itself.

Next up are our views for the Link model, again in plain old Django. We’ll use the generic views provided by Django, for listing, creating and viewing links:

from django.forms.models import modelform_factory
from django.views.generic import ListView, CreateView, DetailView
from .models import Link

class LinkList(ListView):
    model = Link

class LinkDetail(DetailView):
    model = Link

class LinkCreate(CreateView):
    model = Link
    form_class = modelform_factory(Link, fields=("title", "link", "description"))

This is almost no effort thanks to Django’s generic views - the only deviation here is a custom form class in our create view, so that we can limit the fields that will be editable in the form used for submitting new links.

I can’t believe it’s Mezzanine: Threaded Comments and Ratings

Now we’ll start blazing through the Mezzanine features that require almost zero effort to apply to our project. Mezzanine provides the Django application mezzanine.generic, which houses several different utilities that can be generically applied to any Django model. Each of these use Django’s generic relations, and come in the form of a custom Django model field. Provided are things like keyword tagging, ratings, and threaded comments. We’ll start by adding threaded comments and ratings to our Link model:

from django.db import models
from mezzanine.core.models import Displayable, Ownable
from mezzanine.generic.fields import CommentsField, RatingField

class Link(Displayable, Ownable):
    link = models.URLField()
    comments = CommentsField()
    rating = RatingField()

All we’ve done here is added the CommentsField and RatingField fields to the Link model. Below is our link detail template, which will display the description given to the submitted link, and the threaded comment discussion following it. You’ll see it uses the comments_for and ratings_for template tags, to render the comments and rating widget:

{% extends "base.html" %}
{% load comment_tags %}

{% block title %}
<a href="{{ object.link }}">{{ object.title }}</a>
{% endblock %}

{% block main %}
<p>{{ object.description }}</p>
<p>{% rating_for object %} by {{ object.user.username }} {{ object.publish_date|timesince }} ago</p>
{% comments_for object %}
{% endblock %}

These two template tags are Django inclusion tags, that use a corresponding template file to include in the page. This means we can override them in our project, in order to customise the way comments and ratings are rendered. For Drum, we’ll mostly leave the comment templates alone, but for ratings, we want to customise the template to display up and down arrows for voting, rather than the default form with numeric rating choices (1 to 5 by default in Mezzanine). But before we get to the template, we’ll first configure some Mezzanine settings in Drum’s settings.py module:

COMMENTS_USE_RATINGS = True
COMMENTS_ACCOUNT_REQUIRED = True
RATINGS_ACCOUNT_REQUIRED = True
RATINGS_RANGE = (-1, 1)

The first setting COMMENTS_USE_RATINGS is just a basic flag that controls whether individual comments can be rated, which we want in Drum. The next two settings, COMMENTS_ACCOUNT_REQUIRED and RATINGS_ACCOUNT_REQUIRED are fairly obvious - we set these to True to ensure only registered users can add comments and ratings. The RATINGS_RANGE setting is the interesting one - it lets us define the range of valid rating values, which we specify -1 and 1 for, to represent a negative and positive rating. With that in place, we can now customise our rating include template:

<div class="rating">
    <form method="post" action="{% url "rating" %}">
        {{ rating_form }}
        {% csrf_token %}
    </form>
    <span class="arrows">
        <a href="#"><i class="icon icon-arrow-up"></i></a>
        <a href="#"><i class="icon icon-arrow-down"></i></a>
    </span>
    <span class="score">
        {{ rating_sum }}
    </span>
</div>

The template code here is almost identical to the default one used in Mezzanine. It displays the rating form (using the rating_form variable provided by the inclusion tag) and a value for the current rating (we use the rating_sum variable for this, but the inclusion tag also provides rating_count and rating_average, which is used in the default template). The real change we’ve made here though, is to add some links with up and down arrow icons, as well as some extra CSS classes which we can then use to hide the original rating form, and apply some JavaScript to the arrow links that submit the hidden rating form::

$(function() {
    $('.arrows a').click(function() {

        var arrow = $(this);
        var index = arrow.find('i').hasClass('icon-arrow-up') ? 1 : 0;
        var container = arrow.parent().parent();
        var form = container.find('form');

        form.find('input:radio')[index].checked = true;

        $.post(form.attr('action'), form.serialize(), function(data) {
            if (data.location) {
                location = data.location;
            } else {
                container.find('.score').text(data.rating_sum);
            }
        }, 'json');

        return false;
    });
});

As an added bonus, rather than simply submitting the hidden rating form, we post it via AJAX. Both the comment and rating views provided by the mezzanine.generic app know how to deal with AJAX requests, and will return an appropriate JSON response. In the case of ratings, the JSON response will include a location to redirect to, should the user needs to log in, otherwise it will contain the new rating sum, which we then use to update the visible rating sum directly in the web page, without ever leaving it.

Profiling Users

Now that we’ve got ratings and threaded comments hooked up, and restricted to registered users, let’s get users actually registering and logging in. Once again this is a breeze with Mezzanine, which provides the mezzanine.accounts app for implementing public user accounts. The accounts app provides all the features you’d expect, such as forms, views, and templates, for users to register an account, log in, update their account information, and more. Support for different sign-up flows such as requiring email verification, or manual approval by a staff member, are provided via the ACCOUNTS_VERIFICATION_REQUIRED and ACCOUNTS_VERIFICATION_REQUIRED boolean settings.

On top of that, Mezzanine’s accounts app is also fully integrated with Django’s Profile models. This means you can create a profile model associated with each user, and Mezzanine will pick up each of its fields and use them where applicable, such as the registration and account update forms. Mezzanine also defines the ACCOUNTS_PROFILE_VIEWS_ENABLED setting, which when enabled, will provide public profile pages for each user account.

With mezzanine.accounts added to the INSTALLED_APPS setting in Drum, next up we’ll define a simple profile model, that will allow users to provide a short bio and link to their own website:

from django.db import models

class Profile(models.Model):

    user = models.OneToOneField("auth.User")
    website = models.URLField(blank=True)
    bio = models.TextField(blank=True)
    karma = models.IntegerField(default=0, editable=False)

You’ll notice we’ve also defined a karma field on the profile for each user, that can’t be edited via a form. Karma is an interesting feature of sites like Hacker News and Reddit, where the idea is to provide a score for each user that indicates their reputation on the site. Generally a user’s karma will increase as their links and comments are rated positively by other users, and decrease accordingly if the ratings given are negative.

With a karma field in place on our user profiles, all that’s left to do is update the user’s karma each time a rating gets saved, which we can do using Django’s model signals:

from django.db import models
from django.db.models.signals import post_save
from django.dispatch import receiver
from mezzanine.generic.models import Rating

@receiver(post_save, sender=Rating)
def karma(sender, **kwargs):
    rating = kwargs["instance"]
    value = int(rating.value)
    # Since ratings are either +1/-1, if a rating is being edited,
    # we can assume that the existing rating is in the other direction,
    # so we multiply the karma modifier by 2.
    if not kwargs["created"]:
        value *= 2
    content_object = rating.content_object
    if rating.user != content_object.user:
        queryset = Profile.objects.filter(user=content_object.user)
        queryset.update(karma=models.F("karma") + value)

Time-scaled Ranking

All we’ve done so far is set up some basic crud features for our user-contributed links, and configure a handful of features provided by Mezzanine. We have one more requirement though, and I’ve saved the best for last.

One of the key ingredients of sites like Hacker News and Reddit, is the way ratings are used to rank link submissions and comments throughout the site. Links aren’t simply ranked by their overall rating, as this would lead to stale content on the site, where the most popular links of all time would remain at the top of the homepage. Instead, the ranking of links also takes their age into account. Ideally the most popular links should quickly rise toward the top of the homepage if they receive a high number of positive ratings, soon after being posted to the site. Links toward the top of the homepage would then gradually decline in their ranking, as their age increases, and as newer links are submitted that also receive a strong number of positive ratings.

A quick search online for how to implement this style of time-scaled ranking revealed a few different approaches, with varying complexity. I opted for one of the more simple algorithms I found, mostly because it actually works well, but also because it can be easily implemented in SQL, which means we can let the database take care of sorting results, and still make use of Django’s ORM and pagination.

from django.conf import settings
from django.utils.timezone import now

def order_by_score(queryset, date_field):
    """
    Take some queryset (links or comments) and order them by score,
    which is basically "rating_sum / age_in_seconds ^ scale", where
    scale is a constant that can be used to control how quickly scores
    reduce over time. To perform this in the database, it needs to
    support a POW function, which Postgres and MySQL do. For databases
    that don't such as SQLite, we perform the scoring/sorting in
    memory, which will suffice for development. We also have a
    date_field arg for controlling which field on the model represents
    its creation date.
    """

    scale = getattr(settings, "SCORE_SCALE_FACTOR", 2)

    # Timestamp SQL function snippets mapped to DB back-ends.
    # Defining these assumes the SQL functions POW() and NOW()
    # are available for the DB back-end.

    timestamp_sqls = {
        "mysql": "UNIX_TIMESTAMP(%s)",
        "postgresql_psycopg2": "EXTRACT(EPOCH FROM %s)" ,
    }
    db_engine = settings.DATABASES[queryset.db]["ENGINE"].rsplit(".", 1)[1]
    timestamp_sql = timestamp_sqls.get(db_engine)

    if timestamp_sql:
        score_sql = "rating_sum / POW(%s - %s, %s)" % (
            timestamp_sql % "NOW()",
            timestamp_sql % date_field,
            scale,
        )
        return queryset.extra(select={"score": score_sql}).order_by("-score")
    else:
        for obj in queryset:
            age = (now() - getattr(obj, date_field)).total_seconds()
            setattr(obj, "score", obj.rating_sum / pow(age, scale))
        return sorted(queryset, key=lambda obj: obj.score, reverse=reverse)

The above code first defines some different SQL snippets for converting date fields into timestamps, since this varies across different databases like PostgreSQL and MySQL. We also only define these for database back-ends that support a POW() function, as needed by our ranking algorithm. We then provide a fallback for any other database, where we don’t use SQL for ranking, and simply sort results in memory, which allows use to still use a database like SQLite during development, with a limited amount of data.

A feature like this might typically be implemented as a Django model manager, however since Drum doesn’t have control of the model for threaded comments, we implement this as a regular Python function, that we can wrap around calls to the model managers for links and comments. Here’s our previous LinkList view, with time-scaled ranking applied:

from django.views.generic import ListView
from .models import Link
from .utils import order_by_score

class LinkList(ListView):

    def get_queryset(self, *args, **kwargs):
        # Also call select_related on the user profiles, so we can
        # display submitters' usernames and karma against each link.
        qs = Link.objects.all().select_related("user", "user__profile")
        return order_by_score(qs, "publish_date")

Lastly, we want to apply our order_by_score function to the threaded comment discussion as well. At first glance this isn’t seemingly straight-forward, since these are entirely managed by template tags in Mezzanine, but with a little insight into how these work, we can achieve what we want quite easily.

The comments_for template tag we mentioned earlier is the one responsible for rendering a threaded comment discussion. It takes a parent argument, which to begin with is the object that has threaded comments attached to it - the Link model in Drum’s case. It then includes the comment template, which iterates through a single level of comments, displaying each one. For each comment, we then call the comments_for tag again, passing the comment in as the parent. This continues recursively, and is how the entire tree of comments gets rendered.

This might sound like a performance nightmare, if we assume each call to the comments_for tag performs a database query to retrieve a single branch of comments, however the tag is a bit smarter than this, and only queries the database when first called, building up the tree of comments and storing them in a template variable. Subsequent calls to comments_for then check for that variable, and reuse it if found.

Armed with this knowledge, we can easily create a template tag that mimics the initial call to comments_for, retrieving and building the comment tree, and storing it in the template context, prior to comments_for even being called. By using the correct variable name (all_comments), the first call to the comments_for tag will entirely bypass retrieving comments from the database, when it sees they’re already in the template context.

from collections import defaultdict
from django import template
from ..utils import order_by_score

register = template.Library()

@register.simple_tag(takes_context=True)
def order_comments_by_score_for(context, link):
    comments = defaultdict(list)
    qs = link.comments.visible().select_related("user", "user__profile")
    for comment in order_by_score(qs, "submit_date"):
        comments[comment.replied_to_id].append(comment)
    context["all_comments"] = comments
    return ""

Here’s our link detail template again, with a call to the new order_comments_by_score_for tag, prior to comments_for being called.

{% extends "base.html" %}
{% load comment_tags %}

{% block title %}
<a href="{{ object.link }}">{{ object.title }}</a>
{% endblock %}

{% block main %}
<p>{{ object.description }}</p>
<p>{% rating_for object %} by {{ object.user.username }} {{ object.publish_date|timesince }} ago</p>
{% order_comments_by_score_for object %}
{% comments_for object %}
{% endblock %}

Conclusion

That’s a wrap. Even for those unfamiliar with Django, it’s clear from the amount of code presented here how easy a task this was. Here are a few other features I implemented that aren’t covered here:

• Skinned with FlatUI and pjax
• Views for ordering all links by their age, all comments by age or score, and links and comments by individual user (from their profile page)
• Links also display the domain they’re hosted on (a method on the Link model)
• To prevent duplicate links being posted, a duplicate will redirect to the original submission, until an expiry period has passed (a month for example)
• All usernames display their karma score, and link to their profile pages
• A Django management command for importing links from an RSS feed

What’s next for Drum? As I mentioned, the likelihood of the current demo gaining popularity is slim, however I think it’s an interesting step forward in the Mezzanine eco-system. Consider what would be required to turn Drum into a clone of other popular web applications:

• Reddit: Use Mezzanine’s keywords (already applied to our Link model via the Displayable model) to break up the link listing view into individual views by keyword
• Stack Overflow: Remove the link field from our Link model, and allow submitters to mark one of the related comments as correct
• Every forum application under the sun: Remove the link field and ratings altogether, and apply some extra detail to our listing templates, like comment count and last commenter

The potential evolution of Drum becomes quite obvious: a general forum solution for Mezzanine, in the same way Cartridge provides ecommerce for Mezzanine. Are you interested in making this a reality? Check out the Drum source code and let’s make it happen!

I’ve been playing around with Publish/Subscribe queues (or pub-sub queues) for the last few months, which has led me through some research that has been very interesting for me personally. I’ve wanted to write about my experiences for a while now, but unfortunately this post continues to unwrite itself over time, as I refine my research and disprove any assumptions I’ve made along the way. Nonetheless, I’ve now decided to write about what I’ve worked on so far, and with that in mind, I’ll make the disclaimer that this post isn’t an attempt to draw any definitive conclusions, but merely to talk about my learning and experience as a work in progress. We’ll take a look at some pub-sub use cases, using Redis and ZeroMQ, from both Python and Google’s Go language.

Motivation

Real-time web applications have been a great area of interest for me over the years. I use the term real-time quite loosely here, as real-time in software engineering technically refers to a lower level set of system constraints. Instead I’m referring to a style of web application where users can interact with each other in a seemingly instantaneous way, such as a chat room or a multi-player game of some sort.

Applications like these where users interact in real-time will generally require some separate form of pub-sub component, which handles the communication when an event is triggered by one user, and needs to be broadcast to all the other users who should be notified of it. The event might be a character sprite moving on a game screen, or a message written in a chat room. Using a pub-sub component that is separated from the main web application is also as much an architectural requirement as it is a functional one - by moving the communication layer out into a separate component, the responsiveness of publishers within the application is no longer bound to the volume of messages being sent to subscribers. Also of equal importance, our web application layer no longer depends on any shared state, and can be spread across multiple threads, processes or servers.

I’ve built several toy applications like this in the past, such as DrawnBy (shared drawing) and Gamblor (shared gambling, chat and character movement), and in each case I’ve always used Redis as an in-memory store for transient data - temporary shared state across the app, that doesn’t need the persistence guarantees and subsequent performance costs that a traditional database comes with. Now Redis has been described as the Swiss army knife of databases, and it’s a great description for it - not only does it provide a wide variety of built-in data structures, it also offers a pub-sub queue, which has made Redis a well-suited companion for these types of applications I’ve built.

Since building these apps, I’ve been experimenting with some ideas around real-time web games, and character movement throughout them. Suppose we wanted to create a two-dimensional universe, of perpetual width and height, that users could move around within - how would we design it in a scalable way? The basic concept I came up with was to partition the world into a virtual grid, where each square on the grid uses its own pub-sub channel for communicating movement events to users, and each user publishes and subscribes to the grid square they’re on, as well as the ones surrounding them.

Here’s a diagram illustrating the idea, where each coloured area is a player’s screen. The green area might be a wide-screen desktop, the blue area a mobile device, and the red one, well, it’s a box.

Each grid square is a communication channel on a perpetual grid

In the diagram, the red and green players are subscribed to each other’s movements, as are the blue and green, but the red and blue players don’t receive notification of each other’s movements, since they’re not subscribed to any common channels.

I built a working prototype of the above design using Python, gevent and Redis. There were a handful of intricacies that came into play that the above model doesn’t go into. Things like managing extraneous grid subscriptions surrounding the player, to ensure smooth transitions between grid squares, and finding the sweet spot in limiting the volume of events being sent over the network by faking some of the character movement on screen.

Once I had this working, without a literal or figurative end-game in mind, I had the luxury of being able to further focus on the performance and scalability of such a system. My first step down this path of digression was to rewrite the back-end in Google’s Go language. I’d attended a few of the Sydney Go Meetups and had been looking for a chance to dive into the language, so this seemed like a good opportunity to do so.

The port to Go was a fun experience. It looked very similar to the Python version in terms of design and the amount of code required, which is a testimony to Go’s expressiveness and the breadth of its standard library. I’ll talk more about Go later in this post, but for now, the rewrite didn’t yield much difference in the amount of work the grid system could deal with. As with most real-world projects, its limitations were likely to be architectural rather than being bound to the language used.

I then started to look more closely at the pub-sub setup. With the grid design in place, we’d have a straight-forward path ahead for partitioning the pub-sub channels over multiple Redis instances, but how much volume could a single Redis pub-sub instance handle? Was there a tool that could handle more? I decided to explore this question further, and from that point on, I was entirely swept away into the realm of pub-sub benchmarking.

Broker Characteristics

Before we dive into comparisons, let’s look at some of the characteristics we want, given the above requirements.

No persistence

We’re purely interested in message throughput, without concern for reliability. More specifically, a subscription to a channel can be though of as a stream, representing what’s happening right now. If a subscriber has to reconnect, it has no need to receive missed messages - it’s up to the client to determine what’s appropriate.

Reliability via persistence also comes at a cost, as this typically requires writing messages to disk. Given the desire for maximum throughput, our message broker shouldn’t be storing messages at all - they should be sent out to subscribers as soon as they’re received from a publisher.

Trusted clients

A message broker is assumed to reside on the same trusted network as the pub-sub clients, so no form of authentication is required.

Protocol agnostic

We only want to pass string messages (or more specifically, byte sequences) around. Clients are responsible for encoding and decoding any particular format such as JSON, MsgPack, or others, without the broker having any knowledge of the format.

Horizontally scalable

We should be able to add more broker instances into the mix, in order to handle a growing number of clients and messages. I can’t think of a message broker that wouldn’t fulfil this requirement, since as long as we’re using multiple pub-sub channels, we can easily come up with a partitioning scheme such as consistent hashing that divides channels across brokers.

ZeroMQ

ZeroMQ is a piece of software I’d wanted to learn more about for quite some time. I’d heard the term pub-sub being used in conjunction with it, but knew it wasn’t a pub-sub server itself, as its name clearly indicates. Perhaps it would provide an approach that negated the need to use pub-sub entirely - either way, this seemed like a good opportunity to take a closer look.

ZeroMQ has a reputation for being hard to understand, given any single description about it, until you spend enough time with it to hit that point of enlightenment where it just clicks. The main reference documentation for ZeroMQ is the ZeroMQ Guide, which is a lengthy read, but for anyone with an interest in distributed systems, is well worth the time investment, even if you don’t end up using ZeroMQ itself. To avoid doing it a disservice trying to describe it myself, here’s the ZeroMQ description straight from the guide:

ØMQ (also seen as ZeroMQ, 0MQ, zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you sockets that carry atomic messages across various transports like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fanout, pub-sub, task distribution, and request-reply. It’s fast enough to be the fabric for clustered products. Its asynchronous I/O model gives you scalable multi-core applications, built as asynchronous message-processing tasks.

That’s a lot to digest in one quote, but it’s a great description. I’d say the main thing to take away is that ZeroMQ is a software library, that provides the building blocks for building things like pub-sub queues, rather than being an actual pub-sub queue or any other kind of network server itself.

There’s a huge point to be made here around the age-long debate over whether to use existing software for infrastructure, or to roll your own. Developers often lean towards the latter - it’s a path that can offer a lot more flexibility, without the constraint of having to fit square requirements into existing, potentially round, solutions. And let’s be honest, it’s a lot more fun! Inevitably it’s a painful path though, wrought with human error - the mistakes that only become apparent once the software has had time to mature in production. With something like ZeroMQ however, developers can have their cake and eat it too. You can design your network software to precisely match your own requirements, and all of the low-level details such as message buffering and routing strategies are all tucked away neatly in the software library.

Once I’d played around with it a bit, I was able to form a more concrete question: could I use ZeroMQ to build a pub-sub server, and how would it compare to Redis? It turns out the first part of that question can be answered trivially, with very little code, while the second part would require a bit more effort.

Show Me the Code

Without further ado, here’s a basic pub-sub message broker in Python using ZeroMQ, that demonstrates how little is required to get up and running:

import zmq

context = zmq.Context()
receiver = context.socket(zmq.PULL)
receiver.bind("tcp://*:5561")
sender = context.socket(zmq.PUB)
sender.bind("tcp://*:5562")

while True:
    sender.send(receiver.recv())

In order to test both queues as consistently as possible, I wanted to create procedures that use the same code base. To achieve this, I wrote a ZeroMQ pub-sub wrapper for the client, that implements the same API as redis-py, the Redis client for Python.

import zmq

class ZMQPubSub(object):

    def __init__(self, host="127.0.0.1"):
        context = zmq.Context()
        self.pub = context.socket(zmq.PUSH)
        self.pub.connect("tcp://%s:5561" % host)
        self.sub = context.socket(zmq.SUB)
        self.sub.connect("tcp://%s:5562" % host)

    def publish(self, channel, message):
        self.pub.send_unicode("%s %s" % (channel, message))

    def subscribe(self, channel):
        self.sub.setsockopt(zmq.SUBSCRIBE, channel)

    def unsubscribe(self, channel):
        self.sub.setsockopt(zmq.UNSUBSCRIBE, channel)

    def pubsub(self):
        return self

    def listen(self):
        while True:
            channel, _, data = self.sub.recv().partition(" ")
            yield {"type": "message", "channel": channel, "data": data}

And lastly, here’s a slightly watered down version of the test script used. First we set up a configurable number of clients that publish and consume messages. Each client then sneakily leverages the pub-sub queue itself, to provide per-second metrics for the number of messages consumed. Then after a certain time period, we report on the median number of messages consumed per second, per client.

#!/usr/bin/env python

import argparse
import multiprocessing
import random
import time
import redis
import zmq_pubsub


def new_client():
    """
    Returns a new pubsub client instance - either the Redis or ZeroMQ
    client, based on command-line arg.
    """
    if args.redis:
        Client = redis.Redis
    else:
        Client = zmq_pubsub.ZMQPubSub
    return Client(host=args.host)


def publisher():
    """
    Loops forever, publishing messages to random channels.
    """
    client = new_client()
    message = u"x" * args.message_size
    while True:
        client.publish(random.choice(channels), message)


def subscriber():
    """
    Subscribes to all channels, keeping a count of the number of
    messages received. Publishes and resets the total every second.
    """
    client = new_client()
    pubsub = client.pubsub()
    for channel in channels:
        pubsub.subscribe(channel)
    last = time.time()
    messages = 0
    for message in pubsub.listen():
        messages += 1
        now = time.time()
        if now - last > 1:
            client.publish("metrics", str(messages))
            last = now
            messages = 0


def run_workers(target):
    """
    Creates processes * --num-clients, running the given target
    function for each.
    """
    for _ in range(args.num_clients):
        proc = multiprocessing.Process(target=target)
        proc.daemon = True
        proc.start()


def get_metrics():
    """
    Subscribes to the metrics channel and returns messages from
    it until --num-seconds has passed.
    """
    client = new_client().pubsub()
    client.subscribe("metrics")
    start = time.time()
    while time.time() - start <= args.num_seconds:
        message = client.listen().next()
        if message["type"] == "message":
            yield int(message["data"])


if __name__ == "__main__":

    # Set up and parse command-line args.
    global args, channels
    default_num_clients = multiprocessing.cpu_count() / 2
    parser = argparse.ArgumentParser()
    parser.add_argument("--host", default="127.0.0.1")
    parser.add_argument("--num-clients", type=int, default=default_num_clients)
    parser.add_argument("--num-seconds", type=int, default=10)
    parser.add_argument("--num-channels", type=int, default=50)
    parser.add_argument("--message-size", type=int, default=20)
    parser.add_argument("--redis", action="store_true")
    args = parser.parse_args()
    channels = [str(i) for i in range(args.num_channels)]

    # Create publisher/subscriber workers, pausing to allow
    # publishers to hit full throttle
    run_workers(publisher)
    time.sleep(1)
    run_workers(subscriber)

    # Consume metrics until --num-seconds has passed, and display
    # the median value.
    metrics = sorted(get_metrics())
    print metrics[len(metrics) / 2], "median msg/sec"

The main points of configuration in the above code, are the toggle between Redis and ZeroMQ clients, and the number of pub-sub clients used. A client contains two loops, one that publishes messages, and one that consumes them. Each of these loops are run on a separate OS process using Python’s multiprocessing module, so the number of processes saturating the CPU is roughly equal to 1 broker + clients * 2. A handful of other configurable options are also there, such as message size, and the number of pub-sub channels used, but these didn’t really yield any meaningful variation in the results.

Initial Results

Firstly, here are some details around my own setup used:

• OS: Mac OS X Lion 10.7.2 (11C74)
• Processor: 2 GHz Intel Core i7
• Memory: 8 GB 1333 MHz DDR3
• Python: 2.7.1
• Redis: 2.4.2 (with hiredis parser)
• ZeroMQ: 2.2.0

And here are the initial results:

These results look quite grim, and have a couple of stand-out points. Firstly, the volume of messages produced and consumed by our benchmark script when run against the ZeroMQ broker, is fairly bound to the number of CPU cores being consumed. We see an increase in message volume per client, as we increase the number of processes being used, until we’re utilising all available cores, after which point we see a drop off due to contention. No surprises there really.

The second point is more interesting. Relatively speaking, Redis seems quite slow, and more obviously, we see almost no change based on the amount of concurrent work happening. After a bit of digging around, it turns out the redis-py client will send each pub-sub message over the wire individually, while ZeroMQ will internally manage buffering messages for you, sending them out when it deems optimally appropriate.

So Redis hits a wall quite early here, while ZeroMQ obviously has a huge advantage. In a real-world application this distinction may not exist, with high message volume derived from the number of clients, rather than the amount of messages generated per individual client. For our test vacuum though, it’s a real problem. Fortunately the redis-py client provides the ability to pipeline messages in batches, so we can easily get Redis back on even ground with ZeroMQ by providing a custom pub-sub client that makes use of Redis pipelining.

import thread
import threading
import time
import redis


class BufferedRedis(redis.Redis):
    """
    Wrapper for Redis pub-sub that uses a pipeline internally
    for buffering message publishing. A thread is run that
    periodically flushes the buffer pipeline.
    """

    def __init__(self, *args, **kwargs):
        super(BufferedRedis, self).__init__(*args, **kwargs)
        self.buffer = self.pipeline()
        self.lock = threading.Lock()
        thread.start_new_thread(self.flusher, ())

    def flusher(self):
        """
        Thread that periodically flushes the buffer pipeline.
        """
        while True:
            time.sleep(.2)
            with self.lock:
                self.buffer.execute()

    def publish(self, *args, **kwargs):
        """
        Overrides publish to use the buffer pipeline, flushing
        it when the defined buffer size is reached.
        """
        with self.lock:
            self.buffer.publish(*args, **kwargs)
            if len(self.buffer.command_stack) >= 1000:
                self.buffer.execute()

This BufferedRedis client is fairly simple. It holds onto messages published until it hits a certain number of messages buffered, 1000 in the code above, and then sends them off. For the low-volume case of our metrics channel in the previous test code, this isn’t enough though, so we also periodically flush messages, every 200 milliseconds in a separate thread. Next steps would be to allow the buffer size and flush interval to be configurable, but for our benchmarking purposes, these values work well.

Here are the results again, using the buffered client for Redis:

That’s much better, and we can see here that with the new buffered client, our test routine is making better use of concurrency against Redis. But can we see any real difference between Redis and ZeroMQ brokers yet? The fact these results come out quite closely indicates a chance we may have hit another wall in our benchmarking.

Go Go Go

With the slightest notch of Go experience under my belt, this seemed like another good opportunity to give Go a spin. With the closeness displayed by our two brokers so far, the possibility of a limitation occurring with the combination of Python and the hardware being used seemed worth exploring.

The Go version of the benchmarking routine isn’t particularly interesting, as it mimics the Python version very closely, with the main difference being that we use goroutines for concurrency, rather than OS processes. The code for the client libraries however, turned out to be hugely different between the Python and Go implementations, with the main distinction being the type systems - Python dynamically typed, and Go statically typed.

Both languages support duck-typing, whereby calling code can run against different types of data given a common set of members. This is a requirement for our client testing code, in order to be able to swap the Redis and ZeroMQ clients with a single flag. Python’s dynamic typing supports duck-typing in the true sense, in that calling code need know nothing about the types of data its working on until it actually runs. In Go however we need to be more explicit, and Go provides support for this via interfaces. An interface in Go is simply a type, defined by a set of function signatures. With interfaces, we can set up a generic client interface, and create client types that implement it, without calling code knowing about the underlying type being used.

Here’s what our Redis and ZeroMQ clients look like, when given a common interface that the testing routine can run against:

package pubsub

import (
    "fmt"
    zmq "github.com/alecthomas/gozmq"
    "github.com/garyburd/redigo/redis"
    "strings"
    "sync"
    "time"
)

// A pub-sub message - defined to support Redis receiving different
// message types, such as subscribe/unsubscribe info.
type Message struct {
    Type    string
    Channel string
    Data    string
}

// Client interface for both Redis and ZMQ pubsub clients.
type Client interface {
    Subscribe(channels ...interface{}) (err error)
    Unsubscribe(channels ...interface{}) (err error)
    Publish(channel string, message string)
    Receive() (message Message)
}

// Redis client - defines the underlying connection and pub-sub
// connections, as well as a mutex for locking write access,
// since this occurs from multiple goroutines.
type RedisClient struct {
    conn redis.Conn
    redis.PubSubConn
    sync.Mutex
}

// ZMQ client - just defines the pub and sub ZMQ sockets.
type ZMQClient struct {
    pub zmq.Socket
    sub zmq.Socket
}

// Returns a new Redis client. The underlying redigo package uses
// Go's bufio package which will flush the connection when it contains
// enough data to send, but we still need to set up some kind of timed
// flusher, so it's done here with a goroutine.
func NewRedisClient(host string) *RedisClient {
    host = fmt.Sprintf("%s:6379", host)
    conn, _ := redis.Dial("tcp", host)
    pubsub, _ := redis.Dial("tcp", host)
    client := RedisClient{conn, redis.PubSubConn{pubsub}, sync.Mutex{}}
    go func() {
        for {
            time.Sleep(200 * time.Millisecond)
            client.Lock()
            client.conn.Flush()
            client.Unlock()
        }
    }()
    return &client
}

func (client *RedisClient) Publish(channel, message string) {
    client.Lock()
    client.conn.Send("PUBLISH", channel, message)
    client.Unlock()
}

func (client *RedisClient) Receive() Message {
    switch message := client.PubSubConn.Receive().(type) {
    case redis.Message:
        return Message{"message", message.Channel, string(message.Data)}
    case redis.Subscription:
        return Message{message.Kind, message.Channel, string(message.Count)}
    }
    return Message{}
}

func NewZMQClient(host string) *ZMQClient {
    context, _ := zmq.NewContext()
    pub, _ := context.NewSocket(zmq.PUSH)
    pub.Connect(fmt.Sprintf("tcp://%s:%d", host, 5562))
    sub, _ := context.NewSocket(zmq.SUB)
    sub.Connect(fmt.Sprintf("tcp://%s:%d", host, 5561))
    return &ZMQClient{pub, sub}
}

func (client *ZMQClient) Subscribe(channels ...interface{}) error {
    for _, channel := range channels {
        client.sub.SetSockOptString(zmq.SUBSCRIBE, channel.(string))
    }
    return nil
}

func (client *ZMQClient) Unsubscribe(channels ...interface{}) error {
    for _, channel := range channels {
        client.sub.SetSockOptString(zmq.UNSUBSCRIBE, channel.(string))
    }
    return nil
}

func (client *ZMQClient) Publish(channel, message string) {
    client.pub.Send([]byte(channel+" "+message), 0)
}

func (client *ZMQClient) Receive() Message {
    message, _ := client.sub.Recv(0)
    parts := strings.SplitN(string(message), " ", 2)
    return Message{Type: "message", Channel: parts[0], Data: parts[1]}
}

The redigo Redis library for Go used here is quite different from its Python counterpart. Under the hood, it uses Go’s bufio package, which in conjunction with a network connection, provides buffered reads and writes over the network, so there’s no need for a separate API analogous to redis-py’s pipelining, as buffering is a fundamental aspect of the client. As you can see though, in the NewRedisClient function, we still need to set up mechanics for periodically flushing any buffered data in order to support the low-volume case, so it’s not entirely magical.

The astute reader will have noticed we don’t implement RedisClient.Subscribe and RedisClient.Publish methods - this is due to the unnamed embedded redis.PubSubConn within RedisClient, which already contains these methods. By embedding it without a name, its methods are directly accessible from the outer type. This is a really powerful feature of Go, allowing very elegant type hierarchies to be constructed using mixins.

And The Winner Is…

You the reader of course, for making it this far through this post. Seriously though, here are the combined results for both Go and Python versions:

Before we go any further, it’s important to highlight some key differences between the Python and Go test routines. As mentioned, the Go version isn’t particularly interesting, as the code is almost identical to the Python version. The way it runs however, and makes use of the available hardware, is very different. In the Python version, our best shot at making use of all available cores is to run each publisher and subscriber routine in a separate Python interpreter, each running on a single OS process. Go’s goroutines paint an entirely different picture. With only a single pub-sub client, we’re able to consume all available CPU cores using a single OS process - Go manages all of the parallelism for you. So we end up achieving the highest volume in messages with a single client, given that it can consume all available cores without any contention coming into play.

So this isn’t at all a comparison between Python and Go, since we’d be comparing apple pies to orange juice. But that’s fine, as it was never the point. The switch to Go merely allowed us to make better use of the available hardware, in order to reach a point in message throughput where we could potentially see a greater variance between Redis and ZeroMQ.

Conclusion

What can we take away from all of this? To be brutally honest, not much - I feel like I’ve only scratched the surface here, and without really diving in and profiling some of the code used, any conclusions drawn at this point would be fairly superficial. It did provide a good avenue for learning all about ZeroMQ and Go, which was a ton of fun, and something I’m definitely going to spend more time with. I also learnt that the buffering strategies used when dealing with a high volume of messages over the network form a critical piece of the puzzle.

You can find all of the Python and Go code I wrote for this in the repo called two-queues, on both GitHub and Bitbucket.

Update (next day): After publishing this, it was well received in the community, with endorsements from both Pieter Hintjens (creator of ZeroMQ) and Salvatore Sanfilippo (creator of Redis). Some great discussions continued on from there, on Hacker News, Reddit and Twitter - have a read!

Update (next week): Others have come along and made some interesting additions to the source code, check these out too:

• Elixir on the Erlang VM
• Redis 2.6.10 vs ZeroMQ 3.2.3

I’ve just released Gnotty 0.2 with a handful of new features which I thought would be worthwhile writing about. I wrote in greater detail about Gnotty when it was first released, but if you missed that and are wondering what it is, it’s an open source Python web application for managing an IRC channel. It provides a chat interface using WebSockets and searchable message archive - both which support mobile devices using responsive design. It also comes with programmable bots you can use to build services around the IRC channel.

What’s New

Here’s a list of the new features in 0.2:

• Join and leave messages can now be filtered in chat and the message archive.
• Next and previous day links in the message archive.
• Consistent username colouring, now also implemented in the message archive.
• A new gnottify_runserver Django management command, for running both the web and WebSocket servers together during development.
• Bots now automatically reconnect (with exponential backoff) if they’re disconnected from the channel.
• A new timer event type has been added to the bot framework, for running periodic events.

Push and Pull

The new timer events are quite simple on their own, but very interesting for Gnotty as a whole. As a feature, they simply provide the mechanism for bots to run events repeatedly at a given time interval. For Gnotty, this comes full circle in building bots that interact with external services and APIs. Since Gnotty’s first release, it has supported having data pushed to it via webhooks - custom HTTP endpoints used to send data to Gnotty, that a bot then acts upon. Now with timer events, data can be pulled by Gnotty from external sources as well.

Here’s an example bot that implements a timer event. It uses Mark Pilgrim’s feedparser library to consume my blog’s RSS feed, posting new items to the IRC channel as they’re discovered:

from feedparser import parse
from gnotty.bots import BaseBot, events

class RSSBot(BaseBot):

    def __init__(self, *args, **kwargs):
        # Store of retrieved feed items, so we only post them once.
        self.feed_items = set()
        # Consume initial feed items without posting them.
        self.parse_feed(message_channel=False)
        super(RSSBot, self).__init__(*args, **kwargs)

    # Runs every 60 seconds
    @events.on("timer", seconds=60)
    def parse_feed(self, message_channel=True):
        for item in parse("http://blog.jupo.org/atom.xml").entries:
            if item["id"] not in self.feed_items:
                self.feed_items.add(item["id"])
                if message_channel:
                    self.message_channel("%(title)s: %(id)s" % item)
                    return

The above example is actually included in Gnotty with the 0.2 release, with a bit more functionality implemented around error handling, as well as supporting multiple feeds.

Thank you

The best validation an open source project can receive is when other developers find the project useful enough to freely contribute back to it. Gnotty is off to a good start in this regard, and a big thanks goes to Philip Neustrom, Mike Waites, and Ken Bolton for contributing features and fixes to Gnotty since its first release.

Do you have an idea for a cool feature Gnotty could provide? Go right ahead and send a pull request on either GitHub or Bitbucket!

Earlier this year I finally got around to setting up an IRC channel for Mezzanine, where developers and users can collaborate on the development of Mezzanine and related projects. It’s been a good experience so far, facilitating quick questions as well as longer drawn out discussions, both which are sometimes better served by a real-time chat than a Mezzanine mailing list thread.

At the time I thought about how I could provide a web-based interface for the IRC channel, via the Mezzanine website. Mezzanine has always striven hard to be newcomer friendly, from the early days with its one-step project creation, to more recently with its bundled server provisioning and production deployment utilities, so having live support available directly on the project website was a natural evolution.

Existing Projects

The first step was to research existing web-based IRC clients. The main contender was a project called qwebirc, which is used by freenode and many other IRC networks. It’s been under active development for many years, and given some of the projects using it, would appear to be quite mature. One of the core requirements for me would be to customise the interface easily, in order to integrate with the look and feel of the Mezzanine website. Unfortunately after diving a bit deeper, I walked away with a laundry list of issues that turned me off using it.

Firstly there seemed to be almost no documentation, and the code contained almost no comments, so getting an understanding of how everything worked would have involved a decent time investment. Secondly, the transport mechanism between the browser and server was unclear. There seemed to be a reference to XHR polling in one of the 40 or so JavaScript files included, but ideally I wanted something that made use of WebSockets. Again these files weren’t at all commented, and left me with the feeling that the client side of the project was heavily over-engineered. Finally, qwebirc is based on the Twisted networking framework - software that I’m not personally familiar with, and one that’s known to have a fairly steep learning curve.

Another requirement I had for the Mezzanine site was to also include a browsable and searchable message archive. Getting the IRC conversations indexed by Google would mean that new users could potentially find answers to their questions without having to ask them, which would alleviate time spent on support for myself and other members of the community (a big shout out to Ken Bolton and Josh Cartmell for their ongoing efforts here). I’m vaguely aware of some Django projects around that implement this functionality, but wouldn’t it be nice if there was an all-inclusive package for IRC integration that made use of modern technology?

Starting From Scratch

Given all the above, I had a clear idea around what I wanted from web-based IRC client. A simple bridge between IRC and gevent on the server, with WebSocket support on the client.

I’ve got a good amount of experience using WebSockets backed by gevent. In fact I created django-socketio quite some time ago, specifically to integrate gevent-backed WebSockets with Django. That project however has languished behind the latest WebSocket protocol, and one of the longest standing items in my open source work has been to get up to date with the latest WebSocket protocol, and upgrade django-socketio.

So at the risk of drowning in the sea of NIH, I decided to start my own project to build this, as it would give me the perfect opportunity to get back up to speed with the latest WebSocket protocol, as well as giving me the most flexibility around customising the client and including extra features in it, such as the message archive and more.

The result of this endeavour is a project I’ve called Gnotty. At its core, Gnotty is a lightweight bridge between IRC and gevent, with WebSockets on the client via socket.io. Starting from scratch meant that I was able to implement a ton of extra features, which against the UNIX philosophy of doing one thing well, I ended up getting quite carried away with. Nonetheless I’m really pleased with the result so far.

Following is an overview of the extra features I’ve implemented so far in Gnotty, that make it much more than just a web-based interface to IRC.

Responsive Layout

The default interface for Gnotty is entirely based on Twitter’s Bootstrap. This made it easy to ensure that it functions correctly on mobile devices thanks to Bootstrap’s responsive features.

A fully responsive IRC client, on your desktop or mobile device

JavaScript Client

Gnotty provides a JavaScript client that makes no assumptions about the structure of the browser interface. It simply exposes each of the methods and events that occur for interacting with the IRC room. Don’t want to use default interface? The JavaScript client allows you to develop your own from scratch.

Message Archive

As mentioned, one of the key requirements I had was to have all messages archived so that they can then be browsed and searched through. Whatever the context of using Gnotty is, from collaborating on an open source project, to private rooms for teams to use, this form of knowledge reuse is a critical feature. Gnotty provides this feature by combining Django models with Python’s logging module. Logging namespaces are defined for IRC messages as well as several other events, and when deployed as a Django project, logging handlers are configured that store all IRC messages to Django models. The interface for browsing and searching the message archive is then provided using Django views.

Message archive by date or keyword search

Note that Django integration with Gnotty is entirely optional. You can deploy Gnotty as a stand-alone bridge between IRC and gevent without using Django at all. Given the use of Python’s logging module, you can then create your own logging handlers, and store IRC messages in some other format or location should you wish to.

Django Templating

By default, Gnotty’s interface uses Django’s templating language. This opens the door to many opportunities for customisation, from using Django’s template tags, to integrating with your existing Django project’s look and feel. Deploying Gnotty as a Django project opens it up to the entire ecosystem of third party Django applications.

Private Rooms

This is another bonus feature simply available by virtue of Gnotty integrating with Django. By defining the GNOTTY_LOGIN_REQUIRED setting in your Django project, the chat interface and message archive then require an authenticated Django user in order to be accessed. Combine this with a local IRC server such as ngIRCd, you can have your own hosted private IRC room for team collaboration, all set up in a matter of minutes.

Bot Framework

Here’s where things really got carried away. Gnotty includes a mini framework for injecting IRC bots into the room. The bots expose all of the events that can occur in the IRC channel, such as users messaging, entering and leaving. The bots also include an API for creating simple Python commands, where user messages can be mapped to Python methods on the bot. Bots can also have timer events implemented (new in 0.2), that run periodically at a given interval, and are useful for polling external resources. Finally, since Gnotty at its core is a web interface, the bots also allow webhooks to be implemented - custom URLs that allow other services to interact with the bots over HTTP.

With IRC channel events, user commands and webhooks, a ton of interesting possibilities open up for building IRC bots that perform a variety of tasks. Gnotty also provides a handful of default bots to get you started with:

• ChatBot - A bot that demonstrates interacting with the IRC channel by greeting and responding to other users.
• GitHubBot - A bot for relaying commit information from GitHub
• BitBucketBot - A bot for relaying commit information from Bitbucket
• CommandBot - A bot that implements a handful of basic commands that can be issued by users in the channel.
• RSSBot - A bot that watches RSS feeds and posts new items from them to the IRC channel (new in 0.2).
• Voltron - All of the available bots, merged into one super bot.

Next Steps

I’ve given an overview here of the core features that Gnotty implements in its first release. Take a look at the Gnotty documentation where all of the features are described in much greater detail, along with configuration options and code samples to get you started. As with all my open source projects, Gnotty is available on both GitHub and Bitbucket, so if you’d like to work with me on future development then by all means dive right in.

I think there’s a ton of possibilities around future development, particularly around the idea of private rooms for team collaboration. I’d love to see IRC’s DCC send supported, where team members could send files to each other that are uploaded and streamed backed down directly through the browser. Also Gnotty is currently designed to only work with a single IRC room, so support for multiple rooms is another obvious direction for future development to take.

Finally if you’re keen to see Gnotty in action, you can see a skinned version of it that makes use of all of its features, integrated into the Mezzanine project’s website.

Last night I gave my second SyDjango talk, "Django Admin, The Missing Manual". There were quite a few new faces this month, as well as a good mix of regular Djangonauts from previous months.

In this talk I covered the Django Admin, some reasons people might not use it, why you should use it anyway, and how to address some of those possible issues. I walked through a basic project for a blog with a spartan Django Admin interface, and the steps involved in turning it into a much more full-featured and attractive experience for the end user.

Thanks again to all the fellow Sydney Djangonauts who came along and made it a great night. You can find source code for this talk on both GitHub and Bitbucket.

Table of Contents

Table of contents and video/slides sync via vimeo-deck.

Last night I gave a talk at the third SyDjango meet-up where I presented my django-forms-builder project. Below is the video and slides from the talk which went really well, spurring about a dozen questions at the end. Excuse the background noise in the video, unfortunately we didn't have the venue to ourselves. Thanks to all the fellow Sydney Djangonauts who came along and made it a great night.

Table of Contents

Table of contents and video/slides sync via vimeo-deck.

The Django Dash for 2012 is now over, and once again it was a ton of fun! The Dash is a 48 hour coding competition for teams of up to three, competing to see who can build the best Django application over a weekend. To top it off, all of the entries are made available as open source.

This year I went solo for the first time, which made it quite gruelling as I had to handle everything from back-end code, interface development, and visual design. I decided to follow the same recipe as our 3rd place entry from last year, Drawn By, specifically by creating an environment where people can come together and interact in real-time using WebSockets. I named my entry GAMBLOR - it’s an online casino where people can move around and chat with each other, while playing casino games with fake money. Check out GAMBLOR as well as the source code if you’re interested, and you can see the game plugin system I built to power the initial games implemented: roulette and craps.

Apart from the staples of Django and jQuery, here are some of the components I used that made GAMBLOR possible in such a short amount of time:

• gevent-socketio - Provides the Python implementation of socket.io, allowing real-time activity to occur in the browser using WebSockets.
• Redis - A NoSQL database, similar to memcached but with richer data structures.
• CSS3 - The design was implemented almost entirely without graphics, using many CSS3 features such as shadows, background gradients, and even 3D transforms for the gaming chips.
• jquery-transit - A library for advanced JavaScript animations, which powered the roulette wheel rotation.
• jquery-collision - Provides collision detection for browser objects, which allowed for avatar movement to be constrained within the game area and around the game tables.

I’ve put together a table below that lists all of the final entries, with links to their sites and source code. I left out any entries that didn’t contain a working site, or didn’t seem to function. If I’ve missed any or you have any other corrections, please let me know.

It’s interesting to look at some of the recurring themes present within all of the entries. There were multiple entries that fell under each of the following ideas:

• Cloud server management - Cloud Fish, Gungnir
• GitHub mashups - Badger, Heroes of Git & Hub, Mosaic, Try Box
• Tutorial/class creation - Django Tutorial, Try Box, Try Try, Tutor Us
• Photo mashups - Gif Feed, Green Room, Lemidora, Miracles Live, The Busitizer
• Portfolio generation - Folio Bag, Mosaic
• Map mashups - Kirchenreich, Quester

My definite favourites are Heroes of Git & Hub, which lets you battle your open source projects against others in a D&D style game, and The Busitizer which I’ll just leave for you to check out for yourself.

One of Django’s many great features is its powerful template inheritance. A decade ago we would use simple concepts like include files for platforms such as ASP and PHP, allowing us to create reusable template snippets that could be embedded in multiple pages. Later, ASP.NET and Ruby on Rails would improve on this with their master/layout concepts, allowing us to define a base skeleton template for a site, with an area that all other pages would inject their content into. Django takes this approach even further with its template inheritance. It allows templates to extend other (parent) templates, with those parent templates containing named blocks that can be overridden. The blocks in the parent template can contain default content, and when overriding these blocks, the default content can be overridden, left as is, or even prefixed with or appended to, as the child template will have access to the default content in the parent template’s block. This is analogous to object oriented programming, where base classes can be subclassed, and have their methods overridden, with access to the super-class’s methods to be called at whatever point is deemed appropriate.

Overriding vs Extending

Another powerful feature of Django’s is its template loaders. Each loader implements an approach for finding and loading the contents of a template when it’s requested by name. A typical Django project will contain multiple template loaders, and when a template is loaded by name, that name will be passed through each of the loaders sequentially, until one of the loaders finds the template.

The two most commonly used template loaders are the filesystem loader and the app_directories loader. The filesystem loader will look at the TEMPLATE_DIRS setting, and search each of the directory paths in it for the requested template. The app_directories loader is similar, but will look for a directory called “templates” in each of the apps listed in the INSTALLED_APPS setting.

TEMPLATE_LOADERS = (
    "django.template.loaders.filesystem.Loader",
    "django.template.loaders.app_directories.Loader",
)

PROJECT_ROOT = os.path.dirname(os.path.abspath(__file__))
TEMPLATE_DIRS = (os.path.join(PROJECT_ROOT, "templates"),)

Reusable Django apps will often provide a default set of templates where applicable, and the app’s view functions will load these templates by name. With both the filesystem and app_directories template loaders configured, the app’s version of the template will be loaded, unless a template with the same name is found in the project’s templates directory, since the filesystem loader is listed first in the TEMPLATE_LOADERS setting. This allows a project developer to easily override the templates for a third-party app by copying it into their project’s templates directory, in order to customise the look and feel of the app.

A problem arises for the project developer however, when the app’s template contains sufficiently complex features, like many extendible template blocks, template variables, and more. Once they copy the template to their project’s templates directory, they’re essentially forking it, that is, they’ll no longer be able to seamlessly make use of any new features for that template in future versions of the third-party app. Worst case is that an upgrade to the app will break their project, until they copy the new version of the template and customise it again, or upgrade their own copy by hand to be compatible with the latest version of the app.

With a complex template like this, more often than not the project developer may simply want to change it in a very small way, such as modifying the content in one of its blocks. Wouldn’t it be nice if you could use template inheritance to extend the app’s template, and simply override the relevant blocks as desired? Unfortunately this isn’t possible with Django due to circular template inheritance. The app’s view will be looking for the template to load by a given name. If we want our project’s version to be used, we need to use the same template name for it to be loaded. If our project’s version of the template tries to extend a template with the same name, Django will load our project’s template again when looking for the parent template to extend, resulting in an infinite loop that will never complete. Django’s template inheritance isn’t smart (or stupid) enough to ignore the absolute path of the current template being used, when searching for the parent template to extend.

Alternative Approaches

The general approach to dealing with this problem, is for app developers to separate the name of of the template being loaded by their view, from the parts of the template that a project developer may want to customise. This might involved breaking all of the features up into separate include files that can be overridden individually. Another approach is to make each view load an empty template that extends the real template - developers can then override the empty template, end extend the real template as required.

In an effort to make Mezzanine’s templates more customisable, these ideas were recently brainstormed on the Mezzanine mailing list. While these approaches might work extremely well for individual Django apps that only provide a handful of default templates, the question of complexity and maintenance comes up with larger-scale projects like Mezzanine which contains almost 100 template files at the moment. All of a sudden we’re looking at a minimum of doubling the number of template files - even more if we get more granular with includes. We’ve lost the simplicity of simply checking which template a view loads, and copying it to our project for customisation. So the question was proposed as to how could we possibly get circular template inheritance to work - if that was possible, we’d have a fantastic tool for both overriding and extending templates at once, without any wide-sweeping changes to the template structure across the entire project. Read on for the gory details of how it’s quite possible.

Hacks Inside

Fair warning: the rest of this post describes an unorthodox approach that allows circular template inheritance to work. It’s a little bit crazy, it’s a little bit cool. Some would call it a terrible hack, your mileage may vary. If you’re going to use it, consider the actual problem it solves, and whether or not it applies to your situation. Understand what it does, test it, and weigh it up against the alternative approaches described earlier.

The problem can be boiled down to one of state - when Django’s extends template tag is used and the parent template to extend is searched for, the tag contains no knowledge of the extending template calling it. If it did, we could theoretically exclude its path from those being used to search for the parent template. So a possible solution might involve two steps:

• Make the extends tag aware of the path for the template that’s calling it.
• Search for a template with the same name, preferably using Django’s template loaders, and exclude the path of the template doing the extending.

Both of these steps are achievable thanks to the design of Django’s template loaders. While the underlying location of a template isn’t available through the higher level interfaces like get_template and find_template, if you dig down slightly into the template API, you’ll find each loader contains a load_template_source method, which is part of the call chain for finding and loading a template’s source. load_template_source returns both the source for the template it discovers, as well as the absolute path of the template.

From this point we could go ahead and fulfil the second step by searching all other possible file-system paths for a template with the same relative template path, excluding the absolute path we’ve retrieved via load_template_source, but ideally we’d like to leverage the template loaders to do this. Fortunately this is a breeze given the way template loaders work. The load_template_source method for each of the template loaders can accept a list of directories to use, and will fall back to a default if none are specified. The filesystem loader will use the directories defined by the TEMPLATE_DIRS setting, and the app_directories loader will use a list of template directories for all of the INSTALLED_APPS which it builds up when first loaded. I’ve never seen these directory arguments used in practice, but there they are just begging to be exploited as the perfect solution to our template searching problem.

Extending Extends

Now that we have the theoretical hooks needed, it’s time to implement our template tag. Again we find ourselves in the situation where the pieces of Django we need to touch are structured perfectly to do what we need. The extends tag is implemented using the ExtendsNode class. It contains a get_parent method, which is responsible for loading the parent template object that is being extended. So all we need to do is subclass ExtendsNode and override get_parent. We’ll also include our own find_template method, similar to Django’s, that also returns the absolute path of the template that was found. I’ve dubbed this approach “overextending”, since it allows you to both override and extend a template at the same time.

from django.conf import settings
from django.template import TemplateDoesNotExist
from django.template.loader_tags import ExtendsNode

class OverExtendsNode(ExtendsNode):

    def find_template(self, name, dirs):
        """
        Replacement for Django's ``find_template``, that also returns
        the location of the template found.
        """
        for loader_name in settings.TEMPLATE_LOADERS:
            loader = find_template_loader(loader_name)
            try:
                source, path = loader.load_template_source(name, dirs)
            except TemplateDoesNotExist:
                pass
            else:
                return source, path
        raise TemplateDoesNotExist(name)

    def get_parent(self, context):
        """
        Load the parent template using our own ``find_template``, which
        will also give us the path of the template found. We then peek
        at the first node, and if its parent arg is the same as the
        current parent arg, we know circular inheritance is going to
        occur, in which case we try and find the template again, with
        the absolute directory removed from the search list.
        """
        from django.template.loaders.app_directories import app_template_dirs
        dirs = list(settings.TEMPLATE_DIRS + app_template_dirs)
        parent = self.parent_name.resolve(context)
        template, path = self.find_template(parent, dirs)
        if (isinstance(template.nodelist[0], ExtendsNode) and
            template.nodelist[0].parent_name.resolve(context) == parent):
            # Remove the template directory from the available
            # directories to search in, and try again.
            dirs.remove(path[:-len(parent) - 1])
            template, path = self.find_template(parent, dirs)
        return template

All that remains is creating the overextends template tag function that uses OverExtendsNode. For this we can pretty much copy pasta Django’s extends tag function, replacing ExtendsNode with OverExtendsNode.

Diving Deeper - Unlimited Inheritance Levels

Keeping in mind that this approach is is specifically geared towards solving the problem of both overriding and extending a template in a third party app, that is, one we don’t want to modify the source code of, our approach so far works. But what if we wanted to overextend a template that also overextends another template? Say for example our project template overextends a template in third-party app “A”, which is dependent on a template in third-party “B”, that it also overextends. This is quite an edge case, but the code above would fail in this scenario. When the template in app “A” tries to overextend the template in app “B”, it would exclude itself from the search path, and end up loading our project’s version of the template, and we’re back to square one with circular inheritance never completing.

This is less of an edge case and much more likely, with frameworks like Mezzanine, where it’s common for people to create themes as third-party apps. The theme provides a set of templates and static files, and gets added to INSTALLED_APPS just as a regular Django app would. With the approach of overextending introduced into the eco-system, theme developers may overextend Mezzanine’s templates, and project developers may overextend the theme’s templates. Taking this into account, our approach so far looks quite handicapped.

Again we have a state problem - the second and subsequent calls to overextends have no knowledge of the previous calls, so they can’t exclude the chain of template directories that have been so far excluded when overextending.

We can solve this problem by making use of the template context to store the state of directories excluded so far when using overextends. We store a dictionary mapping template names to lists of directories available. In the code above, we build the full list of directories to use each time overextends is called. If we maintain that list in the template context, removing from it each time overextends is used, we can support unlimited levels of circular template inheritance.

Here’s an refactored version of the previous example, along with the overextends tag function, supporting multiple levels of circular template inheritance.

from django import template
from django.template import Template, TemplateSyntaxError, TemplateDoesNotExist
from django.template.loader_tags import ExtendsNode
from django.template.loader import find_template_loader

register = template.Library()

class OverExtendsNode(ExtendsNode):

   def find_template(self, name, context, peeking=False):
        """
        Replacement for Django's ``find_template`` that uses the current
        template context to keep track of which template directories it
        has used when finding a template. This allows multiple templates
        with the same relative name/path to be discovered, so that
        circular template inheritance can occur.
        """

        # These imports want settings, which aren't available when this
        # module is imported to ``add_to_builtins``, so do them here.
        from django.template.loaders.app_directories import app_template_dirs
        from django.conf import settings

        # Store a dictionary in the template context mapping template
        # names to the lists of template directories available to
        # search for that template. Each time a template is loaded, its
        # origin directory is removed from its directories list.
        context_name = "OVEREXTENDS_DIRS"
        if context_name not in context:
            context[context_name] = {}
        if name not in context[context_name]:
            all_dirs = list(settings.TEMPLATE_DIRS + app_template_dirs)
            context[context_name][name] = all_dirs

        # Build a list of template loaders to use. For loaders that wrap
        # other loaders like the ``cached`` template loader, unwind its
        # internal loaders and add those instead.
        loaders = []
        for loader_name in settings.TEMPLATE_LOADERS:
            loader = find_template_loader(loader_name)
            loaders.extend(getattr(loader, "loaders", [loader]))

        # Go through the loaders and try to find the template. When
        # found, removed its absolute path from the context dict so
        # that it won't be used again when the same relative name/path
        # is requested.
        for loader in loaders:
            dirs = context[context_name][name]
            try:
                source, path = loader.load_template_source(name, dirs)
            except TemplateDoesNotExist:
                pass
            else:
                # Only remove the absolute path for the initial call in
                # get_parent, and not when we're peeking during the
                # second call.
                if not peeking:
                    context[context_name][name].remove(path[:-len(name) - 1])
                return Template(source)
        raise TemplateDoesNotExist(name)

    def get_parent(self, context):
        """
        Load the parent template using our own ``find_template``, which
        will cause its absolute path to not be used again. Then peek at
        the first node, and if its parent arg is the same as the
        current parent arg, we know circular inheritance is going to
        occur, in which case we try and find the template again, with
        the absolute directory removed from the search list.
        """
        parent = self.parent_name.resolve(context)
        # If parent is a template object, just return it.
        if hasattr(parent, "render"):
            return parent
        template = self.find_template(parent, context)
        if (isinstance(template.nodelist[0], ExtendsNode) and
            template.nodelist[0].parent_name.resolve(context) == parent):
            return self.find_template(parent, context, peeking=True)
        return template

@register.tag
def overextends(parser, token):
    """
    Extended version of Django's ``extends`` tag that allows circular
    inheritance to occur, eg a template can both be overridden and
    extended at once.
    """
    bits = token.split_contents()
    if len(bits) != 2:
        raise TemplateSyntaxError("'%s' takes one argument" % bits[0])
    parent_name = parser.compile_filter(bits[1])
    nodelist = parser.parse()
    if nodelist.get_nodes_by_type(ExtendsNode):
        raise TemplateSyntaxError("'%s' cannot appear more than once "
                                  "in the same template" % bits[0])
    return OverExtendsNode(nodelist, parent_name, None)

The final step required is to automatically add our overextends tag to Django’s built-in tags. Django’s ExtendsNode uses a feature where it gets marked as having to be the first tag in a template (ExtendsNode.must_be_first is set to True). This means that it (and subsequently our ExtendsNode subclass) need to be available without having to load the template library that implements it. This is as simple as calling the django.template.loader.add_to_builtins function from your project’s settings module, passing it the Python dotted path as a string for the module that contains out overextends tag.

django-overextends

Originally this post contained a similar approach to the one above, but made use of the origin attribute found on template objects. Shortly after publishing it, Tobia Conforto helped me work out that the origin attributes on template objects are only available when DEBUG is True in your Django project. A big thanks goes out to him for bringing this up, allowing me to work out a more solid approach that this post now describes.

Tobia also suggested how badly a solution like this is needed for reusable app developers, and that it really belongs in a publicly available package that people can install and add to their Django projects. So I’ve bundled it into a reusable Django application called django-overextends. It contains documentation, tests, continuous integration, and is available on PyPI, GitHub and Bitbucket.

The actual code written when developing a typical web application, ranks pretty lowly on the complexity scale, when compared to the rest of the software engineering industry. Of course, web development has its own set of interesting complexities around architecture and scaling, but the application code itself is relatively simple. Glue one library to another, update a row in a database, print some more rows out, send out an email, and so on.

As a web developer, I always enjoy the chance to dig deeper into more formal data structures and algorithms when the opportunity arises. At Fairfax I’ve recently had two distinct situations come up where I’ve needed to work through sets of data, with each of the items in the set containing references to other items that they depends on. The desired outcome was to sort the items so that I could process each item, knowing that each time I reached an item with dependencies, those dependent items would already have been processed.

The first scenario was one with serialised Django models. I had a number of model instances that would be instantiated and serialised, with the aim of persisting them to the database at a later point. These models contained relationships with other models that were also being serialised. Temporary primary keys were used in order to build their relationships, allowing all of the models to then be serialised with their relationships intact.

The problem then arose after deserialising these instances, and persisting them to the database. Each model with a foreign key to another couldn’t be processed, until the model it was related to had been persisted to the database first, as real primary keys are required for foreign key relationships to be defined. My first solution was a quick hack that manually sorted the objects correctly, knowing in advance which classes of models I was dealing with. This of course meant the code couldn’t be reused in a general manner, and as more classes of models were added to the mix down the track, this would need to be addressed.

The second case like this came up in an entirely different situation. At Fairfax we’re building an impressive distributed system built on many RESTful APIs. Each of these APIs contains its own schema, and we set out to build a tool that could introspect these schemas, and generate client objects for interacting with them. These schemas also contain relationships between resources, and these resource relationships are also modelled in our client objects. Again this situation was one where I needed to process an unordered set of resources in the correct order, so that their relationships could correctly reference resources already handled.

Shyly fool my bitten shame twice, or something like that. When this problem came around the second time, I didn’t have the luxury of knowing up front what the exact classes of data I’d be dealing with were, as they were due to change quite quickly as we iterated. I then set out to implement the correct solution for the problem, that could be applied to both of these situations.

A quick Google search for “dependency resolution” brought me to the Wikipedia page for topological sorting, which was the solution I was looking for. In both cases, we have graphs. Think of a graph as points on a map called nodes, with each node connected by lines called edges, to other nodes in the graph.

A simple graph

A directed graph is one where each of the edges contain a direction, so each of the lines contain an arrow pointing one way or the other. An acyclic directed graph is where each of the edges only point in one direction, so that it’s not possible to follow the edges from one node to another, returning back to the original node. If this last condition is not satisfied, then the graph is said to contain directed cycles, that is, a path can be followed from one node to others, and back to the original node again.

An acyclic directed graph

In both scenarios I faced, my graph structure was different from those commonly found in examples of topological sorting. I had the outgoing edges defined, rather than incoming edges. The gist of the topological sort I needed, is to repeatedly go through all of the nodes in the graph, moving each of the nodes that has all of its edges resolved, onto a sequence that forms our sorted graph. A node has all of its edges resolved and can be moved, once all the nodes its edges point to, have been moved from the unsorted graph onto the sorted one.

Consider the graph above from left to right, as pairs of nodes and their outgoing edges:

graph_unsorted = [(2, []),
                  (5, [11]),
                  (11, [2, 9, 10]),
                  (7, [11, 8]),
                  (9, []),
                  (10, []),
                  (8, [9]),
                  (3, [10, 8])]

Our expected output from a topological sort function would be as follows, with no nodes containing edges pointing to nodes before themselves:

>>> from pprint import pprint
>>> pprint(topolgical_sort(graph_unsorted))
[(2, []),
 (9, []),
 (10, []),
 (11, [2, 9, 10]),
 (5, [11]),
 (8, [9]),
 (3, [10, 8]),
 (7, [11, 8])]
 

Note that the ordering need not be precisely the same each time. In the result above, node 9 could come before node 2, as neither of these contain edges, so they equally belong first in the sorted graph.

Here’s an implementation of my topological sort in Python:

def topolgical_sort(graph_unsorted):
    """
    Repeatedly go through all of the nodes in the graph, moving each of
    the nodes that has all its edges resolved, onto a sequence that
    forms our sorted graph. A node has all of its edges resolved and
    can be moved once all the nodes its edges point to, have been moved
    from the unsorted graph onto the sorted one.
    """

    # This is the list we'll return, that stores each node/edges pair
    # in topological order.
    graph_sorted = []

    # Convert the unsorted graph into a hash table. This gives us
    # constant-time lookup for checking if edges are unresolved, and
    # for removing nodes from the unsorted graph.
    graph_unsorted = dict(graph_unsorted)

    # Run until the unsorted graph is empty.
    while graph_unsorted:

        # Go through each of the node/edges pairs in the unsorted
        # graph. If a set of edges doesn't contain any nodes that
        # haven't been resolved, that is, that are still in the
        # unsorted graph, remove the pair from the unsorted graph,
        # and append it to the sorted graph. Note here that by using
        # using the items() method for iterating, a copy of the
        # unsorted graph is used, allowing us to modify the unsorted
        # graph as we move through it. We also keep a flag for
        # checking that that graph is acyclic, which is true if any
        # nodes are resolved during each pass through the graph. If
        # not, we need to bail out as the graph therefore can't be
        # sorted.
        acyclic = False
        for node, edges in graph_unsorted.items():
            for edge in edges:
                if edge in graph_unsorted:
                    break
            else:
                acyclic = True
                del graph_unsorted[node]
                graph_sorted.append((node, edges))

        if not acyclic:
            # Uh oh, we've passed through all the unsorted nodes and
            # weren't able to resolve any of them, which means there
            # are nodes with cyclic edges that will never be resolved,
            # so we bail out with an error.
            raise RuntimeError("A cyclic dependency occurred")

    return graph_sorted

I imagine some other uses for topological sorting would be task queues, where certain tasks are dependent on other tasks being completed first. It could also be used by package managers that install software libraries, to ensure each library has its dependencies met before it’s installed.

This weekend I finally launched Mezzanine 1.0 after two years in development.

Rather than talk about Mezzanine itself and the lead up to 1.0, I thought it would be fun to look at the online reach of the release announcement, as well as some of the growth that has occurred over the last 18 months since I first made Mezzanine available. For more detail on Mezzanine and the release itself, check out the 1.0 announcement to the wider Django community, as well as the discussion leading up to the 1.0 release.

Active Growth

At the end of 2010 I wrote my annual year in review post, where I talked about Mezzanine and the activity that had occurred in its first 6 months. With the new 1.0 release, I thought it would be interesting to take a fresh look at some of the statistics I talked about back then. We can see below that growth has continued all the way through at a steady pace.

Combined follower/fork count for GitHub and Bitbucket brought to you by One True Repo

The Launch Party

The launch party we held for the release was a raging success. Just to be clear, by we I mean me, and by party I mean sitting at my computer all day with several browser tabs open, feverishly refreshing them with the hope of some good exposure and interest around the release. And there was plenty!

After first announcing the release to the django-users mailing list, I then posted it to a handful of popular channels in the programming community. All around the responses were positive, ranging from congratulation and praise, to all sorts of questions from curious people in the Django space who hadn’t heard of Mezzanine before.

django-users

As mentioned, I made the initial announcement to the django-users mailing list. Here I gave a general overview of Mezzanine and Cartridge, and went through their core features. The follow-up responses were positive, and included some questions about how Cartridge compares to other more popular ecommerce Django apps. I then gave a short version of my original post from when I first made Cartridge available, which covers that area in more detail.

Hacker News

I’ve posted a few articles to Hacker News before, but they’ve never been promoted to the front page, which occurs after receiving enough votes from the community. So I was delighted when 10 minutes or so after posting the announcement to Hacker News, it had reached enough votes to hit the front page. After that the votes came pouring through, with the announcement eventually making its way to 4th place on the front page, where it remained for most of the day.

When I woke up the next day, the article had reached over 90 votes, and a good range of questions had been posted, from looking for help getting started, to comparisons against other CMS projects. By then the news of the latest Rails exploit had flooded the front page, and the Mezzanine release had been pushed down into oblivion.

Reddit

At the same time, I posted the announcement to Reddit. It also reached the front page of proggit, the programming sub-reddit I posted it to. Again there were a good number of comments with praise and questions. What was interesting about Reddit is that the number of votes both for and against the article are visibly displayed.

Downvotes, really!!?

It was amusing to think about what would cause people to vote it down. Perhaps they were fanatical supporters of another language, framework or CMS. I’ll never know! I think it’s a safe bet though that it had nothing to do with Mezzanine itself, based on the actual comments posted there and elsewhere.

Twitter

Of course I tweeted the announcement once it was made. After a handful of direct and indirect retweets, the tweets started coming through with a lot of lovely praise, particularly for the responsive layout of Mezzanine’s new project homepage, something that all credit goes to the Bootstrap team for. The most humbling moment though was when a tweet came through from Antonio Rodriguez, the former CTO of HP of all people!

This looks like it may be Django’s killer app

Very cool.

After the announcement had hit the front pages of Hacker News and proggit, an army of Twitter bots that are connected to those sites then tweeted links to the announcement, resulting in hundreds of tweets and thousands of new visitors. The final assault was then triggered by Smashing Magazine, who tweeted Mezzanine to their half a million followers.

GitHub

While GitHub wasn’t somewhere I explicitly made the announcement to, like the channels above, the reaction on there was probably the most important. Mezzanine and Cartridge received over 100 new developers following the projects.

This resulted in Mezzanine being the most watched Python project on GitHub for both the day and the week! For the first time it also entered into the top 100 Python projects on GitHub of all time.

A quick burst of small contributions followed, from spelling corrections in the documentation, to patches for getting things running smoothly on Windows. I then released version 1.0.1, and 1.0 became history.

Final Toast

It’s been a hard slog over the last two years. With a full-time job and a family to take care of, the biggest challenge has always been finding the time to build new features and work with the community to ensure Mezzanine stays on the right track for its users. In that regard, Mezzanine wouldn’t be what it is today without the contribution of all the developers who have written features, fixed bugs, and most importantly, helped out new-comers on the mailing list. A special thanks goes out to all of you:

Here’s looking forward to the next two years, and keeping Mezzanine and Cartridge a lean, mean, site building machine.

It was almost a year ago that I took up a new role using Ruby on Rails. I’ve previously talked about my thoughts on Rails, and given my experience with Django I probably wouldn’t consider using Rails for my own projects. What I did explore in my time with Ruby was another framework called Sinatra which I used to build several apps. Firstly I’ll go over Sinatra and some of the related pieces in the stack, and then I’ll cover the apps I built.

Sinatra

Sinatra is a micro-framework, which differs from mega-frameworks like Rails and Django, in that Sinatra is bare-bones. It mostly deals with mapping URLs to request handlers, and not much more beyond that. No templating, no ORM, no middleware. All of these features can be slotted in using third party libraries where required. This makes for a very pleasant development experience with smaller sized apps - instead of having to do everything the Django/Rails way, you’re free to pick and choose the parts you need, and weave them together in the best way you see fit. You’re working at a relatively lower level, with much less scaffolding, and a lot more flexibility and control.

Python has its own counterparts in this space as well, such as Bottle and Flask. However at the time I was looking to dive further into Ruby, and Sinatra seemed like a great way to lean into it.

If you’re new to web development, or an experienced developer coming to Ruby or Python from older stacks like ASP.NET or PHP, I’d highly recommend starting out with a micro framework like Sinatra or Flask, before moving onto their bigger siblings Rails and Django. You’ll get a great feel for their respective languages, without getting bogged down in the frameworks themselves.

DataMapper

You can’t go very far these days developing a web application, without needing some form of persistent storage such as a database, and a library to work with it that goes beyond hand-written SQL. Django has its own ORM which is very powerful, but suffers from lacking a blessed, seamless migration tool. Rails has Active Record, which has grown into the defacto ORM in the Ruby eco-system, and has its own set of problems. The main issue I had with Active Record was that there was no clear definition of what fields a particular model implemented, aside from diving directly into the database itself. It coincides clearly with the notion that Rails contains too much magic. Compare this to Django’s declarative ORM, where each model’s class contains an explicit blueprint of which fields and methods the model implements. The value of this in quickly picking up a new code base is highly under-stated, if the popularity of Active Record is anything to go by.

I began looking for an ORM solution for Sinatra and quickly came across a project called DataMapper, and was incredibly pleased. Not only does it provide the same declarative style that Django’s ORM does, it goes above and beyond that with features that blow both Django’s ORM and Active Record out of the water.

Firstly it provides the ability to automatically migrate models. Arguments against magic aside, this is an amazing feature. Simply change the model, and the changes are migrated to the database when next instantiated.

Secondly, DataMapper completely eliminates the N+1 query problem. When iterating through data and accessing other models via related fields, DataMapper will query the database when the first relationship is accessed, retrieving all required data pertaining to the outer loop, building up all of the instance relationships on the fly prior to accessing them. Yes, it sees into the future and protects you from obliterating your database, a mistake all too common in web development. It’s worth noting that Django introduced this feature in the yet to be released 1.4 with the prefetch_related method. This is another great example of the explicitness of Python compared to the implicitness of Ruby.

Whether you’re building a web application or not, if you’re accessing a database from Ruby, consider using DataMapper. It’s a great piece of software.

Heroku

The explosion in Ruby and Python development frameworks over the last half decade has been a boon to web development. Security, modularity, shelf life, and time to market have all dramatically improved thanks to dynamic languages, the frameworks that have developed around them, and the open source communities that make them possible. It’s not all fun and games however. Deployment of these applications has grown considerably more complex. Gone are the days of using FTP to upload some PHP scripts to a server, and hitting refresh on the web page to test your changes. We now have to deal with a wide variety of deployment tasks, from reloading application processes, database migrations, dependency management and much more.

Naturally the community has risen to solve these problems, a movement sometimes referred to as DevOps, with tools in place such as Ruby’s Capistrano, Python’s Fabric, and configuration management tools to map complex deployments such as Chef and Puppet. While the learning curve is steep, with enough time invested up front, deployments can become as simple as pushing a button, and are more robust and integrated with quality assurance than ever before.

Modern deployments such as these require experts. This is where Platform as a Service (PaaS) offerings come in. PaaS providers are modern hosting companies that take care of all the dirty work in configuring servers and automating deployments for you. Typically they’ll expose a distributed version control server using Git or Mercurial that you can push code commits to. The process of pushing commits then triggers deployment, automatically performing all of the required tasks. PaaS providers will also provide all of the related services required, such as databases, message queues, caching servers and so forth.

Are PaaS providers a magic bullet? Absolutely not. Every application will have a tipping point where it grows beyond the “one size fits all” approach provided by PaaS offerings. You’re also at the mercy of the provider when it comes to uptime, so mission critical applications with strict service level agreements may require much more fine grained control in their hosting environments. Never forget that you are the only one responsible for your service’s availability. However for smaller, stock, non-critical applications, PaaS providers are a dream come true, that remove most of the complexities around service provisioning, configuration and application deployment.

The most well known of these providers is Heroku, who were the first to popularise the PaaS architecture. Not only does Heroku offer a rich variety of add-on services that I was looking for, like Varnish, Memcached, and PostgreSQL, they also provide free hosting for low capacity sites - ideal for the types of applications I ended up building with Sinatra.

So with Sinatra, DataMapper and Heroku combined, I developed several small applications that scratched particular itches for me, in order to build up a good working knowledge of Ruby.

Linked Out

I don’t keep an up to date CV any more. If I pick up a new skill, or start a new role, I’ll update my LinkedIn profile. It’s the quickest and easiest way to keep my professional information up to date. For better or worse though, over the last few years LinkedIn has turned into a mass hunting ground for recruiters. I used to take the time to enter into a dialogue with each and every recruiter that contacted me, after all anything less would be rude, but over time I realised the futility in this, as the practice by recruiters to blast out boilerplate introductions to anyone who matched a keyword search, became more and more common. But I digress. The state of recruitment aside, these conversations would inevitably lead to recruiters asking for a CV they could present to their clients. LinkedIn profiles contain a “download as PDF” feature, which I would always refer recruiters to, but LinkedIn embeds their logo within the PDF, and over time as they’ve added new profile features, the PDF download hasn’t picked these up. What I’d always wanted was an easy way to export my profile as a clean PDF, containing only the information relevant to a CV. I also wanted to be able to share the tools with anyone else who wanted to use it, so a baby Sinatra app seemed like the perfect fit.

There’s a great Ruby library for interacting with LinkedIn’s API, and PDFKit for converting HTML to PDF, which meant that I could format the CV using HTML and CSS, and all I then needed to do was convert that directly to PDF.

The app I ended up building is called Linked Out. It has a very simple flow to it. You first authenticate via LinkedIn’s oAuth service, and then you’re redirected back to Linked Out, where you can create a PDF version of your profile. As an added bonus I hooked into LinkedIn’s connections API, so you also get the option of creating a CV for any of your LinkedIn connections - no need for recruiters to bother people with CV requests, they can create the CVs themselves.

LinkedIn is somewhat lacking when it comes to formatting large blobs of text in profiles. People tend to create all sorts of formatting themselves, typically to create bulleted lists. So Linked Out contains some smarts to look for these different types of free-text formats, converting them into proper lists and headings where appropriate.

Need to fend off pesky recruiters with a nice looking CV? Go and update your LinkedIn profile, and give Linked Out a try.

Klout Feed

If you’re unfamiliar with Klout, it’s a reputation measurement system that gives you a daily score based on your online interactions. It looks at your Twitter account as well as other social media services, and applies an algorithm based on the number of mentions you receive, retweets, and favourites, also taking into account the Klout score for each of the people who trigger these. It then assigns you a score out of 100 which you can measure on a daily basis to gauge how effectively you’re using Twitter. According to Klout at least.

Klout has been described as many things, from a revolutionary game-changer in social media, to a vapid and narcissistic waste of time. Personally I found it to be an entertaining distraction. It did keep me coming back each day to check my score.

Now I’m one of a dying breed that still uses RSS exclusively to keep track of everything going on online. Updates from my LinkedIn connections, contributions and issues for my projects on GitHub, Google Groups mailing lists, and of course various news sites and blogs. They all contain RSS feeds which I can catch up on, in one single interface. But not Klout. You need to log into their site each time you want to check you score. There’s an interesting point here. If you want to keep users coming back to your site and piss them off at the same time, create a great service but make sure you don’t include an RSS feed for the data your users are interested in.

To solve this I put together an app called Klout Feed. It uses Klout’s API to provide an RSS feed for each user, publishing their score and its change each day. The flow isn’t as seamless as Linked Out’s. Klout doesn’t provide any form of application integration the same way LinkedIn, Twitter and Facebook do. Just a per-user API key with daily limits assigned to it. So with Klout Feed you first need to head on over to Klout and grab an API key, then bring that back to Klout Feed to get the URL for your score’s RSS feed.

Ian Anderson has since gone ahead and written a how-to article on combining Klout Feed with the If This Then That service. The result is that you can get email or SMS notifications each time your Klout score changes.

One True Repo

Most open source projects use Git or Mercurial for version control, and are hosted on either GitHub or Bitbucket respectively. Some people like myself host their projects on both sites. I’ve talked about my setup for hosting on GitHub and Bitbucket before. Both sites provide totals for the number of interested developers following the project, and the number who have forked the project. A fork is when someone creates a copy of a project, usually with the intent of adding some news features or fixes, and pushing them back to the original source.

What I’ve always wanted is a combined API for totalling followers and forks across both services for a single project hosted on both sites, and also for all projects for a given user on both sites. This is what I tackled for my next Ruby project, which I called One True Repo (OTR).

OTR’s original form was as a library that other developers could embed in their project, so I built it as a Ruby gem that you can both include in your own project, or simply run from the command line and pipe the data it returns into other programs. The next step was to build a baby Sinatra app that provided a hosted version of the API that people could query. The project itself contains everything for all three of these forms - the library, the command-line tool, and the Sinatra web app.

Querying the GitHub API was trivial and all the information I wanted was provided by it very easily. The Bitbucket API wasn’t quite up to scratch for this however. Remarkably it doesn’t expose follower and fork count on each project. Some screen-scraping was therefore required to get these totals for each of a user’s projects.

Do you mirror your open source projects across both GitHub and Bitbucket? Ever wonder how many people are following all your projects on both services? Give One True Repo a try.

Conclusion

As you can see from some of these apps, the Sinatra on Heroku combination is the perfect fit for small mashups that act as glue between other popular APIs. Free of charge, rapid development, and a great pool of libraries to choose from in the Ruby eco-system.

Another year has gone by and another obligatory year in review post is due. 2011 was a year full of change for me. I changed jobs twice, spent half the year coding in a new language, moved interstate, and switched my primary operating system.

Ruby

After working at Citrus for almost nine years, I was well overdue for a change of scenery, so when I was approached by Impact Data to come and work for them using Ruby on Rails, I welcomed the opportunity. My time there was short and sweet however, as after six months I decided to move back home to Sydney, but it was a really rewarding experience working with an incredibly smart team on a technology stack that was new to me. I had a great time learning Ruby, forming a strong appreciation for its elegance, and it was very interesting along the way making plenty of comparisons between Rails and Django. Naturally I started using Ruby in my own projects, getting to know Sinatra for a handful of apps that I built. I’ll be writing a more detailed post about those soon, so stay tuned!

Django Dash

For the second year in a row I entered the Django Dash hackathon. This time around I wanted to do something that really pushed Django outside of its typical usage patterns. I’d recently read Cody Soyland’s introductory blog post on using WebSockets with Django, and so I came up with an idea I called Drawn By, a collaborative drawing app where people could create sketches together in real-time, save them to an image they can download, and rate others’ sketches in the gallery.

I got to use a variety of technology I hadn’t used before which was really fun. I used gevent as the evented web server for maintaining open socket connections with the browser, the NoSQL database Redis for queuing events and storing temporary pixel data, and the browser’s Canvas API for front-end drawing interaction and rendering. I set a relatively high goal for ourselves this year with what we wanted to achieve, but we pulled it off nicely in the end.

This year we came 3rd place out of around 50 entries, which was a great improvement on the previous year’s result of 8th. The most important result however was the creation of django-socketio, which was extracted from Drawn By and released as open source. It brings together all of the scaffolding for using WebSockets with Django, and implements an events and channels system for building your own applications around it. I previously wrote about django-socketio right after releasing it, and since then it has gained quite a lot of traction, with a handful of developers contributing back fixes and new features.

Open Source

I actually spent less time this year contributing to open source than I did the previous year, but I steamed ahead nonetheless with a lot of new projects, as well as continued development and support for my major works, Mezzanine and Cartridge. Both these projects have reached a very mature level over the course of 2011, thanks to tons of contributions from the Mezzanine and Cartridge community, which continues to grow steadily. Here’s a list of the projects I released as open source over the year:

• Drawn By: Collaborative real-time sketching. (Django / Python)
• django-socketio: WebSockets for Django. (Django / Python)
• Virtualboxing: Comparison utilities for Riak and MongoDB. (Ruby)
• Grillode: Multi-purpose chat server. (Node / CoffeeScript)
• Linked Out: Create PDF resumes for LinkedIn. (Sinatra / Ruby)
• Klout Feed: Daily Klout scores via RSS. (Sinatra / Ruby)
• Babbler: A Twitter bot. (Python)
• One True Repo: Combined GitHub and Bitbucket API. (Sinatra / Ruby)
• hg-github: A Mercurial extension for GitHub. (Mercurial / Python)
• sphinx-me: A Sphinx documentation generator. (Python)

OSX

As I mentioned, towards the end of the year I moved back to Sydney. An opportunity came up to work with Fairfax, the largest media organisation in Australia. Fairfax is building a new publishing platform using Django, so my experience with content management and Django was a natural fit. Mostly though, it was a chance for me to move back home and be closer to my family, after being away from them in Melbourne for a decade.

When I started at Fairfax, I was surprised to find the entire team running OSX. I wasn’t surprised so much by the choice itself, as OSX is very popular in the Django and wider development community, but more so by my own lack of experience with it, having solely used Linux for the last half decade. I decided to give it a go and found it to be on par with Linux as a development environment.

My biggest gripe was breaking down the mental muscle I’d built up around the shortcut keys for wrangling text. At first I was fumbling on OSX, but after a few days of using OSX during the day and Linux at night, I found that I wasn’t efficient with either of them - I needed consistency. I was also long overdue for a new machine. I originally had my eye on some of the MacBook Air clones like the Acer UltraBook and Asus ZenBook, but I couldn’t find any information about running Linux on these, and I wasn’t prepared to go through the pain of working it out if anything went wrong. So I bit the bullet and picked up a 13 inch MacBook Air.

All I need at the software level is visible application shortcuts, keyboard-driven application switching, a decent editor, terminal, web browser and package manager, and I’m good to go. In that regard, OSX and Linux are equivalent for my use, each with their own minor flaws. What I am really loving is the hardware. The keyboard seems laid out in a way that lets me type faster than ever, and the solid state drive means everything is instantaneous - that paired with the best battery life I’ve ever experienced, and it’s a dream machine.

The two front-runners in source code management these days are undoubtedly Git and Mercurial. Distributed version control has clearly proven itself as the superior model over older centralised systems like SVN, particularly in the context of open source development, where the ability to fork repositories and push and pull branches between them facilitates a much more efficient and streamlined work-flow.

My personal opinion about the difference between the two is similar to my take on Django and Rails. Relative to their alternatives, the two are far more similar than different, essentially providing the same concepts and features, but with quite different underlying philosophies in their implementations. For this reason I choose the less popular Mercurial over Git to manage all of my projects. Firstly I feel it has a more natural and intuitive UI (I’m not referring to a graphical interface here, but the actual commands it implements and their arguments). Another key factor is that Mercurial is written in Python, which means hacking on it and building extensions for it is a breeze.

What makes Git more popular than Mercurial? One major factor is their respective online hosting services, GitHub for Git and Bitbucket for Mercurial. If you’re unfamiliar with these sites, they’re like Facebook for programmers who share projects and collaborate together on code rather than post photos and videos. While both of these sites mostly implement the same core features, GitHub has always been several steps ahead of Bitbucket in terms of overall polish, and being first to market with new features. This difference has led GitHub to become far more popular than its Mercurial counterpart, and as a result, Git far more popular than Mercurial.

With the majority of open source activity occurring on GitHub, what then is a Mercurial user to do? Limiting your projects to the audience of Bitbucket means missing out on a lot of potential collaboration. Fortunately some time ago, the team at GitHub developed a Mercurial extension called hg-git. It allows you to transfer code back and forth between a Mercurial repository on your machine to a Git repository on another machine, like GitHub for example, taking care of all the translation required between Git and Mercurial. After several years of using hg-git, it’s one of the only pieces of software that continues to amaze me. Think of Google’s language translator, which on a good day can provide some very quirky translations when converting text from one language to another. hg-git performs the same task, but has no room for error when translating a source code repository from Mercurial to Git, and back again. Admittedly hg-git has a much easier job to do than translating human languages with all their ambiguities. Still, I am constantly impressed by the task it performs.

So hg-git allows me to develop my projects using Mercurial and have them shared on both GitHub and Bitbucket, allowing for maximum collaboration which is fantastic. It’s not entirely seamless however. I still need to perform a couple of manual steps such as adding GitHub paths to my repo configuration, and creating Mercurial tags that map to the Git branches I want to work with. I need to do this each time I set up a new repository, be it for starting a new project, or forking a project of my own or someone else’s. Recently I had to do this about half a dozen times in the space of an hour while working on a few different projects, and I thought to myself that I should be able to automate it. The result is a Mercurial extension I’ve called hg-github which automatically takes care of these manual steps required. It also wraps hg-git, so you don’t need to install both extensions, as hg-github pulls in and takes care of all the hard work that hg-git does.

Overview

Once hg-github is installed, assuming the default remote location of your repository is on Bitbucket, the GitHub path is automatically added and given the name github, so you can push to it with the following command:

$ hg push github

For other named Bitbucket locations, the name github-NAME is given, where NAME is the name of the path located on BitBucket. For example consider the following .hg/hgrc repo config:

[paths]
default = ssh://hg@bitbucket.org/stephenmcd/hg-github
somefork = ssh://hg@bitbucket.org/stephenmcd/hg-github-temp

hg-git will add entries to the config file as follows. Note that the config file isn’t actually written to:

[paths]
default = ssh://hg@bitbucket.org/stephenmcd/hg-github
somefork = ssh://hg@bitbucket.org/stephenmcd/hg-github-temp

github = git+ssh://git@github.com/stephenmcd/hg-github.git
github-somefork = git+ssh://git@github.com/stephenmcd/hg-github-temp.git

hg-github assumes you have the same username on GitHub and Bitbucket. If you have a different GitHub username, you can specify it by adding the following section to your global .hgrc file. For example my GitHub username is stephenmcd:

[github]
username = stephenmcd

As a creator and maintainer of several popular reusable Django applications, one of the most commonly requested features I’m asked for is the ability to customise the fields that a model implements. This topic comes up often on the Mezzanine mailing list, and during this particular thread we researched ways that fields could be dynamically injected into models at run-time.

Other Approaches

It’s worth taking a look at other approaches to the general problem, and what their drawbacks are, in order to provide context for what the final solution needs to achieve.

One approach is to implement as many of the model classes as possible as abstract base classes, so that users can subclass these with their own models. This approach makes sense for certain types of customisation, and it’s what I’ve done with django-forms-builder for example. Some caveats exist with this approach however. Firstly, relationship fields can’t be defined on the abstract models, so these need to be implemented in concrete models either within the same app, or by the user implementing their own subclasses. Secondly, any functionality that references your models, such as views or middleware, needs to either have configurable settings for choosing which models to use, or be reimplemented entirely by the user to make use of their custom fields.

Another approach is to simply recommend that users subclass the models that the app provides using multi-table inheritance. Unfortunately this will introduce unnecessary overhead with the extra database queries required when accessing the instances of the subclasses. Best case is that this amounts to an extra query or two in a view dealing with a single instance. Worst case is that when this approach is used with a queryset in a template, an extra query is performed for each instance returned - the classic N+1 query problem.

Dynamic Injection

The ideal approach would allow users to directly modify models with their own code, outside of the models’ apps, without the models themselves having to implement any special hooks for customisation. The end result being an optimal database design, with no extra API requirements for the relevant models. It just so happens that this is possible by using several features that Django exposes, and combining them together in a particular way.

The approach boils down to three concepts:

• Dynamically adding fields to model classes
• Ensuring Django’s model system respects the new fields
• Getting the load ordering correct for the above to work

Django’s model fields provide an undocumented contribute_to_class method. This method serves as a fancy version of setattr and takes a value and attribute name to use as arguments. Internally it then takes care of all the house-keeping required for a field to be added to a model.

The other feature of Django we’ll use is the class_prepared signal. This signal is emitted each time a model class is declared and loaded for the first time by Django’s model system.

from django.db.models import CharField
from django.db.models.signals import class_prepared

def add_field(sender, **kwargs):
    """
    class_prepared signal handler that checks for the model named
    MyModel as the sender, and adds a CharField
    to it.
    """
    if sender.__name__ == "MyModel":
        field = CharField("New field", max_length=100)
        field.contribute_to_class(sender, "new_field")

class_prepared.connect(add_field)

The final consideration is connecting the class_prepared signal at the correct time. It needs to occur prior to the relevant model class being declared, otherwise the signal will never be triggered when we want it to. A general way of achieving this is to connect the signal from within an app that is listed before the app containing the relevant model, in the INSTALLED_APPS setting. Note that in the above code, we don’t explicitly import the model to use it as the signal’s sender, instead checking for the model’s class name, as importing it would break these load ordering requirements.

Caveats

Like the previously described approaches, dynamic injection also comes with a set of drawbacks. These drawbacks stem from the fact that the apps containing the models being customised don’t contain a definition for the fields being injected. This means that migration tools likes South are unable to detect the new fields, and workarounds such as creating manual migrations are required.

Another related problem is when new admin classes containing references to the custom fields are registered and the fields haven’t yet been injected. A typical requirement for injected fields is to expose them via Django’s admin interface, which can be achieved by unregistering existing admin classes for the models that fields are being injected into, subclassing these admin classes with new references to the injected fields, and registering the new admin classes. Unfortunately if this unregister/register dance occurs in an admin module, the fields may not have yet been injected. A quick work-around for this is to perform the unregister/register calls inside your project’s urlconf.

Mezzanine Support

Drawbacks aside, the field injection technique described above has characteristics that make it incredibly useful. As such the approach has first-class support in Mezzanine by way of the EXTRA_MODEL_FIELDS setting. This setting allows you to define a sequence of all the custom fields you’d like to inject. Each item in the sequence contains four items: the dotted Python path to the model (including the field name to use for injection), the dotted Python path to the field class to use for the injected field, a sequence of the field’s position arguments, and finally a dict of its keyword arguments.

EXTRA_MODEL_FIELDS = (
    # Add a custom image field from the fictitious somelib.fields module
    # to Mezzanine's BlogPost model:
    (
        # Dotted path to field.
        "mezzanine.blog.models.BlogPost.image",
        # Dotted path to field class.
        "somelib.fields.ImageField",
        # Positional args for field class.
        ("Image",),
        # Keyword args for field class.
        {"blank": True, "upload_to": "blog"},
    ),
    # Example of adding a field to *all* of Mezzanine's content types:
    (
        "mezzanine.pages.models.Page.another_field",
        "IntegerField", # 'django.db.models.' is implied if path is omitted.
        ("Another name",),
        {"blank": True, "default": 1},
    ),
)

Mezzanine then uses this setting to inject all of the fields defined, using class_prepared and contribute_to_class as described earlier. It handles getting load order correct by performing the injection within the mezzanine.boot app, which is forced to the front of all apps defined in INSTALLED_APPS. Django’s admin is also patched in the boot app, to defer certain calls to unregister and register, to correct the ordering issues described earlier.

Update, Feb 14: @rkJun has translated this post into Korean.

I recently gave a presentational talk to the development team at my work, on what it’s like contributing to open source. A common perception by people not involved is that the open source community seems like one big socialist hippie commune, whose motivations are entirely altruistic with little reward for the individual. The approach I took was to dispel this myth and show that for many who participate, their motivations are mostly self serving, with many personal gains to be made. Despite some of the awkward truths I focused on, the talk was received really well.

Your Day Job Sucks, Make Programming Fun Again

If you have a regular 9 to 5 job as a software engineer at a typical company, chances are likely that your job sucks in several ways. You work on the same types of projects every day. You work with the same technology every day, and most likely it’s dated. There are more modern technology stacks out there that are much more elegant, yet for a variety of non-technical reasons, you’re not allowed to use them at work. You also have deadlines. Deadlines lead to knocking out features as quickly as possible, which directly conflicts with the quality of your work, so you’re often forced to take ugly shortcuts, leaving little to be proud of.

If this is your only exposure to programming, it gets boring pretty quickly and isn’t much fun at all. Contrary to this, programming can actually be incredibly fun! Imagine your company had hundreds of different projects to choose from, and let you choose whichever one you wanted to work on. Then on top of that, they told you that you could use whichever technologies you wanted to, and that you could take as long as you needed to build it correctly - so long as the end result was the most well designed and beautifully written software you’ve ever produced. As a professional programmer, that sounds like a dream, but that’s exactly how it is when you work on open source projects. Choose your problem domain. Choose your technology stack. Choose your pace. Choose your quality.

Become a Better Developer through Collaboration

There are two ways to become a better developer: writing code and reading code. This clearly doesn’t make sense in a silo on your own. There’s little to learn from reading your own code beyond reminiscent value, and you could write code on your own for years and the only lessons learned would be those you teach yourself. Naturally the real learning is to be had from interacting with other developers, writing code that they can review and give you feedback on, and reading code written by other developers with more experience and different approaches than you have.

Drawing back to your day job, how many developers do you collaborate directly with. A few? A dozen? A small pool that will give you much to learn from, but is still relatively limited in scope. The open source community is made up of thousands of developers, who over many years have produced millions of lines of code. Code of quality much greater than you’d ever be exposed to without stepping outside of your workplace.

Acquire Cash Money

One of the biggest misconceptions abut open source development is that it’s a financially fruitless labour. While true to a certain extent, there are actually several ways in which it can pay a monetary return, both indirectly and directly.

You may work a steady, full-time job, or you’re looking for one, and think of yourself as an employee with a fixed salary, in total contrast to those wild freelancers and entrepreneurs who don’t know where their next meal will come from. Wrong. You are a business, just like your employer is. You have one customer, your employer. Sometimes you might feel as though you’re stuck in your day job, when in actual fact you’re simply stuck with one client.

So how can you gain more clients? How can you gain a better client? By promoting your business of course. Open source provides a great way for you to showcase you and your ability as a developer. Your open source contributions can distinguish your CV from other developers, namely, your business’s competition. Graphic designers have always had the luxury of being able to display their portfolios, and developers can too. I’ve been on both sides of the hiring table, and when choosing between potential hires, someone with significant open source contributions has already demonstrated a clear passion for what they do, and therefore has a major advantage over those who are tasked with convincing people of the same, without having anything to show for it.

Getting your open source projects to the level of quality and usefulness where people are actually building production systems with them is huge achievement in itself. Stemming from that are direct opportunities for paid work. Your corporate users may need your software customised for their use cases. You may even be lucky enough that they’d like features developed into the project that are great ideas - things you might not have thought of, and will take the project to the next level. Who better to develop these than the creator of the software themselves? I’ve had several instances where this has occurred, and companies have sponsored development on features that, had I had the insight to think of, would have developed freely on my own anyway.

Broaden Non-Technical Experience

Hands on coding is only one small aspect of software development. There are many other disciplines that make up the process of shipping a successful product. Project management, community building, formal writing, product development and marketing. While freelancers and one-man commercial studios may get the benefit of dipping their toes into these different areas, as a developer at a typical company your role often doesn’t take you outside of the day to day coding.

Running an open source project properly will allow you to skill up in broad range of non-technical areas. I spend a lot of my time working on my most popular project, however these days only a small portion of this time is actually spent coding. Most of the time is spent in discussion with other developers on their feature ideas, coordinating the inclusion or exclusion of these, and providing support for new users. It’s hard work, but it’s a great and relevant experience.

Corollary to this, many non-technical people who’d like to contribute to open source in some way, believe that they don’t have anything worthwhile to offer. With all that’s involved, some of the most beneficial contributions are those of a non-technical nature. If this describes you, then please don’t hesitate to dive in. Your help means more than you can imagine.

Feels Good Man

The most obvious yet least tangible benefit in contributing to open source, is the idea I first alluded to around altruism and giving to others. As I mentioned, coordinating a non-trivial open source project can be hard work. It has its highs and lows, and sometimes it feels like there’s more of the latter. However every now and then, I’ll get a personal email of praise and gratitude. Not in any public forum where the soapbox can be a conduit for ulterior motives, just a private and genuine expression of thanks. I would never have thought of this as being particularly rewarding, until I actually experienced it. It really fills you with a true sense of pride and satisfaction, which makes it all worth while in the end.

Get Started Now

The Chinese philosopher Lao Tzu said:

A journey of a thousand miles begins with a single step

Getting started might seem like a daunting task. Start small. Ignore the inner voice that whispers in your ear, arguing that your open source idea wouldn’t be of any use to anyone. That might even be true, but it doesn’t matter. Do it anyway. A lot of my projects are only a few dozen lines of code. People have found them useful in ways I couldn’t have imagined.

Ideas can be hard to come by. Most of mine were itches I wanted to scratch. Product gaps in the technology stacks I’m interested in, or mundane processes I went through regularly, upon realising they could be automated and productised.

Perhaps you’re an end user of open source software, and it isn’t perfect. A feature works differently than how you think it should, or something is undocumented and you struggled to find it. These days it seems that the modus operandi is to throw up your arms, and complain as loudly as possible about how the software you’re using for free, isn’t exactly the way you want it to be. Don’t. You have the choice to do something about it. Dive into the source code. Contribute something back. Improve yourself, and be a part of something.

I recently took part in Django Dash, the annual hackathon where teams of up to three compete to build the best Django project they can within 48 hours. This year I worked with Travis White and Josh de Blank to build DrawnBy - a collaborative drawing application that allows people to sketch together in real-time.

DrawnBy makes extensive use of WebSockets which are mostly unheard of in web stacks like Django and Rails, or even antiquated stacks like PHP and ASP.NET, which are all designed around accepting a request and returning a response to the browser as quickly as possible. This is in contrast to WebSockets which allow full duplex communication between the browser and server, and therefore require long running requests per user.

A variety of patterns for dealing with WebSockets in Django emerged while developing DrawnBy, and since the Dash I’ve been working on abstracting these into a reusable Django application called django-socketio which I’ve released today. It’s available on GitHub, Bitbucket and PyPI.

Here’s an overview of the features.

Features

• Installation of required packages from PyPI
• A management command for running gevent’s pywsgi server with auto-reloading capabilities
• A channel subscription and broadcast system that extends Socket.IO allowing WebSockets and events to be partitioned into separate concerns
• A signals-like event system that abstracts away the various stages of a Socket.IO request
• The required views, urlpatterns, templatetags and tests for all the above

Channels

The WebSocket implemented by gevent-websocket provides two methods for sending data to other clients, socket.send which sends data to the given socket instance, and socket.broadcast which sends data to all socket instances other than itself.

A common requirement for WebSocket based applications is to divide communications up into separate channels. For example a chat site may have multiple chat rooms and rather than using broadcast which would send a chat message to all chat rooms, each room would need a reference to each of the connected sockets so that send can be called on each socket when a new message arrives for that room.

django-socketio extends Socket.IO both on the client and server to provide channels that can be subscribed and broadcast to.

To subscribe to a channel client-side in JavaScript use the socket.subscribe method:

var socket = new io.Socket();
socket.connect();
socket.on(connect, function() {
    socket.subscribe(my channel);
});

Once the socket is subscribed to a channel, you can then broadcast to the channel server-side in Python using the socket.broadcast_channel method:

socket.broadcast_channel("my message")

Events

The django_socketio.events module provides a handful of events that can be subscribed to, very much like connecting receiver functions to Django signals. Each of these events are raised throughout the relevant stages of a Socket.IO request.

Events are subscribed to by applying each event as a decorator to your event handler functions:

from django_socketio.events import on_message

@on_message
def my_message_handler(request, socket, context, message):
    ...

Each event handler takes at least three arguments: the current Django request, the Socket.IO socket the event occurred for, and a context, which is simply a dictionary that can be used to persist variables across all events throughout the life-cycle of a single WebSocket connection.

• on_connect - occurs once when the WebSocket connection is first established.
• on_message - occurs every time data is sent to the WebSocket. Takes an extra message argument which contains the data sent.
• on_subscribe - occurs when a channel is subscribed to. Takes an extra channel argument which contains the channel subscribed to.
• on_unsubscribe - occurs when a channel is unsubscribed from. Takes an extra channel argument which contains the channel unsubscribed from.
• on_error - occurs when an error is raised. Takes an extra exception argument which contains the exception for the error.
• on_disconnect - occurs once when the WebSocket disconnects.
• on_finish - occurs once when the Socket.IO request is finished.

Like Django signals, event handlers can be defined anywhere so long as they end up being imported. Consider adding them to their own module that gets imported by your urlconf, or even adding them to your views module since they’re conceptually similar to views.

Binding Events to Channels

All events other than the on_connect event can also be bound to particular channels by passing a channel argument to the event decorator. The channel argument can contain a regular expression pattern used to match again multiple channels of similar function.

For example, suppose you implemented a chat site with multiple rooms. WebSockets would be the basis for users communicating within each chat room, however you may want to use them elsewhere throughout the site for different purposes, perhaps for a real-time admin dashboard. In this case there would be two distinct WebSocket uses, with the chat rooms each requiring their own individual channels.

Suppose each chat room user subscribes to a channel client-side using the room’s ID:

var socket = new io.Socket();
var roomID = 42;
socket.connect();
socket.on(connect, function() {
    socket.subscribe(room- + roomID);
});

Then server-side the different message handlers are bound to each type of channel:

@on_message(channel="dashboard")
def my_dashboard_handler(request, socket, context, message):
    ...

@on_message(channel="^room-")
def my_chat_handler(request, socket, context, message):
    ...

Chat Demo

The “hello world” of WebSocket applications is naturally the chat room. As such django-socketio comes with a demo chat application that provides examples of the different events and channel features available.

I recently started a new role at a Ruby on Rails shop, which as a long time Django specialist was a really interesting opportunity. There’s a lot of competition between the two frameworks’ communities, ranging from friendly rivalry and respectful admiration at the mature end of the scale, to all out fanboy fuelled flame-wars at the other.

After you wade through the rivalry, the common wisdom voiced is that they’re conceptually the same. If you know Python, go with Django and if you know Ruby, go with Rails. After spending several months with Rails I can attest to this being true. At a bird’s-eye view both frameworks contain almost identical concepts, implemented with different philosophies stemming from the ideals expressed by the languages they’re written in. Both frameworks provide a vastly superior approach to security, modularity and rapid development than their predecessors do.

An interesting question to ask would be: which would be the best framework to choose, not knowing either language? It would be naive of me to believe I am unbiased, but I would certainly recommend Django over Rails. The relative strictness of Python and explicitness of each component in Django, compared to the implicit magic in Rails, is simply much more geared towards creating large-scale systems in a sane and transparent way. To Ruby’s credit though, I have developed a real admiration for the language itself, and have continued using it in my own projects - but that’s a topic for another post.

Considering how similar the two frameworks are component-wise, one thing I did miss was a side-by-side cheat sheet for working out what each of the concepts were in Rails that I already knew well in Django. I’ve put one together below to help out anyone who might be picking up either framework while already knowing the other. For clarity, I’ve also included descriptions of each type of component, for those who haven’t used either framework.

Not all of these pairings are a perfect one-to-one match conceptually, but should be good enough to get an overall view of what each concept is within both frameworks.

The question of scalability with regard to Mezzanine and Django recently came on up on the mezzanine-users mailing list, to which I offered the following reply.

Mezzanine and Django itself are fantastic choices for someone concerned with scaling for high traffic.

The Mozilla add-ons site that hosts all Firefox plugins, and Disqus which is currently the world’s highest traffic commenting system, both run on Django. Each of these have been quoted at 500 million hits per day and 1 billion per month respectively.

One of the keys to scaling sites like these is the wealth of options available for caching in the Django ecosystem, and the ability to then scale out the number of both application servers running Django, and caching servers typically running memcached, with very little modification to your application code. Django comes out-of-the-box with the ability to switch on site-wide caching, per page caching and template fragment caching. Beyond that there are also third-party Django applications that implement object level caching such as Django Cache Machine and Johnny Cache, for even finer-grained control with little modification to your code.

Both Mezzanine and Cartridge have been designed from the ground up with scalability in mind. Particular care has been taken to avoid any n+1 queries, for example rendering out multiple instances of Mezzanine’s navigation tree containing any number of nested levels of navigation will only ever run a single database query. Same with Cartridge’s products. What this means is that you can go very far traffic- wise, using a single server without even thinking about caching. Once you do then the ability to add application and cache servers is trivial, and will take you incredibly far using a single database. Once you start needing multiple database servers Django also comes with the built-in ability to route models across different databases, so there are extra options there beyond your typical master/slave database replication scenario.

I’ve only touched on the topic to provide an overview of what’s available, and you should certainly research it further for your particular scenario, but as you can see scalability is a core concern baked into Django and Mezzanine, so you can choose these with confidence.

Mezzanine and Cartridge Hit the Mainstream

This week I had the fantastic opportunity to be interviewed by Rodney Gedda from IDG for their TechWorld and CIO magazines. The interview focused on my open source work with the Mezzanine and Cartridge projects.

Melbourne hacker releases open source Django apps

These projects have been gaining traction at an incredible pace, with a solid community forming around them over the last year. All of this has happened in spite of having no formal promotion outside of the community itself, which makes this type of exposure all the more exciting.

It’s also no secret that I think the Australian web development market is many years behind the United States, so it’s a great step forward to see modern technology getting covered by the mainstream local tech media.

Having remained utterly faithful to Python and Django over the last year, and dedicating all of my available time to open source projects like Mezzanine and Cartridge, with the new year at hand I though it was about time to take a break and add some new technology to my repertoire. During his keynote speech at Djangocon.eu 2010, the creator of Django Jacob Kaplan-Moss states “it will challenge what you think you know about web/server architecture” when referring to Node.js. Since then it has been sitting in the back of my mind as something I definitely needed to check out, so I decided to dive in head first.

Node.js is a general purpose JavaScript development environment geared towards writing network servers. It uses an event-based, non-blocking architecture which allows your web application to scale to thousands of concurrent connections without needing a finite pool of threads or processes. JavaScript is executed using Google’s V8 engine which ranks very highly in speed compared to other dynamic languages, so not only does Node.js scale elegantly, it’s damn fast.

Having recently built Grillo, a console based chat server, I’d been considering what it would be like to put together a web-based version. In fact I had achieved something similar in the past using Python’s BaseHttpServer module, and while functional for a few hundred connections, my approach would never scale, as either a separate thread or process would be required for each open connection. The event driven architecture of scaling a web server for an increasing number of open connections is mostly a solved problem, especially in the Python community with projects like Twisted and Tornado. However Node.js is different in that its non-blocking evented model is a first class citizen by design.

With a useful project at hand to try Node.js on, I set about creating what I’ve named Grillode (yes you guessed it: Grillo + Node). It’s a web-based chat server with a set of configuration options that lets you run it in various modes, such as a customer support queue, or with Chatroulette style random match- ups. I’ve released the source onto GitHub and Bitbucket, and also have a demo up and running.

The process of putting Grillode together led me through many parts of the ecosystem that has developed around Node.js - following is an overview of the pieces I ended up working with.

Node Package Manager (NPM)

NPM is a command line utility that gives you access to a central online repository of packages built for Node.js. It works wonderfully when installed correctly, but on my machine I encountered a handful of issues where it ended up recreating various system directories all throughout my home directory. After setting up various symlinks by hand, I did get it to work after many failed attempts at installing it. This issue was definitely specific to my machine as I was then able to go ahead and install NPM seamlessly on several different servers.

Once everything was working correctly it made deployment of Grillode a breeze. By specifying all of its dependencies in a package.json file, NPM was able to installed everything required in a single step.

Express

Express provides basic URL routing to functions that will typically perform some application logic and hand off data to a template to be rendered. It contains integration points for a handful of different templating libraries and it also contains a simple middleware system. I’d definitely consider it to be a micro framework, but it’s a great start at whipping your Node.js application into a well defined structure.

Socket.io

Socket.io takes all the leg work out of maintaining an open connection to the browser. You start by attaching it to your Node.js server which then automatically makes available the client-side JavaScript required. This provides the communication channel between the client and the server, which attempts to use web sockets when available, and transparently falls back to Flash sockets or even old-school AJAX polling if the former options aren’t supported by the browser.

It then provides all of the methods and event handlers for connecting and sending data over the connection. The beauty behind how this is implemented is that it exposes these methods and events almost identically to both the Node.js server, and the browser client - instantly you have available two-way communication between the browser and the server via an open connection, without requiring any new requests to the server.

CoffeeScript

As many others have done, I’ve often compared JavaScript to Python in that they both share an object model defined by a hash table of names and object members, which can be introspected and dynamically modified. While this is a very elegant model, JavaScript boasts syntax reminiscent of the turn of the century, cluttered with semicolons and braces, and missing a handful of features found in modern languages such as list comprehensions and much more. Well the war on semicolons is over with the explosion in popularity of languages such as Python and Ruby showing this to be true.

CoffeeScript is a language inspired by Python and Ruby which gets compiled directly into JavaScript. It therefore retains all the properties of JavaScript such as its data types, objects and methods, but provides a much more modern and clean syntax with some fantastic sugar, such as list comprehensions, class-based object construction, string interpolation and more.

Experimentally, CoffeeScript can be run directly in the browser in place of JavaScript by including the compiler JavaScript file itself, but much more interestingly it can be used as an execution environment for Node.js, which will perform the compilation to JavaScript when the Node.js application is first started. I found my experience to match reports of up to 30% in reduction of the amount of code required.

Coffeekup

When looking into what was available for templating, I started out with EJS which basically gives you executable JavaScript within your template files. I found this to provide a poor separation between display and application code, something that Django gets right by providing a limited template language that has basic flow control, but makes writing non-trivial logic difficult to do.

I then discovered a Node.js template library called Coffeekup, that tied in very closely with the time I had already spent with CoffeeScript. Coffeekup allows you to define your HTML entirely in CoffeeScript. I’m still undecided on whether this is a thing of beauty or horror. It’s very surreal to work with web page markup expressed entirely in programming code. I guess there’s somewhat of an undeserved feeling of the presentation being too closely tied to programming logic. There is a magical feeling in having your server code, client code and presentation code all in the exact same language, coupled with the given language being CoffeeScript which is incredibly clean.

Conclusion

I really enjoyed working with Node.js and the young ecosystem surrounding it. At this point in time, I wouldn’t consider it for a typical project over a full stack framework like Django with the elegance of Python, however it definitely serves as a fantastic choice for a very specific criteria - scalable, real time web applications.

I recently came across the Rosetta Code Project. It’s a community contributed wiki that contains hundreds of solutions to programming problems, implemented in hundreds of different programming languages, which is great source of entertainment for a programming languages enthusiast such as myself. The main focus of the project isn’t to demonstrate individual solutions on their own, but to provide comparisons between different programming languages and how they approach the same task.

The wiki also contains various dynamic reports, such as which tasks have yet to be implemented for each particular programming language. I took a look at the Python page to see if there were any interesting tasks remaining that I could potentially provide a solution to, and as I expected there were only a few relatively obscure tasks that were yet to have solutions provided.

One task remaining for Python that did catch my eye was to demonstrate a simple chat server using sockets. I’ve always been especially fond of network programming, from web crawlers to XMLRPC to lower level sockets, I seem to really enjoy writing code that runs over the Internet without necessarily being related to web development, so I went ahead and added a Python solution for the chat server task.

I honestly had so much fun working on this that I decided to extend it even further. Firstly I refactored it into a more object oriented approach which allowed for creating both server and client tools which could be run concurrently using separate threads of control. I then added several commands that could be run directly in chat by any user, for example listing the current users who were logged in.

I then decided to publish the code for my chat server and client onto both GitHub and Bitbucket, after giving it the name “Grillo”. It’s named after the Italian phone of the same name, developed in 1965. They both share the common theme of being a very small communications device for their class, while being implemented with relatively basic technology.

Given that there are many richer and more powerful applications available for implementing the features Grillo provides, at the least it serves as a good example of how to do basic socket programming in Python, as well as a demonstrating some simple tricks for controlling threads. At the best case someone will pick up the code base and extend it further in ways I haven’t anticipated - here’s hoping!

2010 was an amazing year for me professionally. I learnt and achieved much more than I’ve historically packed into a single year, so like many other I’ve decided to jump on the year in review bandwagon and put this post together.

Open Source

Prior to 2010 I’d certainly used open source quite heavily, in fact I’d based my entire career as a developer around it, focusing on Python, Django and Linux and the open source ecosystem surrounding this amazing platform. However as far as giving back to the community went, I had only ever contributed small bug fixes and enhancements to a handful of open source projects over the years, and had never been heavily involved in any single project. So in 2010 I dived in head first launching a range of open source projects. Firstly some smaller utilities such as django-forms-builder and gunicorn-console, and then the Django content management platform Mezzanine, along with its shopping cart plug-in, Cartridge.

The reception Mezzanine has received has been nothing short of amazing and well beyond anything I had anticipated. It’s been an incredibly rewarding learning experience, managing product development and working closely with its contributors towards growing its community. Here are a few statistics for the project only 6 months since it was launched:

• 120 followers and 30 forks across GitHub and Bitbucket
• Code contributions from 10 developers
• A mailing list with over 60 members and over 300 messages
• Over 4,000 downloads from the Python Package Index
• Almost 9,000 visitors to the website with over a 50% return rate

Team Leadership

Shortly prior to 2010 I moved into the role of development team lead and 2010 provided me with a wealth of new experience in this regard. Being responsible for a development team producing rock solid work has always been an aspiration of mine and it was incredibly fulfilling to have that come to fruition. It’s been a fairly painless experience due to working with some of the best developers I’ve met in over a decade, and it’s been nothing short of amazing.

Tooling

I added a ton of new software tools to my arsenal, being largely responsible for building out our development and deployment processes. I spent a lot of time getting up to speed with some amazing software such as South, Fabric, NGINX and gunicorn. I’ve recently been putting together a development and deployment guide that covers combining all of these and other parts of our stack which I’ll also publish in the near future, so stay tuned for that.

Django Dash

This was the first year I decided to put my Django skills to the test by entering Django Dash - the 48 hour hackathon where teams of 3 build a Django project from scratch. Teaming up with Josh de Blank and Andrew Fisher, the final result saw us launch Rate My Flight which earned us 8th place out of 40 teams internationally. It was a ton of fun getting to interact with some of the wider Django community and I even picked up a few new Django tricks along the way.

So that was 2010 - what a blast! I can’t imagine what 2011 will bring if it involves as much change and initiative as the previous year has. Bring it on!

Web development technology has come an incredibly long way over the last decade. Unfortunately in local markets like Australia where I reside, the tech sector seems to languish years behind the United States and Europe. We saw this happen with broadband adoption at the turn of the century where most Australians were still on dial-up for years after the rest of the world enjoyed widely available cable and ADSL.

Fast forward to 2010; and the same situation has occurred with web development. Anyone who follows the startup scene focused around San Francisco and NYC will be familiar with the most popular development technologies available - Ruby, Python, Scala and many more. Contrast this to the Australian space where the majority of development houses have been sitting for many years on technologies such as ASP.NET and PHP, which reached their peak in popularity over half a decade ago.

Why are the majority of shops in markets like mine so complacent when it comes to their technology stacks? Is it CTOs that haven’t written a line of code in years and their fear of the unknown? Is it a lack of willingness to invest in keeping their developers’ skill-sets up to date and marketable? On these things I can only speculate. What I can speak about knowledgeably however are some of the reasons these latest technologies far outshine their predecessors, and if only a single technology manager reads this post and decides to act on it then it was well worth the time spent writing it.

We chose Django several years ago so that’s what I’ll be making reference to in the following comparisons, however you could just as easily swap it out with Ruby on Rails or any other modern platform and the points would be more or less equivalent.

Efficiency

I recently came across this fantastic quote:

“IDEs: a form of automation needed when the environment in question erects artificial barriers.”

Have you ever tried writing C# or Java in a plain text editor? It is an exercise in futility as these languages sport incredibly verbose syntax and deeply nested libraries which require specialised tools simply to write code. What about the developer that needs to work with all of these different technologies each day, can they be expected to be experts in several different IDEs and switch between them freely? If only these different programming environments could be used from the same editor - what a joy it would be as a practising polyglot!

On the plus side languages like C# and Java are relatively clean and consistent when compared to abominations such as PHP, which truly is a disorganised mess - functions named using both verb_noun and noun_verb, lots of similar functions with no apparent naming convention (Eg: sort(), arsort(), asort(), ksort(), natsort(), natcasesort(), rsort(), usort(), array_multisort(), uksort()) and a weak type system that can lead to bugs which are difficult to discover.

These languages stand in great contrast to modern dynamic languages such as Python and Ruby. Ask any Python or Ruby developer which IDE they use and the majority of them will tell you they don’t need one. These languages are terse, use flat hierarchies in their libraries and are incredibly expressive.

Security

Aside from the recent Padding Oracle Exploit, ASP.NET has remained fairly secure over the years. Unfortunately the other parts of the stack, IIS and SQL Server, that it’s exclusively tied to have been the punching bag of the network security world throughout the last decade with viruses such as Code Red, SQL Slammer and more, leaving countless websites either defaced or knocked entirely offline. With a track record like this it truly is a wonder how anyone would knowingly choose to build public facing Internet services based on a Windows stack.

Inversely we have PHP that while typically deployed on a LAMP stack built with security in mind, the language itself makes writing secure applications an extremely disciplined task. One need look no further than the ongoing range of security issues that have plagued applications such as Wordpress over the years: SQL injection, cross site scripting (XSS), remote code execution - it’s like an all-you-can-eat smorgasbord of web application exploits.

Django in contrast runs on top of a secure LAMP stack and is designed from the ground up with security in mind. It’s protected by default against SQL injection, XSS and cross site request forgeries. A developer would actually have to make a concerted effort to create an exploitable Django application. Also like many open source projects a security issue in Django isn’t dealt with because a corporation deems it to be the most cost effective decision. In the very few and far between occasions when security issues have inevitably been discovered, turn-around time for resolving these has been within a 24 hour period - not weeks or even months as is often the case with corporate vendors that lack the agility and motivation to act responsibly.

Freedom

A common misconception about open source software is that it lacks the reliability of support that comes with choosing a commercial vendor. This is a short sighted view now plaguing many businesses. When Microsoft introduced .NET it made the skill-sets of thousands of VB developers redundant. What happens when Microsoft announces that .NET is to be deprecated in their next technical adventure? The problem here is that a public company with an obligation to generate as much profit as it can controls the technology path of billions of dollars of software. Sometimes it’s in their best financial interest to create fantastic technology, and sometimes it’s in their best financial interest to tear it all down again.

There is then the issue of acquisition. Companies like Microsoft and Oracle have a long and successful history of acquiring their competitors simply to discontinue their competing technology - let’s hope the vendor you’re in bed with isn’t too good.

Even Java, which for all intents and purposes is an open source technology has recently shown that it isn’t immune to the flaw of corporate ownership with Oracle suing Google over its use on Android phones. Will Android developers find that their time and effort invested in this platform will all be for naught?

Python and Django are both licensed under Permissive Free Software Licenses, which allow anyone to go ahead and do whatever the hell they like with them. They are owned by the Python Software Foundation and the Django Software Foundation respectively. These are non-profit bodies that for the most part exist to enforce the IP rights of each technology. These technologies are community driven with one goal in mind: to create best of breed technology. There is no financial motivation here and so we thus find ultimate reliability with this software stack being impervious to the risks described above - it cannot be made redundant by any financially driven corporate strategy as the licensing and foundations have been specifically put in place to prevent this from being possible. An interesting corollary to this is that these technologies go relatively unheard of without the backing of large corporations promoting them. Next time you’re the target of a technology sales pitch consider the high likelihood that you’re not looking at the best technology in its given application domain - best of breed doesn’t need to be sold at all, it sells itself.

Developers

ASP.NET, Java and PHP developers outnumber Python and Ruby developers by the hundreds if not thousands. This is a great selling point to technical decision makers - the ability to quickly and easily hire developers when the need arises is critical. But what of the quality of these developers? An interesting question to pose is why a developer who specialises in .NET or PHP chose their particular technology. Answer: because that’s what everyone else uses. You certainly won’t find a developer who has gone out and thoroughly investigated a broad range of different languages coming back and choosing .NET over everything else. Those who have done so have chosen their languages on its technical merits. These are the passionate developers with a love for their craft, not those who are merely in it for a paycheck and whose workmanship will reflect as much. Paul Graham referred to this in 2004 in his incredibly insightful essay The Python Paradox.

Mature Applications

The final point I’d like to cover is the maturity of applications developed on top of any given platform. This is where modern languages fall short as by definition they simply haven’t gained enough penetration for mature applications to have been developed yet. This is where there is opportunity. This is where the next generation of mature web applications will be built using web application frameworks like Django and Rails that are designed from the ground up for rapid customization over long software life-cycles while maintaining the original design integrity of your application - something only a very disciplined developer can maintain with something like PHP, which in almost all cases will eventually end up as spaghetti code.

Applications like Wordpress and Magento may work fine off the shelf for an end user, but what type of path have you created for your customer by implementing technology that gets closer and closer to its end of life the more it’s customised?

In conclusion - developers and CEOs, challenge your technical decision makers to overcome their complacency and invest in technology of the future. Push your employer, your peers and most importantly yourself forward. Invest your time in efficient, secure and unencumbered technology.

I’m happy to announce the first release of Cartridge - a Django shopping cart application I started working on back in 2009. The development path that Cartridge has taken has been a strange one. I stopped working on it throughout 2010 in order to get the ball rolling with a project called Mezzanine that I’ve written about previously. Many parts of Mezzanine actually originated in Cartridge and once development of Mezzanine was well under way it made the most sense for continued development of Cartridge to occur as a plug-in for Mezzanine which has now come to fruition.

Beyond creating a kick-ass shopping cart application for Django, the main design goal I originally had for Cartridge was to address some of the mistakes I felt existed in other offerings available both within and outside of the Django community. These areas I’ve aimed to address are as follows:

Performance

The Django ORM is a double-edged sword that while saving you a lot of time can also do a lot of damage when used without regard to the underlying SQL queries being performed. I’ve come across examples in templates where queries are being performed inside loops nested inside more loops resulting in abysmal performance. Fixing these problems wasn’t simply a case of refactoring template logic as these issues were core to the design of how prices and variations were modelled. The only solution was to throw a ton of fine-grained memcached usage at the problem, but this should be a optional approach to scalability - not a minimum requirement for keeping the site from falling offline. Cartridge has been designed with performance in mind from the start with a range of denormalised data structures providing O(n) performance as the number of products and categories grow.

Intuitive Interfaces

An end user should be able to use an admin system for the first time and discover an interface that is logical and intuitive. Having to go through a handful of screens to set up a single product requires users to mentally train themselves to remember a work-flow that isn’t entirely natural. If the former is achievable then the latter is definitely unacceptable. The number of forms and fields in an interface can be described as a conversation between the system and its user, and this conversation should be as quick and painless as possible. Cartridge provides single interfaces for creating products, categories, discounts and other types of shop data, with only applicable fields making up these forms.

Bloated Code

Having a system that implements every single feature that might ever be required in a shopping cart implementation certainly makes for an easy sell, however as this list grows these features become more obscure and less likely to be required in an average implementation. This can result in a convoluted code base that is very difficult to apply customizations to - an inevitable requirement given the unique nature of shopping cart implementations. Cartridge addresses this issue by implementing only the features typically required by all shopping cart implementations, leaving custom features up to the developer who will find that due to a tight feature list that the code base and data models remain very clean and easy to work with.

Aside from these features that distinguish it from other shopping cart applications, Cartridge comes with a standard set of features that you’d expect to find:

• Hierarchical categories (via Mezzanine pages)
• Easily configurable product options (colours, sizes, etc)
• Hooks for shipping calculations and payment gateway
• Sale pricing and discount codes
• Stock management
• Product popularity
• Thumbnail generation
• Built-in test suite
• Separation of presentation (no embedded markup)
• Smart categories (by price range, colour, etc)
• Configurable number of checkout steps

The live demo of Mezzanine now includes Cartridge so go ahead and try it out! If you’re interested in hacking on Cartridge then the source code is freely available under a BSD license at both github and bitbucket.

You’d be forgiven for reading the title of this post and thinking it’s about a crazy approach to project briefing that somehow mimics open source development - as interesting as that sounds, it isn’t the case and my motives are much more simplistic and sinister. What I’d like to do here is put a brief together for an open source project called Mezzanine. This brief isn’t specifically geared towards programmers so if you think this isn’t for you then please continue reading and let me prove you wrong.

What is Mezzanine?

Anyone who follows my updates will know it’s an open source CMS framework I’ve been working on over the last couple of months. It now has a concrete feature set having come remarkably far in a very short amount of time. This might lead you to believe an entire team of people have been working on it but in fact it’s mostly been myself alone - it’s thanks to the incredibly rapid development that using Django brings you that so much has been done so quickly. For those readers who aren’t familiar with it please go ahead and check out the overview in the documentation, play around with the live demo and have a read of my previous article that talks about why I started Mezzanine and what I hope to achieve.

Why would I want to help?

Perhaps you’re an end user of a poorly designed CMS and you’ve often wished you could do something about it. Perhaps you’re a developer that’s had the unfortunate experience of trying to extend a seemingly user-friendly CMS that’s built using archaic technology, and wished you could be working with something that’s much more cutting edge and elegantly designed. Perhaps you’re someone who “gets” open source at a deeper level but always felt as someone who isn’t a coder that you couldn’t contribute. Perhaps you’re in business development and you’re tired of trying to sell “enterprise” crap with completely absurd price tags. If you have anything to do with web development then there’s something in this for you.

What do I get out of it?

As much as you put in of course. The experience of contributing to open source software on paper can often be a competitive advantage over other candidates for a job interview or even development contracts for your business. There’s also the chance of notoriety - imagine being responsible for the user interface or branding of the next Wordpress. Imagine your staff are core contributors to one of the web’s leading development tools. Again the success of the project will only match its contributions so it’s ultimately up to you.

What can I do to help?

A common misconception about open source software is that it’s something that only coders can participate in. Unfortunately the result of this is that the majority of open source software ends up being only contributed to by coders and is incredibly lacking in a variety of areas. I’m talking about visual branding, copy-writing, UI development - all these areas that fall outside of coding but are equally crucial in successfully shipping a professional piece of software. Mezzanine has now reached a point where it can only continue to move forward at a consistent pace by bringing in these skills that I don’t specialise in. So without further ado, here are the specific roles I think need filling and what the focus of each would be.

Graphic Designer

The entire project is desperately in need of some visual love. At the simplest level it could really use some basic branding such as a logo and “powered by” buttons. Then there’s the Mezzanine website, documentation and default site that are all currently quite spartan looking.

Interface Developer

So far the template mark-up for the default site is as minimal as can be. While this is intentional to some extent in order to best serve those that would come along and customise it for their projects, I think this idea could be improved upon with a greater level of modularity. I’m also keen to introduce a CSS framework like Blueprint into the default site. Once that’s all in order then I’d like to address what theming would look like. Is this simply a matter of packaging up copies of the templates as separate themes? A great milestone for Mezzanine would be to have a handful of built-in themes created, as well as documenting the process for creating your own.

UX Designer

I’ve introduced a handful of user interface elements into Mezzanine that could definitely use some ironing out from a usability and accessibility perspective. The main contender is the navigation tree in the admin that’s used for managing the hierarchy of the entire site as well as being the entry point for accessing most of the content management. There’s the dashboard interface for the admin area which is in a very early stage. There’s the overall layout for both the project’s own site and the default site. Lastly and of great importance, there’s the entire system for in-line editing which is featured in the default site - making this feature as user-friendly as possible is critical.

Technical Writer

Mezzanine currently has a good start on documentation but at the moment it’s mostly focused on developers. I’d eventually like to have a lot more material aimed at both end users of Mezzanine as well as marketing material geared towards business decision makers.

Product Evangelist

This is probably the easiest task of all. We simply need the word to be spread. Learn about Mezzanine and use whatever medium you like to let the world know how great it is, be it Twitter, mailing lists or blog posts.

This list isn’t entirely complete and some of the tasks certainly overlap. If you think you fit the bill or know anyone else who would get a kick out of working on Mezzanine then there’s no time like the present to get started. There aren’t any obligations with this so contributions of any size are welcome. If you’d like to get involved but don’t know where to start just post a message to the mailing list and let’s talk!

Ask any developer that has put together a Django admin interface and they’ll tell you that it’s an amazing piece of technology that allows you to whip up an admin system for your web application in a number of minutes rather than days. Unfortunately this power can be a double-edged sword as without enough time and thought up front, a developer can easily end up creating an interface that’s almost impossible for its intended audience to work with.

I’ve seen this issue manifest itself in two ways. The first is the bare-bones case where the options available for creating the admin interface are simply left out, resulting in a spartan admin that does little to guide the user on how to use it. The second could be described as the opposite end of the scale where there is actually far too much going on in the admin interface at the cost of simplicity and intuitiveness. This can often be the result of gluing together a range of resuable apps that each have their own approach to providing admin interfaces with the end result looking like a Rube Goldberg machine. These scenarios are bad for customers and bad for Django. When end-users think of Django, they think of the admin interface - that’s what Django is to them so it’s critical to get this component right.

I recently had Wordpress suggested to me as a solution to this problem and it’s easy to see why. The Wordpress install base alone speaks huge volumes while its admin interface is incredibly user- friendly. It also benefits from not requiring technical expertise to get a simple website with pages and a blog up and running. However I felt this idea overlooked the underlying issue of poorly configured Django admin interfaces, while taking a step backwards by investing in PHP - a relatively inelegant technology with a very limited application scope.

My solution to the problem was to tackle the underlying issue more directly by creating a Django application which I’ve called Mezzanine. The approach I’ve taken is to have functionality on par with Wordpress that can be used as a starting point when developing basic websites. This meant putting a lot of thought into the admin options used, as well as including a custom version of the django- grappelli admin skin to come up with a modern looking and intuitive admin interface. The other key approach I’ve taken is to include as much functionality as possible directly in the application itself for the sake of a consistent and lightweight code base that can easily be hacked on. It’s worth noting that this is in total contrast to other Django website applications such as Mingus and Pinax, and that this difference really comes down to a question of scope. Pinax for example is capable of a much wider range of functionality than what I’m aiming for with Mezzanine out of the box which is to cater for basic websites with the following features:

• Hierarchical page navigation
• Save as draft and preview on site
• Drag-n-drop page ordering
• WYSIWYG editing
• SEO friendly URLs and meta data
• Mobile device detection and templates
• Blogging engine
• Tagging
• Custom templates per page or blog post
• Gravatar integration
• Google Analytics integration
• Twitter feed integration
• bit.ly integration
• Sharing via Facebook or Twitter
• Built-in threaded comments, or:
• Disqus integration

The Mezzanine admin dashboard

I’ve open sourced the initial version of Mezzanine with a BSD license on both github and bitbucket - it still has a long way to go so jump right in and fork away.

Like a lot of Django shops our software stack consists of two layers up front: a public facing web/proxy server and an application server sitting behind it. For a long time we’ve enjoyed success using nginx and Apache to fill these roles respectively, but as an application server the 800 pound gorilla that is Apache can really be overkill, which over time we’ve found can have quite a cost around lack of granular control. So we recently decided to try out the up and coming gunicorn which is currently gaining in popularity throughout the Django community and so far it’s been very smooth.

One of the interesting features it provides is the ability to handle various kill signals which map to functions such as adding and removing worker processes as well as reloading the master process, all on the fly without dropping a single client connection. So after a brief honeymoon period I then came up with the following list of questions that mightn’t be apparent when serving a single application, but really come into play when serving dozens of applications this way on a single server:

• How can we deal with the signals interface without knowledge of process IDs?
• How can we gain visibility around the ports being used?
• How can we gain visibility around the number of worker processes being used?
• How can we gain visibility around the amount of memory being used per application?

All of these can be answered with a small amount of command-line-fu, however I wanted this process to be ridiculously easy for our entire team. For quite some time I’ve wanted to put together a console application using the curses library so a simple management console for gunicorn seemed like the perfect opportunity to do so and as such, gunicorn-console was born.

Firing up a few gunicorn instances with varying parameters
gunicorn-console gives you the following interface in all its 8bit glory

If you’re hosting multiple applications served up via gunicorn then gunicorn- console should make managing them easier. I’ve released it with a BSD license on both github and bitbucket using the amazing hg-git extension, so go ahead and make it better!

Update, May 30: I ended this post with a request for others to contribute and after only a day someone already has. Adam Vandenberg went ahead and forked the project with some patches to get it running on OSX, so a big thanks goes to him.

A project of mine contains a number of third-party apps that are development related and potentially not available on every machine the project will run on. My general approach to dealing with these was to try and import the app in my settings module and if successful, add it to the INSTALLED_APPS setting. However as the number of these apps grew it became a wart within the settings module so I put together this snippet for managing them.

We first create a sequence of dictionaries, each containing information about an installed app such as the module to try and import, an extra potential condition for checking and then the sequences of names to add to INSTALLED_APPS, MIDDLEWARE_CLASSES and TEMPLATE_CONTEXT_PROCESSORS. Let’s start with the settings for optionally including the apps django- command-extensions, django-debug-toolbar and south.

# Define any settings specific to each of the optional apps.
import sys
USE_SOUTH = not (len(sys.argv) > 1 and sys.argv[1] == "test")
DEBUG_TOOLBAR_CONFIG = {"INTERCEPT_REDIRECTS": False}

# Sequence for each optional app as a dict containing info about the app.
OPTIONAL_APPS = (
    {"import": "django_extensions", "apps": ("django_extensions",)},
    {"import": "debug_toolbar", "apps": ("debug_toolbar",), 
        "middleware": ("debug_toolbar.middleware.DebugToolbarMiddleware",)},
    {"import": "south", "apps": ("south",), "condition": USE_SOUTH},
)

Next we simply iterate through the sequence of optional apps and set them up.

# Set up each optional app if available.
for app in OPTIONAL_APPS:
    if app.get("condition", True):
        try:
            __import__(app["import"])
        except ImportError:
            pass
        else:
            INSTALLED_APPS += app.get("apps", ())
            MIDDLEWARE_CLASSES += app.get("middleware", ())

Anyone who has programmed in Python for a considerable length of time will at least have some passing familiarity with PEP 8 - the document that goes into an incredible level of detail in dictating precisely how code should be written. While its primary goal is to ensure that Python code is written in a consistent fashion throughout the community, therefore making it as easy as possible to read, it also provides one of many aspects that makes programming in Python an incredibly efficient process - it negates the need for a lot of decision making around any of the choices one might come across that are already covered in PEP 8.

I’ve recently been spending a lot of time writing technical documentation. While it’s been interesting doing something different for a change, the perfectionist in me is constantly frustrated with finding myself using inconsistent language across different sections when faced with the exact same context, for example:

• contains / has / includes / provides
• discussed / referred to / described / mentioned
• above / earlier / previously
• below / later / next

Are these types of ambiguities in technical writing something that professional editors typically deal with? What I’d love to see is something like PEP 8 for technical documentation.

Earlier this week I had the pleasure of removing my final Windows install after wiping my machine at work and installing Ubuntu on it. It was during the late 90s that I first tried out Linux after getting my hands on a Redhat 6.1 CD from the cover of a magazine I’d bought. I didn’t keep it installed for very long and after a few more tries over the years with Mandrake (now Mandrivia) and Damn Small Linux, it wasn’t until 2005 when I installed Slackware 10.2 as my primary operating system at home and really cut my teeth on it in order to test how cross-platform my Python projects were. It was a great experience learning about all the various sub-systems, compiling software and libraries from source, embracing the command line and modifying some of the internal scripts to get things working the way I wanted.

Fast forward to 2010 and in my workplace the migration from a Microsoft development shop to a Linux/Python shop after several years is finally complete, paving the way for this latest install. I did experience a couple of hiccups that hadn’t happened before. Firstly I have dual wide-screen monitors at work and I rotate one of them 90 degrees in order to maximise the amount of visible code on my screen. The display properties in Ubuntu only gave me the ability to flip the display 180 degrees which seemed quite odd so I then tried to manually rotate the display with the xrandr command which reported my overall virtual screen size as being too small for the rotation. I resolved this with the update below to my x.org configuration to use a virtual screen size large enough to handle the rotation while including the second monitor.

Section "Screen"
    Identifier "Configured Screen Device"
    Device "Configured Video Device"
    SubSection "Display"
        Virtual 2880 1440
    EndSubSection
EndSection

The second issue was more a lack of foresight on my part than a problem with the new install itself. After a vanilla install of any modern operating system you’ll undoubtedly be required to download a series of updates that have occurred since the version you’ve installed was initially released. The difference with most Linux distributions is that almost all of your software is managed in this way from installing to updating, it all goes through the same service known as a package manager - one of the many things with Linux that once you’re used to using you won’t know how you ever worked without it. So away I went with the initial round of updates which left the package manager busy for several hours, during which time some issues arose with a project that immediately required my attention. Unfortunately I needed to install a handful of libraries to get up and running and with the package manager busy I was left in a real bind. Fortunately I was able to use one of our test servers remotely to resolve the issue but the lesson learnt here is that for a new development system it’s best to leave the initial system update until after your development environment is completely set up.

My current project has the common requirement of storing and rendering a hierarchical tree of categories. This project is geared towards potentially junior developers with the expectation of it being hacked at every time it’s used - a set of scaffolding where simplicity isn’t just a loose goal but a fundamental requirement.

Two popular approaches to the hierarchical tree are the Adjacency List (AL) and Modified Preorder Tree Traversal (MPTT) models. The advantage of AL is that it only stores the exact data required for representing the tree while MPTT stores extraneous data for assisting in traversing the tree in an optimal fashion. The simplicity of the AL model makes it much better suited to the requirements I mentioned, however the problem with AL is the recursive nature in which you traverse it.

def show_branch(parent, depth=0):
    # iterating the entire tree for each branch gives quadratic performance
    for node in nodes:
        if node.parent == parent:
            print (" " * depth) + node
            show_branch(node, depth + 1)

Worst case here is O(n²) performance but thanks to Python’s lightning fast hash table implementation we can create a copy of the tree as a dictionary of branches giving us O(n) overall performance when traversing the entire tree.

# copy the tree into a dict of branches
branches = {}
for node in nodes:
    parent = node.parent
    if parent not in branches:
        branches[parent] = []
    branches[parent].append(node)

def show_branch(parent, depth=0):
    # iterating only the nodes in the branch gives linear performance
    for node in branches[parent]:
        print (" " * depth) + node
        if node in branches:
            show_branch(node, depth + 1)

When rendering the entire tree, using this technique will greatly increase performance as the tree grows in size. Be aware though that if your application only ever deals with a single branch in any given view, this technique won’t perform as well as directly querying the database for the nodes in a single branch.

Technology shows style of things to come

The Sydney Morning Herald has an article that talks about some of the work I did for Sportsgirl including their community portal running on Django and a Facebook app written in C# - nice to get recognised!

Many major Linux distributions such as Ubuntu ship with gedit as the default text editor. It has all the standard features you’d expect in an editor such as syntax highlighting, a tabbed interface and the ability to integrate external tools. Most importantly it’s highly extensible with the ability to create plug- ins for it written in Python or C.

One great plug-in that’s been written is gedit-ftp- browser, an FTP client that embeds itself into the editor giving you the ability to remotely edit files over FTP. I’ve just been accepted as a contributor to the project which I’m really excited about. I’ve implemented a “Save As” feature allowing the user to create and upload new files over FTP. Next up I’ll be working on the ability to manage multiple FTP servers via profiles.

Lately I’ve noticed people posting various different takes on making the default Django settings a lot more dynamic. The development and deployment requirements for the projects I work on tend to be far from straight-forward and over time I’ve come up with my own approach to Django settings, so here it is.

The simplest approach typically involves importing all the names from a custom settings module at the end of the project’s standard settings module, providing the ability to override settings on a per machine basis.

try:
    from local_settings import *
except ImportError:
    pass

This still requires modifying local_settings.py on a per machine basis. Another approach that builds on this is to import a custom settings module from a host_settings package that has a unique name derived from the current machine the site is running on - this gives the advantage of being able to specify custom settings per machine and have each of these settings modules reside in the version control system, without the same file having to be modified on a per machine basis.

from socket import gethostname
try:
    exec ("from host_settings.%s import *" % 
        gethostname().replace("-", "_").replace(".", "_"))
except ImportError:
    pass

This simple version of the host_settings approach I’ve seen attempts to deal with the differences between a valid hostname and a valid module name with the two calls to replace, but ignores the fact a hostname could begin with a number which would be an invalid module name. Other versions of this approach handle this correctly and involve calling the __import__ built-in, iterating over and updating each name in the settings module individually, but once we look at some further requirements below and how to deal with them we’ll find this isn’t necessary.

Let’s take a step back for a moment and talk about some of the requirements I mentioned before. Where I work a project can end up deployed in a dozen different locations - a handful of developer machines and various different servers managing the project life cycle. Due to a variety of non-technical reasons it’s often required that various versions of a project run side by side in the same location, so with a project called project_x we end up with project_x_feature_a and project_x_feature_b sitting in the same location - all of a sudden all of our references to project_x are broken. So we end up taking the approach in our code that the actual name of a project’s directory is a moving target and should never be referenced - we never import from package_x and anything in our settings module that would typically reference this we set dynamically.

import os
project_path = os.path.dirname(os.path.abspath(__file__))
project_dir = project_path.split(os.sep)[-1]
MEDIA_URL = "/site_media/"
MEDIA_ROOT = os.path.join(project_path, MEDIA_URL.strip("/"))
TEMPLATE_DIRS = (os.path.join(project_path, "templates"),)
ADMIN_MEDIA_PREFIX = "/media/"
ROOT_URLCONF = "%s.urls" % project_dir

So that removes any hard-coded reference to the project’s directory name, however we sometimes have to go as far as having host specific settings that vary across these different project versions residing on the same machine, such as a different database for example. The ultimate goal here is to not have any files in the project’s version control repository that are manually edited for a specific instance of the project. So using the host_settings approach from earlier on, we continue on from the code above by using the project_dir variable when referencing the machine specific host_settings module so that each of the host_settings modules are named not only after the machine they exist for, but the project directory as well.

from socket import gethostname
host_settings_module = "%s_%s" % (project_dir, 
    gethostname().replace("-", "_").replace(".", "_").lower())
host_settings_path = os.path.join(project_path, "host_settings", 
    "%s.py" % host_settings_module)
if not os.path.exists(host_settings_path):
    try:
        f = open(host_settings_path, "w")
        f.close()
    except IOError:
        print "couldnt create host_settings module: %s " % host_settings_path
try:
    exec "from host_settings.%s import *" % host_settings_module
except ImportError:
    pass
TEMPLATE_DEBUG = DEBUG

As an added bonus, we try to create the host_settings module if it’s missing and warn if we’re unable to create it.

After living the last few years without a mobile phone it finally became a problem for me so recently I decided to get a new one. I only needed something simple for receiving calls, not making them so any kind of plan or contract was out of the question since I could spend a few bucks on pre-paid and theoretically not pay anything after that. I didn’t consider any of the latest smart-phones either like the HTC or iPhone (I wouldn’t buy a crippled device anyway) since they’re so ridiculously priced compared to lower end models that aren’t even a year old.

I ended up getting a Nokia E63 that has a full qwerty keyboard and wireless LAN access which were the main selling points for me. It cost $299 when it was being sold for over $500 at other places around town so it felt like a smart purchase. Apart from those features I looked at, I really knew nothing about the phone and Symbian OS which powers it and after taking a closer look at the software available for it I’ve been really surprised.

Google has created a handful of Symbian apps for things like gmail, reader, youtube and maps which all work great. The other day I found an app called fring which is very similar to Pidgin in that it wraps up all the various IM clients into one, even including Skype with voice working! So it has essentially turned my phone into a Skype handset which is amazing. I’ve also found that there’s a version of putty for Symbian so I can actually SSH onto any of our Linux servers or desktops and access the shell from my phone!

The most incredible thing I’ve found for Symbian however has been a project called PyS60 - Nokia has actually ported the Python run-time to the Symbian OS. I was amazed once I had this installed and was sitting there typing out Python code into an interactive console directly on my phone. The standard library is even included and it’s very interesting - certain pieces aren’t fully ported but it comes with a handful of modules specifically for PyS60 which handle things like locating wireless networks and working with the user interface. It even includes OpenGL bindings which is unbelievable - that’s right, you can develop 3D games in Python for your phone.

I’m well into developing my first app which is a small RPC server for the phone controlled by a wxWidgets client. The idea is to be able to traverse the phone’s file system and create, edit and execute Python apps on the phone from a remote machine. The SimpleXMLRPCServer module isn’t included with PyS60 and broke when I tried to copy it onto the phone and import it manually. I’ve since been able to patch it and get it working which I’ve submitted to Nokia, hopefully they’ll include it in their next release.