Architecture

The new site uses a couple of products from the cloud services at Rackspace. First there is a Cloud Load Balancer in front of two 512 MB Cloud Servers. The two web nodes are running a nginx proxy in front of gunicorn. The web nodes talk to a 512 MB MySQL Cloud Database.

Appengine has a great tool for monitoring performance of your web apps. The dashboard has a number of graphs as well to check on the health of your web app. This is hard to give up, which is why I turned to New Relic. With New Relic you get much more data; I do not regret the switch.

During J(an)ulython I'll be working on my project cannula, which is a deployment tool for websites. Using cannula I can deploy the site like so:

$ git push cannula master

Living on the Edge

Even though Django 1.5 is still in beta, it is worth running to get the new configurable User model. This made the transition super easy from webapp2 user models on Appengine to a custom class to interact with Django Social Auth properly. Webapp2 stores the auth_ids in the format 'provider:uid' in a list property on the User class. It also provides methods to add new auth ids, etc. Now with Django 1.5 you can alter the User Class to add custom fields or methods. Here is the User model from Julthon:

from django.db import models
from django.contrib.auth.models import AbstractUser
from social_auth.models import UserSocialAuth

class User(AbstractUser):
    description = models.TextField(blank=True)
    url = models.URLField(blank=True, null=True)
    # more custom fields here

    def add_auth_id(self, auth_str):
        """
        Example::
            user = User.objects.get(username='foo')
            user.add_auth_id('email:foo@example.com')
        """
        provider, uid = auth_str.split(':')
        UserSocialAuth.create_social_auth(self, uid, provider)

    @classmethod
    def get_by_auth_id(cls, auth_str):
        """
        Example::
            user = User.get_by_auth_id('twitter:julython')
        """
        provider, uid = auth_str.split(':')
        sa = UserSocialAuth.get_social_auth(provider, uid)
        if sa is None:
            return None
        return sa.user

    @property
    def auth_ids(self):
        auths = self.social_auth.all()
        return [':'.join([a.provider, a.uid]) for a in auths]

Now with very little change to the api endpoint for commits it is easy to associate a commit with a user by email address.

class Commit(models.Model):
    user = models.ForeignKey(settings.AUTH_USER_MODEL, blank=True, null=True)
    hash = models.CharField(max_length=255, unique=True)
    author = models.CharField(max_length=255, blank=True)
    name = models.CharField(max_length=255, blank=True)
    email = models.CharField(max_length=255, blank=True)
    message = models.CharField(max_length=2024, blank=True)

    @classmethod
    def user_model(cls):
        """No need to import our custom user model."""
        return cls._meta.get_field('user').rel.to

    @classmethod
    def create_by_email(cls, email, commits, project=None):
        """Create a commit by email address"""
        user = cls.user_model().get_by_auth_id('email:%s' % email)
                # create the commit here

You could accomplish everything just by interacting with Django Social Auth models. It just seems a little cleaner to use methods on the User model to get a user or add properties to it. The commit model in this case really should not care about the Social Auth models.

SQL, Oh How I Missed You

The Appengine Datastore is a NoSQL database which is really easy to work with. There is no configuration needed, and you don't have to create tables or alter your schema in order to make changes. My biggest complaints of the Datastore are the lack of a (fast and complete) count method and if you want to do a query there has to be an Index built for it. While there is more overhead associated with SQL databases, some things are much easier to do, which is one reason we decided to make the switch.

class Location(models.Model):
    slug = models.SlugField(primary_key=True)
    name = models.CharField(max_length=64, blank=False)
    total = models.IntegerField(default=0) # this field is never updated!

SELECT july_user.location_id AS slug,
    people_location.name AS name,
    SUM(game_player.points) AS total
    FROM game_player, july_user, people_location
    WHERE game_player.user_id = july_user.id
    AND july_user.location_id = people_location.slug
    AND game_player.game_id = %s
    GROUP BY july_user.location_id
    ORDER BY total DESC
    LIMIT 50;

>>> query = Location.objects.raw(LOCATION_SQL, [1])
>>> locations = [l for l in query]
>>> print locations
[<Location: Atlanta, GA>,
 <Location: Philadelphia, PA>,
 <Location: Boston, MA, USA>,
 <Location: Austin, TX>,
 ...

>>> atl = locations[0]
>>> print atl.name
Atlanta, GA
>>> print atl.total
1625

Wrap up

Moving away from a complete PAAS solution can be a little scary. Thankfully there are a number of projects that can help your transition. I plan on detailing other aspects of the site in this blog as well. Hit the comment section up if you have any questions!