The Lazy Guide to Websockets

Jan 24th, 2013

by Robert Myers

robert@julython.org

Story

Simple Django View

Views.py

def projects(request, username):
    user = get_object_or_404(User, username=username)
    projects = user.projects.all()
    commits = Commit.objects.filter(user=user)

    return render_to_response('projects.html', {
            'projects': projects,
            'profile': user,
            'commits': commits,
        },
        context_instance=RequestContext(request))

Problems

Solution

Tastypie

Example Resource

class CommitResource(ModelResource):
    user = fields.ForeignKey(UserResource, 'user',
                             blank=True, null=True)
    project = fields.ForeignKey(ProjectResource, 'project',
                                blank=True, null=True)

    class Meta:
        queryset = Commit.objects.all()
        queryset.select_related('user', 'project')
        allowed_methods = ['get']
        filtering = {
            'user': ALL_WITH_RELATIONS,
            'project': ALL_WITH_RELATIONS,
            'timestamp': ['exact', 'range', 'gt', 'lt'],
        }

Example URLs

from tastypie.api import Api
from july import api
v1_api = Api(api_name='v1')
v1_api.register(api.CommitResource())

urlpatterns += patterns('',
    url(r'^api/', include(v1_api.urls)),
)

Example Output

{
    meta: {
        limit: 1,
        next: "/api/v1/commit/?format=json&limit=1&offset=3",
        offset: 2,
        previous: "/api/v1/commit/?format=json&limit=1&offset=1",
        total_count: 11
    },
    objects: [
        {
            created_on: "2013-01-18T06:26:39.349473",
            hash: "61ef4c52d9731b8f03240961834f3b1d4fa6cd53",
            message: "Adding a fabric command to load test commits",
            other: "Fields here"
        }
    ]
}

Backbone and Knockout

Backbone Collection

JULY.CommitCollection = Backbone.Collection.extend({
  model: JULY.Commit,
  url: function() {return '/api/v1/commit/?' + this.params();},
  initialize: function(data, options) {
    this.limit = options.limit || 20;
    this.offset = options.offset || 0;
  },
  params: function() {
    return jQuery.param({limit: this.limit, offset: this.offset});
  },
  parse: function(resp) {
    this.total = resp.meta.total_count;
    this.offset = resp.meta.offset + this.limit;
    return resp.objects;
  }
});

Knockback

Glue between Backbone and Knockout.

JULY.CommitsView = JULY.ViewModel.extend({
  initialize: function(options) {
    this.c = new JULY.CommitCollection(null, options);
    this.c.fetch({add: true});
    this.commits = kb.collectionObservable(this.c);
  },
  hasMore: function() {
    return this.commits.collection().hasMore;
  },
  fetch: function(){
    if (this.hasMore()) {
      this.commits.collection().fetch({add:true});
    }
  }
});

Knockout In Template

<div data-bind="foreach: commits">
  <div class="media">
    <a class="thumbnail pull-left" data-bind="attr:{href: link}">
      <img data-bind="attr: {src: picture_url, alt: username}" />
    </a>
    <div class="media-body">
    <h4 class="media-heading" data-bind="timeago: timestamp"></h4>
    <strong data-bind="text: message"></strong>
    <p class="hash" data-bind="visible: username">
      <a data-bind="text: username, attr: {href: link }"></a> &mdash;
      <a data-bind="visible: url, attr:{href:url }">
        <span data-bind="text: hash().substring(0, 8)"></span>
      </a>
    </p>
    <p class="hash" data-bind="visible: !username()">
      <span data-bind="visible: !url(), text: hash().substring(0, 8)"></span>
    </p>
      </div>
  </div>
</div>

Knockout Continued

Connect the page up to the collection:

<script type="text/javascript">
  ko.applyBindings(new JULY.CommitsView());
</script>

Realtime

Time to get real!

Choices

Nginx Push Stream Module

http://www.nginxpushstream.com/

Benefits

Hard Parts

You have to build it yourself:

# clone the project
git clone http://github.com/wandenberg/nginx-push-stream-module.git
PUSH_PATH=$PWD/nginx-push-stream-module

wget http://nginx.org/download/nginx-1.2.6.tar.gz

# unpack, configure and build
tar xzvf nginx-1.2.6.tar.gz
cd nginx-1.2.6
./configure --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf \
--add-module=../nginx-push-stream-module
make

Building cont.

# install and finish
sudo make install

# check
sudo nginx -v
> nginx version: nginx/1.2.6

# test configuration
sudo nginx -c $PUSH_PATH/misc/nginx.conf -t
> the configuration file $PUSH_PATH/misc/nginx.conf syntax is ok
> configuration file $PUSH_PATH/misc/nginx.conf test is successful

# run
sudo nginx -c $PUSH_PATH/misc/nginx.conf

Configuration

# TOP LEVEL MESSAGING CONFIGURATION
push_stream_ping_message_text '""';

## SERVER CONFIGURATION ##
server {
    push_stream_ping_message_interval 10s;
    push_stream_content_type "application/json; charset=utf-8";
    # simplified Template
    push_stream_message_template "{\"text\":~text~\"\}";

    # LOCATIONS HERE
}

Web Sockets

# Subscription for Websockets via nginx-push-stream-module
location ~ /events/ws/(.*) {
    push_stream_websocket;
    push_stream_websocket_allow_publish off;
    set $push_stream_channels_path $1;
}

EventSource

# Subscription for EventSource with nginx-push-stream-module
location ~ /events/ev/(.*) {
    push_stream_subscriber;
    push_stream_eventsource_support on;
    set $push_stream_channels_path $1;
}

Long Polling

# Subscription for messaging system with nginx-push-stream-module
location ~ /events/lp/(.*) {
    push_stream_subscriber long-polling;
    set $push_stream_channels_path $1;
}

Stream

# iFrame streaming for messaging system with nginx-push-stream-module
location ~ /events/sub/(.*) {
    push_stream_subscriber;
    set $push_stream_channels_path $1;
}

Stats

See how many connections are open and how many messages there are:

# Messaging Channel Stats
location ~ /events/stats/(.*) {
    push_stream_channels_statistics;
    set $push_stream_channel_id $1;
}

Publishing Messages

# Publish interface for messaging system
location ~ /events/pub/(.*) {
    # only allow on the local server (readonly clients)
    allow 127.0.0.1;
    deny all;
    push_stream_publisher admin;
    set $push_stream_channel_id $1;
}

Example Usage

Subscribe:

curl -v 'http://localhost/events/sub/channel'

Publish:

curl -X POST 'http://localhost/events/pub/channel' -d 'Hello!'

Creating Messages from Django

POST a message to the channel(s) you want.

hint: just use Requests

Re-use Tastypie Resources

url = 'http://localhost/events/pub/'
resource = CommitResource()
# Build a bundle from the new object
bundle = resource.build_bundle(obj=commit)
# Run full_dehydrate to run all custom dehydrate methods
dehydrated = resource.full_dehydrate(bundle)
serialized = resource.serialize(
    None, dehydrated, format='application/json')
# Make messages
requests.post(url + 'user-%s' % commit.user.id, serialized)
requests.post(url + 'project-%s' % commit.project.id, serialized)
requests.post(url + 'global', serialized)

Tastypie allready serializes your content to JSON, so just use that. The above code is run after a new commit object is created.

Change Client Code

JULY.CommitCollection = Backbone.Collection.extend({
  initialize: function(data, options) {
    // A reference to this collection
    JULY.collection = this;

    this._pushStream = new PushStream({
      host: "www.julython.org",
      modes: "websocket",
      urlPrefixWebsocket: "/events/ws"
    });


    // Continued...
  }
});

Continued

// on message callback
this._pushStream.onmessage = function(text) {
  JULY.collection.unshift(text);
};

// handle errors/open/close
this._pushStream.onstatuschange = function(state) {
  console.log("-- PushStream state changed: " + state);
};

// Subscribe to the channel 'project-(id)', 'user-(id)'
this._pushStream.addChannel('global');
this._pushStream.connect();

Add Push Stream Javascript Helper

<script type="text/javascript" src="pushstream.js"></script>
<script type="text/javascript" src="commits.js"></script>
<script type="text/javascript">
  ko.applyBindings(new JULY.CommitsView());
</script>

'Live' Live Demo

http://www.julython.org/live/