Groovie

I’ve been doing a lot of comparisons and research into various Python web frameworks lately, mainly focused on MVC oriented frameworks. Some of them have templating languages they come with, some not so much. The thing I have started to notice is how similar the Controller in them all is starting to look. First though, lets see what they all can do when it comes to the Controller.

For comparison’s sake, I’m bringing out some examples using CherryPy, Myghty, Bricks, Aquarium, Ruby on Rails, and Django.

Please note I’m not including frameworks built using any of the ones mentioned as their Controller looks pretty much the same. I’m also not an expert in all these frameworks as I only have so much time in the day, so if I seem vague on something its because I don’t know for sure. Feel free to comment (or correct) and clear up any vagueness I express.

Dispatch and Responder Styles

All the frameworks I’ve mentioned implement a style of object publishing (sometimes referred to as dispatch-style) that maps to the URL coming in. If you have another name you want to call this, go for it, I haven’t seen anything consistent on what to call this stuff.

1. User creates a regular expression to explicitly map a URL to an object (Explicit Mapping)
   • (r'^polls/(?P<poll_id>\d+)/vote/$', 'myproject.apps.polls.views.polls.vote') (from Django tutorial)
2. Object is mapped out according to the URL (Implicit Mapping)
   • /admin/login/hello/—> admin/login.py/hello() (Myghty)
3. Programmatic rule set determines object that responds (Programmatic Mapping)
   • map.connect ’:controller/:action/:id’ (Rails)
   • m.connect(’:controller/:action/:id’) (Routes)

1. Create a class inheriting from a Controller object, methods of the class instance are called to respond (Class-method)
2. Create a callable object that responds (Callable)
3. Create a function thats directly called (Function)

Note that the last 2 are functionally equivalent for the most part, however a callable object has the typical advantages (or disadvantages) of persistence in some web frameworks, as such if a web framework only shows one of those styles I won’t mention the other. Curious what frameworks use what styles from the above two sets?

For this listing, I’ll refer to the mapping scenario as dispatch-style and the method for defining the response as the responder-style, also please note that some frameworks support multiple dispatch-styles and responder-styles.

• Dispatch-Style Mapping 1^:
  • Explicit, Implicit, Programmatic
• Responder-Styles:
  • Callable (Though it looks almost exactly like the Class-method style)

• Dispatch-Style Mapping:
  • Implicit
• Responder-Styles:
  • Class-method

• Dispatch-Style Mapping:
  • Explicit (Not with regexp’s though, each class is attached off the root to determine what URL leads to it)
• Responder-Styles:
  • Class-method (Must be explicitly exposed with @cpg.expose)

• Dispatch-Style Mapping:
  • Explicit
• Responder-Styles:
  • Function (Packaged in the same module if they’re related)

• Dispatch-Style Mapping ^1:
  • Explicit, Implicit, (Programmatic with Routes in upcoming 0.99)
• Responder-Styles:
  • Class-method, Callable, Function

• Dispatch-Style Mapping 1^:
  • Programmatic
• Responder-Styles:
  • Class-method

1^: User can create custom dispatchers to determine responder object

Who did it first?

If we throw in two more frameworks that utilize CherryPy, like Subway, and TurboGear, we can start to see that using a responder-style of Class-method is pretty popular. What’s even more interesting is that all the Class-method Controllers look remarkably similar, and I don’t think its because they’re copying each other.

The Aquarium project appears to be approximately 4 years old now, but from looking at the mailing list had little or no users until only 2-3 years ago (One poster from 2001 describes his aquarium plants turning brown…). From the sourceforge changelogs, the Controller stuff was added with 1.4 release in 2004. CherryPy has been around for 3+ years and I’m presuming it had the Controller stuff in it from the get-go as it seems a pretty core function of CherryPy. Rails has been out for a year or so, and was used in a form internally for BaseCamp for awhile predating that.

If you really want to trace back this style of Controller, you’d most likely find yourself looking at Java Python code (See Phillip J. Eby’s comment below). I’m really just trying to point out here, that even if one of these projects saw something they liked from the other, none of them were the originators. They all took a good concept, and implemented it differently, though they’ve all started to look remarkably similar.

A Controller Line-Up

So lets take a look at some of the examples on the pages of each respective website. Django is the only one that really stands out, as they’re not using a Class to contain the methods, rather they’re grouping them as plain functions inside a module.

Aquarium

class BaseController(Controller):

    def __call__(self, callNext, *args, **kargs):
        """Save a reference to myself in ctx.controller.""" 
        self._ctx.controller = self
        callNext(*args, **kargs)

    def doMessageAddAction(self):
        """Add a new message.
        ...does a bunch of stuff...

I mentioned that it was Callable but functioned in a way like a Class-method. This appears to be because Aquarium puts a context and maybe other initialization stuff per request in the __call__ bit, then has it actually calls a different method to respond.

Bricks

from bricks.controller.crud import CRUD
from bricks import user

class Test(CRUD): 
    # Override the default delete() method
    def delete(self, rowid):
        return CRUD.delete(self, rowid)
    delete.expose = user(level=1, app='app')

controller = Test('todo')

Bricks adds an interesting ability here by attaching function attributes to indicate what authorization is needed to execute the function called. Some of the other frameworks shown here do similar things through function decorators.

CherryPy

from cherrypy import cpg

class HelloWorld:
    def index(self):
        return "Hello world!" 
    index.exposed = True

cpg.root = HelloWorld()

Pretty handy I think, and it beats writing regular expressions (though doesn’t appear as flexible). One of the reasons I prefer the Programmatic dispatch-style. Moving on..

Django

from django.core.extensions import get_object_or_404, render_to_response
from django.models.polls import choices, polls
from django.utils.httpwrappers import HttpResponseRedirect

def vote(request, poll_id):
    p = get_object_or_404(polls, pk=poll_id)
    ... does a bunch more stuff to use Django-supplied functionality...

If you imagined the functions inside a class, instead of a module, it’d look rather similar to the others….

Myghty

class HelloWorld:
    def doit(self, m, **params):
        m.write("hello world!")

helloworld = HelloWorld()

At this point, the similarity between Bricks, CherryPy, and Myghty should be pretty obvious. Aquarium is close as well, while Django is only a wee farther from what’s appearing to be the norm.

Rails

class HelloController < ApplicationController
    def index
        return render :text => "hello world!" 
    end
end

Doesn’t look too different from some of our Python MVC frameworks…

What it All Means

I have no idea. Really, I just thought it was fascinating that these different frameworks started looking so similar. Some of them have been incredibly similar before others, but obviously these approaches catch on and find themselves adopted into other frameworks.

I do start to wonder at what point they’ll be similar enough that the barrier to switching web frameworks is narrowed down to only a few dozen lines of code to change here and there. Sure, going to/from Rails is going to be tough as that also means a language change, but for Python programmers tempted by a feature a different web framework offers I won’t be surprised to see more people hopping back and forth.

With regard to the Python web frameworks, none of them appear to agree on a template language. Several of them use Cheetah, some use Myghty’s template abilities, one uses Kid, and several of them have their very own template language.

It’s also interesting to consider whether or not having more possible dispatch/responder styles is beneficial or detrimental to the popularity of a specific framework. Rails creator David Hansson purposely includes only one style for each as he emphasizes with a concept known as convention over configuration. He provides defaults for pretty much everything, and following conventions makes it easier to write code as it enforces a consistent programming and naming style. I can’t help but notice that of the frameworks I mentioned above, the ones with less (apparent) options and better defaults tend to have more users (With the exception of Bricks as its still in alpha).

You might have noticed I designated Rails as being capable of user-programmed dispatching. While Rails uses its routes system by default, the thing people miss is that they’re just defaults. You can get in there and totally alter the way it works if you want, Ruby doesn’t completely close classes so you could just re-open them and override any aspect of its dispatch system you like.

I think this a valuable lesson that some of the more “verbose” examples above might want to consider. If there’s lines of code in a controller that you have to put in there every-time, find a way to make it the default case so that boiler-plate isn’t needed. It only needs to be a default, an advanced programmer will know how to override it should they need to. There’s obvious limitations to this since things work differently in Python, but there’s still Python ways to do it in a way that feels “right”.

For the Python Web Frameworks I missed

I know, I didn’t cover SkunkWeb, Spyce, Snaklets, Webware and more. Please feel free to add the relevant information regarding your framework in the comments. It’d be helpful if you could indicate what Dispatch-Styles and -Responder-Styles your framework supports.

If I failed to properly cover one of the above frameworks, bringing that to my attention is most appreciated.

The comment system supports the Textile format.

• Added prefix option
• Fixed Python 2.3 bug with thread-local singleton

But hey, its a small package to begin with so what the heck. Though its only 0.2 I’m rather pleased with it so far, its performance is great and quite reliable so I’m using it in production environments already.

If you haven’t been following my blog long, Routes is a feature-complete implementation of Rails routes system. I talk more about my reasons for re-implementing Routes in Python in an earlier post so I won’t repeat them all here.

It’s fairly unique in the Python world as it will do a route lookup search to turn a dictionary back into the URL (URL Generation) that will ensure the same values are created. This allows you to generate URL’s from inside your web pages and easily add new URL schemes without touching all your web pages.

The Routes package is aimed directly at integration with Python web frameworks that support the MVC style paradigm as it returns a controller and action value with the assumption your framework will know what to do with it.

Tired of writing big regexp’s to match URL’s to a class/method for dispatch in your webapp? Pester the framework creator to integrate Routes. :)

• Aquarium has a URL Connector that uses Routes. Not sure if its been added to the core or only exists as an add-on right now.
• Myghty will have Routes integration packaged with it in 0.99 which is on track to be released this week hopefully. It will also bring Python Paste support and integration which will bring a whole bunch of goodies to webapp developers and users. (Note that Python Paste requires Python 2.4)

I’ll be talking more about Python Paste and why you should care later this week. Any comments/suggestions on Routes are greatly appreciated.

On many occasions, its quite useful when developing web applications to have the webapp know whether its in a production/development/testing environment. Rails builds this into the framework and its rather easy to add this toggle throughout a Myghty webapp (or some other Python web framework) as well.

We’ll make use of an environment variable so that the webapp environment can be easily configured from lighttpd or Apache. This way just deploying the application under a different handler can toggle the web applications mode of operation. The other thing we’ll setup while we’re at it, is a variable to keep track of the absolute root of our web application. I’ve found it quite useful in many cases to be able to get at this information without hardcoding it in, this way its always accurate no matter where the program is.

There’s a few rather commands that’ll give you the information we’re looking for. It took me awhile to find this, so hopefully it’ll help someone else out there.

Getting our Absolute Location

webapp/
   templates/
   components/
   scripts/
      dispatch.fcgi

import os

# Set the prefix to our base path for the webapp
myloc = os.path.join(os.getcwd(), __file__)
prefix = os.path.normpath(myloc + '/../..')

In case you’re curious why this happens, __file__ will return the path of the file relative to the working environment its called from. So combining it with the full path name of the current working environment results in the complete absolute path of the module no matter what the current context or working directory.

The prefix assignment uses the normpath call to strip off the filename, and backup one directory to our webapp root. This leaves us with the absolute path to our webapp/ directory.

Setting and Using the Environment variable

import os

MYGHTY_ENV = os.environ.get('MYGHTY_ENV') or 'development'

fastcgi.server = ( 
   ".fcgi" => (
     "www" => (
        "min-procs" => 2,
        "max-procs" => 4,
        "socket" => "/tmp/webapp.socket",
        "bin-path" => "PATH/TO/webapp/scripts/dispatch.fcgi",
        "bin-environment" => ("MYGHTY_ENV" => "production" ),
        "idle-timeout" => 120
      )
    )
)

# Make sure you have mod_env loaded, this line assumed to be in the VirtualHost
# block of your config
SetEnv MYGHTY_ENV production

At this point, you might’ve noticed (if you’ve used Rails) how similar my Fast CGI setup with lighttpd looks when compared to the some of the Rails examples for a lighttpd + Fast CGI setup. This is intentional, as I’m adding a Routes dispatcher to Myghty so it makes sense to layout my web application in a similar directory hierarchy.

Anytime you need to toggle some behavior depending on your webapp’s runtime context, just import os and check it as I showed up above.

Please Note: Reading the last post would help greatly for this one, and this post does end up comparing aspects of Django with Rails in terms of making decisions rather than actual usage.

As I mentioned in my last post, Rails has been greatly helped by the decisions it makes for you. It decides the layout of your application directory, the directories you put stuff in, where you put what parts, etc. These decisions thankfully were done by someone familiar with good application design and had most likely even read Martin Fowler’s Refactoring to Patterns.

What results is a clean, well thought out application that you didn’t even need to think out very much. Before you know it, you fall into these useful and powerful design patterns without even thinking about it (I’d hope you would at least, they’re good patterns for a reason).

A lot of web frameworks make the assumption you are familiar with these patterns already, will setup your application to follow them, and use them in a way that makes sense. Part of the reason for this is because some web framework developers are rather appalled at the thought of making these decisions for the framework user. To be making those decisions for the framework user is looked at as holding their hand, and typically the programmers who write frameworks don’t need any hand holding and don’t expect their users to either (I’m assuming this based on the frameworks I’ve tried).

For example, during Ian Bicking’s test run with Rails he noted that having such a directory hierarchy setup for him put him off a little and mentions people arguing about file layout. I’ve seen very little, if anyone else, even comment on the directory structure that Rails lays out for you. The other irony of this particular post is that a lot of the stuff Ian finds “wrong” about Rails is actually Ruby’s “fault” (Implicit passing of names, mix-in’s, etc.). But moving on….

Ian clearly doesn’t want to lay down the law for users of his software, neither do other Python web framework developers. Having choice in how you lay out your application, with no clear direction what goes where, is considered a good thing. I don’t think there’s anything wrong with this approach, if the users you’re aiming your framework at are expected to be experienced, professional, programmers that are quite familiar with web applications and what works best for them. This does mean the audience will not be that large, and I think thats clearly reflected in the small communities most Python web frameworks have. While their communities are small, I’ve found them to be highly skilled and you get great answers to questions on their respective mailing lists.

The one Python web framework that has almost overnight gone from hardly any users, to at least a few dozen (probably more, but even a few dozen is quite good for a Python web framework), is Django. Django makes a lot of decisions for the user, more so even than Rails by a pretty good amount. While Rails was designed as a web application toolkit, and extracted form a large webapp, Django is designed more as a CMS web application toolkit as its been extracted from newspaper-style web sites. Thus, many of the decisions Django makes for you assume you will want users, user-access to the database objects, etc.

The question is, if someone else is making decisions for you, how important is it that they be good decisions?

One of the decisions the Django team made originally irritated me quite a bit. The Django framework is probably about 95% MVC in nature. It has database files that clearly are acting as Models (which it calls models), logic oriented files traditionally called Controllers (that it calls views), and template files that display the results to the user in a typical View role (though Django calls them templates).

The FAQ on the Django site notes that if you squint, sure enough the observations I just made do pop out, however because of the 5% non-MVC I mentioned they decided the names didn’t work. Rather than calling them by what they obviously were (or closest to), they swapped around names in a way that I think is bound to confuse Django programmers when they go into a real MVC environment and tell someone they edited the view when in fact it was the controller. If they had come up with completely different names that didn’t relate to the MVC paradigm at all, I think that would help to avoid later confusion on the part of a Django user.

Despite the naming of these parts, the decision made for the user is clearly for a MVC-style (or influenced) environment which I think is great ( please change the naming! ;) ). So what is the point of this whole thing?

I think its a good thing for developers of toolkits and especially frameworks to make some more decisions for the end-programmer of their work. This makes it easier for not so experienced programmers to get started quickly, and if you’re using a good design pattern even the experienced dev’s wont have anything to complain about. The two frameworks that currently are getting a lot of attention make a lot of decisions for the programmer using them, and this definitely increases the user-base (assuming you want a large user-base). Even though I know good practices on laying out stuff, it makes my life that much easier when its done for me.

Myghty is a powerful web application framework that builds on the strengths of Perl’s Mason using my favorite language, Python. While Myghty also handles page templating, it has some very powerful features that make it a full fledged framework in my opinion.

Wondering if Myghty is something you should check out?

I would highly recommend looking into Myghty if you’re looking for any of the following:

• Clean, unobtrusive templating language for integrating Python data and HTML
• An easy way to write pages that inherit their look/feel without needing .include statements
• Re-usable code that you can carry from one project to the next
• MVC programming paradigm , or page-driven paradigm (Myghty doesn’t care, it can be anywhere in between as well)
• Advanced caching ability, tunable to just the time intensive sections of a page
• Run as WSGI, FCGI, SCGI, mod_python, or stand-alone with hardly a single change to your code (to switch between them)
• Architecture that scales, from a single developer’s personal project, to a multi-developer’s company website

module_components = [
    {r'myapplication/home/.*' : 'myapp.home:HomeHandler'}, 
    {r'myapplication/login/.*' : 'myapp.login:LoginHandler'}, 
    {r'.*/cart' : 'myapp.cart:process_cart'}, 
    {r'.*' : 'myapp.home:HomeHandler'} 
]