Groovie

As the date edges closer to a Pylons release, I find myself already thinking about future directions of Pylons. I’m obviously rather biased when discussing Pylons as I’m one of its creators, though I still find that thought humorous as the vast vast majority of the code that resulted in an excellent framework is not actually in Pylons, nor did James Gardner (the other Pylons dev) and myself write it.

Matt mentions in a blog entry on Python Web frameworks, maybe its just too easy to write a Python web framework. I didn’t originally see myself writing a framework as such, but it became a fairly logical conclusion to the work I was doing at the time and the framework was originally extracted (like many others) from a large production application. It really is quite easy to write a web framework, and there’s tools available that make it even easier.

WSGI has changed the point at which re-usability is possible in a Python web framework as well, the only possible result I see in the future is more frameworks (Not in the way people think though). Many would argue this is “bad”, however I think due to the different point in re-use made possible by WSGI there will actually be more collaboration and less web developer divisions despite how many ‘frameworks’ are out there (once they’re more WSGI-ish that is). Consider this analogous in some ways to how many linux distributions are out there, and the fact that people can switch between them very easily.

So what went into building Pylons?

The Basics

First, there’s a few things you’re going to need in even the most basic web framework:

• Dispatcher
• Request API/Object

That’s really the utter basics, and there’s frameworks that don’t do much more than this. Obviously many people are going to want more…. like sessions. The question that WSGI raises is why lock a session interface to a framework, when it can be re-used as WSGI middleware? While the term “WSGI Middleware” can be rather intimidating, I previously covered why WSGI is actually quite easy despite the sometimes scary or just annoying term. It’s something to be reckoned with, and there’s no excuse not to get familiar with it if you’re doing web development in Python.

As I needed something rather reliable and sturdy, I went with Myghty’s session and request API. For dispatching, I used a custom resolver utilizing Routes that dispatches to a controller and action (controller method). This results in a fairly MVC’ish style web architecture that’s rather extensible.

So check those off the above list plus sessions:

• Dispatch – Routes based
• Request API – Myghty’s HTTP request object
• Sessions – Myghty’s session interface, which is also now available as WSGI middleware

At this point, we’d have a framework that can interpret a URL, setup an easy to use request object, and call your controller. Your controller would also be able to save data using sessions. We’d have a bit of a problem distributing such a framework though, as it probably wouldn’t be very convenient to setup and install.

In the case of Pylons, so far it means I’ve written just a class to handle dispatch, locating the controllers (actually, Routes will do this), and calling them. A measly hundred or so lines of Python code.

Defaults, Structure Creation, Stand-alone Server

What really helps people get started quickly with a framework is if its easy to install, creates a basic structure of a working web application for them to get started with, and some way to run it. So our new additions:

• Web Framework Installer
• Template for starter Web Application
• Stand-alone Server

This can quickly be a fairly substantial amount of code to write, especially if you want cross-platform installation, and a system that also runs on multiple platforms. Not a problem though, Paste quickly makes the last two of these quite easy, and setuptools handles installing your framework and any requirements it may have. I should note that setuptools isn’t perfect, but it sure beats asking users to go to a half dozen sites and install various packages.

Paste is divided into 3 parts, though it can be tricky to see the relation between them. The core part of Paste contains wsgi middleware parts that will likely appeal to many, and some other basic request handling functionality. PasteScript contains the structure creation bit used in Pylons that generates the starting template. There’s also PasteDeploy which comes in useful later when deploying and running web applications made with a framework using it.

We’re looking at a decent little framework so far, now that we can install it, quickly start a new project, and run it. What’s next?

Templating and Object-Relational Mappers

Some frameworks, typically known as “full-stack” frameworks try and make more choices for you. They aim to fill your needs top-to-bottom, or at least their vision of your needs. While SQLObject is a very popular ORM, it seemed that good use of a layout and basic Python import statements would make using any ORM just as easy.

In Pylons, a models directory is provided, with some commented out suggestions for how you’d go about setting up an ORM. There’s hooks provided in the base application module that’s imported by all your controllers, so its easy to define your ORM classes and use them in your controllers.

One of the other objectives for Pylons was to try and be very ‘Pythonic’. That is, it should re-use as much of a developers Python knowledge as possible. This is used by default for templating since Myghty uses normal Python syntax for its templates in addition to providing powerful caching functionality and great re-use through components.

Not wanting to force anyone into something they couldn’t stand (template languages can be a love-hate thing), we also implemented the TurboGears Template Engines plug-in functionality. This made it very easy to let people use the template language of their choice, in a fairly uniform manner. It was also pretty minimal to implement the template language renderer as the Buffet project provided a great head-start.

Our new list is looking rather complete for a “full-stack” framework:

• Web Framework Installer
• Template for starter Web Application
• Stand-alone Server
• Dispatcher
• Request API/Object
• Sessions
• Templating
• Database Integration

Making the most of Middleware

This is one of the areas where Pylons was able to make some great leaps, with very little actual code. In several areas, thanks to the use of middleware, Pylons is able to offer features other frameworks are still working on.

To start off with, there’s the excellent EvalException middleware which provides an AJAXy exception catching system I have yet to see in any other framework. We’ve formatted it slightly to fit in nicely with Pylons, and it works like a charm. Extremely useful for those times when you want to interactively debug a web application to see what’s what, and it’s a lot quicker than putting print statements all over.

Another important bit, that a lot of frameworks skimp on is unit testing. Using Paste’s fixture middleware, its easy to test your web application. Pylons adds a few objects to the response you can test with, so you can ensure that the session was setup properly, the right template components were called with the right arguments, etc. In the future we’ll likely add some defaults to make using twill an easy option.

The best thing about all of this of course, is that these useful parts can be integrated seamlessly and re-used by other frameworks.

Taking the framework out of Framework

Given how most of these parts are definitely not unique to Pylons, nor are they intended to be, it shouldn’t be long until more frameworks start using the great modularity that Pylons is utilizing. A lot of these parts will likely become standardized to an extent, so that there’s even less barrier to switching frameworks.

At this point, Pylons becomes less of a “framework” in one sense, and more a set of defaults and structure for how a Python web application should be put together. Pylons has more features of course that I haven’t described, the WebHelpers functions are made easily available for use in templates, more Paste middleware is used for slick traceback email’s when you’re in production mode, PasteDeploy makes running your Pylons webapp easy in a variety of situations, etc.

It’s a very easy-to-extend model, with little need to put great amounts of application-y type stuff into the framework itself. It also keeps its components separate for easier testing, out of the actual framework. This means that while Pylons comes with a great set of middleware and parts set up for you, its very easy to swap in your preferred template language, your preferred ORM, a different exception handler, etc. The choice is up to you, but the defaults are set to a good starting point (also called “convention over configuration”).

Upping the Re-usability Ante

Increasing re-usability is what I’d consider the future for Python Web Development. With WSGI middleware driving re-usability, such concepts that “to use Feature X, you must use your Framework Y” just doesn’t apply. Unifying development work on excellent components that can be re-used in any WSGI-compatible framework makes Python Web Development better.

Pylons isn’t alone in aiming for this style of framework, Ian Bicking is working on a project that starts the opposite direction, with just a layout and you add sessions, templating, etc. as you need it. TurboGears is adding more Paste-compatible features that will shortly make it trivial for them to add in the EvalException middleware (assuming they haven’t already, I haven’t checked lately) and other great components. Different frameworks have different levels of re-usability, those that are built with re-usability in mind at the beginning will likely be able to adapt quicker to new demands and requirements, and take maximum advantage of the great middleware being created.

While having more people use Pylons would be great, it isn’t necessary for Pylons to become a better framework. Having more people use the WebHelpers package, or make their framework Paste-compatible, or use Routes, or Myghty’s powerful caching/session API’s all helps Pylons. It also helps any other framework using these components, and that’s what counts the most.

Here’s the final tally of Pylons features and where they came from:

• Web Framework Installer – setuptools
• Template for starter Web Application – PasteScript
• Stand-alone Server – Paste
• Dispatcher – Routes / Myghty
• Request API/Object – Myghty
• Sessions – Myghty
• Caching – Myghty
• Templating – Myghty, or any that support the TurboGears Template Plug-In
• Helper functions/AJAX – WebHelpers
• JSON – simplejson + Pylons decorator
• Global “convenience” objects – Pylons
• Database Integration – SQLObject, SQLAlchemy, anything else
• Interactive Debugging – Paste
• Traceback E-mails – Paste
• Webapp Unit Testing – Paste
• Webapp Deployment – PasteDeploy
• Webapp Distribution/Installation – setuptools

This is just the default set-up, its trivial to add more middleware, which would make this list very, very long and includes such things as OpenID Authentication, authenticated session tickets, along with other great stuff.

The Pylons Code-base: Pylons Module Reference

Zachary.com has a very nice update on Routes with CherryPy covering not only how to integrate them, but why you’d want to use Routes style dispatch instead of the object publishing approach CherryPy uses by default.

In the future, there’ll also be an independent dispatcher for Routes that handles dispatch at the WSGI level. I believe Ian Bicking already has such code, though he hasn’t released it yet. I’m sure it’d make a great Part 2 in his series(?) on working with WSGI up close and personal. If you haven’t read his Do-It-Yourself Framework, I’d highly suggest giving it a read.

Not only does it help demystify WSGI, but it should hopefully make it more obvious why WSGI is changing the point of competition in the world of web programming.

I first heard about Python from a roommate about 5 or 6 years ago. He was putting together a presentation for some Bio-informatics people, and was going to go over the basics of Python programming as its very popular in the scientific community. I didn’t give it much though as I was mainly using Perl at the time for any programming work, and the white space was of course somewhat of a put-off.

Fast forward a few years… I had been programming rather heavily in Perl at the time, and despite having used Perl literally for years, I still felt like I hadn’t achieved “mastery”. The effects of code in Perl would jump out and surprise me every few days, and sometimes hunting down weird bugs due to ambiguous syntax was rather time consuming. In short, Perl just wasn’t fun anymore.

I started looking around for other languages, and took a look at Ruby. Back then, Ruby had the same things that were annoying me about Perl. Implicit variables that seemingly came out of nowhere (@, $, etc), inconsistencies, and since the community was so small it lacked the vibrant and active development Perl had. Ruby also was missing so many of the CPAN modules I had come to rely on. Then browsing through some of Eric Raymond’s writings, I came across an article about Python he wrote titled the same as this blog entry.

White space and Mastery

My white space fears were removed rather quickly.

Of course, this brought me face to face once again with Python's pons asinorum, the
significance of whitespace. This time, however, I charged ahead and roughed out some
code for a handful of sample GUI elements. Oddly enough, Python's use of whitespace
stopped feeling unnatural after about twenty minutes. I just indented code, pretty
much as I would have done in a C program anyway, and it worked.

But this wasn’t what truly piqued my interest. As I mentioned, the feeling that Perl was somehow beyond mastery. To me, mastery means that I can write massive chunks of code with confidence it’ll act exactly as I intended it to. So seeing Raymonds next few paragraphs really got me going.

That was my first surprise. My second came a couple of hours into the project, when I 
noticed (allowing for pauses needed to look up new features in Programming Python) I 
was generating working code nearly as fast as I could type. When I realized this, I 
was quite startled. An important measure of effort in coding is the frequency with 
which you write something that doesn't actually match your mental representation of 
the problem, and have to backtrack on realizing that what you just typed won't 
actually tell the language to do what you're thinking. An important measure of good 
language design is how rapidly the percentage of missteps of this kind falls as you 
gain experience with the language.

This is all summed up very nicely,

I wrote a working, usable fetchmailconf, with GUI, in six working days, of which 
perhaps the equivalent of two days were spent learning Python itself. This reflects 
another useful property of the language: it is compact--you can hold its entire 
feature set (and at least a concept index of its libraries) in your head. C is a 
famously compact language. Perl is notoriously not; one of the things the notion 
``There's more than one way to do it!'' costs Perl is the possibility of compactness.

Being compact is a valuable asset for Python. Not only does it make writing large blocks of code without error possible, but it also means the code is going to be easy to read. After opening up a few other Python projects and scanning through the code, I was hooked and proceeded to buy the Learning Python book (Though the online tutorials would’ve been just fine as well).

Black Magic isn’t so Black

Since then, I’ve been using Python close to non-stop for just a little over a year. In just the first week of using Python, I felt more productive and wrote code I was positive would work (just as ESR said) on the first try. A few months ago, interested in learning even deeper “black magic” of Python, I went to a presentation by Alex Martelli on Python’s Black Magic, Meta-classes and Descriptors.

The most fascinating thing was that meta-classes got their behavior from simple concepts I already knew. It was hard to believe things were so easy. Eric Raymond also noticed this:

It's remarkable enough when implementations of simple techniques work exactly as 
expected the first time; but my first metaclass hack in a new language, six days from 
a cold standing start? Even if we stipulate that I am a fairly talented hacker, this 
is an amazing testament to Python's clarity and elegance of design.

Fields of Use

I have now written quite a few projects in Python, and on several of the open-source ones heard, “I just wanted to add Feature X to this, and I was actually able to figure it out just look at the code. I don’t even know Python!”

These projects span fields: shell scripts, network daemons, web applications, computing, GUI’s, and more. Learning Python is useful for many fields, and I know people learning Python right now so they can tackle problems in different fields with a compact language thats easy to master.

The Choir

I realize that this entry, carried by Planet Python, is mostly preaching to the choir. However, I felt something like this was needed. Partly because some members of the Python community have gotten odd impressions about what we should try and do in Python and what we should just “give up” on, having lost some sort of war (bizarre, I know).

People come to Python for a variety of reasons, they stay for a variety of reasons. Having compelling tools to make tasks easy in many fields is great, and means there’s no need to learn new languages solely to try and make a task in one field slightly easier (Though its always healthy to learn more).

It’s definitely valuable knowledge to know why a task in one field is easy using Language X, and for that alone its good to give it a spin. Learn what was done right, and what was done wrong. In the end though, if that language isn’t a compact language like Python, I’m just not going to enjoy it as much.

I didn’t start learning Python because of Application/Framework Y, I learned it because Python is exceptionally compelling by itself. Others learn it for the same reason, and despite the bizarre claim that book sales is equivalent to language usage/popularity, it should be obvious by now that a book isn’t needed to master Python.

For those demoralized because some other language is getting attention due to a field its had a huge impact on, re-evaluate your feelings. This isn’t the first time its happened, and it won’t be the last. Google uses Python, ILM uses Python, thousands (yes, thousands) of major corporations use Python.

Python doesn’t need to be master of every field, it just needs something compelling enough that you’re as productive using it as you would be using a different tool in a language that isn’t so compact.

I keep seeing blogs and blog comments pop up with some very odd notions. Many of these are summed up in this comment from a recent post by Tim on Ruby books

“It’s doubtful that the Python folks can come up with anything as compelling (or elegant) as Rails. Why? Because Ruby is so good at creating Domain Specific Langauges (DSLs). Ruby’s anonymous code blocks are a big part of what enables DSLs to be written so easily in the language. Python doesn’t have them. Python’s lambda’s (and closures in general) are crippled as well, which also doesn’t help Python’s cause.”

What is a Domain Specific Language (DSL)?

I always like to carefully define my terms before actually talking about them. So first I looked up several common definitions of a DSL. We have Martin Fowler’s entry on a DSL which describes it as:

“The basic idea of a domain specific language (DSL) is a computer language that’s targeted to a particular kind of problem, rather than a general purpose language that’s aimed at any kind of software problem.”

He further clarifies between two types of DSL:

“The most common unix-style approach is to define a language syntax and then either use code generation to a general purpose language, or write an interpreter for the DSL. Unix has many tools that make this easier. I use the term External DSL to refer to this kind of DSL.”

“The lisp and smalltalk communities also have a strong tradition of DSLs, but they tend to do it another way. Rather than define a new language, they morph the general purpose language into the DSL. (Paul Graham describes this well in Programming Bottom-Up.) This Internal DSL (also referred to as an embedded DSL) uses the constructs of the programming language itself to define to DSL.”

To summarize, an Internal DSL satisfies one of the following:

• Uses a general purpose language, then morphs it into a language to fit the domain
• Adds functionality to a general purpose language such that solving a specific domains problem is easier

An External DSL:

• Completely new language syntax
• Typically requires code generation or interpreter for DSL

Martin Fowler espouses on how very dynamic languages like Lisp, Smalltalk, and Ruby are more conducive to internal DSLs as the meta programming capabilities and dynamic nature make it easy to extend and customize the functionality for your domain. I completely agree with his contention that an Internal DSL is limited by the syntax and structure of the language.

DSL’s in Python and Ruby

While the anonymous code blocks in Ruby are useful and syntactically nice, similar functionality can be done in Python through the use of generators and iterators (more powerful generator functionality similar to Ruby’s anonymous blocks are planned for Python 2.5).

Ruby also lets you extend built-in types, which is utilized by Rails to add a few helper functions to core Ruby data types. Python and Ruby both have closures that operate slightly differently, Ruby’s closure is definitely more “full featured”. However this typically isn’t a problem because of list comprehension’s, generator expressions, and generators/iterators.

Zope is a very clear example that its quite possible to build an internal DSL in Python. Whether they built it in a clear, easy to use way is up for someone else to debate. SQLObject makes dealing with databases as easy as ActiveRecord, and there’s many other examples of internal DSL’s built in Python (Take a look at twill, a web application testing language implemented in Python).

Is Rails a DSL?

First, it should be obvious that if Rails is a DSL, its an Internal DSL. The most noticeable strides towards being a web programming DSL in Rails:

• Over a dozen core Ruby classes are extended/modified
• Dozens of helper functions
• New classes (syntactic sugar) defining database access and web paradigms

Given how loose the definition of internal DSL is, I think its clear that Rails qualifies, so does Django, TurboGears, Aquarium, etc.

If you prefer to be more strict about an internet DSL, and argue that a lay programmer needs to be able to do a reasonable amount of ‘work’ without knowing the base language implementing it, Rails starts to move apart. You can do a reasonably large amount of work in Rails without knowing Ruby. A lot can also be done in Zope without knowing Python.

In either case, you can’t really start to make advanced applications in any of the frameworks without knowing the general purpose language its built on. DSL’s can and have been built in Python, and I believe if you go out and start counting them up, you’ll see more DSL’s implemented in Python than in Ruby.

Anyways, hopefully for people that run around re-iterating something they heard somewhere about DSL’s and Python vs. Ruby, this clarified something. It’s easy to create DSL’s in Python, lots of people do, and there’s lots of them. Python and Ruby are both great for making them, outside of Rails I don’t know any other DSL’s in Ruby. Perhaps someone can compile a list of all the internal DSL’s written in both Ruby and Python?

TurboGears has been making some impressive strides in both features, documentation, and possible user acquisition lately. Where it gets somewhat interesting is regarding its user-base though. The approach TG takes – building glue on top of other projects – is not new, as Subway also utilizes this, however the popularity TG has been enjoying has resulted in some interesting by-products.

First, its a bit interesting to take a look at some numbers. It’s hard to be precise, as users don’t exactly announce themselves, but many of them do usually join the mailing lists for the projects.

Django-users:

• Users: 546
• Activity: Medium
• Average messages per month (5 months+): 340

Update: Eugene pointed out that there is Django-developers as well. Many of the Django-devel members are also on the Django-users list, so adding them together wouldn’t be too accurate. Here’s the Django-devel information, along with the total traffic if you take the two combined.

Django-developers:

• Users: 323
• Activitiy: Low
• Average messages per mont (5 months+): 177
• Average messages of Django-devel+users: 517

TurboGears:

• Users: 668
• Activity: High
• Average messages per month (3 months+): 1,124

Information taken from Google Groups as of Nov. 14th, 2005

While each list is bound to pick up users that contribute and lurk who do not actually use the framework, I think this trend is significantly higher with TurboGears. I believe this is because TurboGears builds on several successful projects that are even used in fields outside of web development.

Project/Code Re-Use and Explaining the Mail Traffic

Django takes a somewhat interesting approach, especially when compared to TurboGears, in that re-use is non-existent.The reason given for the amount written from scratch does make sense to a certain extent, especially when you remember that Django was created 2 years ago (some existing projects weren’t too hot back then). It’s also interesting to note that in some cases, the syntax and decisions being applied to Django now, reflect those made quite awhile ago in other projects.

TurboGears pushes re-use to the extreme, and only resorts to writing it from scratch if there’s no clean way to integrate a similar project that does the same thing. Only the new Identity framework, and the crud stuff was written from scratch for example. But many of the largest, most heavily used pieces have large communities of their own, i.e. SQLObject, FormEncode, CherryPy, MochiKit, and setuptools (I likely missed some).

Also interesting to note, several of the top posters to the TG mailing list happen to be the project developers for the individual projects TG is comprised of. I doubt they’re heavy users (or perhaps haven’t even used TG), however they have plenty to contribute as the questions often relate to the individual parts. This creates a powerful community and further fuels development and use of the individual projects TG pulls together.

When considering the use of one framework vs the other, I think this re-use issue is definitely something to be considered. Building on other stable, established projects that are used widely and extensively leads to a more mature platform. The size of each individual community also means more people to work on solutions that benefit the whole (and even those using the parts).

For example, the TG community is starting to tackle some basic CRUD/administration type stuff. I don’t actually use TG myself, but I do use SQLObject and FormEncode which means I can plug my SQLObject classes into a TG project, and use their web based admin tool with my database. This is rather powerful and extremely useful code re-use, and in the future it’ll be even easier to do the same thing (likely using a WSGI app).

Code re-use is important, and the project re-use that TG employs leads to many benefits. Despite the reason cited on the Django page, I think Django is still ripe for re-use cases. The Django syntax is now so close to SQLObject, I wish Django would just switch to it and extended it where needed, rather than continue to re-implement the rest of it.

To be fair, Django does make it possible to use other templating languages, and Zope3 Page Templates are available (along with a Django ZPT error formatter). If part of the argument for OOP and Python is code re-use, projects that re-invent not just the wheel but the entire car aren’t exactly shining examples of what’s possible in Python.