Groovie

Pylons 0.9.7rc1 was released a week ago, unfortunately I haven’t had time to actually blog it so better late than never. This is a big step towards the 0.9.7 release, and contains some major changes over 0.9.6 while still retaining a huge degree of backwards compatibility.

At this point, the thing I get asked the most is:

When will Pylons 0.9.7 be released?

So the short answer, when the new website and docs are ready. We’re going to a lot of effort to totally eradicate that old mantra that “Pylons has no docs”, and we’re doing it big. Most of the docs have already been updated, revamped, and moved to the new Sphinx doc tool (Take a look at the new Pylons docs).

The new website is nearing completion as well, and for those using the 0.9.7 release candidate, when posting a traceback you’ll get a link to it thats on the new beta website. Until then, 0.9.7 is feature-frozen and newer RC’s up to 0.9.7 are bug-fix only.

New Features

Pylons gets the substantial amount of its feature-set from the other Python libraries it uses, and here’s some of the new things these libraries have brought Pylons users:

• Moved to WebOb from Paste for the Request and Response objects
• WebHelpers 0.6 (previous was 0.4) This is a huge update, including safely escaped HTML builders, a literal object to mark strings as safe (vs unsafe) for use in templating languages, and a move away from all the old ported Rails helpers to new ones that in many cases have more features with less bugginess
• Routes 1.9, with minimization turned off. This helps for more predictable route generation and matching which confused many, and in some cases led to hard-to-debug routes being created and matched. The new syntax available also breaks with the Rails’ish Routes form, and lets you easily include regexp requirements for parts of the URL.
• Mako Automatic Safe HTML Escaping
• Simplified rendering setup that doesn’t use Buffet
• Simplified middleware setup with easier customizability
• Simplified PylonsApp for customizing dispatch and URL resolving
• and lots of bug fixes!

There’s a more detailed page covering 0.9.7 changes available as well that can also assist in the rather minimal change needed for a 0.9.6 project to get going with 0.9.7rc1.

Other things in Pylons-land

With TurboGears2 extending Pylons for its foundation, many various parts of TG2 have become usable within Pylons, not to mention existing packages that have been getting better and better.

ToscaWidgets has gotten drastically simpler, no longer requiring the rather confusing RuleDispatch package with its generic methods. This makes the tw.forms package install with a fraction of the packages it used to require, and since it comes with Mako templates won’t incur any speed bumps it used to have from its use of Genshi. The new Pylons tutorials for it also make it a breeze to quickly create large forms with advanced widgets.

Some might have noticed that Reddit released their source code, which happens to be in Pylons. Their code is a good example of some of the customizing possible with a Pylons based project, as they added some custom dispatching to make controllers work in a more similar fashion to web.py controllers that they ported their app from. In a way, its similar to how TG2 has been able to support TG1 users for the most part by customizing Pylons to dispatch in a TG1 style manner.

Profiling an application got a lot easier with repoze.profile, and I’m sure more cool bits of WSGI middleware will be coming out of the repoze project in the future, not including some of the past handy bits like repoze.who which is used in TG2 for its new identity system.

I ported a little app that Robert Brewer wrote to track memory leaks. Being terribly uncreative on names for my new WSGI middleware version, I called it Dozer. It’s a handy little piece of WSGI middleware to throw in when you think you might have a memory leak to try and sort it out.

Pylons is moving along quite nicely, and the amount of WSGI middleware and tools that work with it continue to expand which makes it hard to list all the cool new projects I’ve seen lately that work wonderfully with Pylons.

Mako and SQLAlchemy continue to evolve with Mako having pretty much zero backwards incompatible changes in the past 6+ months, while SQLAlchemy slowly deprecates things as they prepare the 0.5 release. These packages have massive amounts of features and are rapidly becoming very stable easily making Pylons + Mako + SQLAlchemy a tough combination to beat.

Phil Jenvey has been making some great progress getting all the components of Pylons running on Jython, and posted a good write-up of the remaining work being done. It’s interesting to note that one of the big issues will affect any web framework on Jython, not just Pylons. That is, the reload time when used in development to restart the server.

While I don’t plan on deploying Pylons apps in WAR files anytime soon, its nice to see Jython emerging as a candidate for deployment.

I’m thrilled to announce that MarkMail now indexes the Pylons-dicuss and Pylons-devel mail lists. For those looking for a great way to search and browser the Pylons mail lists, the MarkMail interface is top-notch.

For those looking for detailed Pylons docs…. there’s some very exciting developments in the documentation front coming up shortly that will make you rather happy. :)

Sprints!

Wow, they were great. A lot got done, quite a bit more than I was expecting. I’d like to give a big shout-out to the Pylons sprinters (in random order, sorry if I missed anyone):

• Karen Lo
• Mike Orr
• David Montgomery
• Mike Verdone
• Wes Devauld
• Ian Bicking
• Phil Jenvey
• and many more TG2 sprinters

WebHelpers saw some significant gains, including the addition of a literal object for safe HTML escaping, and default Mako auto-escaping in Pylons 0.9.7.

There will definitely be more Pylons/TG2 sprints in the future, in multiple locations around the country.

Tutorials

Ouch… setuptools came through, unfortunately the network didn’t, nor did my first built egg of Pylons (unfortunately I got a new laptop recently that was missing some things…). Mark Ramm and I did get the show under way, but the delay in starting up definitely affected how far the tutorial attendee’s made it in the basic Wiki project and proved frustrating for all.

I made it through all the Pylons information I had in the Mastering Pylons/TG2 section, but unfortunately had no time left to go into more detail on various aspects of it in a more hands-on style. I definitely won’t be doing any more tutorials without ensuring lots of USB thumb-drives are handy with virtualenv Pylons/TG2 ready to go.

Sessions and the Conference

They seemed ok, several of them seemed very, “And this person is giving this talk…. why?”. Note to session presenters, honesty is great but I really do wonder why you’re presenting when your opening remarks are, “What I’m about to show you, anyone who’s used this already knows, so there’s nothing to see here.” Show us something with some zing!

The lightning talks, despite a first day of some awkward sponsored ones, were actually the highlight for me. I almost spit water laughing on the “Speech Recognition does war”, followed up by “His base of God would have I done now” (Originally intended to be ‘Speech recognition does work. oh god, what have I done now?’). Unfortunately Ian Bicking failed to give us an update on ZhangoPyloGears, I can only imagine it has fully attained consciousness and freed itself of his clutches.

Coming up…

Pylons 0.9.7 is close at hand, just waiting for some final bits of WebHelpers to drop into place and it should be ready to pop. And one last side-note, I think I finally found the documentation tool of my dreams… Sphinx!

There’s been a bit of talk on the Pylons devel list regarding the recommended way to use SQLAlchemy with Pylons mainly regarding how to use SA (SQLAlchemy) in a fashion that is well documented and easy to work with (and maintain!).

Prior to Pylons 0.9.6 and SQLAlchemy 0.4 it was a bit of a mess, with the framework needing to load the config (since thats where your db settings are), then setup globals for SA…. eek. Mike Orr had a good intermediary solution for SA 0.3 called SAContext that handled many of the tricky parts. Unfortunately, this actually caused even more confusion as more ways of doing the same thing came about. SAContext solved some of the global config grabbing issues, but the additional layer of indirection made trouble shooting even harder (despite how small of a library it was).

Less is More

So the fix? Less intermediary layers, less indirection… essentially, KISS. Despite how much Pylons was attempting to help a user to get the db going, the additional layers in the end actually caused more problems then they solved. Of course, I shouldn’t have been too surprised…. Mike Bayer did warn me about many of these things at the beginning. Being overly eager to make things “easier” for new users, I ignored him. :)

This is why Pylons does not recommend Elixir, and with SA 0.4 the recommended usage of SA is to use its plain mapper functionality should you need an ORM layer. Yes, that’s right, despite almost every web framework out there pushing its ORM on you (or someone else’s ORM), there are many times when an app doesn’t even need a full-blown ORM.

Declarative vs Basic SA

For a better look at why one might consider additional layers on SA a bad thing lets compare a fairly basic table setup consisting of users and groups. Each user can be in multiple groups, and lets use proper referential integrity to ensure that groups aren’t deleted when users are still in them.

Compare the following two ways of setting up a basic many to many relation and the tables:

SQLAlchemy 0.4

from sqlalchemy import Column, ForeignKey, MetaData, Table, types
from sqlalchemy.orm import mapper, relation

metadata = MetaData()

person_table = Table('person', metadata,
    Column('id', types.Integer, primary_key=True),
    Column('name', types.String, nullable=False),
    Column('age', types.String)
)

group_table = Table('group', metadata,
    Column('id', types.Integer, primary_key=True),
    Column('name', types.String, nullable=False)
)

persongroups_table = Table('person_groups', metadata,
    Column('person_id', types.Integer, ForeignKey('person.id', ondelete='CASCADE'), primary_key=True),
    Column('group_id', types.Integer, ForeignKey('group.id', ondelete='RESTRICT'), primary_key=True),
)

class Person(object):
    pass

class Group(object):
    pass

mapper(Person, person_table, properties=dict(
       groups=relation(Group, backref='people', lazy=False)
))
mapper(Group, group_table)

Elixir

from elixir import *
class Person(Entity):
    has_field('name', String)
    has_and_belongs_to_many('groups', of_kind='Group')

class Group(Entity):
    has_field('name', String)
    has_and_belongs_to_many('people', of_kind='Person')

On first glance, its pretty obvious that everyone should love Elixir vs the obviously more tedious SA approach of layout out your tables, then mapping them to the class objects. However, look at these two examples, and try to quickly answer the following questions:

• How do you add a column to the many to many table to store an additional bit of info for the join?
• Do they both enforce referential integrity?
• How do you control whether SA is eager loading the relation? Can you restrict it to just one column of the relation?
• What are the table names used?
• How many tables are in your database?
• Where do you change the id column name?
• Which one is closer to the Zen of Python?

I think the explicit setup makes many of these questions easier to answer just at a glance. Those with enough Elixir experience can fairly easily answer most of these questions, but consider what that implies. Not only do you need to know SQLAlchemy options and parameters, but you need to know Elixir options and how they map to the SQLAlchemy functions. The desire to reduce the up-front setup of the ORM actually increases the amount of knowledge a user has to have in order to use it, and the most worrisome aspect… how to troubleshoot it.

Setups that Grow with You

With Pylons, a goal has been to provide out of the box recommendations that grow with you. That is, using the set of recommended tools may not be as apparently “easy” as some other frameworks. However, the pay-off is that you don’t hit a wall in 2 months when your application inevitably gets a little more advanced and needs to do something the simple tools either can’t do at all, or it’s incredibly difficult to do even slightly complicated things (eager load 2 columns off a related table, but not all of them). This way, the toolset you learned, you can keep using as you get more advanced and you don’t “outgrow” your tools.

While Elixir definitely appears to be easier at first glance, when you need to get more complicated you can’t exactly turn to the SA docs since Elixir has put a layer between you and SA. This can be very crippling when you eventually hit a wall, and so much ‘magic’ is wrapped up in the declarative layer that you have to troubleshoot additional layers of code when something goes wrong. The result of this is that to effectively use Elixir in complicated setups with SQLAlchemy, you need to really really know both of them which actually requires more work for a user than plain SQLAlchemy.

The SA example clearly requires a little more up-front setup, however, are you really adding tables to your database every day? How often are you going to be actually touching the table and mapper code, vs just adding domain model methods to your Person/Group class? Did the layer make it easier or harder to use multiple databases and/or put more between you and advanced SA functionality you might need later?

Adam Gomaa pointed out some interesting issues with Django’s ORM and Elixir but unfortunately tries to do the same thing Elixir and TurboEntity do…. add more layers. More layers more indirection more to wade through when you need to do something that should be pretty basic with SQLAlchemy (and is probably nicely documented on the SA site, which won’t help with these layers until you dig through the layer to find the basic SA objects the SA site refers to…).

What really makes a lot of this even more trouble-some with SA, is that when setting up complex relationships, the order of declaring table objects becomes important, since relations need to refer to them and the ORM classes. This usually results in some interesting metaclass hackery when you have these Entity’s in multiple modules, importing each other, and doing other fairly common stuff.

SQLAlchemy 0.4

In the end, I’ve been using plain SQLAlchemy 0.4 (at beta5 now, but quite stable) a lot lately, and its really great. Yes, setting up the tables (generally a one-time thing) took me probably 15 mins longer than it would’ve with Elixir. But I’m fairly certain I’ve saved myself significantly more time in the long run since I won’t need to worry about diving into Elixir code to try and find SA objects when I need a complex query, or trying to figure out how to hack Elixir’s connection should I need multiple db connections at once, etc.

So please, new users to SQLAlchemy, use just SQLAlchemy. It definitely seems daunting at first, but the flexibility and comprehensive documentation give you a solution that scales to meet your needs, with no walls in sight.

On a side-note, its interesting to compare my position on this issue to the Django team lack of AJAX helpers. The Django team rightfully claims that Javascript isn’t that hard, so “get over it” and learn a nice Javascript library so you can do powerful things. Also note that by including AJAX helpers, Pylons is encouraging one part that doesn’t scale… as the AJAX helpers will have you hitting a wall sooner or later.