Groovie

I released Routes 1.9 today, which is another step on the Road to Routes 2.0. Some of the highlights that people will be most interested that I had previously blogged about now available:

Minmization is optional

Pylons 0.9.7 will default to turning minimization off (projects are free to leave it on if desired). This means that constructing a route like this with minimization off:

map.connect('/:controller/:action/')

will actually require both the controller and the action to be present, and the trailing slash. This addresses the trailing slash issue I wanted to fix as well.

Named Routes will always use the route named

This is now on by default in Routes 1.9, which results in faster url_for calls as well as the predictability that comes with knowing exactly which route will be used.

Optional non-Rails’ish syntax

You can now specify route paths in the same syntax that Routes 2 will be using:

map.connect('/{controller}/{action}/{id}')

Or if you wanted to include the requirement that the id should be 2 digits:

map.connect('/{controller}/{action}/{id:\d\d}')

Routes automatically builds the appropriate regular expression for you, keeping your routes a lot easier to skim over than a bunch of regular expressions.

Routes 2 will be bringing redirect routes, and generation-only routes, making Routes 1.9 a great way to transition to Routes 2 when its ready.

In the latest 0.9 release of Beaker, I’ve finally added cookie-side session storage. I was a little bit moved to finally do this by seeing that Rails 2.0 had added cookie-side session storage, and heck if I was going to miss out!

A few changes from how Rails 2 did it though, I was definitely not content to store all the data in an end-user visible form in the cookie. That only left encryption as the next logical choice, and that quickly led me down a path of quite a bit of cryptography research.

The world of cryptography is a constantly evolving and rapidly progressing field. New papers are coming out all the time with new research on ways to break, or ‘wound’ a particular encryption scheme. Providing a weak form of encryption in Beaker would be worse then not having it at all, since it’d lead someone to falsely believe the session data was secure.

After initially going with an RC4 cipher implementation, I got ahold of some crypto people that are actually in the field, and the unanimous opinion was to use AES encryption in Counter Mode, also referred to as AES-CTR along with a signature to prevent tampering (you’d be amazed what you can do to encrypted data, and it’d still technically decrypt). This led to a slight increase in requirements unfortunately, as pure Python based AES encryption is a bit slow. This means that using cookie-based sessions in Beaker requires the installation of PyCrypto, which includes a C extension (making cookie-based sessions faster than file-based, memcached, and db-based sessions).

The final solution in Beaker uses 256-bit AES-CTR with a 256-bit HMAC for authentication purposes. It’s fast, secure, and scales across a cluster without a problem. It’s not for everyone of course, cookies are rather severely limited in size, so if you’re just storing a few small tidbits of information in a session, for example:

• a user id
• some flags about the users status (logged in, etc)
• a flash message

Then cookie-stored sessions might be perfect for you.

Update: Forgot to mention, in the future, Beaker will probably use pycryptopp instead of PyCrypto since the PyCrypto library’s AES-CTR implementation isn’t as efficient as it could be, and will be using VMAC’s instead of HMAC’s for even more speed. Plus, apparently Andrew Kuchling isn’t maintaining PyCrypto, as there’s quite a few patches for it sitting unanswered on the sourceforge and launchpad bug trackers.

It’s with great pleasure that I announce the release of Pylons 0.9.4. This release has quite a few bug fixes and enhancements, the most since the 0.9 milestone. It’s also likely one of the last big updates before a 1.0 release candidate (there will be some small changes in 0.9.5 and possibly a 0.9.6).

First, the most important changes for those upgrading from an existing Pylons application:

• WARNING: Removed the lang_extract and lang_compile commands. They used pygettext.py and its associated msgfmt.py, which lacked the ability to extract ngettext style function calls and had issues with unicode strings. The new I18NToolBox project aims to provide this functionality (and more) via the gettext command line utilities. http://i18ntoolbox.ufsoft.org
• WARNING: Myghty’s allow_globals config var has changed, causing the following when running pre-compiled templates: Error(TypeError): do_run_component() takes exactly 13 non-keyword arguments (5 given) Delete the compiled Myghty templates directory (specified by cache_dir or myghty_data_dir in the config file) to resolve the error.
• WARNING: The localization function ’_’ now uses ugettext (returns unicode strings) instead of gettext. To preserve the old behavior, append the following line to your project’s lib.base and lib.helpers imports: from pylons.helpers import gettext as _
• WARNING: Removed 0.8.x legacy code and backwards compatibility functions.

Please note that since some i18n functions have moved, your helpers.py will need to be updated to import _, and ungettext from pylons.i18n.

Also:

- The XMLRPC Controller got a significant update so that it now provides the full range of XML-RPC Introspection facilities for your service methods.

- SQLAlchemy convenience functions have been added to pylons.database for use with the SessionContext plugin, and to create and retain SA engines.

- Paste dependency was updated to 1.1.1, Routes to 1.6.1 (important update for map.resource functionality)

- Pylons special objects (g, c, h, request, session) now available in interactive debugger without _attach_globals.

- Controller actions can now be generators

- Pylons base WSGI app uses wsgi.org routing_args spec for easier swapping of URL resolvers.

Install

Please see http://pylonshq.com/docs/0.9.4/install for installation details.

Full Changelog

• WARNING: Removed the lang_extract and lang_compile commands. They used pygettext.py and its associated msgfmt.py, which lacked the ability to extract ngettext style function calls and had issues with unicode strings. The new I18NToolBox project aims to provide this functionality (and more) via the gettext command line utilities. http://i18ntoolbox.ufsoft.org
• All Pylons special objects are now available within paster shell (not just h and g).
• WARNING: Myghty’s allow_globals config var has changed, causing the following when running pre-compiled templates: Error(TypeError): do_run_component() takes exactly 13 non-keyword arguments (5 given) Delete the compiled Myghty templates directory (specified by cache_dir or myghty_data_dir in the config file) to resolve the error.
• Changed i18n functions in templates to use proxy objects so that using set_lang in a template works right. Fixes #153.
• Now allowing any template plugin to overwrite global PYLONS_VARS (such as c, g), not just pylonsmyghty.
• Adding SQLAlchemy support to the database.py file. Saves the session engine to g to maintain it during the apps lifetime. Uses SessionContext plugin for management of the current session.
• Updated config object so that init_app can take an optional template engine argument to declare the default template engine.
• Updated Myghty plugin to use extra_vars_func when passed in.
• Fixed Buffet to use extra_vars_func properly.
• Fixed the validate decorator when there are validation errors and variable_decode=True: now passing the original params to htmlfill.render instead of the varable_decode’d version. Patch by FlimFlamMan.
• Added ungettext function for use with pluralized i18n, and the N_ function (gettext_noop) to mark global strings for translation. Added ungettext, N_ and translator objects to be globals for templates. Refs #126.
• WARNING: The localization function ’_’ now uses ugettext (returns unicode strings) instead of gettext. To preserve the old behavior, append the following line to your project’s lib.base and lib.helpers imports: from pylons.helpers import gettext as _
• Pylons special objects are now available within the interactive debugger (deprecating _attach_locals).
• Added setup-app run before unit tests run so that webapp has proper setup tasks handled. Fixes #113.
• Added paste.deploy.CONFIG setup to middleware.py, websetup.py and testing files in the Pylons project templates. Closes #112.
• Added security policy doc to index for use as Pylons security policy. Closes #91.
• Improved the repr() of the c context object to show attributes.
• Set environ[‘paste.testing_variables’] whenever that key is available, not just in testing mode.
• Added capability to have an action be a generator function.
• Added introspection capability to XMLRPCController and signature checking.
• Updated Controller to use additional arg lookup scheme so that the source of the function args for _inspect_call can be easily overridden.
• Updated RPCController, renamed to XMLRPCController. XMLRPCController now functions properly and will automatically return proper xmlrpc responses.
• Added test configuration ini file to default template. Closes #114.
• Fixed problem with pylons.database.PackageHub.get raising errors other than AttributeError when the database isn’t configured. Added new UnconfiguredConnectionError exception, instead of just KeyError or TypeError (depending on what part of the configuration failed).
• Fixed default g init, since bare object has no init method. Reported by Ian Bicking.
• Fixed issue with SQLObject method override having wrong name. Reported by climbus with patch. Fixes #133.
• Moved log function to pylons.helpers and translation functions to pylons.i18n. using pylons.util purely for Pylons internal util functions.
• WARNING: Removed 0.8.x legacy code and backwards compatibility functions.
• PylonsApp now has option to not use Routes middleware, default resolving uses new wsgi.org routing_args spec.
• Refactored routes dispatching to use new Routes middleware.
• Fixed paster shell command to properly acquire mapper object without relying on the template being configured in a specific manner.
• Added keyword argument pool_connection to pylons.database.PackageHub; if set to false then SQLObject connections won’t use pooled database connections (a new connection will be opened for each request).

Many thanks to Phil Jenvey, Ian Bicking, James Gardner, and all the other active members of the Pylons community!

Cheers, Ben

This is slightly old as Routes 1.4 was released about a week and a half ago, but I thought it deserved some attention. There were a handful of fixes and some slightly major feature enhancements in 1.4.

From the changelog:

• Fixed bug with map.resource related to member methods, found in Rails version.
• Fixed bug with map.resource member methods not requiring a member id.
• Fixed bug related to handling keyword argument controller.
• Added map.resource command which can automatically generate a batch of routes intended to be used in a REST-ful manner by a web framework.
• Added URL generation handling for a ‘method’ argument. If ‘method’ is specified, it is not dropped and will be changed to ‘_method’ for use by the framework.
• Added conditions option to map.connect. Accepts a dict with optional keyword args ‘method’ or ‘function’. Method is a list of HTTP methods that are valid for the route. Function is a function that will be called with environ, matchdict where matchdict is the dict created by the URL match.
• Fixed redirect_to function for using absolute URL’s. redirect_to now passes all args to url_for, then passes the resulting URL to the redirect function. Reported by climbus.

Web Resources

The map.resource command is based directly off the Simply Restful Rails plugin which adds support for various verb-oriented controller actions in a RESTful service style approach. The Simply Restful layout is more or less the exact service style laid out in the Atom Publishing Protocol.

It’s a great approach and it also meant providing a few other features to Routes that I hadn’t implemented previously, the most important being able to limit matching of a URL based on the HTTP method used. This is present in the new conditions clause for a Route:

map.connect('user/:id', controller='user', action='edit', 
    conditions={'method', ['GET', 'HEAD']})
map.connect('user/:id', controller='user', action='update',
    conditions={'method', ['PUT']})

The conditions clause can also accept your own function should you want to restrict the route to matching based off some other criteria (sub-domain, IP address, etc).

def stop_comcast(environ, match):
    if 'comcast.net' in environ['REMOTE_HOST']:
        return False
    return True

map.connect(':controller/:action/:id', conditions={'function':stop_comcast})

David Heinemeier Hansson recently posted an entry about Resources on Rails discussing how important web services are. The other key point was to make it easier to write controllers that could not only give you easy browser access to your resources, but provide a web service API as well.

The two snippets shown above give you an edit and update capability that restricts matching based off the HTTP method (verb). Writing a huge mess of those for the rest of the functions needed for a full web service API like Atom is a bit of busy-body work, so in the opinionated style of Rails a single command wraps up the whole thing. In Routes it looks like this:

map.resource('user')

That will make the two routes at the top of this entry in addition to routes that handle PUT, and DELETE. It maps them out to a set of actions in the controller, and provides the capability to easily add more methods for specific verbs.

The map.resource command is still getting tuned up, and we’re integrating the additional functionality it provides back into Pylons as well. Josh Triplett also wrote some Python code that will parse HTTP Accept headers fully so that we can add some nice functionality to use in the controller to return the appropriate data given what the client is expecting (HTML, XML, JSON, etc.)

If you’re using a Python web framework that doesn’t use Routes… maybe its time to put a request in. :)

It’s been a busy month for Pylons, with lots of changes for the big internal API change in 0.9. The great news for those making Pylons apps right now with 0.8.2 is how few backwards compatibility issues there are. Most of the big changes take place under the hood and compatibility objects are present to mitigate the massive breakage that I’ve seen happen in other framework upgrades.

This is going to remain a big focus on future development in Pylons too, and the API in 0.9 is solid enough that I don’t see anything but minor tweaks in the future (1.0 and beyond). Pylons based apps will be easy to maintain and upgrade thanks primarily to WSGI.

No NIH Syndrome Here!

We’ve taken some cues from the Django project that we believe make for a cleaner request-cycle. In Pylons 0.9, controllers are expected to return a response object, and for convenience a method is included that renders a template to a new response object and returns it (This will look very familiar to Djangonauts).

The command was integrated with a slightly more enhanced version of Buffet along with the Beaker Session/Caching middleware. The end-result is a powerful command that not only can render templates in any Template-Plugin compatible template language, but the rendered result can also be cached. It’s a great way to utilize template languages which might not be all that quick by themselves.

Sample controller in Pylons 0.9:

from myproj.lib.base import *

class UserController(BaseController):
    def show(self, id):
        # Cache based on id in the URL, for 30 seconds
        return render_response('/user/show.myt', cache_key=id, cache_expire=30)

    def index(self):
        # Just for fun, use Kid to render the index
        return render_response('kid', 'user.index', cache_expire=15)

Being able to easily cache any template in any template language makes it very easy to sprinkle in caching when you need to handle massive loads yet stay dynamic.

Another new feature present in the latest Routes and Pylons is resource mappings, which automatically generate routes for you with HTTP method restrictions. This makes it easy to setup controllers and their actions for specific HTTP verbs (aka, REST-ful URL’s and web services). The implementation of this feature was directly inspired by the Simply Restful Rails plug-in that was also demo’d by David Hansson in his Resources on Rails talk. Being able to discriminate valid routes based on HTTP method was brought up in the past and I’m happy to have seen an implementation that solves the issues I originally had with the idea.

The one feature still in development is to easily discern the content type requested, which was also inspired by Rails. Josh Triplett has written some code that deals with the ugly task of parsing the HTTP Accept headers, and I’m working on adding it into Paste for easy re-use by all. Combined with Routes, it’ll provide a clean and easy way to setup web applications that can serve multiple forms of content from a single action.

Conferences

Pylons was represented at EuroPython by the other lead developer of Pylons, James Gardner. For those still trying to grasp WSGI, he gave a talk on WSGI and middleware that looks like it was quite interesting (I wasn’t there unfortunately.)

Other talks involving Pylons at EuroPython:

• Dr. Rob Collins gave a presentation on Factory monitoring with Pylons, XML-RPC and SVG
• James Gardner’s full talk on Pylons, and slides of his Pylons talk
• Web Framework Shoot-out between Django, Zope, TurboGears, and Pylons (If more were covered, please let me know)

I’ll be at OSCON for those interested in chatting about Pylons, Python, or Python web frameworks later this month.

Pylons 0.9 Release

We’re currently wrapping up new features in 0.9, to make sure the resource mapping capability is present before a feature-freeze for release. Hopefully the release will be out within the next 2 weeks, if you haven’t checked out Pylons before I’d highly suggest taking a look at 0.9!