Groovie

Sometimes, it amazes me whats possible fully utilizing WSGI middleware in an application stack. While it likely isn’t something totally unique to the framework, the relative ease with which it can be done still sometimes gets me to grin.

Tonight, a Pylons user on the IRC channel (irc.freenode.net, #pylons) asked if it was possible to get a URL laid out so that /s/SOMETHING would map into their ’s’ controller, with the second part passed in as a variable. That alone is pretty easy, however the additional requirement was that the controller action would change depending on the user’s “type”.

There’s two ways to deal with this, the first of which is the only possible way in many frameworks. Have every request to the URL map to a single function, and in that function load up the session and call the appropriate function to handle the request based on their user type. This way works fine in Pylons too, but thanks to Routes and WSGI middleware we have another option.

Routes has a lot of capabilities to it, there’s been numerous additions to the Python implementation that the Rails version is not capable of. One of them is the ability to alter the resulting URL match dictionary in various conditional functions. To toggle the controller action used, we’ll be using the ability to pass in a function to Routes conditions that can alter the resulting match.

This condition checking function has full access to the WSGI environ so if you wanted to restrict a specific controller/action combination to people referred from Slashdot, no problem! You can carefully fine-tune the conditions required for dispatch at the same place you define your URL resolution.

Since Pylons uses Beaker for session handling via WSGI middleware, the session object will already be available when our Pylons app gets called. Beaker loads the user session into environ['beaker.session']. Given this knowledge, we can write a conditional function for use with Routes like so:

def check_user(environ, result):
    session = environ['beaker.session']
    user_type = session.get('type')
    if not user_type:
        result['action'] = 'index'
    elif user_type == 'admin':
        result['action'] = 'view_action'
    else:
        result['action'] = 'not_logged_in'
    return True
map.connect('s/:domain', controller='s', conditions=dict(function=check_user))

Viola! Now Routes will run the function provided to see if it returns True before accepting that as a valid match. In the process, the action used will be set as desired. I’ve always thought a good sign something is well designed is when people can use it in ways you didn’t originally anticipate. If that’s the criteria, I think Routes succeeds and then some.

Disclaimer: Yes, I wrote Routes, and a good chunk of Beaker and Pylons, so I might be biased and tooting my own horn. :)

In light of Phillip Eby’s recent post concerning WSGI Middleware as harmful, I’ve had more than a few thoughts on the issue. None of them are all that new, but given the post I think its useful to get some of them out there.

First, I agree 100% with PJE’s post. The issue it raises results in two lines of thought. Without a doubt these objects using the WSGI specification should not be called WSGI Middleware or WSGI Applications. This means that either: 1) People should stop using the WSGI spec for non WSGI application/middleware objects.

Or… 2) WSGI needs new terminology for this application of the specification, and should not be muddling up the WSGI middleware/application definitions and environ namespace with meta-framework API’s.

To answer either of these possibilities it helps to evaluate why things are developing like this right now, and almost all of it comes down to one thing… tool developers are incredibly picky and opinionated.

To avoid further muddling up WSGI definitions, I’ll be using the following term:

WSGI Framework Component (WFC) – A WSGI specification based component that acts possibly as either a WSGI application or WSGI middleware, or some mix of both. Example, a WFC that ensures users are logged in before accessing your WSGI application (thus acting as WSGI middleware), but will render its own form and go through its own login procedure should they need to login (thus acting like a WSGI application). This is referred to as a WFC because using WSGI is seen as a way to avoid binding it to a specific framework, while its clear that an application using it actually requires it to be there to operate (thus its not WSGI middleware).

Dealing with Disagreement the WSGI Way

WSGI makes it a lot easier to disagree, yet still harness the code and development efforts of those that disagreed with you. WFC’s allow re-usable code that isn’t utterly dependent on your framework of choice as long as its WSGI compatible. Thus the fact that many frameworks are WSGI compatible at various levels makes it very enticing to build re-usable components at the WSGI level instead of using framework-bound API’s.

A thought that started cropping up, and hitting the Web-SIG mail list, and which I believe one target of Phillip Eby’s post, regards putting ‘standard’ keys into the WSGI environ for applications to utilize. This would to an extent allow you to swap WFC’s that do similar things, but in different ways. Maybe you want to swap two resolving middleware, so you use the wsgi.org routing spec to determine how the URL was resolved then dispatch appropriately. You can now swap WFC’s that do routing to an extent since there’s a further specification in place.

There are other wsgi.org specifications underway, and lots of various WFC’s being developed. If I’m using Pylons, and someone using CleverHarold or some other WSGI type application makes a WFC that does something cool, I can use it as is without having to agree with the design of their application of WSGI based framework.

Compare that to a CherryPy2 filter, CherryPy3 tool, or Django middleware. To use any of those, I need to use the whole framework. For CherryPy, this may not be the case in the future should it allow a CherryPy tool to act in the middle like middleware. Robert Brewer has said in the past he wants CherryPy to be the end-point and not continue dispatching elsewhere which would rule out its use as a library for a WFC. Thus, I’m labeling CherryPy as a framework in the context of WFC creation, while Paste and Yaro are libraries usable both in WSGI apps/middleware and in WFC’s (Note that CherryPy3 is almost capable of being used as a WFC, except it can only dispatch to non-CP3 WSGI apps).

Going Overboard with WFC’s and WSGI

The other aspect to these new WFC’s that I think Phillip hit on the head, is that there’s quite a few being pushed into this layer that really don’t belong there. No one has put out a solid checklist to know when something should be in a library, a plugin API (possibly using setuptools entrypoints like the TG Template Plugin API), actual WSGI middleware, or a WFC. As a result, there are WFC’s that do very little, and in some cases have no reason to be operating at the WSGI layer.

So, I’m going to propose some guidelines, a rough draft as I’m sure there’ll be plenty of useful feedback, on when something should be considered for a WFC and when it should be a library. It’s also useful to note that libraries can operate on things from WSGI, vs WFC’s which get plugged into a framework/app as if it was WSGI middleware.

The guidelines for WFC’s should roughly follow the same guidelines you’d want for any WSGI middleware. There’s some conditions that make it more obvious than others on where some functionality belongs and of course there’s always exceptions to the rules.

Signs some code would be a good candidate for a WFC (It’s assumed that if you’re thinking of making a WFC, you will be wrapping your actual ‘application’ with it):

• A set of operations needs to always occur before and after the application is called, and requires knowledge of the incoming and outgoing headers
• Modifications are done to the HTTP headers and/or content being returned to the client (cookies, HTTP caching, content transformation)
• The application may not be called at all (authentication, authorization, conditional dispatching)

Note that the first condition doesn’t apply to functionality that merely requires something to setup. It’s overkill using WSGI just to run a function at the start of every request - even if it needs environ - there’s no reason you couldn’t just put the function call in your app, call it every request, and put the function in its own module/package (thus easy to re-use).

A lot of the Paste functions operate like this, and many of them just take the environ as their call giving you a nice API without requiring a WFC (which Phillip Eby advocates as well):

    request = paste.wsgiwrapper.WSGIRequest(environ)
    print request.cookies, request.path_info

There’s no reason a variety of WFC’s I see on the WSGI middleware and utils list couldn’t operate like this as well. Take wsgiakismet for example, which parses the form submission and screens it against Akismet. The example as a WFC actually looks more involved than I could see a library based version looking:

# theoretical library version of wsgiakismet
from akismetverify import verify_akismet

def app(environ, start_response):
    # Wordpress API Key and website name are required arguments
    usersub = verify_akismet(key='3489012ab121', site='http://blog.example.com/', environ)
    start_response('200 OK', [('Content-type', 'text/plain')])
    return ['Comment is %s' % usersub['comment'][0]]

Note that using it like this as a function that takes environ and the other 2 keys actually makes it easier to use than the original sample requiring you to import cgi and re-parse the form vars.

So some good ways to know you might be on the wrong track with a WFC:

• Only a few things are being done on setup, and stuffed into environ
• Some environ keys are manipulated
• Your code never alters or does anything with the status codes, headers, or content
• ... or none of the conditions to know when it should be a WFC exist

I’m sure there’s more criteria I’ve missed, and it’d be great to have a page possibly on wsgi.org regarding design decisions to hopefully avoid having anymore functionality pushed into the WSGI layer when there’s no good reason for it.

It’s with great pleasure that I announce the release of Pylons 0.9.4. This release has quite a few bug fixes and enhancements, the most since the 0.9 milestone. It’s also likely one of the last big updates before a 1.0 release candidate (there will be some small changes in 0.9.5 and possibly a 0.9.6).

First, the most important changes for those upgrading from an existing Pylons application:

• WARNING: Removed the lang_extract and lang_compile commands. They used pygettext.py and its associated msgfmt.py, which lacked the ability to extract ngettext style function calls and had issues with unicode strings. The new I18NToolBox project aims to provide this functionality (and more) via the gettext command line utilities. http://i18ntoolbox.ufsoft.org
• WARNING: Myghty’s allow_globals config var has changed, causing the following when running pre-compiled templates: Error(TypeError): do_run_component() takes exactly 13 non-keyword arguments (5 given) Delete the compiled Myghty templates directory (specified by cache_dir or myghty_data_dir in the config file) to resolve the error.
• WARNING: The localization function ’_’ now uses ugettext (returns unicode strings) instead of gettext. To preserve the old behavior, append the following line to your project’s lib.base and lib.helpers imports: from pylons.helpers import gettext as _
• WARNING: Removed 0.8.x legacy code and backwards compatibility functions.

Please note that since some i18n functions have moved, your helpers.py will need to be updated to import _, and ungettext from pylons.i18n.

Also:

- The XMLRPC Controller got a significant update so that it now provides the full range of XML-RPC Introspection facilities for your service methods.

- SQLAlchemy convenience functions have been added to pylons.database for use with the SessionContext plugin, and to create and retain SA engines.

- Paste dependency was updated to 1.1.1, Routes to 1.6.1 (important update for map.resource functionality)

- Pylons special objects (g, c, h, request, session) now available in interactive debugger without _attach_globals.

- Controller actions can now be generators

- Pylons base WSGI app uses wsgi.org routing_args spec for easier swapping of URL resolvers.

Install

Please see http://pylonshq.com/docs/0.9.4/install for installation details.

Full Changelog

• WARNING: Removed the lang_extract and lang_compile commands. They used pygettext.py and its associated msgfmt.py, which lacked the ability to extract ngettext style function calls and had issues with unicode strings. The new I18NToolBox project aims to provide this functionality (and more) via the gettext command line utilities. http://i18ntoolbox.ufsoft.org
• All Pylons special objects are now available within paster shell (not just h and g).
• WARNING: Myghty’s allow_globals config var has changed, causing the following when running pre-compiled templates: Error(TypeError): do_run_component() takes exactly 13 non-keyword arguments (5 given) Delete the compiled Myghty templates directory (specified by cache_dir or myghty_data_dir in the config file) to resolve the error.
• Changed i18n functions in templates to use proxy objects so that using set_lang in a template works right. Fixes #153.
• Now allowing any template plugin to overwrite global PYLONS_VARS (such as c, g), not just pylonsmyghty.
• Adding SQLAlchemy support to the database.py file. Saves the session engine to g to maintain it during the apps lifetime. Uses SessionContext plugin for management of the current session.
• Updated config object so that init_app can take an optional template engine argument to declare the default template engine.
• Updated Myghty plugin to use extra_vars_func when passed in.
• Fixed Buffet to use extra_vars_func properly.
• Fixed the validate decorator when there are validation errors and variable_decode=True: now passing the original params to htmlfill.render instead of the varable_decode’d version. Patch by FlimFlamMan.
• Added ungettext function for use with pluralized i18n, and the N_ function (gettext_noop) to mark global strings for translation. Added ungettext, N_ and translator objects to be globals for templates. Refs #126.
• WARNING: The localization function ’_’ now uses ugettext (returns unicode strings) instead of gettext. To preserve the old behavior, append the following line to your project’s lib.base and lib.helpers imports: from pylons.helpers import gettext as _
• Pylons special objects are now available within the interactive debugger (deprecating _attach_locals).
• Added setup-app run before unit tests run so that webapp has proper setup tasks handled. Fixes #113.
• Added paste.deploy.CONFIG setup to middleware.py, websetup.py and testing files in the Pylons project templates. Closes #112.
• Added security policy doc to index for use as Pylons security policy. Closes #91.
• Improved the repr() of the c context object to show attributes.
• Set environ[‘paste.testing_variables’] whenever that key is available, not just in testing mode.
• Added capability to have an action be a generator function.
• Added introspection capability to XMLRPCController and signature checking.
• Updated Controller to use additional arg lookup scheme so that the source of the function args for _inspect_call can be easily overridden.
• Updated RPCController, renamed to XMLRPCController. XMLRPCController now functions properly and will automatically return proper xmlrpc responses.
• Added test configuration ini file to default template. Closes #114.
• Fixed problem with pylons.database.PackageHub.get raising errors other than AttributeError when the database isn’t configured. Added new UnconfiguredConnectionError exception, instead of just KeyError or TypeError (depending on what part of the configuration failed).
• Fixed default g init, since bare object has no init method. Reported by Ian Bicking.
• Fixed issue with SQLObject method override having wrong name. Reported by climbus with patch. Fixes #133.
• Moved log function to pylons.helpers and translation functions to pylons.i18n. using pylons.util purely for Pylons internal util functions.
• WARNING: Removed 0.8.x legacy code and backwards compatibility functions.
• PylonsApp now has option to not use Routes middleware, default resolving uses new wsgi.org routing_args spec.
• Refactored routes dispatching to use new Routes middleware.
• Fixed paster shell command to properly acquire mapper object without relying on the template being configured in a specific manner.
• Added keyword argument pool_connection to pylons.database.PackageHub; if set to false then SQLObject connections won’t use pooled database connections (a new connection will be opened for each request).

Many thanks to Phil Jenvey, Ian Bicking, James Gardner, and all the other active members of the Pylons community!

Cheers, Ben

For those that haven’t gotten on the WSGI bandwagon, are still confused about Paste, how it fits into Pylons, and how its used in frameworks; I gave a Google Techtalk last week that hopefully can clear a few things up. The talk is now up on Google Video and the slides are available as PDF. You’ll want to get the slides and follow along if you’d like to read the code samples as the Google Video compression has turned them into large colorful blurs.

The main focus of the talk is on WSGI, with a bit on Paste and Pylons as well, and runs about 51 minutes. I could easily fill an hour or more just on Pylons, which I plan on doing at some point.

Disclaimers about the talk

• When I mention how the Rails routes code does extensive code generation, this mainly applied to the 1.0 and prior version of Rails routing. The Rails routing system got an overhaul around 1.1 that made it significantly easier to read, though that was also when the security bug in the routing crept in.

• Pony’s are valuable, though I don’t know why.

• Yes, I realize that hot pink and its friends are not the best presentation colors.

Enjoy!

This is slightly old as Routes 1.4 was released about a week and a half ago, but I thought it deserved some attention. There were a handful of fixes and some slightly major feature enhancements in 1.4.

From the changelog:

• Fixed bug with map.resource related to member methods, found in Rails version.
• Fixed bug with map.resource member methods not requiring a member id.
• Fixed bug related to handling keyword argument controller.
• Added map.resource command which can automatically generate a batch of routes intended to be used in a REST-ful manner by a web framework.
• Added URL generation handling for a ‘method’ argument. If ‘method’ is specified, it is not dropped and will be changed to ‘_method’ for use by the framework.
• Added conditions option to map.connect. Accepts a dict with optional keyword args ‘method’ or ‘function’. Method is a list of HTTP methods that are valid for the route. Function is a function that will be called with environ, matchdict where matchdict is the dict created by the URL match.
• Fixed redirect_to function for using absolute URL’s. redirect_to now passes all args to url_for, then passes the resulting URL to the redirect function. Reported by climbus.

Web Resources

The map.resource command is based directly off the Simply Restful Rails plugin which adds support for various verb-oriented controller actions in a RESTful service style approach. The Simply Restful layout is more or less the exact service style laid out in the Atom Publishing Protocol.

It’s a great approach and it also meant providing a few other features to Routes that I hadn’t implemented previously, the most important being able to limit matching of a URL based on the HTTP method used. This is present in the new conditions clause for a Route:

map.connect('user/:id', controller='user', action='edit', 
    conditions={'method', ['GET', 'HEAD']})
map.connect('user/:id', controller='user', action='update',
    conditions={'method', ['PUT']})

The conditions clause can also accept your own function should you want to restrict the route to matching based off some other criteria (sub-domain, IP address, etc).

def stop_comcast(environ, match):
    if 'comcast.net' in environ['REMOTE_HOST']:
        return False
    return True

map.connect(':controller/:action/:id', conditions={'function':stop_comcast})

David Heinemeier Hansson recently posted an entry about Resources on Rails discussing how important web services are. The other key point was to make it easier to write controllers that could not only give you easy browser access to your resources, but provide a web service API as well.

The two snippets shown above give you an edit and update capability that restricts matching based off the HTTP method (verb). Writing a huge mess of those for the rest of the functions needed for a full web service API like Atom is a bit of busy-body work, so in the opinionated style of Rails a single command wraps up the whole thing. In Routes it looks like this:

map.resource('user')

That will make the two routes at the top of this entry in addition to routes that handle PUT, and DELETE. It maps them out to a set of actions in the controller, and provides the capability to easily add more methods for specific verbs.

The map.resource command is still getting tuned up, and we’re integrating the additional functionality it provides back into Pylons as well. Josh Triplett also wrote some Python code that will parse HTTP Accept headers fully so that we can add some nice functionality to use in the controller to return the appropriate data given what the client is expecting (HTML, XML, JSON, etc.)

If you’re using a Python web framework that doesn’t use Routes… maybe its time to put a request in. :)