Groovie

Finally pushed out release 0.8 of Pylons, a bunch of great features bundled up in a slick package that makes web development easy and brings re-use to new levels. I’m not going to repeat the full announcement here, but I’d suggest checking it out if you’re the slightest bit interested in a framework leveraging WSGI for re-use and ease of use.

Now that 0.8 is out, its already time to get to work on 0.9 which is going to have a slightly more stream-lined WSGI stack but other than that won’t be very different from 0.8 from a usability perspective.

Ian Bicking noted that I was considering building the next Pylons on RhubarbTart which is sort of true. Julian Krause, Ian and myself are working on WSGI and associated components that will find themselves in all of our “frameworks” and most likely quite a few others too. That’s really one of the best things about writing solid pieces of web code in a more component style vs. a hulking framework, portability is great and the re-use is fantastic.

So far, Routes has found itself embedded in at least a half-dozen different frameworks and I’ve heard some rumors its to be ported to PHP for the Zend framework as well. The WebHelpers package is also picking up users and is being used by some Djangonauts which is great as Django is generally a fairly ‘helper’ friendly framework.

Ah yes, back to the topic, Pylons has tied together these pieces of code in a very seamless fashion that make it easy to write what you want without forcing you down a path that may confine you later. On the other hand, Pylons currently doesn’t have some of the friendlier elements that frameworks like TurboGears and Django both have. Such as a pretty Admin interface, or an extensive Toolbox app, but that stuff is in the works where appropriate.

Our main desire with this release was to get solid footing for the integration of these parts, and a good architecture for building web applications. To that end, we think Pylons is a very solid performer and will adapt easily to a wide variety of needs. You’ll also get a framework that makes it a snap to plug-in new middleware whenever something striking your fancy appears on the radar, and you can keep using the parts you like best, even if you stop using Pylons.

I got Routes 1.2 out the door today, it’s a fairly important update for 2 reasons:

• A bug crept in with 1.1 using the default controller directory scanner. The scanner wasn’t properly retaining the directory prefix which causes mismatches when using controllers underneath a sub-directory.
• url_for can (and should) be used for all your URL needs, including static files

The second one is pretty important if you’re at all interested in creating portable web applications that can be used along-side other applications. While some frameworks provide generation of URL’s that lead to other web pages, hard-linking the other ‘static’ content will cause problems if you try and use the application under a different mount point.

Instead of using a url like /css/source.css it should instead be generated with url_for('/css/source.css'). This way Routes can ensure that if you’re running under a WSGI environment and there’s a SCRIPT_NAME present (this indicates the applications location), it will be pre-pended to your absolute URL’s. When used like this, additional keyword arguments passed in will be used as query args on the URL making url_for a handy way to create URL’s that are properly URL encoded.

Another useful feature that made it into 1.2 allows you to alias URL’s you might want to use throughout your web application, I call these static named routes. An example can be found in the named routes section of the Routes Manual.

Enjoy!

Ruby’s anonymous block capability is probably the main feature I find myself wishing Python had on more than one occasion. While the upcoming Python 2.5 PEP 323 provides for the “with” statement which will enhance generators to get a bit closer. Though as Ryan Tomayko notes, this still doesn’t make the block available within the “generator”.

There are of course, many uses for having the block (with its closure scope), available in the function/generator you’re using. I’ve seen this used to easily register code call-backs with a CleanUp/Initializaiton manager, and other cases where its preferable to actually retain the block in its entirety for later execution. It’s also very useful when you want the function to control execution of the block, and return its output in a modified form.

In these ways, it would appear that the generator enhancements won’t quite be bringing the full power of Ruby anonymous blocks to Python. However, I recently found out that a relatively recent feature in my favorite Python template language, Myghty, implements something quite close to this.

Myghty’s Component Calls with Content

In a previous entry on Formencode and SQLObject I noted how useful the component call with content can be. What I failed to note (I didn’t know at the time as well), was that not only is the function/component able to get the content of the content “block”, but its also able to execute it again and again.

Consider this example from my prior entry:

Hi, lets translate the content under:
<&| MODULE:mylib:translate, lang='es' &>
    This entire block of content will be sent in as a 
    single variable to translate.myt
    for use. This includes any <b>HTML tags and such</b> as well.
</&>

and the function it calls:

from mytranslater import translated

def translate(m, lang):
    body = "<p>The translated text is:</p>" 
    body += translated(lang, m.content())
    m.write(body)

The m.content() call can be called as many times as you want the output of it, and it retains the scope of its original location. This in many ways emulates how Ruby can yield to the block and capture its output, however it is not possible to stash the block itself (in Myghty/Python).

Next up is to actually be able to use values from the function inside the block. In Ruby this is done in a very elegant fashion letting you declare what you’ll call them, then use them, like so:

SomelistOfints.each do |item|
    item += 2
end

To emulate this behavior, our block has to make a special function call and know the name of the value in advance. Here’s an example:

% randomvar = 423
<&| MODULE:mylib:translate, lang='es' &>
    This entire block of content will be sent in 
    as a single variable to translate.myt for use. 
    This includes any <b>HTML tags and such</b> as
    well. Here's something supplied by
    translate: <% m.content_args['value'] %>.
    Of course, <% randomvar %> is still in scope too.
</&>

And just for fun, our translate function will call the block with some different values:

def translate(m, lang):
    body = "<p>The translated and repeated text is:</p>" 
    for val in range(0,4):
        body += m.content(value=val)
    body += "<p>That's it, nothing else.</p>" 
    m.write(body)

The important thing to remember that makes the concept powerful is that the block above is called in the scope you saw it in the template. Whatever variables were available there are used as normal.

Close…

This method brings us close to Ruby’s anonymous blocks, as close as might be possible in Python. Unfortunately its only usable within Myghty (if not, please let me know), and its still not true anonymous blocks. At the very least, its close enough to make me happy for now. While I could just switch to Ruby entirely, there’s still way too many things about Python that I’d miss.

From the details of PEP 343, it appears that this full capability to pass the block in was purposely avoided as having flow control in the “macro” makes the code inscrutable. Hopefully someday the utility and power such functionality provides will result in it being available in Python. Or at the very least, some clever person can try a variant in PyPy and see what sticks.

And yes, I know nested functions can be used and passed around with their closure, but its pretty annoying to be nesting functions back and forth solely for that purpose. It feels like all we have is a hammer…

Further reference:

Myghty docs on Component-Call-With-Content Arguments

I’ve released Routes 1.1 today after extending the unit tests for the new Groupings syntax and updating the docs.

The new syntax is quite powerful and will hopefully make everyone using Routes rather happy. While I’m not about to encourage anyone to use URL’s with .html at the end, there’s plenty of times when you want extensions to mingle with dynamic parts. You can also get some useful abilities like being able to pull out the extension like so:

map.connect('archives/:category/:(section).:(format)', controller='archive', action='by_extension')

This makes it easy to toggle the response depending on the extension, and the regexp business is handled for you.

Integration Enhancements

An additional feature, suggested by a Routes user was to make integration easier in WSGI environments. Earlier, at the beginning of each request you would have to populate the Routes config with the host, protocol, and match result. Now, merely passing the WSGI environ to the Routes config object will run a match, and populate those attributes for you.

The Routes Mapper now can take a function that when called returns a list of valid controllers. If you want to use the directory scanner Routes comes with, all you need to do is pass in the directory you’d like to scan and Routes will scan it for you.

These two new integration improvements make it rather simple to integrate Routes, here’s a basic WSGI app showing this off:

#myapp.py

from routes import *

map = Mapper(directory='/my/directory/of/controllers')
map.connect(':controller/:action/:id')
map.connect('home', '', controller='home', action='splash')

class WSGIApp(object):
    def __init__(self, mapper=map):
        self.mapper = mapper

    def __call__(self, environ, start_response):
        config = request_config()
        config.mapper = self.mapper
        config.environ = environ

        if not config.mapper_dict:
            start_response("404 Not Found", [("content-type","text/html")])
            return ["No match"]
        else:
            start_response("200 OK", [("content-type","text/html")])
            return ["Match with the following dict: %s" % str(config.mapper_dict)]

That’s it! If you’re not using WSGI, there’s been no backwards breakage so the old style of setting up all the attributes of the config will work fine as well.

So, now I have to figure out if there’s anything else Routes should possibly have… or is the only space for improvement at this point further optimization and perhaps usability improvements?

The major feature I was waiting on for Routes 1.1 is for the most part done, mainly adding more unit tests for the new syntax now. As I mentioned previously when announcing Routes 1.0, this feature is the one quite a few people have been waiting for. The ability to split the route path somewhere other than on a /.

Here’s what a few Routes using the new feature look like:

map.connect(':category/:(page).html', controller='stories', action='view')

map.connect('feeds/:(section).:(extension)', controller='feeds', action='formatter')

map.connect('archives/:(year):(month):(day).html', controller='archives', action='view',
          requirements=dict(year=r'\d{4}',month=r'\d{2}',day=r'\d{2}'))

The new section dividers, :(something) can be used side-by-side as the last example above shows, however in such cases each path part must have a rigid regexp requirement placed on it to ensure proper collection of the variables. Typically you will have some characters in between each dynamic part so this issue doesn’t arise.

I’ve retained full backward compatibility with the old syntax as well, if you don’t designate the dynamic part using the () modifier it will fall back to looking for the next / instead. So far, all the existing (and extensive) unit tests are passing, in addition to new tests I’ve been adding.

Routes generates a full regular expression for URL matching based off the route path you give it. This makes it a great way to setup URL routing with the power of regexp’s yet avoiding the hassle of writing a large complex regexp yourself. The other very powerful ability you gain is that Routes can turn the keywords back into the URL, like so:

>>> url_for(controller='archives', action='view', year=2004, month=10, day=12)
'/archives/20041012.html'

>>> url_for(controller='feeds',action='formatter',section='computers',extension='xml')
'/feeds/computers.xml'

If you’re interested in giving it a spin, it can be checked out and installed easily from the svn repository using setuptools:

sudo easy_install -U http://routes.groovie.org/svn/branches/newsplit

Feedback / Suggestions / Bug Reports greatly appreciated.