Handling Form data w/Formencode+SQLObject Redux 2
In a prior post on handling form data I covered how to easily populate a form using FormEncode/SQLObject and validate/save the data.
One of the things I noted was that I had to take the HTML form output first, and use htmlfill with it to populate it with defaults and errors before displaying. This required some extra lines I wasn’t to thrilled with in the controller. After an evening of chatting with Ian Bicking about whether FormEncode could somehow be cleaned up to make this easier, Ian suggested I use a Myghty feature I had forgotten about.
Component Calling with Content blocks in Myghty
Somewhat ironic since I use Myghty all day and Ian doesn’t (afaik). It’s a very useful feature called Component Calls with Content. It’s probably easiest to understand the cool ability this provides by seeing it. In Myghty, each template is known as a component, and complex arrangements can easily be put together by component inheritance and component calls between them.
Normal Python function calls and expressions in a Myghty component (template) looks like this:
Hi everyone, 2 + 2 = <% 2+2 %> and your name is <% lookup_name() %>
To include content from other components, you can either call the component through the function call m.comp('/some/template.myt') or you can use the component call syntax:
Hi everyone
<& /sidebar.myt &>
So here’s how we will use a component call with content:
Hi, lets translate the content under:
<&| /translate.myt, lang='es' &>
This entire block of content will be sent in as a single variable to translate.myt
for use. This includes any <b>HTML tags and such</b> as well.
</&>
And the translate.myt template might look like:
<p>The translated text is:</p>
<% translated(lang, content) %>
<%args>
lang
</%args>
<%init>
from mytranslater import translated
content = m.content()
</%init>
This is probably a lot to absorb, as it utilizes a few different Myghty concepts. Components can have arguments they expect, here the translate.myt component expects a lang arg which is passed in. The content block is then available as m.content() inside the template. The <%init> block is called first and variables defined there are available inside the template.
So how does this help us with the original problem?
Module Components
So far, all of these abilities are possible in Myghty’s ancestor, Mason. One more powerful construct available in Myghty however is the ability to not only call components, but Module Components.
I’ve been familiar with these for awhile as I use them for Controllers. It had never occurred to me though, that they can call any function in a module as if it was a component, not just ‘Controllers’.
Module components allow you to do a variety of interesting things, mainly, calling functions, classes, or objects that are in Python modules. Let’s take a look at the last translate.myt example using this approach instead.
Hi, lets translate the content under:
<&| MODULE:mylib:translate, lang='es' &>
This entire block of content will be sent in as a single variable to translate.myt
for use. This includes any <b>HTML tags and such</b> as well.
</&>
And the mylib Python module (assumed to be in the search path):
from mytranslater import translated
def translate(m, lang):
body = "<p>The translated text is:</p>"
body += translated(lang, m.content())
m.write(body)
Myghty will examine the function signature to determine what variables it wants and will search the current scope to make sure they’re passed in. Very handy. :)
Putting it All Together
So let’s see how this helps us out, first, rather than having to take the form and render it in the controller, we’ll push this into the template using a component call with content. So our new myform.myt looks like:
# myform.myt
<html>
<head><title>basic form</title></head>
<body>
<&| MODULE:mylib:formfill, defaults=defaults, errors=errors &>
<form action="/mypage" method="post">
Username: <input type="text" name="username" size="26" />
<form:error name="username">
Age: <input type="text" name="age" size="3" />
<form:error name="age">
<input type="submit" value="Send it" />
</form>
</&>
</body>
</html>
<%args>
defaults
errors
</%args>
mylib module:
from formencode import htmlfill
def formfill(m, defaults, errors):
form = m.content()
m.write(htmlfill.render(form, defaults, errors))
And the new controller:
from ourschema import UserInfoSchema
from oursqlstuff import UserInfo
def mypage(m):
errors, defaults = {}, m.request_args
if m.request_args:
form, errors = UserInfoSchema(), {}
try:
form_result = form.to_python(m.request_args)
except api.Invalid, e:
errors = e.unpack_errors()
if not errors:
UserInfo(**form_result) # database insert
return m.subexec('/thankyou.myt')
m.subexec('myform.myt', defaults=defaults, errors=errors)
We’ve cut it down 1 line, from 12 to 11, however we no longer have to go through the slightly awkward maneuver of capturing the rendered form and running htmlfill over it in the controller. This is now handled easily in the template using a component call with content. The module component we use is extremely portable and can be used to wrap any form anywhere with automatic fill-in of errors and default values.
FormEncode 0.4 also deprecates HTMLForm which we were using before, so this will help out when thats no longer available.
Module component calls are quite useful, and make it easy to group together powerful function libraries that interact with Myghty parts. Being able to pass them content allows all sorts of new constructs to be created.
If you’re using Myghty with FormEncode and SQLObject, I’d highly suggest switching to this method instead of the one I blogged about before.
Routes 1.0 Released 1
I’ve finally finished the documentation for Routes and as I mentioned earlier regarded releases am now ready for a 1.0 release. If you’re curious about Routes and want to get up to speed, I’d suggest jumping straight to the Routes Manual.
Routes is currently used in Myghty with the routes Paste template, and has been integrated for use both in Aquarium as well as CherryPy (though CherryPy 2.2 should allow better integration).
Now that Routes is feature-equivilant to the Rails version, its time to start planning for new stuff. The first and most obvious is to allow for more advanced configurability of URL’s by allowing for a new separator. This would allow you to get as creative as you like with URL’s, so you could do something like this:
m.connect('archives/:(article)-:(page).html', controller='blog', action='view')
m.connect('feeds/:section/:(format).xml', controller='feeds',
action='xml', format='atom')
This should make for a nice 1.1 feature. For those familiar with the Rails system of Routes, has there been anything you’ve found lacking or were just itching for?
How to Use Database Agnostic SQL in SQLObject 4
One of the advantages to using SQLObject is that the code you write in it can easily be constructed in a way that ensures it’ll work without a problem in all the databases SQLObject supports. This is a tremendous advantage that is most useful when writing web applications intended for wide-spread deployment on a variety of systems.
The most common SQL expressions you’ll likely want to use are Update, Select, and Delete. You can directly issue all of these in SQLObject using the connections sqlrepr method. The easiest way to see this is to play with it on the interactive prompt.
Before we begin, let’s setup a simple database layout using sqlite to try out the examples with:
from sqlobject import *
from sqlobject.sqlbuilder import *
from datetime import datetime
import sys, os
db_filename = os.path.abspath('data.db')
if os.path.exists(db_filename):
os.unlink(db_filename)
connection = connectionForURI('sqlite:' + db_filename)
sqlhub.processConnection = connection
class Comments(SQLObject):
name = StringCol(length=50)
date = DateTimeCol(default=datetime.now)
comment = StringCol()
Comments.createTable()
# Create some boring comments
Comments(name='fred', comment='Hello everyone')
Comments(name='joe', comment='Hi fred')
Comments(name='smith', comment='Good day')
SQLObject supplies a batch of classes for us that generate our database agnostic SQL expressions. While there isn’t too much documentation for using these classes (the mailing lists help), you can get a good idea where to start by looking at the help for them.
Using Select
If we want to take a look at the documentation for the Select class (which was imported above from sqlobject.sqlbuilder), we’ll get the following information:
Help on class Select in module sqlobject.sqlbuilder:
class Select(SQLExpression)
| Methods defined here:
|
| __init__(self, items, where=<class sqlobject.sqlbuilder.NoDefault>,
groupBy=<class sqlobject.sqlbuilder.NoDefault>,
having=<class sqlobject.sqlbuilder.NoDefault>,
orderBy=<class sqlobject.sqlbuilder.NoDefault>,
limit=<class sqlobject.sqlbuilder.NoDefault>)
|
| __sqlrepr__(self, db)
First, we need to pull the connection used for the class. We will then use the connections sqlrepr method to construct our SQL, and the connections query method to actually run it. Let’s take a look at getting all the names only from our Comments:
conn = Comments._connection
nameselect = conn.sqlrepr(Select(Comments.q.name))
results = conn.queryAll(nameselect)
# >>> results
# [('fred',), ('joe',), ('smith',)]
This will populate results with an array of tuples, one tuple for each result with the tuple values in the order you specified for the select (it’d be nice to have a way to get dicts instead…). Let’s take a look at a few more examples of using Select:
fields = [Comments.q.name, Comments.q.date]
namedateselect = conn.sqlrepr(Select(fields, where=(Comments.q.date < datetime.now() )))
results = conn.queryAll(namedateselect)
# >>> results
# [('fred', <DateTime object for '2005-11-01 11:14:39.00' at 843ea68>),
# ('joe', <DateTime object for '2005-11-01 11:21:33.00' at 843ea30>),
# ('smith', <DateTime object for '2005-11-01 11:21:33.00' at 826a3a0>)]
namedateselect = conn.sqlrepr(Select(fields, where=(Comments.q.date < datetime.now()),
limit=2, orderBy=Comments.q.date))
results = conn.queryAll(namedateselect)
# >>> results
# [('fred', <DateTime object for '2005-11-01 11:14:39.00' at 843ea30>),
# ('smith', <DateTime object for '2005-11-01 11:21:33.00' at 826a3a0>)]
Joins, and additional fields can be specified using the normal Class.q notation to let SQLObject generate the proper SQL necessary in the same manner as the documentation explains.
Updating Fields
Doing a large update of a sub-set of fields is definitely something best left to a manual Update command. First, let’s take a look at what the help for the Update indicates:
Help on class Update in module sqlobject.sqlbuilder:
class Update(SQLExpression)
| Methods defined here:
|
| __init__(self, table, values, template=<class sqlobject.sqlbuilder.NoDefault>,
where=<class sqlobject.sqlbuilder.NoDefault>)
|
| __sqlrepr__(self, db)
I’ll admit right now I’m not actually sure what template is for, nor have I used that keyword argument. If someone would like to chime in on the comments, that’d be appreciated greatly.
Updating the table gets a little tricky since we need to specify all of the fields in a database agnostic manner. To avoid very long statements, I’ve broken it down into sections to build the query.
Let’s look at updating all the dates of our Comments table:
datecol = Comments.q.date.fieldName
updatedates = conn.sqlrepr(Update(Comments.q, {datecol:datetime.now()}))
conn.debug = True # So we can see the query execute
conn.query(updatedates)
conn.cache.clear()
# >>> conn.query(updatedates)
# 1/Query : UPDATE comments SET date='2005-11-01 11:57:31'
# 1/QueryR : UPDATE comments SET date='2005-11-01 11:57:31'
# 1/COMMIT : auto
Updating multiple fields is just as easy, merely add more key/val’s to the dict you pass in for the values variable. To update values using the original value of the field in the update, ie, adding something to the existing field we specify that using the Class.q.field format used in where clauses. Also, note that we need to clear the object cache after running the update so that SQLObject fetches the row again before using it.
updatedates = conn.sqlrepr(Update(Comments.q, {datecol:Comments.q.date + 20}))
conn.query(updatedates)
conn.cache.clear()
# >>> conn.query(updatedates)
# 1/Query : UPDATE comments SET date=(comments.date + 20)
# 1/QueryR : UPDATE comments SET date=(comments.date + 20)
# 1/COMMIT : auto
This adds 20 seconds to the existing dates for all the rows. Using the .q notation with the class is necessary for the key value because we need to ensures that Python doesn’t try to add 20 to a string which is what it would’ve tried if we had said {datecol:'date' + 20}.
Deleting
Issuing a Delete on the database is very similar to the update command, the class help looks like this:
class Delete(SQLExpression)
| To be safe, this will signal an error if there is no where clause,
| unless you pass in where=None to the constructor.
|
| Methods defined here:
|
| __init__(self, table, where=<class sqlobject.sqlbuilder.NoDefault>)
By now, the signature for the init method should be fairly familiar as well as what input’s its expecting. Here’s a quick example:
delquery = conn.sqlrepr(Delete(Comments.q, where=(Comments.q.name == 'smith')))
conn.query(delquery)
# >>> conn.query(delquery)
# 2/Query : DELETE FROM comments WHERE (comments.name = 'smith')
# 2/QueryR : DELETE FROM comments WHERE (comments.name = 'smith')
# 2/COMMIT : auto
That’s pretty much all there is to deleting, probably the easiest operation to do with a SQLExpression class.
Transactions, Notes, and Gotchas
When using these techniques in large programs, it can be tricky to ensure that the object cache is cleared out and up to date. If you’re going to use a lot of these commands extensively it might be prudent to turn cacheValues off, or wrap the commands in a function that calls the cache.clear() command.
Using transactions (not all databases support it) can still be done if you want to wrap a batch of these manual expressions into a single transaction. You just need to start the transaction and commit it when done:
trans = conn.transaction()
delquery = conn.sqlrepr(Delete(Comments.q, where=(Comments.q.name == 'smith')))
conn.query(delquery)
updatedates = conn.sqlrepr(Update(Comments.q, {datecol:Comments.q.date + 20}))
conn.query(updatedates)
trans.commit()
If you’d like to use database functions (bottom of the SQLBuilder docs), this is easy to pass in as well but since they’re more database specific you begin to lose portability.
Please feel free to contribute any experiences or further examples of working with SQLObject expressions in a manual fashion as I’ve described here.
Python Web Framework Niches 13
I’ve come to the belief lately that the web frameworks available in Python are increasingly fine-tuned to specific application requirements. Of course, anyone reading the ‘About’ sections for these frameworks should realize this as well. I wonder how many people actually read that section as I’ve seen people latch onto web frameworks without knowing the task it was originally made for.
Without knowing the reason the framework was created, its common for many people to leap to the conclusion that its another Rails wanna-be just because its a ‘full-stack’ web framework. I was playing around with a nice full-stack framework called WebObjects years ago which made it easy to setup database objects, generate CRUD, etc. Zope’s been doing the same stuff for what now seems like eons as well, yet I don’t see people declaring RoR a Zope clone (It obviously isn’t).
In light of that, I’m inclined to agree with Ian Bicking’s response about the lessons Python web people did learn from RoR.
The concept I want to focus on is that people create these new frameworks because they make their task easier than any of the other frameworks already out there. While they might pick up features from other frameworks, most of them aren’t aspiring to be “Python on Rails”. Sometimes this task is easier when other tools can be integrated to avoid code replication, as is the case in one framework I cover here.
Many people have declared the amount of Python web frameworks a “problem” that should be “solved” somehow, perhaps a Highlander fight with swords to the death (There can be only one!). I’d like to suggest the opposite, there’s a lot of Python programmers and I think there’s room for even more web frameworks. The variety is a strength because they make it easier to get specific web applications done.
TurboGears
The TurboGears site has a nice about page describing its purpose, though I feel it doesn’t completely explain the rationale for its creation. There’s some interesting and unique decisions made in TurboGears, like using Kid instead of Cheetah or Myghty for templating. Then there’s the inclusion of Mochikit and the TurboGears decorators for returning output as JSON for use with Mochikit.
So what kind of applications is this web framework geared for? (Please excuse the pun)
The best way to answer this is to look at the application this framework was created for, Zesty News, and the abilities of some of the tools being used. Zesty News is in a rather interesting category of web applications in that the end-users themselves will be installing it, quite likely on their home computer rather than a server. Being able to package it up and easily distribute/upgrade it becomes a key issue along with database portability and code thats database agnostic.
Two tools assist here, setuptools for distribution/packaging and SQLObject for portable database code. Zesty News deals extensively with RSS and XML, so it makes sense that the templating language chosen was actually created for dealing with web services.
These design decisions behind TurboGears should make it fairly obvious when to consider it for your next project. The cohesive toolset you get when you choose TurboGears is ideal for developing portable, easily deployable AJAX-enabled web applications that likely deal with XML frequently and need to stay database agnostic. Even if your web application doesn’t deal with XML frequently, the decisions TurboGears makes for AJAX integration will make it easy to add heavy dynamic interaction to a TG webapp.
Django
Django was created to deal with the requirements of working in the web development department of a news publisher. As such, the framework was created specifically to deal with the requirements placed upon the author. What’s rather interesting is the lack of re-use in Django when it comes to doing things that have been done before in other projects (Database mapper, form validation, etc).
The tools and parts of Django were specifically built to work as one package, and using Django makes that very obvious. One of the things most common when in a newsroom or publishing environment is dealing with CRUD. That is, there is a lot of content and ways to get content into the system and administrate the content is a high priority. As a web framework built for dealing with Content, many of the design decisions reflect the common tasks present in CMS’s (Content Management System).
To start with, you get a slick administration interface for your conntent, that’s miles beyond any of the generated CRUD type stuff in other web frameworks. This differs from the philosophy of other web frameworks that give you basic CRUD (Scaffolding in Rails-speak) in that Django’s admin interface is aimed directly at being production-ready with no modification at all.
Django also makes it fairly easy to make a Django ‘application’ like a Forum or Blog, then slot it into other Django application environments. Again, this makes a lot of sense given the original requirements placed on the creator of Django. If a company has 4 websites, and wants them all to have the new Forum/Classified ability it makes a lot of sense for this task to be optimized.
So what web applications are you going to want to use Django for?
Quite a few, as it turns out dealing with Content is a very common task. If you’re writing a web application heavy on content, that needs a full featured web interface for managing the content it’d be really hard not to recommend Django. It’s easy to get started, and in almost no time you have very powerful functionality running that gives you a lot of usability.
Don’t be Everything for Everyone
Part of the reason I picked these two projects to talk about is that they’re both extracted from a working project (as Rails was). I also haven’t seen many people mention the fact that frameworks developed in such a manner are also inherently going to be optimized for the use-cases that brought them into existence.
Open-sourcing the project lets them grow to an extent, but their design is largely baked in and a useful limitation. Too much expansion past the initial design requirement will make them generic, and with that comes a lot of complexity (sometimes worth it though for the extra re-usability).
Note that the specific things Django gives you don’t help that much if you’re trying to write a Zesty News style application. The same goes in reverse as well, since building an Admin interface of your own isn’t fun and can be time consuming. While it’s possible to make web applications that do this in either framework, compensating for framework design will require extra time when you try to use one framework for everything.
If you’re using one framework for everything, maybe its time to take a look around.
WSGI and WSGI Middleware is Easy 2
Really, its quite simple.
If you already knew that, this post isn’t going to enlighten you at all. If WSGI had been one of those things you kept saying to yourself, “Oh, yea, I’ll learn that someday”, consider this a short intro to all the info a web developer and to some extent a framework author, will likely care about.
All the gory details of WSGI along with a nice overview are available in PEP 333. As I’m mainly interested in it as a web developer, or as the PEP refers to it, a “application developer”, I’ll be focusing on that aspect.
Basic Usage
WSGI is intended to bridge the gap in deploying Python web applications that might use different frameworks in a more uniform manner. A web application that implements the WSGI interface can be called in the same way as any other web application that implements WSGI. The basic call of a web application using WSGI is quite straight forward (example straight from the PEP):
def simple_app(environ, start_response):
"""Simplest possible application object"""
status = '200 OK'
response_headers = [('Content-type','text/plain')]
start_response(status, response_headers)
return ['Hello world!\n']
That really shouldn’t scare anyone who remembers the ‘good’ old days when your CGI scripts had to write every little bit of the content to the web browser. environ is merely a dict thats loaded with some common CGI environment variables that anyone who has worked with CGI applications should recognize. The start_response is a Callable that the web application uses to start the response, send the status, as well as the typical response headers.
Pretty basic stuff so far, but as web developers, we’re not even likely to need to know that much. This is because we don’t need to muck around with setting the status and dumping our content out, our web framework takes care of it for us (the majority of which now support WSGI).
WSGI Middleware
Despite the acronym and somewhat ambiguous ‘Middleware’ term, WSGI Middleware isn’t any big deal either. Take a good look at the code fragment above… its just a function. It could be any Callable, since the server (which we don’t care about) is just going to call it. We could wrap that Callable inside another one without much hassle. I could even show you an example of some WSGI middleware code, but I’ll leave that for you to explore.
The thing to remember here, is that your entire web application can be called with that one simple command up above. As such, other functions or classes can wrap around your web application object and do other things before and/or after your web application gets called. That’s really all WSGI middleware is.
There is currently WSGI middleware that will:- Handle web application errors
- Provide session support
- Profile your web application
- Deal with Login authentication
- and Gzip the output
from paste.exceptions.errormiddleware import ErrorMiddleware
wrapped_app = ErrorMiddleware(simple_app, global_conf, debug=False,
error_email='fred@example.com',smtp_server='localhost')
That’s it, we’ve now used WSGI Middleware. If you wanted to add profiling on top of that, you’d just pass the wrapped_app in as an argument to another WSGI Middleware class.
Now instead of telling your server to use simple_app as your WSGI web application, you just have it use wrapped_app. Notice that we just passed in simple_app as an argument to the ErrorMiddleware class. global_conf is a dict that Paste builds for us based on the Paste config file, so we don’t have to worry about that here either. The other 3 keyword arguments are visible in the ErrorMiddleware docs and I think its pretty obvious what they all are for.
Other WSGI Middleware, like the session middleware, will add extra keys to the environ dict that gets passed into your web application. That way you can get to objects or variables the middleware has setup for your use. There’s nothing hidden or mysterious here, just one Callable calling another.
PEP 333 calls this the “middleware stack”. I like to think of it as function wrappers, or object wrappers since each one is a Callable. We’re just passing an object into a new object as an argument, and so on.
Running a WSGI Callable
Ah yes you’re wondering, great, we got this function that takes two arguments and throws out the response, now how do we make it run? There’s several packages that adapt a WSGI Callable to some more common forms of running a web application.
Probably the easiest of these to use, and the one that Paste has in it, is the flup package. It contains several different WSGI servers that will take your WSGI Callable, and make it available either as Fast CGI, SCGI, or AJP. I’d highly suggest checking out that page as it also contains an example of using several bits of WSGI Middleware back to back.
If you were hesitant or unsure what WSGI and WSGI Middleware was before, hopefully this post has helped. It really is pretty basic, and by putting common tasks like error handling into WSGI Middleware it can help people using other frameworks avoid repeating existing work.
