Comments and Web Services
I’m trying out Disqus for comments on the blog, for one main reason, I just didn’t feel like implementing comments myself. I am somewhat wary of services that seem integral to a blog, like where the comments are, which is why I’ve been leery of using services like this for some time.
What happens if Disqus runs out of VC and goes belly-up? I’ve seen this with a few other web services, and no one using it is very happy when it occurs. On the other hand, I did find some irony in the fact that one of the features Disqus pitches users is, “Don’t lose your comments if the blog disappears”.
On the other hand, I really appreciate the capabilities a central service-based comment system brings. I’ve inadvertently used it on other blogs, and was very pleasantly surprised to get updates and actually be able to keep up on what was happening on the blogs I commented on.
Are other people worried about using services by new companies or am I just overly paranoid?
I’d almost feel better about it if I could pay five bucks a year or something, as I’d at least have some better reassurance that the company is actually making money like I do with Flickr. Or maybe it’d be nice if companies that are profitable to say so, though it occurs to me that even that isn’t a guarantee, as someone could come along and buy them up, then decide to terminate their functionality (like that one site that people used and were pissed about when Six Apart bought them and shut it down, but for some reason I can’t remember the name of it at all).
Blog upgraded! Now running on Pylons + MongoDB
I’ve now updated the blog software powering my blog, which is very long overdue. In the past, this blog was run off Typo, which apparently now hosts their home site off the github (its moved a dozen times in the past 3+ years).
Typo always worked moderately well for me, however, I found it sluggish (Prolly Rails there), and incredibly horrid on ram usage. It was not at all unusual to see it running past 700 megs of RAM after running for just a few weeks, which is a bit annoying as the machine only has 4GB total and is running quite a few things.
After last weeks SF Python Meetup on MongoDB I figured it was time to get a little actual MongoDB usage under my belt. I also inadvertently implemented enough of the MovableType XMLRPC API as I didn’t want my app to be too extensive, just enough for me to post to my blog.
So in the end, I had a small set of requirements for the replacement:
- Not be horribly slow
- Not take up huge amounts of RAM
- Retain all existing URL’s (It really annoys me when people break their old links)
- Compatible with my blog software (MovableType / MetaWeblog XMLRPC API)
- Not screw up the RSS/Atom feeds and cause Planet Python to show all my posts as if they were new (I’ve seen this happen to a few people on occasion)
I wanted to build it myself, because of course, the world definitely needs more blog apps, and I wanted one that used MongoDB. So for those curious, here’s the source code to the blog app.
It’s rather rough, as its fairly custom built just for my needs, nor do I have any plans to expand it into some general purpose blog engine, with themes, etc. The only other thing pending at the moment is to add the ability to comment again, as I haven’t quite gotten that feature in yet. For those trying it out, the README should help get started, but its very rough (thus the name of the package).
Strings, Unidecode, and Slugs
When copying some functionality I needed from the Rails app (to retain URL compatibility), I noticed two things it did which I thought was handy. To convert a title into slug for the article, it used a fairly sophisticated scheme relying on two other packages.
First, was the use of a Ruby port of a Perl package called Text::Unidecode which is pretty cool, and converts UTF-8 chars into their closest ASCII version. I figured someone must’ve ported it to Python as well, and sure enough, someone did! It wasn’t on the cheeseshop though, which was unbearable for me, so I’ve posted it to the Cheeseshop so others can easy_install it.
Next up, was a Ruby library called stringex, which add’s a few things to Rails, including a string method called 'to_url’. That method does a variety of transformations to remove all those funny characters from a title, and do a bunch of other neat changes of common characters to human readable versions (source for those conversions).
I ported the key module of stringex to Python, and it resides in my blog app. If someone would like to extract it and make it into its own package, or even better, if I somehow missed the fact that someone else has ported it already, let me know (tweet me @benbangert!).
I’ll be writing up my thoughts on making a small app with MongoDB, and how it differs from my experience working with CouchDB for PylonsHQ in a later post for those curious.
Making a better TextArea
I’ve been working on some Javascript to enhance the TextArea elements on the PylonsHQ Snippets section, and have noticed that… well, TextArea’s suck.
The hack I’ve seen is to use one of the newer features of browsers, the editable or 'design’ attribute’s for div elements I believe. This lets one build a very snazzy amount of features, such as syntax highlighting, code completion, etc., but I don’t think I needed to go that far.
I only have one main design goal, this TextArea is for the user to enter RestructuredText so it’d be awesome if the TextArea acted in a way that made rst a bit nicer. The obvious two things that came to mind:
- Tab key indents 4-spaces
- Hitting return on an indented line, will retain the indentation on the next line
I’ve actually gotten some Javascript, hobbled together from various parts of the net, along with an 'enter’ key handler I wrote myself on bitbucket.
Course, it’d also be nice to have a button or key combo, that will indent/unindent a selection in the TextArea as well.
Anyone else have any Javascript they’ve hobbled together in the past to make TextArea a little nicer for restructured text?
Beaker 1.3 is juicy caching goodness
Beaker 1.3 is out, actually, its been out for awhile and I’m just not getting around to blogging the fact. It’s a shame I’ve been a bit too busy lately to blog this earlier because in addition to some bug fixes it has some nice new features that make it even easier to use in any Python script/application/framework.
First, to air my dirty laundry, the important bug fixes in Beaker 1.3:
- Fixed bug with (non-cookie-only) sessions not being timed out
- Fixed bug with cookie-only sessions sending cookies when they weren’t supposed to
- Fixed bug with non-auto sessions not properly storing their last accessed time
The worst thing with the first two of these is that they were regressions that snuck in despite unit tests that exercised the code fairly decently. They’re fixed now along with more comprehensive tests to help prevent such regressions occurring again.
Beaker has always had session’s, and caching, but except for Pylons I’ve yet to see anyone actually use Beaker’s caching utility. I’ve seen the SessionMiddleware used in other WSGI based frameworks, but not the caching, which is kind of a shame since it:
- Supports various backends: database, file-based, dbm file-based, memcached
- Has locking code to ensure a single-writer, multiple reader model (This avoids the dreaded dog-pile effect that caching systems such as the one in Django experience!)
For clients that hit the cached function while its already being regenerated, Beaker serves the old copy until the new content is ready. This avoids the dog-pile effect, and keeps the site snappy for as many users as possible. Since the lock used is disk-based though, this does mean you only avoid the effect per machine (unless you’re locking against NFS or a SAN), so if you have 20 machines in a cluster, the worst the dog-pile effect can get is that you’ll have 20 new copies generated and stored.
Now, in Beaker 1.3, to try and encourage its use a bit more, I’ve added a few decorators to make it easier to cache function results. Also with Mike Bayer’s suggestion, there is now cache regions to make it easier to define various caching policy short-cuts.
Cache Regions
Cache regions are just pre-defined sets of cache instructions to make it easier to use with your code. For example many people have a few common types of cache parameters they want to use:
- Long-term, likely to a database back-end (if used in a cluster)
- Short-term, not cached as long, perhaps to memcached
To set these up, just tell Beaker that about the regions you’re going to define, and give them the normal Beaker cache parameters for each region. For example, in this Pylons app, I define 2 cache regions in the INI:
beaker.cache.regions = short_term, long_term
beaker.cache.short_term.type = ext:memcached
beaker.cache.short_term.url = 127.0.0.1:11211
beaker.cache.short_term.expire = 3600
beaker.cache.long_term.type = file
beaker.cache.long_term.expire = 86400
Note: For those wondering about multiple memcached servers, just put them in as the url with a semi-colon separating them.
If you want to use the caching outside of Pylons without middleware (ie, as a plain library), that’s a bit easier now as well:
from beaker.cache import CacheManager
from beaker.util import parse_cache_config_options
cache_opts = {'cache.data_dir’: './cache’,
'cache.type’: 'file’,
'cache.regions’: 'short_term’, 'long_term’,
'cache.short_term.type’: 'ext:memcached’,
'cache.short_term.url’: ’127.0.0.1:11211’,
'cache.short_term.expire’: '3600’,
'cache.long_term.type’: 'file’,
'cache.long_term.expire’: '86400’,
}
cache = CachManager(**parse_cache_config_options(cache_opts))
And your cache instance is now ready to use. Note that using this cache object is thread-safe already, so you just need to keep one around in your framework/app (Can someone using Django explain where you’d keep a reference to this object around so that you could get to it in a Django view?).
New Cache Decorators
To make it easier to use caching in your app, Beaker now includes decorators for use with the cache object. Given the above caching setup, lets assume you want to cache the output of an expensive operation:
# Get that cache object from wherever you put it, maybe its in environ or request?
# In Pylons, this will be: from pylons import cache
from wherever import cache
def regular_function():
# Do some boring stuff
# Cache something
@cache.region('short_term’, 'mysearch’)
def expensive_search(phrase):
# Do lookup with the phrase variable
return something
return expensive_search('frogs’)
The second argument to the region decorator, 'mysearch’. That isn’t required unless you have two function’s of the same name in the same module, since Beaker records the namespace of the cache using the function name + module + extra args. For those wondering what a Beaker namespace is, its a single cache 'block’. That is, lets say you wanted to cache 4 versions of the same thing, but change them differently depending on the input parameter. Beaker considers the thing to be a namespace, and the things that change the thing being cached are the cache keys.
Only un-named arguments are allowed on the function being cached. These act as the cache keys so that if the arguments change, a new copy is cached to those arguments. This way you can have multiple versions of the function output cached depending on the argument it was called with.
If you want to use arbitrary cache parameters, use the other decorator:
# Get that cache object from wherever you put it, maybe its in environ or request?
# In Pylons, this will be: from pylons import cache
from wherever import cache
def regular_function():
# Do some boring stuff
# Cache something
@cache.cache('mysearch’, type='file’, expire=3600)
def expensive_search(phrase):
# Do lookup with the phrase variable
return something
return expensive_search('frogs’)
This allows you to toggle the cache options per use as desired.
If there’s anything else I can do to make it easier to use Beaker in your application, be sure to let me know (Yes, I know more docs would help, this blog post was a first attempt to help out on that front, more docs on the way!).
Pylons 0.9.7 Released
I’m pleased to announce after a rather lengthy release candidate period, that Pylons 0.9.7 is finally out. Pylons 0.9.7 brings a good amount of changes to Pylons from 0.9.6 while still retaining a fairly hefty amount of backwards compatibility to ensure a mostly painless upgrade.
Some helpful documentation on the new release:
- Upgrading to Pylons 0.9.7
- Pylons 0.9.7 Documentation
- Pylons 0.9.7 Docs in PDF
- Pylons Book (Will be updated soon with the final copy from the printed book)
Major changes in 0.9.7:
- Switched to using WebOb for the request/response object
- Various performance improvements to object initialization
- Beaker and Routes updates
- Middleware improvements, and optimizations
This is a huge step forward for Pylons, and I’d like to thank all of the contributers who have helped make Pylons what it is today. We’ve knocked off more bugs for this release than any before, which shows just how far the Pylons community has come:
- 0.9.5 tickets: 45
- 0.9.6 tickets: 64
- 0.9.7 tickets: 160
And we have finally made a huge dent in the historical “lack of docs” problem that Pylons previously suffered from with the new Sphinx generated docs and a comprehensive Pylons book.
The full changelog which describes the major changes (Look for the bits marked with WARNING that might affect backwards compatibility).
0.9.7 (February 23, 2009)
- WARNING: A new option is available to determine whether or not an actions
- WARNING: Fixed a minor security hole in the default Pylons error page that
- WARNING: Fixed a security hole in the default project template to use the
- WARNING: Refactored PylonsApp to remove legacy PylonsApp, moved
- Changed to using WebTest instead of paste.fixture for app testing.
- Added render_mako_def to render def blocks within a mako template.
- Changes to cache_decorator and cached_template to support Beaker API
- Fix HEAD requests causing an Exception as if no content was returned
- Fix a crash when returning the result of ``etag_cache`` in a controller.
- “response” flag has been removed from pylons.decorators.cache.beaker_cache,
- “invalidate_on_startup” flag added to beaker_cache, which provides a
- Updating host to use 127.0.0.1 for development binding.
- Added option to specify the controller name with a controller variable
- setup.py egg_info now restores projects’ paster_plugins.txt,
- The paste_deploy_config.ini_tmpl template is now located at
- Project’s default test fixtures no longer hardcode test.ini; the ini
validate now defaults to translating FormEncode error messages via Pylons' gettext catalog, then falls back to FormEncode's. fixes #296. Thanks Max Ischenko. * Fixed SQLAlchemy logging not working in paster shell. Fixes #363. Thanks Christoph Haas. * Added optionally engine initialization, to prevent Buffet from loading if there's no 'buffet.template_engines' in the config. * Updated minimal template to work with Tempita and other new templating changes. * Fixed websetup to parse location config file properly when the section isn't 'main'. Fixes #399. * Added default Mako filter of escape for all template rendering. * Fixed template for Session.remove inclusion when using SA. Fixed render_genshi to properly use fragment/format options. Thanks Antonin Enfrun. * Remove template engine from load_environment call. * Removing template controller from projects. Fixes #383. * Added signed_cookie method to WebOb Request/Response sub-classes. * Updated project template to setup appropriate template loader and controller template to doc how to import render. * Added documentation for render functions in pylons.templating. * Adding specific render functions that don't require Buffet. * Added forward controller.util function for forwarding the request to WSGI apps. Fixes #355. * Added default input encoding for Mako to utf-8. Suggested in #348. * Fixed paster controller to raise an error if the controller for it already exists. Fixes #279. * Added __init__.py to template dir in project template if the template engine is genshi or kid. Fixes #353. * Fixed jsonify to use application/json as its the proper mime-type and now used all over the net. * Fixed minimal template not replacing variables properly. Fixes #377. * Fixedvalidate decorator to no longer catch exceptions should they be
- Fixed paster shell command to no longer search for egg_info dir. Allows
- Added mimetype function and MIMETypes class for registering mimetypes.
- WARNING: Usage of pylons.Response is now deprecated. Please use
- Removed use of WSGIRequest/WSGIResponse and replaced with WebOb subclasses
- Fixed missing import in template controller.
- Deprecated function uses string substitution to avoid Nonetype error when
- E-tag cache no longer returns Content-Type in the headers. Fixes #323.
- XMLRPCController now properly includes the Content-Length of the response.
- Added SQLAlchemy option to template, which adds SQLAlchemy setup to the
- Switched project templating to use Tempita.
- Updated abort/redirect_to to use appropriate Response object when WebOb is
- Updated so that 404’s properly return as Response objects when WebOb is in
- Added beaker_cache option to avoid caching/restoring global Response values
- Adding StatusCodeRedirect to handle internal redirects based on the status
- Refactored error exceptions to use WebError.
- WSGIController now uses the environ references to response, request, and
- Added optional use of WebOb instead of paste.wsgiwrapper objects.
- Fixed bug with beaker_cache defaulting to dbm rather than the beaker
- The —with-pylons nose plugin no longer requires a project to have been
- The config object is now included in the template namespace.
- StaticJavascripts now accepts keyword arguments for StaticURLParser.
- Fix pylons.database.AutoConnectHub’s doInTransaction not automatically
