TurboGears Tutorial: Social Bookmarking Application

Anyone interested in web development should have heard about the Model-View-Controller [1] software pattern by now. This pattern emphasizes on separation of application's data model, UI (view), and control logic. The concept itself isn't new and has been around since 1979, but recently there have been many successful implementations of the pattern in the domain of web development.

Like any self-respecting web developer, I started to explore the available frameworks, and quickly settled on Python [2] as a language and TurboGears [3] as a framework. I viewed many webcasts and documents, then decided to practice my newly gained knowledge by writing a basic del.icio.us [4]-like social bookmarking application in TurboGears. Now I'm documenting my work as a tutorial, hoping that other TurboGears newcomers would find something useful in it. The tutorial covers the following concepts:

• Designing data models.
• Developing controller Logic.
• Creating view templates.
• Working with forms and widgets.

There are also sections for the reasons behind my Python/TurboGears choice, and the installation process.

Please read on for the tutorial itself and an attachment of the project's source code at the end. I recommend viewing the code while working with the tutorial, as I didn't include imports or other tiny bits. In addition, the tutorial assumes good knowledge of Python.

Why Python

Python is a mature general purpose programming language, it has a large library of standard and contributed packages, dynamic typing makes it ideal for web development, and its clean syntax and object oriented model make it suited for MVC application development. The language enjoys a large active community of developers and users. It also supports Unicode natively, which is an advantage for a web developer who often works with Arabic websites like me.

• Advantages over Perl and PHP: Cleaner syntax and object oriented model.
• Advantages over Ruby: Larger standard and contributed library of packages, native Unicode support.

Why TurboGears

The main philosophy behind TurboGears is integrating a set of Python packages together to create a MVC web framework. This concept of code re-use is in the spirit of MVC and good software engineering. The packages are SQLObject [14] for the model, Kid [15] for the view, and CherryPy [16] for the controller. Those projects are already mature, have thorough documentation and active developers and users. In addition, it's still possible to use many other packages instead of one or more of the components listed above, such as SQLAlchemy [17] and Markup [18].

The other obvious candidate for a Python MVC web framework is Django [19]. While it's a very interesting project, I picked TurboGrears because of the above, and the facts that TurboGears has built-in support for Ajax and JavaScript widgets. Ajax support in Django is still in its early stages.

Installation

OK, so you are still interested. Let's start with the installation process. To install TurboGears you first need Python and SetupTools [20]. If you are using Linux, chances are that you already have Python (and probably SetupTools) installed. If not, it's almost always a matter of searching for and downloading python and setuptools through your package manager. Windows users can find a Python installer at the Python website [21], and a script that does SetupTools' job at ez_setup.py [22].

To install TurboGears under Linux:

$ easy_install -f http://www.turbogears.org/preview/download/index.html [23] TurboGears

$ ez_setup.py -f http://www.turbogears.org/preview/download/index.html [23] TurboGears

After that and if you don't receive any error messages, you should be ready to create a TurboGears application. If not, please consult the documentation [24] for further details.

I'm using TurboGears 0.9a8 which is the most current version as of this writing. APIs are "almost" frozen so the tutorial should remain valid for later versions.

Creating tgLinks Project

TurboGears uses a script called tg-admin for various administrative tasks like creating projects and databases. I'll call our social bookmarking application tgLinks. To create a skeleton for the project, open the commandline, change the current directory to where you want tgLinks to be created, and issue the command:

 $ tg-admin quickstart
Enter project name: tgLinks
Enter package name [tglinks]:
Do you need Identity (usernames/passwords) in this project? [no] yes

The only configuration we need to do is specify the database we want to use. This can be done by editing dev.cfg in the newly created project directory:

sqlobject.dburi="notrans_sqlite://%(current_dir_uri)s/devdata.sqlite"

I use SQLite so I left the line as is (sqlite and pysqlite packages are required. Make sure you have notrans_ in the protocol). if you use MySQL you may want to use something like:

sqlobject.dburi="notrans_mysql://username:password@hostname:port/databasename"

That's it! TurboGears should be ready now.

Designing The Data Model

The first component in the MVC pattern is the data model. A social bookmarking application contains the following entities:

• User: id, username, email, password, display name.
• Link: id, URL.
• Tag: id, name.
• Bookmark: id, title

And we have the following information:

• each bookmark has one user, one link, and zero or more tags.
• Each user has zero or more bookmarks.
• Each link has zero or more bookmarks.
• Each tag has zero or more bookmarks.

The relationships between entities are:

• User-Bookmark: One-to-many.
• Link-Bookmark: One-to-many.
• Tag-Bookmark: Many-to-many.

To convert our data model into code, we need to create a class for each entity, and entity attributes will become class attributes. Data model classes reside in tglinks/model.py. When we enabled identity support, TurboGears generated required classes for users, permissions and the like. We will have to edit the User class to add our custom fields, but first we will create the other classes:

class Tag(SQLObject):
  name = UnicodeCol(alternateID = True, length = 64)
  bookmarks = RelatedJoin('Bookmark')

As you see, the code is quite straightforward. A model class inherits from SQLObject, and each attribute is a property. The Tag entity contains one property which is name</code. It is defined as <code>UnicodeCol which stores Unicode strings. alternateID = True means that this column is going to be unique, and length = 64 is pretty self-explaining.

To create a many-to-many relationship, we define a RelatedJoin property in each class. We will see how to create a similar property in the Bookmark class.

class Link(SQLObject):
  url = UnicodeCol(alternateID = True, length = 255)
  bookmarks = MultipleJoin('Bookmark')

Quite similar to Tag but with one difference, instead of RelatedJoin we use MultipleJoin, which means a one-to-many relationship, and we will see how Bookmark will contain a ForeignKey property on the other side of the relationship.

class Bookmark(SQLObject):
  user = ForeignKey('User')
  link = ForeignKey('Link')
  tags = RelatedJoin('Tag')
  title = UnicodeCol(length = 64)

Bookmark is the joining class in our design; it contains one reference to User (one-to-many or Foreign Key), one reference to Link (again, one-to-many or Foreign Key), and one or more references to Tag (many-to-many, or RelatedJoin). There is also the title property, which is a Unicode string.

Finally, we will modify the User class. If you take your time to explore the generated code, you'll notice that it already contains the following properties: user_name, email_address, display_name, password. We only need to add a reference to Bookmark:

class User(SQLObject):
  # [...]
  bookmarks = MultipleJoin('Bookmark', joinColumn='user_id')
  # [...]

joinColumn is a column in Bookmark used to join the two tables. Usually we don't need to specify it, but here we do because User is a generated class and its SQL table name will be different (tg_user instead of user).

And this should be it. Our data model is ready, to check the generated SQL you can use the following command:

$ tg-admin sql sql

To create the actual tables:

$ tg-admin sql create

Later you may want to drop the tables and start fresh by issuing:

$ tg-admin sql drop

SQLObject provides many other column types such as IntCol, BoolCol, and StringCol. For the full documentation please check SQLObject's pages [25].

Now that we have our database ready, we may populate it with some objects. I'll create a new user by using TurboGears interactive shell:

$ tg-admin shell

>>> user = User(user_name='ayman', password='xxxxx', 
  email_address='blog at aymanh dot com', display_name = 'Ayman Hourieh')

The database now contains one User object. We may query the table in a variety of ways:

>>> User.get(1) # Get user object where id = 1
<User 1 user_name=u'ayman' email_address="u'blog at aymanh ...'" display_name=u'Ayman Hourieh' password=u'xxxxx' created='datetime.datetime...)'>
 
>>> User.by_user_name('ayman') # Get user object where user_name = 'ayman'
 
>>> result = User.select() # Select all objects
>>> for u in result:
      print u
 
>>> result = User.select(User.q.display_name.startswith('A'))
    # All users whose display name starts with an 'A'
 
>>> result = User.selectBy(display_name='Ayman Hourieh')
    # All users whose display name equals 'Ayman Hourieh'

And we may access user bookmarks by user.bookmarks. Of course, it's still empty. Let's add a link and tag, and then a bookmark:

>>> link = Link(url='http://www.google.com/')
>>> tag = Tag(name='search')
>>> bookmark = Bookmark(title='Google', userID=user.id, linkID=link.id)
>>> bookmark.addTag(tag)

You may check the values of: user.bookmarks, link.bookmarks, and tag.bookmarks to see how SQLObject auto-populated them.

Now user ayman has one bookmark to Google. Exit the shell, answering yes to commit changes to the database.

TurboGears also provides a web-based interface to modify and query data. To use it run:

$ tg-admin toolbox

Creating Controller Logic and View Templates

With our data model ready, it's time to move to controller code, but first, let's see how to launch our web application:

$ ./start-tglinks.py

Now point your browser to http://localhost:8080/ [26] and you will see a welcome page. How does this work? Let's check tglinks/controllers.py and see:

# [...]
class Root(controllers.RootController):
    @expose(template="tglinks.templates.welcome")
    def index(self):
        import time
        log.debug("Happy TurboGears Controller Responding For Duty")
        return dict(now=time.ctime())
    # [...]

Each request is translated by the controller component of TurboGears (CherryPy) into a method call. When we request "/", index method above is called. It returns a dictionary. This dictionary is then passed to the template tglinks.templates.welcome to create the final page. If you check the code of tglinks/templates/welcome.kid you will find that it's an XHTML document with namespaced attriubtes. Kid, TurboGears' view component, processes this template variable and handles those attributes to generate the final page. The attributes may use the values passed in as a dictionary from the controller, for example:

<span py:replace="now">now</span>

Other attributes that we will make use of later:

<span py:if="value">text</span>

<ul><li py:for="item in values">${item}</li></ul>

<span strip="" />

For more info on Kid attributes, please read Kid's documentation [27].

Now that we have enough info to write actual code. Let's build our first controller/view combo!

We will start with the index page, modify its controller to the following:

@expose(template="tglinks.templates.index")
  def index(self):
      result = Bookmark.select()[:10]
      return dict(bookmarks=list(result))

We changed the template to index, and made the controller select the most recent 10 bookmarks and return them to the view template.

Let's see what the view does (You don't have to create it from scratch, just copy welcome.kid to index.kid and modify it):

<body>
  <h1>Welcome to TG Bookmarks</h1>
  <h2>Recent Bookmarks</h2>
  <span py:replace="render_bookmarks(bookmarks)" />
</body>

The template simply calls a function named render_bookmarks to render a list of bookmarks with their tags and users. The function is defined in master.kid (which is the master template that other templates extend). It may look a bit complicated first, but it's built upon constructs we explored earlier:

<div py:def="render_bookmarks(bookmarks, show_tags=True, show_user=True, show_edit=False)" py:strip="">
  <ul py:if="bookmarks">
    <li py:for="b in bookmarks">
      <a href="${tg.url(b.link.url)}">${b.title}</a>
      <span py:if="show_edit" py:strip="">[<a href="${tg.url('/edit', url=b.link.url)}">edit</a>]</span>
      <br />
      <div py:if="show_tags" py:strip="">Tags: <ul class="tags">
        <li class="tags" py:for="tag in b.tags">
          <a href="${tg.url('/tag/' + tag.name)}">${tag.name}</a>
        </li>
      </ul><br /></div>
      <span py:if="show_user" py:strip="">Posted by: <a href="${tg.url('/user/' + b.user.user_name)}">${b.user.display_name}</a></span>
    </li>
  </ul>
  <p py:if="not bookmarks">No bookmarks found.</p>
</div>

The function takes a list of bookmarks and checks whether the list is or empty or not. If it's empty, a message is printed saying so. Otherwise, the function iterates over bookmarks and outputs each bookmark's link, user, and tags.

So why I took the route of writing such a function instead of simply iterating over and printing bookmarks in index.kid? A social bookmarking site will contain bookmarks lists all over its pages. For our basic implementation, we will add user and tag pages. Both will also contain bookmark lists similar to the main page, so instead of repeating the same code over, we define it in a function and reuse it.

The render_bookmarks function takes three parameters that control what sections of the list are printed, show_user for the user of the bookmark, show_tag for the tags section, and show_edit is used on the current user's page to show an edit link and let them edit their own bookmarks.

The function assumes the following URLs: /user for user pages, /tag for tag pages, and /edit for editing bookmarks. We will create controllers and views for those URLs, but first, let me explain some concepts that appeared in the previous code snippets.

You may have noticed that index method in our root controller has a decorator named expose. To make a method available to requests, one should assign this decorator to it. The decorator takes several possible parameters, one of them is template which is used to assign a Kid template to the controller method.

To access identity data, one may use an array of functions and properties provided by turbogears.identity. Here are some of the most useful ones:

identity.not_anonymous() # User is logged in
identity.current.user # Logged in user object

And identity.require() which is a decorator that takes a condition, and limits its controller method access to users who satisfy this condition, for example:

@identity.require(identity.not_anonymous())

Please check this page [28] for more info on the identity module.

You may have noticed the call to tg.url in render_bookmarks. This TurboGears function creates a URL out of the passed path and parameters.

OK, now we have an index page that lists the most recent bookmarks. Let's continue with the tag page. This page will list the latest bookmarks under this tag. Here is the controller code:

@expose(template='tglinks.templates.tag')
def tag(self, name=None):
  if not name:
    raise NotFound
  try:
    tag = Tag.byName(name)
  except SQLObjectNotFound:
    raise NotFound
  return dict(tag=tag)

Unlike index which doesn't take any parameters, this controller method takes one called name. It will be called when a request such as http://localhost:8080/tag/linux is made. In this example, name's value will be 'linux'. I think the code is easy to understand by now. raise NotFound fires a 404 "Page not found" error. We do so if the request doesn't specify a tag name. Otherwise, we look for this tag in our database, and pass it to the view template. In case the tag isn't found, a 404 error is fired as well.

Let's see what the view looks like:

<body>
  <h1>Tag: ${tag.name}</h1>
  <span py:replace="render_bookmarks(tag.bookmarks, show_tags=False)" />
</body>

It simply calls render_bookmarks, but tells it not to show tags, given that we are already on a tag page.

The user controller method is quite similar. It only contains some extra code to deal with anonymous users. Let see how it works:

@expose(template='tglinks.templates.user')
def user(self, user_name=None):
  if not user_name:
    if identity.not_anonymous():
      return dict(user=identity.current.user)
    else:
      redirect('/login')
  else:
    try:
      user = User.by_user_name(user_name)
    except SQLObjectNotFound:
      raise NotFound
    return dict(user=user)

The method first checks whether user_name is specified. If not, it checks whether the current user is logged in and displays their page, or redirects them to the login page. Finally, if a user_name is specified, it displays the user's page or a 404 error in case the user doesn't exist.

The related view is hardly different from tag's view:

<body>
  <h1>${user.display_name}'s Bookmarks</h1>
  <span py:replace="render_bookmarks(user.bookmarks, show_user=False, show_edit=tg.identity.user is user)" />

tg.identity.user is user checks whether the current user is viewing their page, and displays an edit link in this case.

Finally, let's add some links to the top of the page next to "login" to simplify navigation. This can be done by editing master.kid:

<span py:if="not tg.identity.anonymous">
  Welcome ${tg.identity.user.display_name}.
  <a href="/">Home</a>
  <a href="/user">User page</a>
  <a href="/post">Post link</a>
  <a href="/logout">Logout</a>
</span>

We will also add a CSS stylesheet to our project, and make the tag list inline. The project folder contains a directory called static. Here is where images, CSS, and JavaScript go. Create a file named main.css under static/css, with the following content:

ul.tags, li.tags {
  display: inline;
  margin: 0;
  padding: 0;
}

To link the stylesheet to our pages, edit master.kid and add the following in the head section before the title tag:

<link [29] rel="stylesheet" type="text/css" href="/static/css/main.css" />

That's it. Our application now has user and tag pages. It's time to create the "Post bookmark" page, but first, let's see how the frontpage will look with a couple of bookmarks added:

Working with Forms and Widget

To simplify working with forms, TurboGears introduces the concepts of widgets and validators. A widget is a form element that can be as simple as a textfield, or as complex as a JavaScript date/time picker. Validators take care of validating data to make sure it's actually what the application expects, and convert it back to a format Python understands. To see those concepts in action, let's create a form to enter or edit bookmarks. It should contain fields for URL, title, and tags:

from turbogears import widgets, validators as v, error_handler
 
bm_form = widgets.ListForm(
  fields=[
    widgets.TextField(
      name='url', label='URL',
      attrs={'size': 64, 'maxlength': 255},
      validator=v.All(v.NotEmpty, v.URL(add_http=True))
    ),
    widgets.TextField(
      name='title', label='Title',
      attrs={'size': 64, 'maxlength': 64},
      validator=v.All(v.NotEmpty, v.UnicodeString)
    ),
    widgets.TextField(
      name='tagstr', label='Tags',
      attrs={'size': 64, 'maxlength': 255},
      validator=v.UnicodeString
    )
  ],
  submit_text='Save'
)

We instantiate the ListForm class to create a form, using a constructor that takes two parameters, fields contains a list of widgets to be displayed, and submit_text is the text of the submit button.

name and label are the name and title of the widgets respectively. attrs is a dictionary that will be converted into HTML attributes. validators.All validates input only if all of its validators validate input as well. validators.NotEmpty requires non-empty input. validators.URL checks input for a valid URL (add_http adds protocol to URL if it's not present). And validators.UnicodeString converts input to Unicode if it's not already so. This way, once data reaches the processing method, we are sure that it's valid.

Forms and widgets are callable objects. To display this form in a template, we call it passing action, method and a dictionary containing initial values. Let's see the post controller method:

@expose(template='tglinks.templates.post')
@identity.require(identity.not_anonymous())
  def post(self, url='', title='', tagstr='', tg_errors=None):
    bm_data = {'url': url, 'title': title, 'tagstr': tagstr}
    return dict(bm_form=bm_form, bm_data=bm_data)

It simply passes the form and initial values to the template. tg_errors is a variable related to error handling. I will explain its purpose later.

Now for post.kid code:

<body>
  <h1>Post New Bookmark</h1>
  ${bm_form(bm_data, action='/save', method='post')}
</body>

As you see it's pretty straightforward. The result would be our form with passed attributes and initial values.

We now have two remaining functionalities. save and edit methods:

@expose()
@validate(form=bm_form)
@error_handler(post)
@identity.require(identity.not_anonymous())
def save(self, url, title, tagstr):
  user = identity.current.user;
 
  try:
    link = Link.byUrl(url)
  except SQLObjectNotFound:
    link = Link(url=url)
 
  result = Bookmark.selectBy(user=user.id, link=link.id)
  if result.count():
    bookmark = result[0]
    bookmark.title = title
    for tag in bookmark.tags[:]:
      bookmark.removeTag(tag)
  else:
    bookmark = Bookmark(user=user, link=link, title=title)
 
  for name in set(tagstr.split()):
    try:
      tag = Tag.byName(name)
    except SQLObjectNotFound:
      tag = Tag(name=name)
    bookmark.addTag(tag)
 
  turbogears.flash('Bookmark has been saved successfully.')
  raise redirect('/user')

I'll go through the most interesting parts of this method. We don't pass any template to expose because we are always redirecting back to some page. The validate decorator validates input, and error_handler specifies the method called when errors occur. We are re-using post as an error handler. This explains its tg_errors parameter which contains information about errors. We don't do any actual error handling because the default behavior of displaying an error message next to fields is enough.

Finally, there is edit. It takes a URL, retrieves its bookmark data, and redirects to post:

@expose()
@identity.require(identity.not_anonymous())
def edit(self, url):
  user = identity.current.user
 
  try:
    link = Link.byUrl(url)
  except SQLObjectNotFound:
    raise NotFound
 
  result = Bookmark.selectBy(user=user.id, link=link.id)
  if result.count():
    bookmark = result[0]
  else:
    raise NotFound
 
  return self.post(
    url=bookmark.link.url,
    title=bookmark.title,
    tagstr=' '.join([tag.name for tag in bookmark.tags])
  )

And this should be it, but let's first style our form before seeing it in action. Add the following to the stylesheet:

.listform label {
  display: block;
}
.listform li {
  list-style: none;
}

And now the post/edit form itself:

For more info on forms and widgets, you may view the devcasts [30], or read the following pages: 1 [31], 2 [32], 3 [33].

Final Word

I tried to cover the concepts I learned about TurboGears in this tutorial. In each section there is a list of links to further readings, and below you will find the source code of the project. I'm thinking of adding del.icio.us style AJAX edit-in-place for bookmarks, but this may be a separate tutorial.

Thanks for reading until the end and hope you enjoyed it as much as I enjoyed writing it!