Lisp for the Web, Part II

29 July 2011

In this article, I hope to extend Adam Petersen’s article Lisp for the Web, bringing up for discussion some of the aspects of web programming that most of us are familiar with.

What to Expect

Adam Petersen wrote an awesome article called Lisp for the Web. In it, he walks readers through creating a simple web application using Common Lisp (hereinafter Lisp). It’s a fantastic article, and I recently sat down with it (and some other resources) to finally write some code in Lisp.

So what I’d like to do here is take some of the things that were left out (with good reason) of the original article, and flesh out the experience of developing for the web with Lisp. At the end of the article, Mr. Petersen writes that

Due to the dynamic and interactive nature of Lisp it’s a perfect fit for prototyping. And because Lisp programs are so easy to evolve, that prototype may end up as a full-blown product one day.

It’s my hope that in continuing his tutorial in this way I can show readers how that process from prototype to production may take place; in an organic way that showcases Lisp’s ‘evolutionary’ abilities.

The Original Program

The original program can be found at the Lisp for the Web article. I made some changes to it in order to fit some specific criteria:

  • The original article recommends using BerkeleyDB for its backend, accessed through the Lisp library Elephant. I’m still using Elephant, though I believe there may be more “Lispy” serialization libraries available — Elephant requires a lot of dips into foreign functions. BerkeleyDB, however, has been swapped out for SQLite. This makes deployment a lot easier, and avoids the licensing hassles that seem to follow BerkeleyDB around.
  • We are binding the Elephant store to a variable, namely *store*, in case we want to change databases during top-level interaction.
  • The syntax for starting a server in Hunchentoot has changed since the original article. We now make an instance of a Hunchentoot acceptor in order to get the server bound and running.
  • The original article uses URLs which end in .htm. For example, the index page is located at /retro-games.htm. There’s no reason for this to not simply be /retro-games, and I’ve changed all URLs to follow this convention. Remember, kids, cool URIs don’t reveal their implementation!
  • To be a bit more inclusive, we are no longer just voting on retro games. Now you can add votes for any game you wish! (This is not really a necessary change, but I don’t know many very old games, so at the very least it makes coming up with examples easier.)

So the first thing we’re going to do is make these transformations to the original file, and then we can continue working from there.

If you’d like the diff so you can start off with these changes, I’ve posted it to Gist: game-voter.patch

main-page.png

Your Lisp Environment

The original article, quite wisely, avoids discussing the actual environment in which its Lisp is set up. It’s not useful to anyone to have an obsolete description of a Lisp installation. In this case, however, I’d like to take a snapshot of the state of Lisp in certain ways; I am going to describe, to some extent, my setup.

Running OS X 10.6.7, with Steel Bank Common Lisp installed through the excellent homebrew package management system. My editor, REPL, and general Lisp interaction environment is Emacs 23.3, compiled for the Mac by David Caldwell.

The package manager I’m using is the fantastic Quicklisp. I’ve tried installing packages for Common Lisp before with ASDF, and Quicklisp is so much more simple, capable, and promising it’s scary.

Within a REPL, the following command did everything I expected and nothing I didn’t: installed the libraries (often called systems) I needed without complaining or breaking, and providing the packages in the top-level.

CL-USER> (ql:quickload '(cl-who hunchentoot parenscript elephant fiveam css-lite cl-json))

Smarter Routes

There was one thing I noticed almost immediately as I was playing around with the program. If I wanted to make a change to a ‘controller’, i.e. any of the functions defined by define-url-fn, re-evaluating it would simply push another route onto the hunchentoot:*dispatch-table*. This isn’t a huge problem per se, but it is unnecessary computation, and after having pushed a hundred or so different versions of a controller dispatch onto the *dispatch-table*, I desired a more practical way of describing the routes in my site. In addition, using a macro such as define-url-fn to describe the routes means that Emacs doesn’t recognize that controller function as a valid symbol. It’s a silly convenience, but the more I use defmacro and defun, the easier I can find the code I’m looking for with Emacs’ symbol lookup.

So for example, the controller for adding a game goes from this:

(define-url-fn (game-added)
  (let ((name (parameter "name")))
    (unless (or (null name) (zerop (length name))) ; In case JavaScript is turned off.
      (add-game name))
    (redirect "/games"))) ; Display the front page.

To this:

(defun controller-game-added ()
  (let ((name (parameter "name")))
    (unless (or (null name) (zerop (length name)))
      (add-game name))
    (redirect "/retro-games")))

Now with a handful of functions, I manually set the *dispatch-table* to the list of routes I desired, using Hunchentoot’s dispatcher creation functions.

hunchentoot:create-regex-dispatcher is a function that takes a regular expression for an endpoint, and runs the function named by the second argument upon execution.

(setq *dispatch-table*
 (list
  (create-regex-dispatcher "^/index" 'controller-index)
  (create-regex-dispatcher "^/vote" 'controller-vote)
  (create-regex-dispatcher "^/game-added" 'controller-game-added)
  (create-regex-dispatcher "^/new-game" 'controller-new-game)))

Writing CSS in Lisp

The original tutorial never supplies the CSS it used to style the website. As you can see, I provided some of my own CSS to pretty things up a bit – but you won’t see that if you’ve made it this far into the article. Why? Because we’ve overwritten the contents of *dispatch-table* with the four routes we’ve defined above. What we’re going to do is define another route for the CSS, but provide the CSS in a Lisp form that then gets translated into CSS for us. This provides us a little bit of flexibility and can help reduce some of the redundancy that CSS tends to accumulate, without resorting to some CSS extension (like Sass or LESS).

The CSS goes in a controller function, just like the other dynamically created pages on the site.

(defun controller-css ()
  (setf (hunchentoot:content-type* hunchentoot:*reply*) "text/css")
  "...")

We set the content-type header to “text/css”, which lets web browsers know that the file we’re sending is meant to be interpreted as a stylesheet. I’ve put together a style, and I’ll be using some aspects of it to exemplify some points with Lisp, but you’re free to style up the site however you’d like. Either way, replace the ellipsis in the code sample above with your style. You can use your CSS verbatim, explicitly using multiple newlines in one string. Lisp strings understand multiple lines without any special indicator:

(setq *some-string* "This is a string that
spans
multiple
lines.")

My CSS style is reproduced at the following link: game-voter.css

We’ll be using the css-lite library to generate our CSS. This library exposes one primary method, css, which takes a list of lists, and from them, generates our CSS. Within this list parameter, every other list describes a set of selectors, and the following list is an associated list of css attributes to values. My first translation into css-lite‘s format looks something like this:

(defun controller-css ()
  (setf (hunchentoot:content-type* hunchentoot:*reply*) "text/css")
  (css-lite:css
    (("body")
     (:width "70%" :margin "0 auto" :font-family "sans-serif"
      :border-left "1px solid #ccc"
      :border-right "1px solid #ccc"
      :border-bottom "1px solid #ccc"))
    (("h1")
     (:font-size "140%" :text-align "center"))
    (("h2")
     (:color "#000" :background-color "#cef" :margin "0 auto" :padding "4px 0"))
    (("#header")
     (:background-color "#cef" :padding "8px"))
    (("#header .logo")
      (:display "block" :margin "0 auto"))
    (("#header .strapline")
     (:display "block" :text-align "center" :font-size "80%" :font-style "italic"))

    ;; and so on ...

So the first thing that stands out is the repetition in describing the border for the body selector. I’m sure there’s some CSS way to describe the border, but as an example, we’re going to use a variable to encapsulate the border description.

(let ((border "1px solid #ccc"))
  (css-lite:css (("body")
                 (:border-left border
                  :border-right border
                  :border-bottom border))))

Simple enough, but what if we wanted to reduce some more complicated piece of CSS? For example, some CSS3 attributes that are initialized differently for each browser?

The gradient property currently has several different implementations. Here are the selectors provided by a popular CSS3 gradient generator, Alex Sirota’s excellent and comprehensive Ultimate CSS Gradient Generator:

background: #1e5799; /* Old browsers */
background: -moz-linear-gradient(top, #1e5799 0%, #2989d8 50%, #207cca 51%, #7db9e8 100%); /* FF3.6+ */
background: -webkit-gradient(linear, left top, left bottom, color-stop(0%,#1e5799), color-stop(50%,#2989d8), color-stop(51%,#207cca), color-stop(100%,#7db9e8)); /* Chrome,Safari4+ */
background: -webkit-linear-gradient(top, #1e5799 0%,#2989d8 50%,#207cca 51%,#7db9e8 100%); /* Chrome10+,Safari5.1+ */
background: -o-linear-gradient(top, #1e5799 0%,#2989d8 50%,#207cca 51%,#7db9e8 100%); /* Opera11.10+ */
background: -ms-linear-gradient(top, #1e5799 0%,#2989d8 50%,#207cca 51%,#7db9e8 100%); /* IE10+ */
background: linear-gradient(top, #1e5799 0%,#2989d8 50%,#207cca 51%,#7db9e8 100%); /* W3C */

What we’d like to do is describe our CSS as having a gradient, and having Lisp do the work of translating it into all the browsers that we want to support.

The first thing to observe is that a lot of the rules use the same basic format – that of the W3C recommendation. It should be easy to apply those rules while just changing the name of the CSS function. Two other things that should jump out at you are:

  • We should use a fallthrough background of the first color in the pattern for browsers that don’t support the gradient at all.
  • The -webkit-gradient rule reverses the order of the arguments: instead of taking a color and a percentage, it takes a percentage and a color.

The function should return a flattened list of CSS selectors and values. We then pass it to an internal css-lite function, make-css-function, with an identifier that we can use in the css macro.

(defun css-linear-gradient (colors-and-positions)
  (let ((default-background (caar colors-and-positions)))
    (flet ((w3c-spec (name)
             (format nil "~A(top, ~{~{~A ~A%~}~^, ~})" name colors-and-positions))
           (webkit-gradient ()
             (format nil "-webkit-gradient(linear, left top, left bottom, ~{color-stop(~{~A%, ~A~})~^, ~})"
               (mapcar #'reverse colors-and-positions))))
      (list
       :background default-background
       :background (w3c-spec "-moz-linear-gradient")
       :background (webkit-gradient)
       :background (w3c-spec "-webkit-linear-gradient")
       :background (w3c-spec "-o-linear-gradient")
       :background (w3c-spec "-ms-linear-gradient")
       :background (w3c-spec "linear-gradient")))))

(css-lite::make-css-func
  linear-gradient (colors-and-positions) (css-linear-gradient colors-and-positions))

This works roughly the way we expect, generating the different combinations of gradient methods above.

(css-lite:css
  (("body")
   (linear-gradient '(("#1e5799" 0) ("#2989d8" 50) ("#207cca" 51) ("#7db9e8" 100)))))

Using jQuery

The original article used some Javascript to prevent the user from submitting the Game submission form if they tried entering the empty string for the name of the game. It was written in parenscript, a library that translates Lisp code into the appropriate JavaScript. It was a fairly rudimentary implementation, reproduced below:

GAME-VOTER> (ps-inline
             (when (= name.value "")
               (alert "Please enter a name.")
               (return false)))
"javascript:if (name.value === \"\") {
    alert(\"Please enter a name.\");
    return false;
}"

Modal alert boxes such at this one aren’t a horrible idea, but there’s a more elegant, expected way to handle this sort of behavior: display an inline message to the user on form submission; and what better way to do that than jQuery?

Parenscript is a fantastic library, but it may require some thought and time to get accustomed to its idioms. Most of them make sense in retrospect, but not before you’ve used them for a while. The good news is, it works perfectly with jQuery once you’ve put this work in, and Lisp macros can help bridge the gap and make your life easier.

The primary macro used with Parenscript is ps, especially if you’ll be injecting the contents of your code directly into <script> tags. Most of Parenscript consists of ‘translators’ of a kind, turning lisp forms into the appropriate Javascript verbiage. The Parenscript reference is the canonical bible of these translations, succinctly explaining all of these rules and how they may be applied. For now we will take a look at the ones that are relevant to us.

Simple functions have their obvious counterparts.

GAME-VOTER> (ps (+ 1 2))

"1 + 2;"

Attempting to call a function by name is similarly obvious. It’s also here that we’re pointing out one of Parenscript’s transformative properties: to take hyphenated symbols and turn them into a valid Javascript identifier, capitalizing the letters right after the hyphens. This is a necessary change as symbols in Lisp are interned in all capitals.

GAME-VOTER> (ps (makeFunnyNoise))

"makefunnynoise();"

GAME-VOTER> (ps (make-funny-noise))

"makeFunnyNoise();"

Most jQuery programming is accomplished by the passing of anonymous functions. In Parenscript, these functions are created through – how else? – the use of lambda.

GAME-VOTER> (ps (lambda (foo bar) (+ foo bar)))

"function (foo, bar) {
    return foo + bar;
};"

Variables may be defined in various ways; the way that makes the most sense to us is the use of a let binding, which works in much the way you’d expect. This is the best part of Parenscript: it is predictable. You don’t have to alter many habits to start using it.

GAME-VOTER> (ps (let ((a 1) (b 2)) (+ a b)))

"var a = 1;
var b = 2;
a + b;"

Objects can be created with the create operator.

GAME-VOTER> (ps (create :url "http://example.com"))

"{ 'url' : 'http://example.com' };"

If we had a (Javascript) Object named foo, and wanted to call its bar function, we would have to use the ‘chain’ operator in Parenscript, denoted by the @ symbol. We then surround the chain operator with another set of parentheses to indicate that we are calling the result. This is where we also show you that any not-understood symbol is left unmolested, such as foo and bar.

GAME-VOTER> (ps (@ foo bar))

"foo.bar;"

GAME-VOTER> (ps ((@ foo bar)))

"foo.bar();"

This is one of the ways that we would need to access jQuery core function, i.e. to call functions hanging directly off of it, such as $.ajax(). Parameters can be put into the outer parentheses of the chained call.

GAME-VOTER> (ps ((@ $ ajax)))

"$.ajax();"

GAME-VOTER> (ps ((@ $ ajax) (create :url "http://example.com")))

"$.ajax({ 'url' : 'http://example.com' });"

The other way is to call the jQuery selector, for manipulating elements on the page.

GAME-VOTER> (ps ((@ ($ "div.foo") fade-in)))

"$('div.foo').fadeIn();"

With all these things in mind, let’s take a look at the jQuery we want to produce on the page:

$(document).ready(function () {
    $('input.btn').click(function () {
        var text = $('input.txt').val().trim();
        if (text == '') {
            $('#error').fadeIn();
            return false;
        };
    });
});

This is my first go at translating it into Parenscript.

GAME-VOTER> (ps ((@ ($ document) ready)
                 (lambda ()
                   ((@ ($ "input.btn") click)
                    (lambda ()
                      (let ((text (chain ($ "input.txt") (val) (trim))))
                        (cond ((equal text "") 
                               ((@ ($ "#error") fade-in))
                               (return false)))))))))

We have some duplication with the code required to prepare an event handler: two lines for document.ready and two lines for $("input.btn").click. Using Parenscript, we can write a Lisp macro and ‘install’ it into Parenscript’s translator, like so.

(defmacro $$ ((selector event-binding) &body body)
  =((@ ($ ,selector) ,event-binding) (lambda () ,@body)))

(import-macros-from-lisp '$$)

import-macros-from-lisp is described in the macro section of the Parenscript documentation. This allows our second attempt to be a lot cleaner.

GAME-VOTER> (ps ($$ (document ready)
                  ($$ ("input.btn" click)
                    (let ((text (chain ($ "input.txt") (val) (trim))))
                      (cond ((equal text "") 
                             ((@ ($ "#error") fade-in))
                             (return false)))))))

Writing a Test Suite

By this point, your little web application should be looking pretty fantastic. It’s simple, but powerful and extensible. The refactoring we’ve done, while fairly cursory, has gone a long way to making sure that more work on the system is possible and easy. There’s one section where we’re lacking, however, and that’s a test suite. Most programmers today have gotten onboard the unit testing bus in some capacity or another. It is possible to go overboard – my feeling is that test suites should contain absurdly small amounts of logic, and be comprised of one or two assertions per unit test. Even then, it’s very easy to start testing the wrong behavior, or writing specious tests.

With these thoughts in mind, we’re going to write a small, efffective test suite. We will be doing so using the FiveAM library. FiveAM is attractive for a few reasons:

  • It encapsulates tests into logical units, called suites, much like functions, macros, and classes are broken up into packages.
  • The assertion operator is simple. is takes a form and if the form evalutates to non-nil, then the assertion passes. If the form evaluates to nil, the assertion fails.
  • It allows for the use of a wrapping macro called a fixture. While other testing suites use the term ‘fixture’ to describe pre-constructed data, such as records in a database, FiveAM extends the notion to include the setup and teardown of tests.

The first thing you should do is create a place to put the tests. Off of my main directory, I’ve simply touched test/test-game-voter.lisp. test-game-voter will be another package, so declare it:

(defpackage test-game-voter
  (:use :cl :game-voter :elephant :fiveam))

(in-package :test-game-voter)

And we’ll write a test to make sure everything works as expected. Make sure to enter the package.

TEST-GAME-VOTER> (test sanity
                   (is (= 4 (+ 2 2))))

TEST-GAME-VOTER> (run!)
.
 Did 1 check.
    Pass: 1 (100%)
    Skip: 0 ( 0%)
    Fail: 0 ( 0%)

Pretty nice, eh? Extremely simple. The single period in the output after the run! command is the single passed test, similar to what you’d see in xUnit derivatives. Now let’s see what happens when a test fails.

TEST-GAME-VOTER> (test more-sanity
                   (is (= 4 (+ 2 4))))

TEST-GAME-VOTER> (run!)
.f
 Did 2 checks.
    Pass: 1 (50%)
    Skip: 0 ( 0%)
    Fail: 1 (50%)

 Failure Details:
 --------------------------------
 MORE-SANITY []: 
      (+ 2 4) evaluated to 6, which is not = to 4..
 --------------------------------

Again, a huge amount of detail. Now, remove those tests from the suite with (rem-test 'sanity) and (rem-test 'more-sanity). We have real tests to write.

The first unit tests we’re going to tackle are on the model level – making sure the game class we created in the first tutorial works the way we expect. However, it’s a persistent class, and we want to make sure we’re saving test instances somewhere else other than our public database. So that’s the first step: defining an Elephant configuration and controller to house this new data. The second step is making sure we compartmentalize our model tests into a model suite.

(defparameter *test-database-config*
  '(:clsql (:sqlite3 "test-store"))
  "The connection information and filename of the database used in the
test suite.")

(defparameter *test-database-controller*
  (open-store *test-database-config*)
  "The database controller for the test suite. We open one here so
that every test run doesn't open more.")

(def-suite :suite-game-voter-model)
(in-suite :suite-game-voter-model)

The next part is going to be a little tricky. We don’t want every single test adding its own results to the database, and then using that dirtied database in the next test. Elephant exposes database transaction management to us, but that’s not quite good enough. We need to be able to specify that the transaction should be aborted. This is going to involve diving into the internals of what makes Elephant go: the clsql-sys and db-clsql packages.

Without getting too much into it, I will say this. *test-database-controller* is our instance of the data store. Stored on it is the information used to track transactions, including the status of the transaction. This field can be set manually, and if Elephant attempts to close a transaction and finds that it has failed, it rolls it back entirely. How does it know if it’s failed? It checks to see if the transaction status is :aborted.

With that explanation in mind, here’s what we’re looking at in terms of code. This may not be the most elegant way of dealing with it, but it works nicely for our purposes.

(def-fixture db-fixtures ()
  (with-transaction (:store-controller *test-database-controller*)
    (&body)
    (setf (clsql-sys::transaction-status
           (clsql-sys::transaction
            (db-clsql::controller-db *test-database-controller*))) :aborted)))

def-fixture is a FiveAM macro. It works almost exactly like defmacro, instead placing the described macro into FiveAM’s own repertoire of fixtures. with-transaction is within Elephant’s namespace, and takes an associated list of options. In this case we want to make sure the transaction is occuring on our test controller. The &body tag is implicit with def-fixture, so we don’t need to provide it in the argument list. Finally, we setf the status of the transaction we’re in to :aborted. When with-transaction hits the end of this form, it will see the transaction ‘failed’, and roll back everything we’ve done in the body.

Now let’s begin writing some tests! Off the top of my head, I can think of a couple things we’d like to test: adding a game, voting for a game, getting a game based on its name, and ensuring we can tell if a game is stored by passing its name. We’ll start with adding a game.

(test test-add-game-name
    (with-fixture db-fixtures ()
      (let ((game (game-voter::add-game "test game")))
        (is (equal "test game" (game-voter::name game))))))

(test test-add-game-votes
    (with-fixture db-fixtures ()
      (let ((game (game-voter::add-game "test game")))
        (is (= 0 (game-voter::votes game))))))

Call run! again and check out the results. Did you get two passing checks? Notice that run! reports the total number of checks, or assertions, and not the number of tests. This is because FiveAM will continue through a test even if a check fails, and report the passing status of all the checks in a test.

Note that we have to use the double-colon notation for our game functions, since we haven’t made them externally visible from our game-voter package.

This is looking good, but we can remove just a sliver of duplication by writing a macro for tests we know will be using the database. To wit:

(defmacro db-test (test-name &body body)
  `(test ,test-name
     (with-fixture db-fixtures ()
       ,@body)))

(db-test test-add-game-name
  (let ((game (game-voter::add-game "test game")))
    (is (equal "test game" (game-voter::name game)))))

Write some more tests. Make sure that game-from-name returns correct values for games that are or aren’t in the system. Make sure adding a game with a duplicate name fails. Then, write something like this:

(db-test test-sanitized-game-name
  (game-voter::add-game "Foo")
  (game-voter::add-game "  Foo  ")
  (is (= 1 (length (game-voter::games)))))

Well, that fails! We aren’t sanitizing our input! This isn’t even caught on the client side. This is what writing good tests is about – finding small pieces of behavior that fail in unexpected ways.

We can ensure that the server takes care of extra padding around the game name by trimming it. string-trim takes a string called a char bag, and then the string you wish to trim. Since all strings are array of characters, string-trim walks through the char bag and removes instances of each character from both sides of your argument.

We can modify the add-game function as below:

(defun add-game (name)
  (let ((sanitized-name (string-trim " " name)))
    (with-transaction ()
      (unless (game-stored? sanitized-name)
        (make-instance 'persistent-game :name sanitized-name)))))

Once the new function is evaluated, running the test suite again should pass.

Conclusion

There’s a lot of places to go from here. The CSS we wrote gets generated on every load. A caching mechanism would be nice. There’s no way to delete entries, or to make sure unscrupulous people aren’t voting more than once. It would be cool if we added some information to the games, such as publisher, year, and so on – maybe even box art and screenshots, which would mean dealing with file uploads. We don’t have tests for our controller functions.

However, we’ve brought our little game voter a little closer to a solid, robust modern web application. I hope I’ve managed to show off how easy it is to extend and refactor a small Lisp project, and have piqued your interest into giving it a try for web development. You’ll be amazed how easy it is to start thinking in Lisp.