Testem: Interactive JS Test Runner

About a week ago I released my Javascript test runner - Testem. It provides a text-based UI and auto-re-run-on-save for the test driven development(TDD) usecase as well as a simple command-line interface for continuous integration (CI). It supports QUnit, Jasmine, and Mocha to boot. To understand how the UI works, you really have to see it in action, I've made a screencast:

There's also plenty of info on the Github README page which should be adequate for getting you started with it.

How I've been Using it

I have been using Testem regularly for client work for a few months. I use the TDD mode to add new features and fix bugs to a project that consists of about 2.5k lines of Javascript. I have ~230 tests in Jasmine which takes ~50 seconds to run on Chrome (I know that's quite slow). I use the odescribe branch of Jasmine to easily narrow my tests so that most of the time, my tests run sub-second. On check-in, a Windows Server 2008 EC2 instance running Jenkins will run the tests on IE7, IE8, IE9, Firefox, Chrome and Safari and notify me when there is breakage.

The Tech

Testem was built using node.js and along the way of building it I've encountered some very interesting techonology and have had to solve some very interesting problems.

Text-Based UI

The text-based UI

has evolved over time and was rewritten at least 4 times. At first the text was simply displayed mostly using console.log. There was one trick I used which employed printing a carriage return /r to move the cursor back to the beginning of the current line, allowing me to replace the text of the current line:

process.stdout.write('Some text')
process.stdout.write('\rSome other text') // 'Some text' will now be replaced
                                          // by 'Some other text'

After that, I decided I wanted a rich UI and so required the full power of ncurses and rewrote it using Mscdex's node-ncurses. This is when I concieved of the tabbed interface, more or less like what I have now.

Then at some point Substack wrote Charm and I rewrote it in that because ncurses takes a long time to compile (for users who have to install Testem), it was a little buggy, and there was no way it was going to work for Windows, and as Substack says, nowadays everyone is ANSI anyway. Charm is so lightweight in comparison, it is basically ~300 lines of Javascript.

Backbone Rewrite

When I got the UI working and looking the way I wanted, I encountered one more problem: flickering. When repainting the tabs, the UI would blank out the space it needs to paint to before painting the text(to get rid of old traces of text), and it refreshed all the tab headers everytime the test results for any browser changed, as a result, I was seeing some unwanted flickering. I decided I needed to use the MVC pattern, where when some state on a model changed (i.e. the test results for a paricular browser), I only have to update the view which represents that model (the tab for that browser). I could have written this on my own, but with all the talk about Backbone.js I was beginning to feel left out - like how everyone's talking about the new movie that you haven't seen and so you finally give in and go see it. The Backbone rewrite worked out very well, and is what I have stuck with now.

Tidbits

Some more interesting tidbits about the UI:

The tabs are rendered using box drawing characters
The spinners (one on each tab for when tests are running) are rendered using the circular arc unicode characters found on this page - I stole that from Mocha
To support window resizing by the user, the UI queries the terminal size(# of lines and columns) twice every second, and redraws the entire UI if it has changed
Because unicode characters does not work by default on Windows, I fall back to ascii characters there.
Although Windows support ANSI terminal codes for the most part, it does not certain features like text scrolling, which I had to work around.

Server Side

Testem uses Express and Socket.IO (stables for Node programs) for the HTTP server that serves up the necessary HTML, JS, CSS, and WebSocket goodness require to run the tests in the browser. The serve will basically serve any file in the current directory via HTTP. It sets Cache-Control to No-cache and ignores headers if-modified-since and if-none-match in order to avoid the browser caching anything.

To serve the client side testem.js used to integrate with the test framework on the page and report back test results, it concatenates all the required Javascript files on the fly. Why? I am lazy and prefer not to have to worry about a build step.

exp.get('/testem.js', function(req, res){

    res.setHeader('Content-Type', 'text/javascript')

    res.write(';(function(){')
    var files = [
        __dirname + '/../public/testem/socket.io.js'
        , __dirname + '/../public/testem/jasmine_adapter.js'
        , __dirname + '/../public/testem/qunit_adapter.js'
        , __dirname + '/../public/testem/mocha_adapter.js'
        , __dirname + '/../public/testem/testem_client.js'
    ]
    async.forEachSeries(files, function(file, done){
        fs.readFile(file, function(err, data){
            if (err){
                res.write('// Error reading ' + file + ': ' + err)
            }else{
                res.write('\n//============== ' + path.basename(file) + ' ==================\n\n')
                res.write(data)
            }
            done()
        })
    }, function(){
        res.write('}());')
        res.end()
    })
    
})

File Watcher

There are some existing file watcher libs out there but I wrote by own because most of them were still not using the fast fs.watch at the time that I looked. I am also using node-glob to allow specifying glob patterns in your testem.yml file. Unfortunately, I found out later that fs.watch is not reliable when using vim to edit the files in question, for that reason, I also use fs.watchFile in addition to fs.watch as a fallback.

Browser Launcher

Getting the browser launcher to work for various browsers on various platform was just a tedious process of installing and testing. For some browsers I had to use extra tricks like telling it to create a new user profile directory and then deleting that directory after it's done. To test code on IE7 and IE8, I would use IE9's compatibility mode, but rather than using a <meta> tag, I used the X-UA-Compatible header, which means I can use the same HTML markup.

I found Caolan's async library to be great for doing mapping and filtering for arrays in an async manner, it dramatically simplified my code browser launcher code.

What's Next?

Getting feedback from other folks using Testem in the real world, and then improve the user experience as much as possible.
BrowserStack integration - after seeing Bunyip I was like oh snap! I need to get on this.

For more go to my roadmap