Better TDD Workflow via Exclusive Tests in Jasmine and Mocha

This unit testing tip brought to you by Andy Murray: congrats Andy on winning your first grand slam title!

The Problem

Once you do test driven development at a scale that's beyond the smallest of projects, your test suite's runtime will inevitably grow. Best practices say that you should do everything in your power to keep the runtime low, because if it takes a long time to run your tests, chances are you are not going to want to run them. Making the majority of the tests tests of small isolated units (that's why we call them unit tests) - sometimes with the help of mock objects - will do a lot to help your test suite's runtime.

Still, it is hard to avoid long runtimes. Because

Integration tests are necessary, even though they should not be the majority of your tests.
It's simple mathematics: the more tests you have, the longer they will take to run. I have never seen the test suite of any project of significant size run under one second.

The Importance of the Runtime for TDD

When you are doing TDD, you are constantly re-running the tests to guide what to do next. Because of that, every second you can shave off of your runtime will speed up your workflow noticeably: every second counts. How much? Okay, let's say if you can reduce the test suite's runtime from 10 seconds to 1 second; and it takes you on the average of 5 seconds in between - the time it takes for you to think and edit the code between test runs. The total duration of your write->run loop has been reduced from 15 seconds to just 6 seconds, and so you've just increased your productivity by 2.5 times.

Breaking it Down

Now, what if I told you that you can have sub-second runtime during your TDD workflow at all times, no matter how big your project gets?

How? Simple: by running only a subset of the tests in the test suite. What's that? The title of the article already gave it away? Well, shoot!

Of course, which subset of the test suite you want to run depends on the module you are working on. If you are working on moduleA, then you probably want to run test_moduleA. Sometimes you may even want to only run one test: this is particularly useful when trying to isolate a bug. But, this technique is nothing new. It's been done™, and I've seen different people do it different ways:

for command-line test runners: you specify a specific file or module or function to run on the command line
if you are code-editor-savvy, you bind a hot key to the test runner, and change the specificity of the runner command that's currently bound to the hot key as you want
if you are even more savvy, you write a text editor extension (or grab one someone else made) which figures out what tests to run based on where your cursor is in your editor. For example, if your cursor were in the function test_foo_bar, and you hit a certain hot key, then it would run only the test: test_foo_bar.
if you use an IDE like Eclipse, you can right click on your code, and click on a menu item like "run this test" and it will only run that individual test.
if you are using an in-browser testing framework like Jasmine, the test reporter generates a nice HTML report with links. You can click on these links to drill down and run only an individual test or sub-suite.

The technique I use with my Javascripts isn't any of these - and that's what I want to talk about. For the lack of a better name, it is called exclusive tests.

Exclusive Tests

Exclusive tests is source-based - in that to change the subset of the suite you want to run, you have to edit the source code to do it.

`ddescribe` and `iit` for Jasmine

For Jasmine, there is ddescribe and iit. With this fork of Jasmine, when any ddescribe is registered, only specs withing these exclusive suites will be run

describe('normal', function() {
  ddescribe('exclusive', function() {
    // all specs here will be run
  });

  // nothing here will run
  it('should not run', function() {});
});

When any iit is registered, only these exclusive specs will be run (precedence over ddescribe)

it('should not run', function() {});
iit('should run', function() {});
it('should not run 2', function() {});
iit('should run as well', function() {});

The ddescribe and iit fork has not yet made it into Jasmine core, hopefully this will change in the future. But until then, you can peruse the fork and you can get the pre-built source at https://raw.github.com/vojtajina/jasmine/ddescribe-iit-build/lib/jasmine-core/jasmine.js

`describe.only` and `it.only` for Mocha

ddescribe and iit just recently landed in Mocha. Only it's been renamed to describe.only and it.only, which while is more verbose, makes 100% more sense. Plus, for Mocha, you can use it with non-bdd dialects too, i.e. suite.only and test.only.

The Workflow

With exclusive tests, the workflow goes like this

Edit the test file for the module you want to work on to narrow it down to only running tests for that module.
Sometimes, you may use iit or it.only to restrict it down to one test to debug a particular problem, but you revert back once you are done fixing the problem.
When you are ready to commit, you always revert the exclusive test declarations so that 1) you see that you hadn't broken anything unexpectedly, and 2) the final checked-in version of the code will run the full test suite.

Check Before You Commit

What if you forget to revert and accidentally check-in the exclusive test declarations?

That's actually happened to me a few times. So, to prevent that from happening again, I use pre-commit git hooks to run a grep on the source code to make sure I don't check that in by accident. FWIW, here's my pre-commit hook script that does that

#! /usr/bin/env ruby

commands = [
    'grep -n ddescribe test/tests.js',
    'grep -n iit test/tests.js',
    'grep -n debugger $(find test -name "*.js")',
    'grep -n "console.log(" $(find src -name "*.js")',
    'grep -n debugger $(find src -name "*.js")',
]

commands.each do |command|
    output = `#{command}`
    if output.size > 0 then
        puts "*** ABORT COMMIT ***"
        puts command
        puts "#{output}"
        exit 1
    end
end

It's written in Ruby: I find that 9 times out of 10 if I start writing a shell script, I end up writing it in Ruby. The ruby should be pretty self explanatory: I have set up an array of grep commands to be run pre-commit (the command inside upticks actually runs the command and returns its output), if any of them return any output (and therefore something was matched), then we abort the commit by exiting with code 1. Bonus: you will also notice that I have grep commands that prevent accidentally leaving debugger and console.log statements in my code.

Advantages

How is this technique better than the other methods I've listed of narrowing your test suite?

It's all about speed!: you don't have to leave your editor to do it - just edit and save. This works especially great when you have a tool that reruns your tests on file-save (like testem).
Text-editor agnostic: this technique works independent of your choice of editor and therefore can be easily be shared among a team of different backgrounds/tastes.

Discussion: Is This Bad?

Up to this point, I suspect that I have rubbed some of you the wrong way: this whole business of putting your code into a temporary state and then having to revert it back probably seems unnatural - is this bad? After all, this is the reason most test frameworks/runners opt-for using something that's external to the source code - the command-line, the IDE, the browser, etc.

There are two opposing viewpoints here. The first viewpoint is that the source code is sacred - it's an artifact. you are not supposed to just change it willy-nilly, it's your secret sauce, you've spend probably countless hours debugging it, and you don't want to risk accidently screwing it up. Debugging is hard.

The second viewpoint is that in order to remain agile and stay in control of your code base, you must constantly be flexing your coding muscles: the worst thing that could happen is being afraid to make changes to your code - at which point your code is in danger of slipping into legacy code status. Unit tests and pre-commit hooks and other checks you put in place act as a safety net to elevate your bravery.

So, if your opinion aligns more with the first viewpoint, then exclusive tests are probably not for you, but if you subscribe to the second viewpoint, then writing transient code probably isn't at all a big deal.