Debug School 5: Test Case Simplification

This is part 5 of the Debug School video series. It introduces the concept of test case simplification. Test case simplification is the first of two power tools in debugging (#2 is the next lesson). It will enable you to corner bugs the same way the CIA is able to corner Will Smith: he can run, but he can't hide.

Transcript

The following transcript was automatically generated by an algorithm.

00:00 : hello welcome to debug school this time
00:03 : test case simplification part one
00:07 : sometimes you encounter a bug that's
00:09 : really hard to diagnose the error
00:11 : message was not helpful or non-existent
00:15 : but that behavior is decidedly wrong and
00:18 : you have no idea where to look next this
00:22 : is the kind of situation where I would
00:24 : turn to test case simplification the
00:26 : technique that I'm going to show you in
00:28 : this lesson this really is my secret
00:32 : weapon it's what allows me to
00:34 : successfully diagnose the problems that
00:36 : often no one else on my team can figure
00:38 : out it has routinely saved the day for
00:41 : me when I encounter the most difficult
00:44 : bugs to diagnose to explain test case
00:47 : simplification first let me define the
00:49 : concept of relevant code in the context
00:53 : of a bug relevant code is any code whose
00:56 : presence is required in order for the
00:58 : bug to exist in its current form
01:01 : removing relevant code makes the bug go
01:04 : away now that doesn't necessarily mean
01:07 : you've fixed the problem because
01:08 : removing some piece of code could create
01:11 : yet another bug entirely but within the
01:14 : context of the bug in question if by
01:17 : removing a piece of code that makes the
01:19 : bugs behavior go away then that code is
01:23 : relevant conversely if you remove a
01:26 : piece of code and the behavior of the
01:28 : bug remains the same then that code is
01:32 : irrelevant given that test case
01:36 : simplification is the process of
01:38 : systematically removing irrelevant code
01:40 : until all that is left is code that's
01:43 : relevant back in the day Netscape
01:47 : Navigator had this bug where if the user
01:50 : attempted to print out a page the
01:52 : browser would crash spectacularly what's
01:56 : interesting about this story though is
01:57 : that the person who found the cause of
02:00 : this problem was not a Netscape engineer
02:03 : but rather an open-source enthusiast
02:06 : with very little programming experience
02:09 : let's call him John John went and
02:12 : volunteered at one of Netscape's
02:14 : bug-a-thon events to help hunt down some
02:16 : bugs instead of telling you what he did
02:19 : I'm going to sort of reenact it for you
02:24 : so the first thing that John does is he
02:27 : opens up the problematic page I actually
02:30 : don't have the version of Netscape which
02:32 : still contain this bug so I'm just going
02:36 : to have to use Firefox as a stand-in for
02:39 : this demo ironically the problematic
02:43 : page happens to be the issue page on
02:45 : Mozilla's issue tracker called Bugzilla
02:48 : so John opens up an issue in Bugzilla
02:52 : and now he tries to print that page when
02:56 : he does the browser crashes great bug
03:01 : reproduced which is always the first
03:03 : step in trying to fix a bug then the
03:07 : next thing he does is he reopens the
03:09 : browser and does a view source on that
03:11 : page he copy and paste the code into a
03:15 : file called issue dot HTML to speed up
03:19 : this demo have prepared a little command
03:21 : line program that John can use to
03:23 : quickly test out what will happen if he
03:25 : tried to print out a particular HTML
03:28 : page this program is called a try print
03:31 : and he would give it the name of the
03:33 : file to print as the argument the
03:36 : program will output either program
03:38 : crashed or okay if the page prints out
03:42 : without problems so he tries to print
03:44 : issue that HTML and it crashed just as
03:48 : we saw happen in the actual browser just
03:52 : as a sanity check
03:53 : he gets the source code of another web
03:55 : page this time the Bugzilla front page
03:58 : and he runs try print on it there he
04:02 : gets an okay so now he knows that the
04:05 : problem is specific to the issue page so
04:09 : he drills down into issue dot HTML he
04:13 : knows that if he removes some code from
04:15 : this file and if it still crashes then
04:19 : the code that he removed is irrelevant
04:21 : ie it
04:23 : did not contain the cause of the bug so
04:26 : he removed some link tags and runs try
04:28 : print and it still crashes and he
04:32 : removed some more and it still crashes
04:34 : so he's working his way to narrowing the
04:37 : cause of the problem but this file is
04:40 : huge it's close to 2,000 lines doing
04:43 : this by hand bit by bit is going to take
04:46 : forever but John has an idea he's going
04:50 : to use a technique called divide and
04:52 : conquer
04:53 : which works like this he divides the
04:56 : file into two halves the top half and
04:59 : the bottom half and he deletes the
05:01 : bottom half
05:02 : he runs try print and it still crashes
05:05 : so now he knows the bottom half must be
05:09 : irrelevant
05:10 : he's just eliminated one half of the
05:13 : file this is huge and he can do the same
05:17 : thing again he splits the rest of the
05:19 : code again into two halves and he
05:22 : deletes the bottom half this time when
05:24 : he runs try print the browser didn't
05:27 : crash that means that he removed the
05:29 : cause of the book along with the bottom
05:31 : half of the code so he needs to bring
05:35 : back the bottom half and then he deletes
05:37 : the top half
05:41 : this time the browser crashes so that
05:44 : means that the top half of the code is
05:47 : irrelevant and we still have relevant
05:50 : code so that is how dividing conker
05:53 : works and John just repeats this process
05:56 : over and over again he deletes the
06:01 : bottom half gets an OK so he puts it
06:04 : back and deletes the top half and gets a
06:06 : crash ok still have relevant code
06:10 : subdivides again and deletes the bottom
06:12 : half gets a crash ok subdivides yet
06:16 : again and deletes the bottom half
06:18 : another crash he does the same thing
06:21 : once again and this time gets an ok so
06:24 : he puts back the bottom half and deletes
06:26 : the top half and now he gets a crash
06:29 : again this the same thing deletes the
06:32 : bottom half and gets a crash
06:34 : we still have relevant could now all of
06:38 : this work is repetitive and tedious but
06:43 : look now very quickly john has narrowed
06:46 : it down to less than a page of code he
06:50 : continues and he deletes the top half
06:52 : this time and he still gets a crash and
06:56 : now the problem is coming into view he
06:59 : is looking at this select tag here and
07:02 : to sort of zoom into it he removes the
07:05 : containing div tag around that select
07:08 : tag and it still crashes so now he's
07:11 : thinking the Select tag is the culprit
07:13 : but maybe it has to do with some of the
07:16 : options within the tag so he removes the
07:18 : option tags and it's to crashes they are
07:22 : not relevant he wonders where the
07:25 : multiple attribute inside the Select tag
07:27 : is the culprit so he removes that and it
07:31 : still crashes he removed the remaining
07:34 : attributes in the tag and it still
07:37 : crashes he then removes even the closing
07:41 : select tag and it's due crashes as a
07:45 : sanity check
07:46 : he removes the Select tag altogether and
07:48 : now he gets an OK so sure enough the
07:53 : presence of a select tag
07:55 : any SelectTech at all is all that's
07:57 : needed to cause the Netscape Navigator
07:59 : to crash what a great ending to a
08:04 : mystery John was able to pinpoint the
08:07 : cause of the bug by simplifying the test
08:10 : case John updates the description of the
08:13 : issue to simply printing any page
08:16 : containing a select tag causes browser
08:19 : to crash this saved in escaped engineers
08:23 : lots of time not only in pointing them
08:26 : to exactly what they needed to know to
08:28 : make the fix but also in helping them
08:31 : triage and prioritize their slew of bugs
08:35 : now that you've seen the technique in
08:37 : action I'm going to talk about its pros
08:40 : and cons let's start with the cons as
08:44 : you have seen this technique can be
08:46 : tedious even with the divide and conquer
08:49 : technique however this technique has got
08:52 : pros in spades first of all it's simple
08:56 : and straightforward you don't even need
08:59 : to know programming in order to use it
09:01 : it's simply the process of elimination
09:03 : which is a skill you've already learned
09:06 : in high school while you were taking
09:07 : multiple-choice tests second it always
09:11 : works it will always help you narrow
09:14 : down the location of the problem by
09:16 : eliminating places where it's not third
09:20 : this technique is universally applicable
09:22 : you can use this technique with any
09:24 : programming language platform or
09:26 : framework and finally at the end of
09:30 : going through this process you get a
09:33 : small reproducible test case which you
09:35 : can use to understand the root cause of
09:37 : the problem at a more basic level and
09:39 : also you can use it to communicate the
09:42 : problem efficiently to others if you're
09:45 : trying to add information to a bug
09:47 : report or if you're trying to get
09:49 : programming help either in person or
09:51 : online places like Stack Overflow or IRC
09:55 : chat rooms
09:57 : when should you use this technique well
10:00 : here are some scenarios where test case
10:03 : simplification is especially helpful one
10:06 : when you don't know where to start
10:08 : looking
10:09 : ie you can't even come up with a
10:11 : hypothesis two when you have the
10:15 : opposite problem there are too many
10:17 : hypotheses and none of them seem any
10:20 : better than the others three
10:22 : when you have a heisenberg a heisenberg
10:26 : is a bug that disappears the moment you
10:28 : put in probes to try to observe its
10:30 : behavior now this is rare but it
10:33 : sometimes happens when you have a
10:35 : complex system and there are intertwined
10:38 : variables and dependencies and when
10:41 : there are multiple things going on at
10:43 : the same time for when you're dealing
10:46 : with a declarative language as opposed
10:48 : to a sequential one because it's not
10:51 : sequential you can't simply put in
10:53 : probes to inspect the state of the
10:55 : program between any two adjacent lines
10:58 : of source code five when you need a
11:02 : simple test case to communicate a
11:03 : problem to other people for purposes
11:06 : that i've already previously just
11:08 : mentioned that's the end of this lesson
11:12 : I have prepared a couple of exercises
11:15 : for you to get some hands-on experience
11:17 : with this technique and that's going to
11:20 : allow you to get a better feel for it
11:22 : and also to retain it better so that you
11:24 : can actually summon this technique for
11:26 : your own use when you're in the heat of
11:27 : a tough debugging session in part two of
11:31 : test case implication I'm going to
11:34 : demonstrate how I solved a problem using
11:36 : this technique with a JavaScript source
11:38 : file in JavaScript the syntax rules are
11:42 : less forgiving than in HTML and that
11:44 : means just cutting out half of the
11:47 : source code is not going to fly until
11:51 : next time