rr Time-Traveling Debugger

In this video I give an introduction and a demo to the rr debugger. rr (https://rr-project.org/) is a revolutionary time-traveling debugger. It was initially developed by Robert O'Callahan and others at Mozilla for the purpose of debugging difficult to reproduce bugs in Firefox. It uses the deterministic replay technique to enable time-traveling. The ability to reverse time in debugging is a game changer. It has saved me hours of debugging time and allowed me to solve mysteries I would not otherwise have been able to solve. The ability to reverse time gives rise to some new debugging techniques. I cover one technique here called "Backup and Drill Down" and I plan to cover more in future videos.

Transcript

The following transcript was automatically generated by an algorithm.

00:00 : hi this is toby and i'd like to
00:02 : introduce you to the rr debugger
00:04 : you can find the project information on
00:07 : their
00:08 : project page right here rr is a time
00:11 : traveling debugger
00:12 : for the languages c c plus plus
00:15 : and rust why is rr interesting
00:19 : so if you think about how developers do
00:21 : debugging
00:22 : today i would say the techniques that we
00:25 : use are very primitive
00:27 : most of us probably use some version
00:30 : of print debugging uh whatever
00:34 : the variant of the print statement is in
00:37 : your programming language
00:38 : you're probably doing that the problem
00:40 : is print
00:41 : debugging really sucks you have to
00:44 : litter your code with all of these
00:46 : print statements which are temporary in
00:49 : nature
00:49 : you wouldn't really want to check it
00:51 : into your code base
00:52 : so you have to go back afterwards and
00:55 : clean them up
00:56 : and you know sometimes you forget and
00:58 : all your teammates get annoyed another
01:00 : problem with print debugging is
01:02 : when you have too many print statements
01:04 : then your logs
01:05 : become noisy the signal to
01:08 : noise ratio gets worse and worse and
01:11 : it's
01:12 : really hard to see what's going on
01:14 : anymore so
01:15 : what about breakpoint debugging why
01:17 : don't people use a debugger
01:18 : well the reason that even though
01:21 : debuggers exist
01:22 : developers are not necessarily using
01:25 : them
01:26 : there must be a reason for that and i
01:28 : think the reason is that
01:30 : breakpoint debuggers are actually
01:32 : inefficient
01:33 : uh in many circumstances when compared
01:37 : to print debugging
01:38 : there's also a learning curve associated
01:40 : with debuggers but
01:42 : but i think one huge problem with
01:45 : traditional breadcrumb debuggers
01:47 : is the price is right problem trying to
01:49 : set a breakpoint
01:51 : is like playing the prices right you
01:53 : have to try to get
01:54 : as close as you can without going over
01:58 : if you use a breakpoint debugger before
02:00 : you probably had the experience of
02:02 : single stepping
02:03 : through a function but then you step
02:06 : over a function
02:08 : and you realize that you've gone too far
02:10 : we now
02:11 : have seen the problem but to drill
02:14 : down even more to the actual cause of
02:17 : the problem
02:18 : we actually would have had to drill down
02:20 : to the function
02:21 : that we just stepped over in order to do
02:23 : that what you would have to do is
02:25 : basically
02:26 : restart the entire debugging session and
02:29 : in many cases you have to re-perform the
02:32 : steps
02:33 : that you have painstakingly done one
02:35 : after another
02:37 : just in order to be able to get back to
02:39 : the point
02:40 : where you can drill down to that
02:41 : function again and this kind of thing
02:44 : actually can happen very often and is
02:47 : why
02:47 : using a breakpoint debugger can become
02:49 : very tedious
02:51 : sometimes people just say you know what
02:53 : i don't want to bother doing that i'm
02:54 : just going to
02:55 : do print statement another thing about
02:57 : debugging
02:58 : that are tough if you can't even
03:00 : reproduce the bug
03:01 : you're not going to be able to fix it
03:03 : and if you can only reproduce the bug
03:06 : one time out of ten times then that
03:08 : means nine times out of the ten
03:10 : you're going to sort of be wasting your
03:12 : time to put it in another way you're
03:14 : gonna be spending a lot of time
03:16 : just reproducing the bug that's just a
03:19 : very tedious process
03:20 : rr solves all three of these problems
03:24 : by allowing you the ability to travel
03:27 : back in time one way to evaluate a
03:30 : debugger
03:30 : is does a developer still revert back to
03:34 : print debugging or not
03:35 : and in my opinion rr passes the test in
03:38 : addition to those three things
03:40 : um one thing i found while using rr
03:43 : for more and more of my work is that it
03:46 : really helps me
03:48 : understand complex code easier than ever
03:51 : before
03:51 : the reason that's true is because the
03:53 : debugger now is so efficient
03:55 : at helping me answer questions about the
03:58 : code
03:59 : that i can really use it as an
04:02 : exploratory tool
04:03 : currently rr works on linux only and it
04:06 : is integrated with the gdb debugging
04:08 : tool
04:08 : it can also work with ides that
04:10 : integrate with gdb
04:12 : how does rr work well rr works using a
04:15 : technique called
04:16 : deterministic replay and the
04:18 : deterministic replay concept
04:21 : is based on the idea that programs
04:24 : usually are very deterministic if you
04:27 : have a simple program
04:29 : if you run this program a hundred times
04:31 : it'll do the same thing every time
04:33 : so programs are largely deterministic
04:36 : now there are sources of non-determinism
04:39 : where do they come from well any time
04:41 : you use random number
04:43 : that's a source of non-determinism or
04:45 : anytime you take
04:47 : input from the user that's also a source
04:49 : of non-determinism
04:51 : what rr does is take out the
04:54 : non-deterministic bits it'll capture
04:58 : them
04:58 : in a recording run so there's a concept
05:01 : of the
05:02 : recording run versus a replay run
05:05 : during the recording run it's going to
05:07 : run the program in record mode maybe
05:10 : let's let's take some user input
05:12 : i'll ask the user for what the top
05:14 : number is
05:15 : source of non determinism is what the
05:18 : user is going to
05:19 : input so what rr does in in its
05:22 : recording run it's going to record
05:24 : what the user entered and it saves that
05:28 : into a file for the replay phase
05:31 : and then when we run the program in
05:33 : replay phase
05:34 : it's going to basically read back the
05:37 : value that the user
05:38 : put in from that file it
05:42 : secretly swaps in that pre-recorded
05:45 : value
05:45 : and puts it back as the return value of
05:48 : this input function
05:49 : now given that how does time traveling
05:52 : work
05:52 : in this scheme if i'm on this line for
05:55 : example
05:56 : how do i go backwards if i run this
05:58 : program again
06:00 : what we get is a series of say
06:03 : instructions that are executed one after
06:05 : another and if you have a for loop like
06:07 : this
06:07 : what happens in reality is we get this
06:10 : loop iterated a number of times so we
06:13 : could uh sort of visualize it like this
06:16 : we can number each one of these
06:18 : instructions if i'm
06:20 : currently inside the debugger and i'm
06:22 : paused on instruction 23
06:24 : if i wanted to walk backwards in time
06:26 : back to instruction 22 what i could say
06:29 : is
06:29 : hey let's restart the program from the
06:31 : top but i'm going to set a breakpoint at
06:34 : instruction 22
06:35 : so when it gets to instruction 22 that's
06:38 : where i want to pause
06:39 : and that achieves the same effect as
06:42 : walking backwards in time by one
06:44 : instruction so that is how time
06:46 : traveling works with rr
06:48 : on the right hand side i have a very
06:50 : small program that calculates the
06:52 : circumference of a
06:54 : triangle and on the left here i have a
06:57 : terminal i'm going to
06:59 : run my triangle program and you can see
07:02 : it gives the answer as 8.4 something
07:07 : for the discerning viewers if you're
07:10 : hip to the pythagorean theorem
07:13 : you recognize this is the three four
07:15 : five triangle
07:17 : and therefore the circumference should
07:20 : be
07:20 : three plus four plus five which equals
07:24 : twelve
07:24 : so that's the wrong answer we're gonna
07:26 : try to debug that
07:28 : if we wanna use rr to debug this we run
07:30 : this program
07:31 : within rr in rr's record mode
07:36 : it again gives the answer rr will
07:38 : generate a trace into this
07:40 : subdirectory which contains all the
07:42 : information that rr needs
07:44 : to replay this exact same code execution
07:48 : if we run it in replay mode now we can
07:51 : use the back in time features do rr
07:54 : replay
07:54 : as a command line argument the path to
07:58 : the directory that the record mode
08:00 : recorded to
08:01 : like this but if we're just interested
08:04 : in replaying the most
08:06 : recent recording then we can leave that
08:09 : off so we'll just say our replay
08:12 : and now we are inside the gdb
08:15 : cli interface but it's using rr as a
08:19 : back end
08:20 : for those of you who are not familiar
08:21 : with gdb i'll cover some of the basic
08:24 : commands that you might use in gdb
08:26 : listed here if you're already familiar
08:28 : with this stuff you can skip to the next
08:30 : part of the video
08:31 : normally in gdb when you start up the
08:34 : debugger
08:35 : you actually have to call the run
08:37 : command to tell your program
08:39 : to start running but rr has already done
08:42 : that for us
08:43 : we're already paused at the beginning of
08:46 : the program the begi but the beginning
08:47 : of the program is
08:48 : not the beginning of your main method
08:51 : instead it's the beginning of some
08:53 : bootstrap code
08:55 : which i believe is in lip c and the
08:57 : bootstrap code is
08:59 : going to do some setup code and then
09:02 : call into your main function and then
09:03 : it'll do some cleanup code
09:05 : so what we have to do is if we want to
09:08 : start stepping within our main function
09:10 : we first have to set a breakpoint
09:12 : this command it has a abbreviation which
09:15 : is
09:15 : b so we could type brick main
09:18 : to stop at the first thing in the main
09:21 : function
09:21 : the the break command has a couple of
09:24 : different syntaxes
09:25 : you can say break the name of the
09:27 : function
09:28 : f or something or if you want to stop at
09:31 : a specific line
09:33 : you would write the specific line number
09:36 : first you would write the name of your
09:38 : file which is triangle.c and then you
09:41 : write
09:41 : colon and then you write the name of the
09:44 : line
09:45 : so those are the two ways of writing a
09:47 : breakpoint we're going to use the first
09:49 : way
09:50 : which is brick main
09:53 : okay or you could have also done m b
09:55 : main
09:56 : to use the abbreviation it would have
09:58 : done the same thing it looks like i've
10:00 : just
10:01 : made two breakpoints at the same
10:02 : location
10:04 : so if i want to clean that up i can
10:06 : always use
10:07 : the info break command
10:11 : and that will show all the breakpoints i
10:13 : have currently
10:14 : currently i have two of them i can
10:16 : delete one of them by using the delete
10:18 : command
10:19 : like so and now
10:23 : i only have one command so anyway now we
10:25 : have a breakpoint
10:27 : at the beginning of the main function
10:29 : i'm going to say continue
10:32 : to continue the program until it hits
10:34 : the next breakpoint
10:35 : which will be here so now we're at the
10:38 : beginning of the main function
10:40 : i can say step or next some of you might
10:42 : be more familiar with
10:44 : debuggers that have more ui
10:47 : you might be familiar with these buttons
10:49 : so i'll map the
10:50 : gdb commands to these buttons so this
10:53 : continue button
10:54 : that looks like a play button in gdb
10:56 : that's also called
10:57 : continue and it has an abbreviation c
11:01 : for this step over button which you're
11:04 : on
11:04 : this line and if this line has a
11:06 : function called you want to step
11:08 : over this function call and you want to
11:11 : end up on the
11:12 : next line whatever that line is this
11:14 : step over in gdb
11:16 : is called next if you
11:19 : want to instead of stepping over you
11:21 : don't want to do that you want to step
11:23 : into it ending up within the body of the
11:26 : function
11:27 : then that's the step in function in gdp
11:30 : that's just
11:31 : step finally if you're already inside
11:34 : the function you want to come out of the
11:36 : function
11:37 : basically skip to the end of the
11:39 : function and then come out
11:41 : ending up here in gdb that is called
11:44 : finish or f for short
11:48 : for all four of these there's a reverse
11:51 : version of that
11:52 : as well which is enabled uh by rr
11:55 : basically stepping backwards in time
11:58 : so for example i can step over next
12:01 : next next next next next next
12:04 : all the way until here line 41
12:08 : and now i can actually see what the
12:10 : answer is
12:11 : by using the print command down here
12:15 : and i can see the answer and at this
12:17 : point
12:18 : i could actually step back in time i'll
12:20 : say reverse next
12:21 : or rn for short
12:25 : and now i actually step back in time
12:28 : before calling this triangle
12:30 : circumference function and i can
12:32 : actually
12:33 : step into the function using the step
12:37 : command and now i'm inside the triangle
12:40 : circumference
12:41 : function now that we have the basics out
12:43 : of the way let's try to debug this
12:44 : program
12:45 : one strategy that traveling back in time
12:49 : enables is something called the backup
12:52 : and drill down
12:53 : which is more or less what we just did
12:55 : we basically step
12:56 : over the execution of this function
12:59 : and then we verify that the answer was
13:01 : wrong because we have the ability to
13:03 : travel back in time we don't
13:05 : have to be so worried about going over
13:08 : because
13:08 : if you go over all you do is you step
13:10 : backwards
13:11 : and then after we're there we can just
13:14 : go
13:14 : inside and then we can end up here so
13:16 : now we've just backed up and we're going
13:18 : to drill down
13:19 : we're going to use the same strategy
13:20 : we're going to just step
13:23 : over each of these lines we're going to
13:25 : verify the answers
13:27 : of the sub problems which is calculating
13:29 : the distance
13:30 : of each side we're going to verify if
13:33 : they are all correct
13:35 : one of them is probably wrong and then
13:37 : when we find out which one is wrong
13:39 : we'll then drill down to the
13:40 : corresponding one okay
13:45 : again we don't have to be worried about
13:48 : going over
13:49 : because we can always step backwards
13:51 : let's print them all
13:57 : whoops okay so
14:00 : four point two four and then
14:04 : zero and then four point two four the
14:06 : correct answers should have been
14:08 : three four and five so all of them look
14:11 : wrong to me i might just pick the zero
14:14 : one
14:15 : because that looks like the most
14:17 : suspicious one so i'm gonna
14:19 : go into the call that calculated side
14:23 : two
14:26 : i just jump back to before we made this
14:29 : distance call
14:31 : and then we're gonna go in by using
14:34 : the step command so now we're inside
14:37 : this
14:37 : distance call and what the distance call
14:40 : is gonna do
14:41 : is calculate two distances square them
14:45 : and then take the square root of the sum
14:48 : of those squares
14:50 : let's see what numbers we're working
14:53 : with
14:53 : now one thing i can do is print them
14:56 : again
14:57 : using the p command but another thing i
14:59 : can actually do to see the values is
15:01 : use the back trace which i want to show
15:04 : you
15:05 : the backtrace command or bt for short
15:08 : either way works actually show you the
15:11 : all the frames in your stack frame so
15:14 : this is actually an
15:15 : easy way to see your function parameter
15:18 : values
15:19 : so you can see in this case we're
15:21 : calculating the distance between the
15:23 : point
15:24 : 0 0 and 3 0.
15:27 : the answer should be 3. if we jump to
15:29 : the end of this function
15:31 : using the finish command it will also
15:34 : show you the return value
15:36 : the return value is 0 which is clearly
15:39 : wrong
15:40 : um and at this point i can inspect the
15:43 : intermediate values so how can zero be
15:45 : one
15:46 : what's this what's the sum of the
15:48 : squares what's the first square
15:53 : uh i might have to go backwards by one
15:56 : so
15:56 : say reverse step yeah now we're
15:59 : backwards
16:00 : into that function now and now i'm gonna
16:03 : print out
16:04 : sd1 is zero
16:08 : uh i think that's correct what about st2
16:12 : is
16:12 : also zero so the y minus the y that
16:16 : should be the zero but the other one
16:18 : should be
16:19 : three or the square of three so that's
16:22 : clearly wrong what's the ah okay
16:26 : both of them we use d1 for the squaring
16:29 : whereas one of them should use d2 for
16:32 : the squaring
16:33 : so i've isolated the bug and this is
16:36 : likely to fix here i rebuilt the program
16:40 : run it and this looks like the correct
16:42 : answer this time
16:44 : so as you can see we can really get a
16:45 : lot of mileage out of
16:47 : rr's time traveling feature for basic
16:50 : debugging tasks
16:51 : like these there are even more powerful
16:54 : ways to use time
16:55 : traveling a couple of new techniques
16:57 : that rr enables
16:59 : really changes the game for me and i'll
17:02 : show you those techniques
17:03 : in upcoming videos in general rr really
17:06 : has been a droid for me
17:08 : to use and if you haven't used it before
17:11 : i hope you give it a try