Time-Travel Debugging with Robert O'Callahan

I have the great pleasure of speaking to Robert O'Callahan (roc), the creator of the rr record/replay debugger and co-founder of Pernosco. We talk about the origins of his various projects, how they work, what they mean for the software industry, and more.

roc's blog - https://robert.ocallahan.org/
rr - https://rr-project.org/
roc's rr talk - https://www.youtube.com/watch?v=H4iNuufAe_8
Pernosco - https://pernos.co/
Chromometer paper - https://www.cs.purdue.edu/homes/xyzhang/fall07/Papers/Amber.pdf
My videos about rr - https://tobyho.com/video-series/rr-debugger.html

Transcript

The following transcript was automatically generated by an algorithm.

00:00 : hi there this is toby i recently had the
00:02 : pleasure of speaking to robert
00:05 : o'callaghan he is a former mozilla
00:08 : distinguished engineer and he is the
00:11 : creator of rr
00:13 : short for record replay a time traveling
00:16 : debugger for
00:18 : c c plus plus rust and some other
00:21 : languages
00:23 : for the linux platform
00:24 : he also is the founder of pronoscope an
00:28 : omniscient debugger built on the
00:30 : foundation of rr that features a rich
00:33 : user interface currently he is working
00:36 : for google research so without further
00:38 : ado
00:39 : this is our conversation
00:42 : i'm really honored to have this
00:43 : conversation with you so time traveling
00:46 : debugging is something i've been
00:48 : uh really digging into for the last
00:51 : couple of years and uh yeah
00:53 : uh i i kind of made
00:56 : from a couple of versions of time
00:58 : traveling debugger for myself one one
01:00 : for a toy language and then another one
01:02 : for python
01:04 : and i didn't i didn't find out about
01:06 : your work until later by accident first
01:09 : time i watched your talk and you were
01:11 : talking about like what the future could
01:12 : be like
01:14 : with
01:15 : omniscient debugging i was like yeah yes
01:18 : you're you're the person who sees the
01:20 : future and uh a possible future
01:23 : so i was inspired then i started using
01:27 : rr for some of my work every time i use
01:29 : it i feel like i'm living in the future
01:32 : it's like everybody else is in the past
01:34 : and i want to say like how did people
01:36 : even debug
01:38 : before this yeah yeah thanks for taking
01:40 : the time i feel like i'm still learning
01:43 : about this technology and as i
01:45 : understand it you have built three
01:47 : debuggers is time traveling debugger a
01:50 : good term to use it's okay
01:52 : um if you have an innocent debugger then
01:55 : it's a bit different from time travel
01:57 : because you're not so much moving around
01:59 : in time as kind of seeing across time
02:02 : um but it's but it's certainly true that
02:04 : rri is a time traveling debugger so
02:07 : um yeah you mentioned three things so
02:09 : yeah i've built i bought this cronin
02:11 : chronicle chronomancer uh debugger as a
02:14 : kind of an experiment um before rrr is
02:17 : that amber yeah so that was the same
02:19 : thing i called it amber for a while and
02:21 : then i changed the name okay um so so
02:24 : basically
02:25 : that project was
02:27 : very much a sort of prototype um and it
02:29 : was an innocent debugger but it was
02:32 : very slow
02:33 : because it didn't have
02:35 : the separated record and replay stages
02:38 : you said to kind of run your program and
02:40 : build the omniscient database as your
02:41 : program ran which meant the slowdown was
02:44 : pretty enormous
02:45 : i think my prototype is actually
02:48 : in that exact same place currently right
02:52 : i i hacked a python interpreter to spit
02:55 : out a log
02:56 : as it's executing
02:58 : and then i would run a separate
03:00 : recreation script that reads back the
03:03 : log and then it'll generate the
03:06 : omniscient
03:07 : database if you have a long execution
03:10 : that can take a really long time yeah i
03:12 : mean
03:13 : basically i was interested in targeting
03:15 : large complex applications like firefox
03:18 : yeah it needed to be
03:20 : reasonably efficient and you know the
03:22 : chrome answer approach you know it was
03:24 : not
03:25 : not that not not required for that what
03:28 : was your original motivation
03:31 : how did you come to this conclusion that
03:34 : i need to build an ammunition debugger i
03:37 : was working on firefox and i spent a
03:39 : fair bit of time debugging firefox bugs
03:41 : and the tools for doing that back then
03:44 : were basically the same as what people
03:46 : are using today unfortunately most you
03:47 : know which is you know gdb basically um
03:51 : well i had an intuition that you know we
03:53 : could do a lot better right like if we
03:54 : gathered more the debugger
03:57 : so we ran the program and gathered lots
03:58 : of information about the execution
04:00 : we could build a better debugging
04:02 : experience with a lot of data that we
04:04 : could collect
04:05 : um and that's what i wanted to
04:07 : experiment with with amber
04:09 : chronicle chronomancer
04:12 : and so that's kind of what i did in the
04:13 : most direct way that i could get at uh
04:15 : so i was i used valve basically modified
04:18 : file grind to uh collect lots of
04:20 : information
04:21 : and it did i mean it worked and that it
04:23 : gave a interesting user experience
04:27 : that was built using it was an eclipse
04:29 : plugin was the interface for that
04:30 : because i was using eclipse as my ide
04:32 : and i uh so did an eclipse plug-in for
04:35 : the debugger interface and you know it
04:37 : worked but the slowdown was enormous so
04:39 : it's like 100x maybe slow down
04:42 : thereabouts and so
04:43 : it made it difficult to use with firefox
04:46 : because you would struggle to reproduce
04:47 : lots of bugs and of course this then the
04:49 : large slowdown made the firefox
04:51 : execution look rather different in some
04:53 : cases so at some point later you know we
04:55 : we were looking at getting some interns
04:58 : uh for missouri research and
05:01 : um one of the mozilla guys andreas gail
05:03 : actually he's mozilla a senior mozilla
05:06 : guy and so he thought that working on
05:07 : debugging would be interesting i already
05:09 : had the idea that it'd be cool if we
05:10 : could record a program execution and
05:12 : replay it that alone will be helpful for
05:14 : debugging bugs that are
05:15 : non-deterministic that are hard to
05:17 : produce so if you just record it once
05:18 : and then replay it and if you do that
05:20 : efficiently that'd be really helpful and
05:22 : i at the back of my mind i also thought
05:24 : hey you know if we can do this recording
05:26 : and replay then eventually we could run
05:30 : that omniscient debugging stuff that i
05:32 : had already implemented we could do it
05:33 : on the replay instead of the recording
05:34 : so that would make that approach more
05:36 : practical because then you can do the
05:37 : recording once and you can process and
05:39 : analyze that recording offline to build
05:42 : this on this database in your mind if
05:44 : you did the record replay method
05:47 : you can get around the performance
05:49 : penalty that you were seeing that's
05:51 : right well but you can you can shift the
05:53 : performance issues to happen at a
05:54 : different time so they're not actually
05:56 : messing up the recording right
05:58 : um also the back of my mind was the idea
06:00 : hey maybe we could parallelize it so if
06:02 : you throw lots of machines a lot of cpu
06:04 : calls are the problem maybe we can do
06:07 : that if we have a recording that we're
06:08 : analyzing instead of having to run the
06:10 : program online anyway so that was the
06:13 : back of my mind but the immediate goal
06:15 : was to implement some chord replay
06:16 : mechanism so the question is what's the
06:17 : most efficient way to do that and you
06:20 : know essentially we came up with the way
06:22 : that rr works right like a computer is a
06:24 : deterministic so if we just capture the
06:26 : non-deterministic inputs then we can
06:28 : rerun the program with those inputs get
06:30 : it to run again the same way that it ran
06:32 : before one of the things at the back of
06:34 : my mind also when i think about that
06:36 : this is a very old project by run down
06:39 : by a guy called michael elizabeth
06:40 : chastain
06:41 : who implemented a very very very simple
06:44 : version of rr essentially just using p
06:47 : trace and it didn't handle multiple
06:49 : threads and it didn't handle more than
06:51 : linux and it didn't handle asynchronous
06:53 : signals and things but for very very
06:55 : basic things it did work that project
06:57 : was interesting and kind of like well if
06:59 : we could just take that approach and
07:00 : make it handle asynchronous advanced
07:03 : and all the modern system call staff
07:06 : then maybe we can come up with a
07:07 : solution that works that was essentially
07:09 : the genesis of rr and
07:11 : um it wasn't too super original like
07:14 : apart from chastain's project there were
07:16 : other projects doing similar sorts of
07:18 : things but but no one had really done
07:20 : the engineering to make it practical and
07:22 : then the real question is if you build
07:24 : it will they come is it actually useful
07:26 : if you build this thing i think that's a
07:27 : really interesting question so we had
07:29 : one intern um albert knoll he started
07:31 : work on it and bought a prototype
07:34 : um and then we actually got pretty far
07:36 : with that and were able to show that a
07:38 : lot of the basic techniques worked one
07:40 : thing we found with that was that the
07:42 : overhead of handling system calls with
07:44 : that approach is pretty high
07:45 : um every time you do a system call uh
07:48 : you add to a context switch to the rr
07:50 : process and basically got a p trace
07:52 : event
07:53 : and then it has like
07:55 : um record some information and then it
07:57 : has to advance the program through the
07:59 : system call and then record some more
08:01 : information in linux switching context
08:04 : to another process is way more expensive
08:07 : than doing a system call and we're doing
08:09 : four context switches for every system
08:11 : call programs to do lots of system calls
08:13 : which many programs do especially
08:14 : firefox those programs got much much
08:18 : slower and so we had another intern come
08:20 : who was a guy called nimrod patos great
08:23 : guy we implemented essentially the
08:25 : system called buffering optimization
08:26 : which was the sort of the major key
08:28 : optimization that makes our work kind of
08:30 : reasonably well and that's basically the
08:32 : idea is that when certain system calls
08:34 : happen instead of taking these p trace
08:36 : traps and doing these context switches
08:38 : to handle it you can essentially handle
08:39 : it without context switching you can
08:41 : handle it essentially in the context of
08:43 : the pro of the process that you're
08:44 : debugging it's it's it's pretty hairy to
08:47 : be honest can i try to explain how that
08:49 : works as i understand it uh yeah i read
08:53 : that paper
08:54 : it's a very very dense paper but uh yeah
08:57 : that that was one of the key parts of
08:58 : the paper the way i understand it is the
09:02 : first time that system call
09:05 : is made uh rr
09:09 : the very first time you actually
09:11 : would probably have to incur that
09:13 : contact switch but then rr traps it and
09:16 : rewrites the calling code to code that
09:19 : would not switch to rr anymore it
09:22 : rewrites the instruction
09:24 : to something that would directly make
09:26 : the system call and then when it i mean
09:28 : what you've said is awkward but but it's
09:30 : a bit more complicated like there's a
09:31 : lot of complexity that that goes on um
09:34 : yeah we rewrite the system call
09:35 : instruction so that instead of doing
09:37 : system call we call into a library
09:39 : that's been loaded into the tracy
09:42 : process and that library
09:45 : um analyzes the system call that's
09:47 : trying to be performed and and if we can
09:49 : if it's simple enough and we can we know
09:50 : how to handle it um then we we have a
09:53 : way of doing the system call
09:55 : uh directly in a way that doesn't cause
09:58 : a p-trace trap and then we can record
10:00 : the results of that system call in a
10:02 : buffer which is accessible to the rr
10:06 : supervisor our supervisor can then like
10:08 : use that buffer to get the results of
10:10 : all those system calls that are being
10:11 : handled in this accelerated way
10:14 : but it's pretty complicated because
10:16 : during replay we need to replay
10:18 : everything that this tracing process did
10:20 : including everything in this special
10:23 : library that has to get actually get
10:24 : replayed and so we have to replay that
10:26 : in a way that it picks up the results
10:28 : from the system call buffer
10:30 : and it gets hairy i would like to try to
10:33 : explain in my own words
10:35 : how time traveling debuggers work uh
10:38 : yeah sure and and then have you either
10:41 : validate me or correct me in some way
10:43 : sure
10:44 : okay the way i understand it map it out
10:47 : in my mind is there are three approaches
10:50 : the first way is state snapshots you
10:53 : snapshot all of the intermediate states
10:56 : the second way is change tracking
10:58 : where
10:59 : you track all the changes that happen to
11:01 : the system over time and log it in some
11:04 : way and the third way is deterministic
11:07 : replay which is what rr does
11:10 : um yeah so this is what would happen if
11:13 : you represent
11:14 : the state of your program over time so
11:16 : this is a timeline and you have state 0
11:19 : state 1 state 2.
11:21 : and
11:23 : and each
11:24 : state change is prompted by this update
11:28 : right this u1 is update one u2 is update
11:32 : two so
11:33 : if you if you took the state snapshot
11:36 : approach you're saving every single
11:38 : state
11:39 : into a database or a log somewhere
11:42 : yeah um and this obviously this would
11:45 : probably be prohibitively expensive for
11:48 : most ceremonials
11:50 : but um but git does do this as i
11:53 : understand
11:55 : so i mean
11:56 : yeah i don't like this is the logical
11:59 : model that git provides but git actually
12:02 : stores differences in most cases rather
12:04 : than explicit states
12:06 : um
12:07 : so for debugging like you could
12:10 : potentially do this for like really toy
12:12 : examples but like no one actually tries
12:14 : to do this
12:15 : in practice because it as you said you
12:18 : know it just doesn't it wouldn't scale
12:20 : because this is going to be a lot of
12:22 : state if you have a larger program
12:24 : the second approach is change tracking
12:27 : where
12:28 : you store the initial state
12:30 : and you store every update that comes
12:34 : into the system that can
12:36 : mutate the state in some way
12:38 : and i want to say that is what your
12:41 : first prototype did is that that'd be
12:44 : correct um
12:47 : yeah
12:48 : i guess i mean
12:50 : there's a combination of things it's a
12:52 : bit like git action just implementation
12:54 : in the sense that you do need to be able
12:56 : to efficiently reconstruct any piece of
12:59 : state so
13:00 : you can't just use updates if you only
13:03 : have updates then figuring out what a
13:05 : whole page of memory looks like at a
13:07 : certain point in time for example might
13:09 : require you to sort of look at all the
13:11 : updates that page
13:13 : going back to the beginning of time
13:16 : and that could be slow so typically what
13:19 : you do what we did in um chronomancer
13:22 : and what we do in finasco is
13:24 : we actually have a mix of changes and
13:28 : snapshots so you want to take some
13:30 : snapshots periodically
13:33 : between those snapshots you record all
13:34 : the changes
13:36 : it's like a hybrid between this and yeah
13:39 : yes yeah so that there isn't too many
13:42 : update events elapsed between that's
13:45 : right so you don't have to
13:47 : be able to get to this particular state
13:48 : without applying too many snapshots and
13:50 : you know that conceptually that is
13:52 : exactly how good is implemented get has
13:55 : like snapshots and updates gotta i see
13:58 : okay
13:59 : now i want to get to uh record replay
14:02 : and i stole this slide from your
14:04 : presentation
14:06 : um
14:06 : which is the idea that
14:08 : the cpu on the right here
14:12 : is a deterministic machine so if you
14:14 : throw the same code
14:16 : at it with the same initial
14:18 : state
14:19 : it should do the same thing
14:22 : every time
14:23 : even if it weren't for the sources of
14:27 : non-determinism on the left
14:29 : so
14:30 : if you can repeat the sources of
14:32 : non-determinism the exact same way every
14:35 : time then it should also give you the
14:37 : right answer every time
14:39 : is that a good way to say it so yeah
14:42 : okay so this is a timeline uh
14:46 : the dot on the timeline are points in
14:48 : time now
14:49 : and then the arrows are events might be
14:52 : like a mouse event or a keyboard event
14:55 : or a network event
14:57 : so the events are coming in at certain
15:00 : points in time of the program but if we
15:03 : think about a program as having state
15:06 : changes and if the pink dots were state
15:08 : changes the program is going to change
15:11 : state as it's going but in between some
15:14 : of the state changes it'll get
15:16 : interrupted by these events
15:19 : and i think the trick with record replay
15:23 : is if you want to get the program to do
15:25 : exactly what it did the first time you
15:27 : want to deliver those events in between
15:31 : the correct state changes that's in
15:34 : place within the timeline as it did the
15:36 : first time yes so so you would save like
15:39 : the event and the time at which that
15:42 : event was delivered to the to the
15:44 : process into a log somewhere
15:47 : yeah
15:48 : so i did this okay
15:50 : you think yeah yeah i mean what you said
15:52 : was all correct cool thank you so much
15:54 : for that could you tell me some like
15:56 : success stories i think there's certain
15:59 : kinds of difficult to reproduce
16:01 : non-humanistic bikes and i've had people
16:02 : in mozilla say you know this bug could
16:04 : not have been fixed without rr so one
16:07 : particular class of bugs where rr has
16:09 : been
16:10 : extremely useful is for people who are
16:12 : debugging garbage collectors garbage
16:14 : collective bugs are often
16:15 : non-deterministic involve memory
16:17 : corruption it's often very difficult
16:19 : to go from where you detected the bug to
16:22 : where the bug was caused
16:24 : and rr makes those bugs actually
16:25 : relatively easy to fix one guy i think
16:27 : was doing research on garbage collection
16:29 : he told me that rr had saved them like
16:31 : months literally months of work
16:34 : which is always a good thing to hear um
16:36 : and uh certainly at mozilla there have
16:38 : been people who have fixed garbage
16:39 : collection bugs uh and there are i know
16:41 : there are other people
16:42 : um using recording replay debugging
16:45 : sometimes involving rr um to fix bugs
16:48 : big projects that use garbage collectors
16:50 : so that is an interesting area yeah i
16:53 : can add my own data point there too i
16:55 : was uh debugging the python interpreter
16:58 : and indeed it was a garbage collector
17:02 : issue that i was dealing with where
17:04 : new object was using memory that was
17:07 : coming from a previously used object it
17:10 : was i wouldn't even know how to debug
17:12 : something like that but but with rr you
17:14 : just say who is the last person or last
17:17 : code that wrote to this and go back so
17:20 : here's the watch point that that's such
17:22 : a key
17:23 : pattern to use with rr you set a watch
17:26 : point on that memory location and then
17:28 : what reverse continue
17:30 : to whoever wrote yeah
17:33 : that is kind of our superpower like yeah
17:36 : when people discover that and they start
17:38 : using it effectively
17:40 : yeah every time something incredible
17:42 : like that happens i'm like oh my god
17:44 : like why why don't we use this all the
17:47 : time have you ever had like a eureka
17:49 : moment where you're really amazed that
17:52 : you made something like this work
17:55 : yeah i mean obviously when we people
17:57 : started discovering that trick with
17:59 : reverse watch points that was a clear
18:01 : breakthrough moment i'd say another
18:03 : interesting breakthrough moment probably
18:05 : one of the best things that i've ever
18:06 : had happened with someone using my work
18:09 : was
18:09 : a mozilla javascript developer they'd
18:12 : already used rr for bird and they were
18:13 : using rr and then they they called a
18:16 : program function from gdb they wanted to
18:18 : pretty print some javascript values so
18:20 : they did like call and then you know js
18:23 : pretty print and then some object
18:25 : address
18:26 : and it worked because that's what they
18:27 : would do when they're using gdb
18:30 : after it worked they said hang on
18:33 : what just happened
18:35 : because it's during the replay and
18:39 : that function
18:41 : did not run during recording right so
18:43 : the question was like how is it in rr
18:45 : they're able to get the function to
18:47 : actually work now we have an
18:49 : implementation that does this involves
18:52 : basically cloning the current state of
18:54 : the program and then allowing it to what
18:56 : we call diverge you clone the current
18:58 : state of the entire program then the
18:59 : function executes from that state that
19:02 : changes the state of the program it
19:03 : changes registers and possibly memory
19:06 : and so we actually throw away the
19:07 : divergent session once you've finished
19:09 : with the function and go back to the
19:11 : replay
19:12 : but that all happens under the hood from
19:13 : the user's point of view it's just magic
19:15 : you call this function that never
19:16 : happened during recording and it just
19:18 : works what was brilliant about that from
19:20 : my point of view is that it's the
19:21 : perfect feature the perfect feature is
19:24 : the feature that it's incredibly
19:25 : difficult to implement and it's
19:27 : incredibly powerful and cool but users
19:29 : don't even realize that they just use it
19:32 : and everything works as they expected
19:34 : and magic it's magical but the magic is
19:37 : completely predictable and hidden
19:39 : to me that is the perfect application
19:41 : feature so i was really excited when
19:43 : when i got when that guy said that
19:45 : because it was like it's like what just
19:46 : because he said what just happened it's
19:48 : like that's when he suddenly realized
19:50 : that some magic had actually happened
19:52 : without him even noticing before yeah so
19:54 : it's amazing yeah that's really cool
19:56 : why do you think debugging is not a
19:59 : first-class citizen in many tech
20:02 : companies perhaps um yeah so that's a
20:05 : really good question so one issue is
20:07 : that traditional debuggers are just
20:08 : pretty bad
20:09 : um the user experience of the debugging
20:12 : is pretty bad but also the problem
20:15 : that what they actually do is also
20:16 : pretty bad like not being able to
20:18 : reverse this cute and embargo is a
20:19 : pretty bad deficiency since debugging is
20:22 : about tracing effects back to causes
20:24 : which means you actually do want to go
20:25 : backwards in time and so traditional
20:27 : debuggers only let you approximate that
20:28 : by essentially doing debugging session
20:31 : after debugging session setting a
20:32 : breakpoint is like playing the prices
20:35 : right where right yeah and you try to
20:37 : get to the as close as possible without
20:40 : going over yeah one strategy is like you
20:42 : don't want to go over you just guess one
20:45 : dollar right and
20:47 : debugging people sometimes do that it's
20:49 : like i just want to stop at the first
20:51 : line because i don't know what's going
20:54 : to happen
20:55 : it's a good analogy another problem with
20:57 : digital debuggers though is that they
20:58 : require you to sort of connect to where
21:00 : the program is running and then stop it
21:02 : and a lot of environments these days
21:04 : like the cloud you can't really do that
21:06 : like if your program is actually like
21:08 : hundreds of processes running on
21:09 : hundreds of machines you don't
21:11 : necessarily know which one to attach to
21:13 : ahead of time and if you stop it you
21:15 : kind of break this whole system
21:17 : uh so these things have to keep running
21:20 : um and so a record and replay approach
21:24 : is actually
21:25 : a much better fit to that sort of thing
21:27 : but of course most people don't have
21:28 : access to record repair technology maybe
21:30 : rr doesn't work for them or they've
21:32 : never heard of rr which is still the
21:33 : case amazingly often for various reasons
21:35 : people have gotten out of the habit of
21:37 : using debuggers and once you fall out of
21:39 : the habit of using debuggers you kind of
21:42 : fall into a downward spiral where you
21:44 : forget how to use them your
21:46 : infrastructure stops even supporting
21:48 : interactive debugging because people
21:50 : aren't using it so yeah so whatever
21:51 : support there was kind of rots away i
21:54 : agree um
21:55 : and people just get to the point where
21:56 : like well i'm you know i only know how
21:58 : to do log base debugging now print
22:00 : debugging which is the thing everyone
22:01 : falls back to that's all you know how to
22:03 : do that's all you're going to use and
22:05 : people come to believe
22:07 : rightly or wrongly that dedicated
22:09 : debugging tools just don't work for them
22:11 : and they come to a super eventually they
22:12 : can't they can't work
22:14 : um
22:15 : so there's this downward spiral that
22:17 : it's very difficult to fight against and
22:19 : i think as an industry
22:20 : some people some of us are trying to
22:22 : like swim upstream and try to show
22:23 : there's a better world but it is
22:25 : difficult yeah sometimes i see bugs in
22:28 : the debuggers and i myself just like i
22:31 : don't want to deal with a buggy debugger
22:33 : so i'm just going to do christmas
22:34 : statements so i mean there's part that's
22:36 : the other side of the another part of
22:38 : the sort of the vicious cycle is that
22:41 : with people as people give up on using
22:43 : debargos the incentive to actually
22:45 : invest in debuggers and the
22:46 : infrastructure around them like the
22:47 : compiler infrastructure and so on
22:50 : those incentives decrease every time
22:51 : people stop using debuggers and um and
22:54 : so the debuggers and that infrastructure
22:56 : get worse um but or can't keep up with
22:58 : other parts of the industry so this is
23:00 : all the bad news i mean the good news is
23:02 : there are people who care you know
23:03 : obviously i care but there are lots of
23:05 : other people who care so
23:06 : you know we just got to keep pushing um
23:09 : basically what i've been trying to do
23:11 : yeah one of the design goals of rr is
23:13 : you want to avoid using code
23:16 : instrumentation by code instrumentation
23:18 : do you mean like something that a tool
23:20 : like valgrind would do yes right coder
23:22 : sprint what i mean specifically for rr
23:24 : here is when the program runs during
23:27 : recording we want it to run the original
23:30 : machine code without modification as
23:32 : much as possible that's what i mean
23:34 : precisely by that the reason is
23:36 : basically that's a way to minimize
23:38 : overhead
23:39 : um as soon as you say you know i'm going
23:41 : to look at every instruction executes
23:44 : during recording and possibly modify it
23:46 : you know you create problems for
23:47 : yourself by doing that there's here
23:49 : every time new instructions execute you
23:51 : have to analyze them you have to parse
23:52 : them potentially modify them
23:55 : it also means that certain things like
23:56 : self-modifying code things like jit
23:59 : compilers they become much harder to
24:00 : handle suddenly you know so rr's
24:02 : approach was we're not going to do that
24:03 : also much more to implement so these you
24:06 : know you can't implement these binary
24:07 : modification things that
24:09 : like valgrind or dynamo rio that
24:11 : actually look at all instructions
24:13 : but it's considerably more work to
24:15 : implement you want to avoid doing the
24:17 : work and doing all that maintenance work
24:18 : it means for example when intel releases
24:20 : like new instructions
24:21 : which they do all the time then you need
24:23 : to add support for those to your to your
24:25 : system
24:27 : and you know in rr's case you know there
24:29 : are a few instructions that cause
24:31 : problems for rr sometimes but but most
24:34 : almost no instructions do so like you
24:35 : know every time intel releases like 1
24:37 : 000 new like
24:39 : um vector instructions you know they
24:41 : make no difference to rr basically so
24:44 : that's really nice
24:46 : how did you come to the realization that
24:48 : you wanna like rr was not enough and you
24:51 : needed to create pronounce call
24:54 : with ri you can reproduce any program
24:56 : state you want right it's just a matter
24:57 : of starting at the beginning and running
24:59 : some state we can essentially from our
25:01 : get access to all program states but we
25:04 : can't do it super efficiently right like
25:06 : if you tell me what was the value of
25:07 : this memory location at this point in
25:09 : time i can go get it but it's going to
25:11 : take a while because i have to
25:12 : re-execute a bunch of code but we can
25:14 : get that information so then the
25:15 : question is if i have access to all
25:17 : program states efficiently in your
25:19 : program state any point of time what
25:21 : would be the ideal debugger interface
25:23 : like what kind of debugger interface can
25:25 : you build on top of that
25:26 : um and this is something i started to
25:28 : explore with chronomancer right and then
25:30 : kind of stop
25:31 : because we couldn't produce that data
25:33 : efficiently but what if we could produce
25:35 : that efficiently what kind of interface
25:36 : could we build you know it won't look
25:38 : like a traditional debugger so i wanted
25:40 : to explore what is the best debugging
25:42 : experience we can we can produce that's
25:44 : a really interesting question and and i
25:46 : think that was why we we started doing
25:48 : finosko and one of the reasons anyway um
25:50 : and pronunciation has some answers to
25:52 : those questions or at least you know
25:54 : some possible answers to those questions
25:55 : the other question is of course how do
25:57 : you implement it can you efficiently get
26:00 : access to all those states and can you
26:02 : make that actually work from an
26:03 : engineering point of view so that you
26:04 : get results in front of users you know
26:07 : quick enough to make it to matter right
26:09 : yeah um and and so that's sort of a
26:11 : massive engineering problem which we
26:13 : also have to solve
26:14 : um and which i think we have solved uh
26:17 : and i think the results are quite quite
26:18 : surprisingly good that's really the
26:20 : thing so it's like rr we could use it as
26:23 : the basis for doing a lot more and so i
26:25 : wanted to pursue that right like because
26:27 : basically what i gives you is gdb
26:30 : interface plus reverse execution which
26:32 : as we've talked about is incredibly
26:34 : powerful but like you're still basically
26:36 : stuck at looking at a single program
26:38 : state at a particular fixed point in
26:40 : time and you can just move that point of
26:42 : time backwards which okay great but
26:44 : that's kind of limiting still like what
26:46 : if we could look across time what what
26:48 : can we do with you know instant access
26:51 : to everything that's what i wanted to
26:52 : explore the engineering side in your
26:55 : talk you did in israel you said that
26:58 : whereas the ambition database is just
27:00 : too slow to generate in your previous
27:03 : attempt now that you have rr
27:06 : now you can leverage parallel execution
27:10 : and and execute multiple dynamic
27:12 : analysis at the same time
27:15 : so if you have a recording if you have
27:17 : an rr recording in particular we can do
27:19 : a replay of that recording and we can
27:20 : instrument the replay so we can actually
27:22 : modify the machine code that's executing
27:24 : during the replay so it does the same
27:26 : thing but also collects extra
27:28 : information so this is basically what
27:29 : we're doing with chronomancer before
27:32 : but we're doing it only during the
27:34 : replay okay so i might run a section of
27:37 : code and it will run like a hundred
27:39 : times slower but we're gonna collect a
27:40 : lot of information like what memory
27:42 : locations are modified and like which
27:44 : exactly which instructions are executed
27:46 : i can put all that in some giant
27:47 : database and how you do how you put it
27:49 : in the database is also very interesting
27:50 : but i can talk about that
27:52 : so um
27:53 : but basically you're running this code
27:55 : is gathering this information and and it
27:57 : is quite slow but thing is that i can do
28:00 : this on any segment of code right i can
28:02 : pick some time range of my program and i
28:05 : can just say let's instrument just that
28:07 : time range and and the rest i can just
28:09 : run normally okay and so what i can then
28:11 : do is i can if i have lots of computers
28:13 : right or lots of cpus on one computer i
28:16 : can have them all look at a different
28:18 : time range so let's just end with that
28:20 : time range and dig out catch all the
28:22 : information from that time range so even
28:24 : if each particular time range is running
28:26 : like 20 times slower
28:29 : if i'm throwing like 50 cpus at it i'm
28:32 : actually winning
28:33 : um
28:34 : and this is more or less what we do
28:37 : i see
28:38 : and it's not you know
28:40 : and i know that not only can you look at
28:42 : different time ranges but you can take
28:43 : the same time range and you can analyze
28:46 : it different ways you know one cpu i'm
28:48 : going to you know just track exactly
28:49 : what gets executed and what happens to
28:51 : register values and on a different cpu
28:53 : i'm going to track what happens to
28:54 : memory for the same time so it can
28:56 : actually
28:57 : split my my ranges up i can parallelize
28:59 : in another axis
29:02 : how feasible is it to make rr work on
29:05 : other operating systems or or maybe cpu
29:08 : architectures
29:09 : that's a great question uh so we do
29:12 : actually have an experimental
29:15 : rr implementation for arm for a
29:17 : particular arm chip okay um so it is
29:20 : possible um we need certain hardware
29:22 : features and we need to avoid certain
29:25 : other hardware features
29:26 : um which makes things complicated i
29:28 : don't have time we don't have time now
29:29 : to go into what those are but um but
29:31 : that is there's sort of written up in
29:32 : some of the papers um basically yeah you
29:35 : need but basically there's no there's no
29:37 : intrinsic reason why it has to be x86
29:39 : only it's just making but we need we
29:41 : need cpu designers to sort of add the
29:43 : right features what about other
29:44 : operating systems
29:46 : other operating systems um
29:49 : it's hard but it is doable like linux
29:51 : happens to have like a combination of
29:53 : features that makes r work we added one
29:56 : or two things to the kernel or one
29:57 : really um to help but
30:00 : um but basically it worked out of the
30:02 : box other operating systems might not
30:04 : have those features and i don't really
30:06 : look too hard to figure out if they do
30:08 : or not the main thing is you need the
30:09 : source code of the operating system
30:11 : because you need to understand exactly
30:13 : what system calls do and how they work
30:15 : in absolute complete detail and we just
30:18 : don't have that information for windows
30:20 : or mac for example
30:23 : or ios so um
30:26 : it could be like it's not not saying it
30:27 : couldn't be done but you would need
30:30 : either lots and lots of money to do lots
30:32 : and lots of reverse engineering or you
30:33 : would need to be apple or microsoft to
30:36 : actually have access to all the
30:38 : information and potentially change
30:39 : things that are causing problems so it
30:41 : can be done
30:43 : the approach would work
30:44 : but you would need
30:46 : deep deep deep knowledge of the os which
30:48 : people outside those companies don't
30:50 : have so i wouldn't want to work on
30:51 : would it be feasible to implement or
30:53 : time travel debugging in the programming
30:56 : language level is that something you
30:59 : would do
31:00 : i mean people have done omniscient
31:02 : debugging at the virtual machine level
31:04 : for java and for
31:06 : something yeah i think you've done it
31:07 : for python um and other people have i
31:09 : think there's another one as well um
31:12 : and some people have actually done this
31:14 : and it does work um you can do it you
31:17 : know there's trade-offs obviously you
31:19 : know if you're doing it at language
31:20 : level you miss thing cases where people
31:23 : are doing machine code that's not part
31:25 : of your language
31:26 : um there could be like you know
31:27 : handwritten machine code or libraries
31:30 : where you don't have source code or it
31:32 : could also be
31:33 : just in time compilers generating
31:35 : machine code that you don't know about
31:36 : so it's hard to get complete visibility
31:38 : into everything that's going on if you
31:39 : do things at the language level or the
31:41 : compiler level but a number of projects
31:44 : have done that and do do that though and
31:45 : it is it can be practical and some there
31:48 : can be some advantage to do that like
31:51 : for firefox it wouldn't have worked
31:53 : because firefox has a jet compiler and
31:55 : it has all this
31:56 : really low level code that won't debug
31:58 : so that would not have worked for
31:59 : firefox but i can understand why it
32:01 : would work for some other applications
32:03 : got it what do you think are the next
32:05 : steps in time travel debugging i think
32:09 : we need to get more people using these
32:11 : tools um
32:13 : just get more people using all right
32:14 : there's still a lot of people out there
32:16 : who could benefit from our who haven't
32:17 : used it and i know because sometimes
32:19 : recently people come to me and say i've
32:20 : been you know i've just discovered ira
32:22 : it's amazing i could have been using it
32:23 : for years but i never knew about it i
32:25 : mean that is still happening so more
32:27 : publicity these kinds of interviews
32:29 : great awesome
32:30 : um and you know obviously pinasco more
32:33 : users for nosco will be good um more
32:35 : users for the technology that exists
32:37 : will be really helpful
32:38 : um and more because that one of things
32:40 : that generates more awareness from the
32:42 : operating systems people and the
32:43 : hardware guys who actually we need the
32:44 : help
32:45 : um so that's really my main wish got it
32:48 : thank you so much for your time see you
32:50 : toby thanks a lot really enjoyed it