Series: Time Traveling Debugger

1. Time Traveling Debugger - Part 1

Jan 10th 202029:14

In this video I demonstrate the time traveling debugger I built over my vacation. A time traveling debugger is a debugger that can step backwards in time. I devised a scheme to implement this concept using immutable data structures within the programming language. The Fun Programming Language was used as the test bed for implementing this concept.

Watch “Time Traveling Debugger - Part 1”

2. Time Traveling Debugger - Part 2

Jan 10th 202023:17

I continue demonstrating the time traveling debugger I built over my vacation. The debugger relies on a history file which contains snapshots of every state of the program’s execution. In this video I demonstrate how the debugger works with heap objects: arrays and dictionaries.

Watch “Time Traveling Debugger - Part 2”

3. Time Traveling Debugger - Part 3

Jan 13th 202028:49

In this video I’ll dig into the internals of the code generator of the Fun language to show you where most of the magic happens to make the time traveling debugger work.

Watch “Time Traveling Debugger - Part 3”

4. Dream of the Time Machine

Feb 21st 202029:30

In this video I will explain the motivation behind my work on the time traveling debugger, the prior art surrounding this idea, and give a status update on the project.

Watch “Dream of the Time Machine”

5. Introducing the Deep Zoom Debugger

May 22nd 202005:17

I give a brief introduction to the deep zoom debugger. The deep zoom debugger is a new concept of what a debugger for programs could be. It uses a zooming based user interface a la Google Earth to allow the programmer to inspect the program from differ levels of detail.

Watch “Introducing the Deep Zoom Debugger”

6. Deep-Zoom Debugger Demo with Huiqi Zhou

May 16th 202017:47

I demo the deep-zoom debugger to my colleague Huiqi Zhou.

Watch “Deep-Zoom Debugger Demo with Huiqi Zhou”

7. Time Machine Status Update 1

Aug 19th 202036:22

I give a status update on the project I've been working called the "Time Machine".

Watch “Time Machine Status Update 1”

8. Time Machine Status Update 2

Aug 28th 202027:43

This time on Time Machine Status Update I talk about my failed attempt at improving the performance of the $stringify serialization function and the effort to update the zoom debugger UI to work with the sqlite-based database scheme.

Watch “Time Machine Status Update 2”

9. Design Problem: Data Fetching Scheme of Zoom Debugger

Aug 31st 202037:40

I present a design problem I encountered while working on the zoom debugger and how I resolved it, at least for the moment. I hope this can be instructive for some developers towards solving problems of their own.

Watch “Design Problem: Data Fetching Scheme of Zoom Debugger”

10. How Time Traveling Works

Nov 16th 202031:55

In this video, Huiqi and I talk about the time traveling debugging I've been working on. We plan to explain the architecture and inner workings of the debugger as well as covering some important computer science concepts along the way over a series of episodes.

Watch “How Time Traveling Works”

11. What it's Like to Code with a Time Traveling Debugger

Jan 11th 202147:52

In this video I want to allow Huiqi and you the viewers to experience what it's like to code with a time traveling debugger. We do a medium difficulty challenge from called Add Two Numbers: At the end we also go over how to get setup if you want to take the debugger for a spin yourself. You can find that info at

Watch “What it's Like to Code with a Time Traveling Debugger”

12. Time Traveling Debugger for Python

Feb 2nd 202135:08

I started working on a time-traveling debugger for Python. This a video documenting it.

Watch “Time Traveling Debugger for Python”

13. Python Debugger Update

Feb 2nd 202118:54

Here I give an update about the progress of the time traveling debugger for Python.

Watch “Python Debugger Update”

14. rr and Time-Traveling Debuggers

Mar 10th 202129:00

In this video I tell Huiqi about rr: a report/replay debugger for C and C++. We talk about how record/replay debuggers work using Redux as a starting point.

Watch “rr and Time-Traveling Debuggers”

15. Time Travel Debugging (in Python)

Mar 24th 202105:44

This video introduces the time traveling debugging for Python I've been working up to today. It show what it's like to code using time traveling debugging, how my implementation works, as well as a proof of concept debugging UI called the zoom debugger. The time traveling debugger project can be found at:

Watch “Time Travel Debugging (in Python)”

16. Omniscient Debugging (ODB) With Bil Lewis

Apr 1st 202145:24

Bil Lewis published an influential paper in 2003 called "Debugging Backwards in Time". It inspired me and others to build time traveling debuggers of their own. The debugger he built was called ODB, or the Omniscient debugger, implemented for Java. I was very lucky to get to chat with him over zoom, ask him some questions, and even have him demo the ODB to me. We had an amicable chat. The source code of ODB can be found at, and his paper can be found at Also, he once did a talk at Google which was recorded and published here:

Watch “Omniscient Debugging (ODB) With Bil Lewis”

17. TT Debugger Data Storage Optimization

Apr 7th 202122:46

In this video with Huiqi I describe a proposal for a change in the design of the time travel debugger's state storage mechanism, in hopes of improving the performance of the recreate program and the disk usage of the resulting history file.

Watch “TT Debugger Data Storage Optimization”

18. Data Visualization in Debugging

Apr 7th 202117:46

In this video Huiqi and I talk about the role of data visualization in debuggers.

Watch “Data Visualization in Debugging”

19. Questions Developers Ask When Debugging

Apr 14th 202146:07

In this episode, I share a status update on the time-traveling debugger project and then Huiqi and I do a brainstorming session on possible visualizations for debuggers.

Watch “Questions Developers Ask When Debugging”

20. Let's Talk About Performance

Apr 21st 202138:37

I tell Huiqi about a performance war story that took place last week.

Watch “Let's Talk About Performance”

21. Learning C/C++ vs Go or Rust

Apr 21st 202106:39

I talk briefly with Huiqi about learning C and C++ vs more modern languages like Go and Rust.

Watch “Learning C/C++ vs Go or Rust”

22. Let's Talk about Performance! (2)

Apr 28th 202116:33

We talk about performance again! We follow up on the previous week's plan to speed up the "recreate" program. Short version: it worked as predicted! Long version: watch the video. The Stackoverflow answer written by Mike Dunlavey that was cited can be found here:

Watch “Let's Talk about Performance! (2)”

23. Python vs C: Side by Side (Plus Macros!)

Apr 28th 202112:02

I rewrote some Python code in C last couple of weeks to improve the performance of the "recreate" program. Here I show a brief side-by-side comparison of the code before and after with Huiqi.

Watch “Python vs C: Side by Side (Plus Macros!)”

24. rr Time-Traveling Debugger

May 19th 202117:13

In this video I give an introduction and a demo to the rr debugger. rr ( is a revolutionary time-traveling debugger. It was initially developed by Robert O'Callahan and others at Mozilla for the purpose of debugging difficult to reproduce bugs in Firefox. It uses the deterministic replay technique to enable time-traveling. The ability to reverse time in debugging is a game changer. It has saved me hours of debugging time and allowed me to solve mysteries I would not otherwise have been able to solve. The ability to reverse time gives rise to some new debugging techniques. I cover one technique here called "Backup and Drill Down" and I plan to cover more in future videos.

Watch “rr Time-Traveling Debugger”

25. Making a Terminal-Based UI Framework

May 26th 202119:23

I am making a terminal-based UI framework for the time-traveling debugger project. Here I share what I did so far with Huiqi.

Watch “Making a Terminal-Based UI Framework”

26. Debugging UI Widget with Time Travel

Jun 18th 202115:58

I recently got my time-traveling Python debugger working again after some major reconstruction work and it felt really good to use it for actual work again. In this video I should you an actual debugging session where I solve a couple of bugs in a text field widget I am working on.

Watch “Debugging UI Widget with Time Travel”

27. Building a UI Framework with Time Travel

Jun 18th 202119:14

In this video I implement keyboard focus for text fields in a UI framework I've been working on. The implementation was aided by the time-traveling Python debugger.

Watch “Building a UI Framework with Time Travel”

28. An Interesting Bug

Jun 27th 202113:49

I tell the story of an interesting bug I encountered while working on a scroll view widget, how I used the time-traveling debugger to tackle it, and the lessons I learned.

Watch “An Interesting Bug”

29. Time Machine Status Update 3

Aug 26th 202109:16

This is the first status update on the time-travel debugger in a while, and I do it in Huiqi's presence. I made a windowing system on top of a UI framework I call oui. I implemented a tree widget which is now able to display composite data structures like dictionaries and object with the expand and collapse feature.

Watch “Time Machine Status Update 3”

30. Seeing Code As Fractals

Aug 26th 202118:50

I show Huiqi an interesting performance bug and the interesting way I visualized the execution of the code to understand the problem.

Watch “Seeing Code As Fractals”

31. Time Machine Status Update 4

Sep 2nd 202104:40

Another status update for the time-traveling debugger.

Watch “Time Machine Status Update 4”

32. Time Machine Status Update 5

Sep 11th 202111:40

I give another status update about the time-traveling debugger with Huiqi. We slipped into a little bit of dynamic programming talk in the middle of the conversation. Enjoy!

Watch “Time Machine Status Update 5”

33. Time Machine Status Update 6

Sep 16th 202113:55

I give Huiqi another status update of the time-traveling debugger. We talked about sticky line numbers, file selection menu, hit indicators for file lines, the launch/goto bar, and future features.

Watch “Time Machine Status Update 6”

34. Time Machine Update and Design Discussion

Sep 23rd 202142:53

I give Huiqi an update on the debugger project and we have a discussion about different approaches.

Watch “Time Machine Update and Design Discussion”

35. Time-Travel Debugging with Robert O'Callahan

Feb 13th 202232:56

I have the great pleasure of speaking to Robert O'Callahan (roc), the creator of the rr record/replay debugger and co-founder of Pernosco. We talk about the origins of his various projects, how they work, what they mean for the software industry, and more. roc's blog - rr - roc's rr talk - Pernosco - Chromometer paper - My videos about rr -

Watch “Time-Travel Debugging with Robert O'Callahan”