rr's chaos mode for hard to reproduce bugs

219 views
Skip to first unread message

Jason E. Aten

unread,
Jun 4, 2025, 7:18:45 AMJun 4
to golang-nuts
This is a fascinating approach to finding hard to
reproduce event-interleaving related bugs.

I'm particularly interested in this approach 
because rr record and replay plus chaos 
mode is directly applicable to 
Go programs -- whereas deterministic simulation
testing (DST) is next to impossible in a Go program 
using more than 4GB of memory (like most of my
programs) because this rules out wasm.

In contrast to DST, the rr+chaos approach
accepts you will be randomly 
sampling executions, but by recording all of them you
can still get reproducibility when you do hit the issue.  

rr is very efficient at recording. Green test runs can be quickly
discarded.

In a blog from 2016, Robert O'Callahan, one of the principal rr authors,
talks about the design of rr's chaos mode for provoking hard to find 
concurrency bugs:

> To cut a long story short, here's an approach that works. 
> Use just two thread priorities, "high" and "low". Make 
> most threads high-priority; I give each thread a 0.1 
> probability of being low priority. Periodically re-randomize 
> thread priorities. Randomize timeslice lengths. 
>
> Here's the good part: periodically choose a short random interval, 
> up to a few seconds long, and during that interval do not 
> allow low-priority threads to run at all, even if they're 
> the only runnable threads. Since these intervals can 
> prevent all forward progress (no control of priority inversion),
>  limit their length to no more than 20% of total run time. 
>
> The intuition is that many of our intermittent test failures 
> depend on CPU starvation (e.g. a machine temporarily 
> hanging), so we're emulating intense starvation of a few 
> "victim" threads, and allowing high-priority threads to 
> wait for timeouts or input from the environment 
> without interruption.
>
> With this approach, rr can reproduce my bug in 
> several runs out of a thousand. I've also been able 
> to reproduce a top intermittent (now being fixed),
> an intermittent test failure that was assigned to me, 
> we've been chasing for a while. A couple of other 
> people have found this enabled reproducing their 
> bugs. I'm sure there are still bugs this approach 
> can't reproduce, but it's good progress.
> 
> I just landed all this work on rr master. The 
> normal scheduler doesn't do this randomization, 
> because it reduces throughput, i.e. slows down
> recording for easy-to-reproduce bugs. 
> Run rr record -h to enable chaos mode for 
> hard-to-reproduce bugs.

Links to more info and background on rr:


https://fitzgen.com/2015/11/02/back-to-the-futurre.html

https://www.percona.com/blog/replay-the-execution-of-mysql-with-rr-record-and-replay/


Robert's talk, "Taming Non-determinism" from 9 years ago is
a good technical introduction to rr.

NB The Delve debugger for Go supports rr, so you can get goroutine stack traces.

Enjoy.

- Jason

Jason E. Aten

unread,
Jun 5, 2025, 4:28:07 PMJun 5
to golang-nuts
Hmm. Maybe rr found a bug in runtime GC code(?)... or
maybe it will be hard to use rr on Go. Let's see what the runtime folks say

cpasmaboiteaspam

unread,
Jun 6, 2025, 2:05:35 AMJun 6
to golang-nuts
Hello Jason,

Intuitively,  to play with timing of goroutines,
to manually inject delays where none exists,
is what new comers do, 
to experiment with deadlocks,
or simply build better understanding, 
at least I did that very often.
Therefor your suggestion definitely makes sense,
to me, a non low level programmer perspective.

(not a useful email, i only want to show some support 
to your various posts which i read with lots of interest,
others too... but physically speaking 
it is hard to follow everybody,
I liked the last post from Robert Griesemer
I like all the post of the Go blog anyway...... 
and the coroutines.... c'est interminable...= )

Thank you,
Thank you all.

Jason E. Aten

unread,
Jun 6, 2025, 3:32:46 AMJun 6
to golang-nuts
Thanks for your note of appreciation, cpasmaboiteaspam.
Reply all
Reply to author
Forward
0 new messages