Undo (where I'm CTO) has existed for longer than RR and its real benefit is that it scales to use cases where RR (for one reason or another) isn't a fit.
Technically:
* Doesn't need hardware performance counters - runs on more CPUs and on cloud systems (where performance counters are often blocked).
* Can attach and detach at any time - means you get to record just a subset of program execution that's interesting.
* You can our ship recording tech with your application and control it by API, so you can grab crash recordings on customer systems.
* Supports programs that share memory with non-recorded processes.
* Supports direct device access (e.g. DPDK).
* Accelerated debugging features - searching with recordings using parallel processing, accelerated conditional breakpoints a few thousand times faster than native GDB.
* We provide a stable, patched fork of GDB that we're occasionally told is more stable than the default.
For many people's use cases none of these really matter - they should use RR if they're not already.
But if you need any of these things then Undo can give you time travel debugging. In practice, it's usually big software organisations that we deal with because they have development pain and the extreme requirements we can match.
Undo has cool features like Live Recording that we don't have in rr.
They don't need access to the hardware PMU which is a big advantage in some situations.
They can handle accesses to shared memory in cases where rr can't.
https://undo.io/resources/undo-vs-rr/ is a good resource.
AFAIK it records multithreaded applications on multiple threads and CPU, rr records them on a single OS thread, AFAIK. Not sure about replay. Never used undo though, so not sure how much better it is.
rr does support multithreaded and multi-process applications, via, like Undo[1], allowing only a single thread to run at a time. (edit note - that's only about multithreading; Undo might have parallel multi-process recording)
Undo founder here. I just got a slack from one of our marketing team who is thrilled you appreciate their work. :-)
Our customers are top tier tech companies and our users are among the smartest engineers on the planet. I'm proud of our marketing, but no-one is going to spend a bunch of money with us just because of that.
WinDbg's time travel debug is really cool and more people should know about it. I'm always a little sad that it's not (so far!) officially integrated in something like VS Code.
Before it was released publicly I believe Microsoft had been using it internally to share recordings on bug reports against massive pieces of software like Office. So it's a serious piece of tech.
It does, but it is really sad by comparison with rr and UndoDB. You could use it to record a few function calls or perhaps if you’re lucky a whole frame of your game but not a whole program.
Time travel debugging on embedded ARM has been available for over 20 years via trace probes [1].
The category namer of time-travel debugging, TimeMachine, (hence time-travel debugging in contrast to other attempted names such as reversible, bidirectional, record-replay, etc.) was available in 2003 and supports/supported the ARM7 [2]. Note, that is not ARMv7 architecture, that is the ARM7 chip [3] in use from 1993-2001.
From what I know, the ARM7 was one of the first ARM designs implementing the Embedded Trace Macrocell (ETM) which could output the instruction and data trace data used to support trace probe-based time travel debugging.
What's limiting us is that Undo does need a Linux kernel - so traditional embedded programming wouldn't be a fit. Embedded Linux could work and we do support ARM64.
I've thought I bit about how you might support time travel on bare metal embedded - but actually there are hardware-assisted solutions (Lauterbach's Trace32 was one we came across) there sometimes.
Undo co-founder here. rr is indeed awesome. If it works for your use-case, you should use it!
Undo is mostly used by companies whose world is complex enough that rr doesn't work for them, and they understand how powerful time travel debugging is.
There has now been a LOT of engineering invested by a lot of very smart people into Undo, so it does also have a lot of polish and nice features.
But honestly, if rr is working for you, that's great. I'm just glad you're not doing printf debugging the whole time :)
I was in talks with them recently because I kept running into limitations with rr. The main advantages for my use case were that undo doesn't have the same dependency on hardware timers, which means the ARM support is much better, you can run it in a VM (e.g. a cloud machine) and you can do replays on different systems.
- If your program is very light on syscalls (i.e. basically entirely in-memory computation), rr can go to a basically 1.0x slowdown. In particular this means you can run benchmarks in it at full capacity, provided that I/O is outside of the repeated part (e.g. if sometimes the bench is noticably slower, you can replay and see if some important loads/stores crossed a cacheline/page). You can even "perf record" / "perf stat" a replay if you want to! (none of this is too useful, but it's fun! Gathering repeated stats over the same execution for more resolution might be useful with proper tooling though)
- rr does have an in-memory buffer of recording data.
- rr recordings should be portable within the architecture, as long as the replay hardware has the extensions the recorder did (or if replayer-unsupported features are disabled at record-time).
I regularly deal with 3 different architectures. I can go and spin up a cloud instance every time I want to run rr (and in fact that's the solution I've been working with), but it's just annoying enough to justify spending a couple hours in sales calls.
Well, if you have a Google L5 making ~365k [1] then it would need to make them ~2.2% more productive overall to be worth it when just considering direct pay. If we consider a Google L3 at ~187k then it would need to make them ~4.2% more productive overall.
This, of course, ignores employee benefits and overhead which usually amount to ~100% extra costs over direct pay. So that is now ~1.1% and ~2.1%, respectively.
And that ignores the fact that you need to pay people less than they produce to be profitable which probably drops us down to ~0.5% and ~1.0%, respectively.
No-one is going to spend $8K out of pocket to A/B test this on themselves. Of all the things you could be doing to improve your productivity, this is some high hanging fruit.
If you have a US employer who is unwilling to spend 8 k$ on software engineering productivity then they are pennywise, pound foolish. It literally costs 10x that for a single junior engineer. And, as I pointed out, the net productivity improvement you need to see to justify that expense is miniscule.
If your employer really is skeptical, then they can run a A/B test over a small group of engineers to prove out changes in productivity. But not even being willing to run that test when it is so cheap is just management incompetence.
Engineers are ridiculously expensive. In electrical engineering, where the engineers are generally less well-paid than in software, employers routinely spend multiple hundreds of thousands of dollars per engineer per year in tooling. Not being willing to spend 8 k$ on a test of well known technology and attempting to identify mere single digit percentage improvements is just stupid.
Not everyone is Google. Some people work for themselves, or have very small teams, or live in a developing country, and don't have lots of spare cash laying around.
Please try to understand that the world is not as simple and black and white as you'd like.
Undo founder here. We've been at this for getting on 20 years now. Originally it cost $295 for a perpetual license. Eventually we understood that the majority of developers (actually employers of developers) will pay $0. But some are happy to pay for tooling, as long as they're confident they'll get a many multiples return-on-investment. Hence our pricing. Happily, enough do that we can run a modest but profitable business (40+ people). Customer churn is practically zero.
Why do people pay for Undo when they can get rr -- which is also really good -- for free? Those whose code or environment is big enough and complex enough that rr doesn't work for them, and they understand how powerful time travel debugging is. If rr works for you, you should use it. This includes most independent developers.
If rr can work for you and you're still not using any kind of time travel debugging, you have effective tied one hand behind your own back! If you're independent (incl student or academic) and rr doesn't work for you, get in touch -- we give free licenses for academic and certain other use cases.
There is a wider thing here about software companies paying for dev tooling. So many companies over the years who made really cool things who couldn't make their business work.
if someone's bringing "google payscale" for comparison... well that's not some average joe...
why not just bring Bill Gates and say "everything -- including private jet -- is dirt cheap" ?
$8k per year simply doesn't make sense for 95% of the programmers. For a lot of developing countries, that's more than a well-paying programmer's annual salary...
I presented a conditional calculation using numbers contextually-relevant to a forum run by a company that funds US tech startups.
IF you are such a company, THEN you only need very minor productivity improvements to justify such a price difference. IF you are such a company, THEN such costs are well within the cost of doing business. IF you are NOT then you are free to apply your own numbers to your situation.
However, the overwhelming majority of companies that fall into the applicable category outright refuse to even consider the possibility of a 8 k$ capital expense on software development tooling out of the absolutely foolish belief that such expenditure CAN NOT be justified as such a cost could not possibly be recouped in benefits or out of the absolutely foolish belief that a product that costs 100$ must be 100x better than a product that costs 1$ to be justified, and thus a product that costs money must be infinitely better than a product that is free to be justified.
It is absolutely management incompetence for US employers to just shut down upon hearing about a 8 k$ capital expenditure instead of doing a proper cost-benefit analysis to determine if such expenditures would be justified.
I agree with you. At most companies your manager would never even think of spending that much money on you, even if it was completely justifiable. You’re lucky if you can get permission to buy a $50 book.
That said, if you’re going to spend money on a debugger then I recommend rr + Pernosco instead. Pernosco is a completely different kind of debugger that will make you much, much more productive at debugging than any traditional debugger where you start by adding breakpoints and then stepping through your code. Costs a lot less too.
Ugh, it appears I was not utterly pedantic enough even though I did, quite clearly, make my statements conditional in a way far beyond what is normally expected when talking about generalities that any normal person would assume are meant to apply in cases contextually-relevant to a forum run by a company that funds US tech startups.
Very well, a software engineering employer in the US who employs over 10 software engineers in the US at above the 20th percentile of wages, which constitutes the employers of a significant fraction of total software engineers above the 20th percentile of wages, would be foolish to not spend 8 K$ on software development tooling that would result in a 10% productivity improvement. It would be further foolish to not investigate such potential improvements given a reasonably credible belief that such a productivity improvement is possible. Outright dismissal without even considering the potential cost-benefit or making a incorrect cost-benefit analysis requiring significantly in excess of 10% is also foolish.
Please try to understand that sometimes when people are not being utterly pedantic to the point of absurdity it is because they assume people will use their judgement to interpret the applicability to their situation rather than because they can only see in black and white.
What's the difference with RR?
Undo (where I'm CTO) has existed for longer than RR and its real benefit is that it scales to use cases where RR (for one reason or another) isn't a fit.
Technically:
* Doesn't need hardware performance counters - runs on more CPUs and on cloud systems (where performance counters are often blocked).
* Can attach and detach at any time - means you get to record just a subset of program execution that's interesting.
* You can our ship recording tech with your application and control it by API, so you can grab crash recordings on customer systems.
* Supports programs that share memory with non-recorded processes.
* Supports direct device access (e.g. DPDK).
* Accelerated debugging features - searching with recordings using parallel processing, accelerated conditional breakpoints a few thousand times faster than native GDB.
* We provide a stable, patched fork of GDB that we're occasionally told is more stable than the default.
For many people's use cases none of these really matter - they should use RR if they're not already.
But if you need any of these things then Undo can give you time travel debugging. In practice, it's usually big software organisations that we deal with because they have development pain and the extreme requirements we can match.
> * You can our ship recording tech with your application and control it by API, so you can grab crash recordings on customer systems.
That’s actually pretty neat.
[rr developer here]
Undo has cool features like Live Recording that we don't have in rr. They don't need access to the hardware PMU which is a big advantage in some situations. They can handle accesses to shared memory in cases where rr can't. https://undo.io/resources/undo-vs-rr/ is a good resource.
If you don't have access to the hardware PMU then you can try https://github.com/sidkshatriya/rr.soft (which is a modification of the rr debugger).
It may not be commercial quality but its open source and free :)
[I built rr.soft]
Undo also support Java and Scala: https://docs.undo.io/java/index.html
AFAIK it records multithreaded applications on multiple threads and CPU, rr records them on a single OS thread, AFAIK. Not sure about replay. Never used undo though, so not sure how much better it is.
rr does support multithreaded and multi-process applications, via, like Undo[1], allowing only a single thread to run at a time. (edit note - that's only about multithreading; Undo might have parallel multi-process recording)
[1]: https://undo.io/resources/undo-performance-benchmarks/ - "Undo serializes their execution"
I stand corrected, not sure where I heard this then.
https://undo.io/resources/undo-vs-rr/ does note parallel recording for multi-process (not multi-threaded), so perhaps that.
[flagged]
Undo founder here. I just got a slack from one of our marketing team who is thrilled you appreciate their work. :-)
Our customers are top tier tech companies and our users are among the smartest engineers on the planet. I'm proud of our marketing, but no-one is going to spend a bunch of money with us just because of that.
So there must be something else in it I guess ;-)
Free Windows equivalent: WinDbg Time Travel Debugging (https://learn.microsoft.com/en-gb/windows-hardware/drivers/d...).
WinDbg's time travel debug is really cool and more people should know about it. I'm always a little sad that it's not (so far!) officially integrated in something like VS Code.
Before it was released publicly I believe Microsoft had been using it internally to share recordings on bug reports against massive pieces of software like Office. So it's a serious piece of tech.
FOSS alternative: https://rr-project.org/
gdb should already have a similar feature?
GDB does:
https://sourceware.org/gdb/current/onlinedocs/gdb.html/Proce...
But it's limited. It's really cool that it's integrated by default but it doesn't scale to big applications / workloads.
RR and Undo both use GDB as a user interface, though, so any skills you have there will carry over.
A lot people don't know and don't use GDBs reverse debugging. It is an awesome and hidden feature, which more developers should know :)
All these Oh wait. I missed it...debugging sessions. and these What exactly changed over there? are answerable.
It does, but it is really sad by comparison with rr and UndoDB. You could use it to record a few function calls or perhaps if you’re lucky a whole frame of your game but not a whole program.
if you could get this working on embedded arm cpus, I think you'd be surprised how many customers there would be.
Time travel debugging on embedded ARM has been available for over 20 years via trace probes [1].
The category namer of time-travel debugging, TimeMachine, (hence time-travel debugging in contrast to other attempted names such as reversible, bidirectional, record-replay, etc.) was available in 2003 and supports/supported the ARM7 [2]. Note, that is not ARMv7 architecture, that is the ARM7 chip [3] in use from 1993-2001.
From what I know, the ARM7 was one of the first ARM designs implementing the Embedded Trace Macrocell (ETM) which could output the instruction and data trace data used to support trace probe-based time travel debugging.
[1] https://jakob.engbloms.se/archives/1564
[2] https://www.ghs.com/products/probe.html
[3] https://en.m.wikipedia.org/wiki/ARM7
What's limiting us is that Undo does need a Linux kernel - so traditional embedded programming wouldn't be a fit. Embedded Linux could work and we do support ARM64.
I've thought I bit about how you might support time travel on bare metal embedded - but actually there are hardware-assisted solutions (Lauterbach's Trace32 was one we came across) there sometimes.
Let me save you a click:
Pricing & Licensing
A UDB floating license costs $7,900 per year.
If you’re going to spend money, then you would be better off using rr and paying for Pernosco. Pernosco is amazing.
Thanks for the Pernosco tip. It really looks amazing and you can try it for free as github user
You’re welcome.
rr is awesome and is free and open and all that. How much better could this possibly be?
Undo co-founder here. rr is indeed awesome. If it works for your use-case, you should use it!
Undo is mostly used by companies whose world is complex enough that rr doesn't work for them, and they understand how powerful time travel debugging is.
There has now been a LOT of engineering invested by a lot of very smart people into Undo, so it does also have a lot of polish and nice features.
But honestly, if rr is working for you, that's great. I'm just glad you're not doing printf debugging the whole time :)
They have a comparison page: https://undo.io/resources/undo-vs-rr/
I was in talks with them recently because I kept running into limitations with rr. The main advantages for my use case were that undo doesn't have the same dependency on hardware timers, which means the ARM support is much better, you can run it in a VM (e.g. a cloud machine) and you can do replays on different systems.
A couple minor notes:
- If your program is very light on syscalls (i.e. basically entirely in-memory computation), rr can go to a basically 1.0x slowdown. In particular this means you can run benchmarks in it at full capacity, provided that I/O is outside of the repeated part (e.g. if sometimes the bench is noticably slower, you can replay and see if some important loads/stores crossed a cacheline/page). You can even "perf record" / "perf stat" a replay if you want to! (none of this is too useful, but it's fun! Gathering repeated stats over the same execution for more resolution might be useful with proper tooling though)
- rr does have an in-memory buffer of recording data.
- rr recordings should be portable within the architecture, as long as the replay hardware has the extensions the recorder did (or if replayer-unsupported features are disabled at record-time).
I regularly deal with 3 different architectures. I can go and spin up a cloud instance every time I want to run rr (and in fact that's the solution I've been working with), but it's just annoying enough to justify spending a couple hours in sales calls.
Well, if you have a Google L5 making ~365k [1] then it would need to make them ~2.2% more productive overall to be worth it when just considering direct pay. If we consider a Google L3 at ~187k then it would need to make them ~4.2% more productive overall.
This, of course, ignores employee benefits and overhead which usually amount to ~100% extra costs over direct pay. So that is now ~1.1% and ~2.1%, respectively.
And that ignores the fact that you need to pay people less than they produce to be profitable which probably drops us down to ~0.5% and ~1.0%, respectively.
[1] https://www.levels.fyi/companies/google/salaries/software-en...
edit: Incorrectly linked to product designer instead of software engineer levels.
OK... Most of us don't know what a "google l5" is, so I guess we can safely ignore this. Heh.
No-one is going to spend $8K out of pocket to A/B test this on themselves. Of all the things you could be doing to improve your productivity, this is some high hanging fruit.
If you have a US employer who is unwilling to spend 8 k$ on software engineering productivity then they are pennywise, pound foolish. It literally costs 10x that for a single junior engineer. And, as I pointed out, the net productivity improvement you need to see to justify that expense is miniscule.
If your employer really is skeptical, then they can run a A/B test over a small group of engineers to prove out changes in productivity. But not even being willing to run that test when it is so cheap is just management incompetence.
Engineers are ridiculously expensive. In electrical engineering, where the engineers are generally less well-paid than in software, employers routinely spend multiple hundreds of thousands of dollars per engineer per year in tooling. Not being willing to spend 8 k$ on a test of well known technology and attempting to identify mere single digit percentage improvements is just stupid.
Not everyone is Google. Some people work for themselves, or have very small teams, or live in a developing country, and don't have lots of spare cash laying around.
Please try to understand that the world is not as simple and black and white as you'd like.
Undo founder here. We've been at this for getting on 20 years now. Originally it cost $295 for a perpetual license. Eventually we understood that the majority of developers (actually employers of developers) will pay $0. But some are happy to pay for tooling, as long as they're confident they'll get a many multiples return-on-investment. Hence our pricing. Happily, enough do that we can run a modest but profitable business (40+ people). Customer churn is practically zero.
Why do people pay for Undo when they can get rr -- which is also really good -- for free? Those whose code or environment is big enough and complex enough that rr doesn't work for them, and they understand how powerful time travel debugging is. If rr works for you, you should use it. This includes most independent developers.
If rr can work for you and you're still not using any kind of time travel debugging, you have effective tied one hand behind your own back! If you're independent (incl student or academic) and rr doesn't work for you, get in touch -- we give free licenses for academic and certain other use cases.
There is a wider thing here about software companies paying for dev tooling. So many companies over the years who made really cool things who couldn't make their business work.
> If […] you're still not using any kind of time travel debugging, you have effective tied one hand behind your own back!
Completely agree about that! I’ve used rr to debug things that I never could have without it. People are resistant to change though.
this
if someone's bringing "google payscale" for comparison... well that's not some average joe...
why not just bring Bill Gates and say "everything -- including private jet -- is dirt cheap" ?
$8k per year simply doesn't make sense for 95% of the programmers. For a lot of developing countries, that's more than a well-paying programmer's annual salary...
I presented a conditional calculation using numbers contextually-relevant to a forum run by a company that funds US tech startups.
IF you are such a company, THEN you only need very minor productivity improvements to justify such a price difference. IF you are such a company, THEN such costs are well within the cost of doing business. IF you are NOT then you are free to apply your own numbers to your situation.
However, the overwhelming majority of companies that fall into the applicable category outright refuse to even consider the possibility of a 8 k$ capital expense on software development tooling out of the absolutely foolish belief that such expenditure CAN NOT be justified as such a cost could not possibly be recouped in benefits or out of the absolutely foolish belief that a product that costs 100$ must be 100x better than a product that costs 1$ to be justified, and thus a product that costs money must be infinitely better than a product that is free to be justified.
It is absolutely management incompetence for US employers to just shut down upon hearing about a 8 k$ capital expenditure instead of doing a proper cost-benefit analysis to determine if such expenditures would be justified.
I agree with you. At most companies your manager would never even think of spending that much money on you, even if it was completely justifiable. You’re lucky if you can get permission to buy a $50 book.
That said, if you’re going to spend money on a debugger then I recommend rr + Pernosco instead. Pernosco is a completely different kind of debugger that will make you much, much more productive at debugging than any traditional debugger where you start by adding breakpoints and then stepping through your code. Costs a lot less too.
You’re being pedantic for no reason.
Ugh, it appears I was not utterly pedantic enough even though I did, quite clearly, make my statements conditional in a way far beyond what is normally expected when talking about generalities that any normal person would assume are meant to apply in cases contextually-relevant to a forum run by a company that funds US tech startups.
Very well, a software engineering employer in the US who employs over 10 software engineers in the US at above the 20th percentile of wages, which constitutes the employers of a significant fraction of total software engineers above the 20th percentile of wages, would be foolish to not spend 8 K$ on software development tooling that would result in a 10% productivity improvement. It would be further foolish to not investigate such potential improvements given a reasonably credible belief that such a productivity improvement is possible. Outright dismissal without even considering the potential cost-benefit or making a incorrect cost-benefit analysis requiring significantly in excess of 10% is also foolish.
Please try to understand that sometimes when people are not being utterly pedantic to the point of absurdity it is because they assume people will use their judgement to interpret the applicability to their situation rather than because they can only see in black and white.
Another more powerful option: Intel SDE / PinPlay