Here's another good example of how this makes things better: https://crash-stats.mozilla.org/report/index/3921dc62-a712-43a7-9b70-0ce970220514 vs. https://crash-stats.mozilla.org/report/index/1fbb67cf-dec1-4729-96d5-061f10220913
In the first crash report we get an unwrap() happening someplace under Renderer::update_texture_cache(): https://hg.mozilla.org/mozilla-central/file/466ab31800308d9a75db091c5cde3eda636cf77d/gfx/wr/webrender/src/renderer/mod.rs#l2423 but no real indication where. In the second crash report with inline stacks we get copy_into_staging_buffer(): https://hg.mozilla.org/mozilla-central/file/c7037dbd2de37f147a445b6dae361671ef71896a/gfx/wr/webrender/src/renderer/upload.rs#l389 That's the expression that has the actual unwrap() Thanks for making it happen! -Jeff On Fri, Sep 9, 2022 at 5:50 PM Markus Stange <[email protected]> wrote: > > Hi all, > > I am happy to announce that crash reports just became a lot more useful! > > We now recover inlined function calls and display them in the crash stack. In > many cases, this allows you to identify the crashing code much faster than > before. In the past, many crucial stack frames would have been omitted due to > inlining. This made it borderline impossible, or at least extremely > cumbersome, to find out where the crash actually happened. > > Here's an example: > Before (inline calls lost): > https://crash-stats.mozilla.org/report/index/ffddd880-b288-49af-bfd7-1dd640220901 > After (inline calls recovered): > https://crash-stats.mozilla.org/report/index/829c31ce-e2d8-4102-9188-d35fa0220908 > In the "before" stack, you can see that the crash is inside > TenuringTracer::collectToObjectFixedPoint(), but that's where the trail runs > cold: The line number just shows you a call to traceObject, but you don't > where inside traceObject the crash occurred. > In the "after" stack, the stack contains two more frames inside > TenuringTracer::collectToObjectFixedPoint(): We now know about the inlined > traceObject frame, along with the line number inside that function; and > another frame is recovered: getDenseInitializedLength(). So the crash really > happened in getDenseInitializedLength() all along. > > This kind of multi-level inlining happens all the time, and it happens even > more so in Rust code than in C++ code. > And now you can see right through it. > > Supported versions: > The fix to emit inline information in symbol files landed last Friday in bug > 1779631 and is now riding the 106 train. This means that, as of today, inline > stacks are only available for crashes from Firefox Nightly. > > Follow-up work: > Crash signatures do not include inline frames yet; this work will happen in > bug 1788269. > > Acknowledgements: > Most of this work happened in various github repos full of Rust code, > specifically in mozilla/dump_syms, getsentry/symbolic and > rust-minidump/rust-minidump. I'd like to thank all my reviewers for the fast > reviews, especially Swatinem and loewenheim from Sentry, and gsvelto, Gankra > and willkg from Mozilla. > Furthermore, I'd like to thank Gankra and willkg for implementing the final > pieces to make the crash report infrastructure take advantage of the new > symbol information. > It's also worth noting that I didn't have to touch any C++ code when doing > this work. The Rust rewrites of dump_syms and rust-minidump have been in > production for a few months now; if I had attempted this project before that, > it would have involved a lot of C++ Google breakpad code and it would have > been much less fun. > > Background: > I worked on this project because symbol files are part of the shared > infrastructure that's used both by the crash reporter and by the profiler. My > main motivation was to get inline stacks into the profiler; we already had > them for local builds, but not for "official" builds (i.e. builds for which > symbol information comes from the symbol server). After doing all the > necessary work for the profiler, making it work for crash reports was really > straightforward and will surely pay off immensely. > > Happy crash analysis! > > -Markus > > -- > You received this message because you are subscribed to the Google Groups > "[email protected]" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/CAL2j%2B_MyEwBey-NRSD2A_BNq8MbfAr2Ko72T-MU0DS1ycwANBw%40mail.gmail.com. -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/CAKb2zs9bwLgujcZKg1YvGSz-3sZGRNEwV_ipbGf%2BX9%3DVmiSuFg%40mail.gmail.com.
