On Sep 9, 2022 at 6:23:38 PM, Chris Peterson <[email protected]> wrote:
> Very cool! I was comparing the two stack traces and did a double take when > I saw the stack frame numbers repeating. But then I realized that the > inlined functions wouldn't have their own stack frames. 🙂 > Hah, yes, we went back and forth on whether to adjust the counting so that every inline frame gets its own frame index, or whether we just repeat the original frame number for each inline. In the end, repeating the frame index was simpler. Are inlined function calls also available in crash reports for Firefox > Android and iOS? > Android: Yes, for crashes in our own native code (i.e. C++ / Rust etc). And the Java stacks have always been complete. iOS: Probably not, I don't know how crash reporting works on iOS. I know gsvelto is working on the oxidized minidump writer for Android. I > don't know if your work depends on his work here. > The expansion of frame addresses into function names and inline stacks happens on the server side, when the minidump is processed, with the help of symbol files which also come from a server. So no extra information in the minidumps is needed, and this work is independent of minidump writing. Cheers, Markus > > thanks, > chris > > > On 9/9/2022 2:49 PM, Markus Stange wrote: > > Hi all, > > I am happy to announce that crash reports just became a lot more useful! > > We now recover inlined function calls and display them in the crash stack. > In many cases, this allows you to identify the crashing code much faster > than before. In the past, many crucial stack frames would have been omitted > due to inlining. This made it borderline impossible, or at least extremely > cumbersome, to find out where the crash actually happened. > > Here's an example: > Before (inline calls lost): > https://crash-stats.mozilla.org/report/index/ffddd880-b288-49af-bfd7-1dd640220901 > After (inline calls recovered): > https://crash-stats.mozilla.org/report/index/829c31ce-e2d8-4102-9188-d35fa0220908 > In the "before" stack, you can see that the crash is inside > TenuringTracer::collectToObjectFixedPoint(), but that's where the trail > runs cold: The line number just shows you a call to traceObject, but you > don't *where inside traceObject* the crash occurred. > In the "after" stack, the stack contains two more frames inside > TenuringTracer::collectToObjectFixedPoint(): We now know about the inlined > traceObject frame, along with the line number inside that function; and > *another* frame is recovered: getDenseInitializedLength(). So the crash > really happened in getDenseInitializedLength() all along. > This kind of multi-level inlining happens all the time, and it happens > even more so in Rust code than in C++ code. > And now you can see right through it. > > Supported versions: > The fix to emit inline information in symbol files landed last Friday in > bug 1779631 <https://bugzilla.mozilla.org/show_bug.cgi?id=1779631> and is > now riding the 106 train. This means that, as of today, inline stacks are > only available for crashes from Firefox Nightly. > > Follow-up work: > Crash *signatures* do not include inline frames yet; this work will > happen in bug 1788269 > <https://bugzilla.mozilla.org/show_bug.cgi?id=1788269>. > > Acknowledgements: > Most of this work happened in various github repos full of Rust code, > specifically in mozilla/dump_syms <https://github.com/mozilla/dump_syms/>, > getsentry/symbolic <https://github.com/getsentry/symbolic/> and > rust-minidump/rust-minidump > <https://github.com/rust-minidump/rust-minidump/>. I'd like to thank all > my reviewers for the fast reviews, especially Swatinem and loewenheim from > Sentry, and gsvelto, Gankra and willkg from Mozilla. > Furthermore, I'd like to thank Gankra and willkg for implementing the > final pieces to make the crash report infrastructure take advantage of the > new symbol information. > It's also worth noting that I didn't have to touch any C++ code when doing > this work. The Rust rewrites of dump_syms and rust-minidump have been in > production for a few months now; if I had attempted this project before > that, it would have involved a lot of C++ Google breakpad code and it would > have been much less fun. > > Background: > I worked on this project because symbol files are part of the shared > infrastructure that's used both by the crash reporter and by the profiler. > My main motivation was to get inline stacks into the profiler; we already > had them for local builds, but not for "official" builds (i.e. builds for > which symbol information comes from the symbol server). After doing all the > necessary work for the profiler, making it work for crash reports was > really straightforward and will surely pay off immensely. > Happy crash analysis! > -Markus > -- > You received this message because you are subscribed to the Google Groups > "[email protected]" <[email protected]> group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/CAL2j%2B_MyEwBey-NRSD2A_BNq8MbfAr2Ko72T-MU0DS1ycwANBw%40mail.gmail.com > <https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/CAL2j%2B_MyEwBey-NRSD2A_BNq8MbfAr2Ko72T-MU0DS1ycwANBw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > > -- You received this message because you are subscribed to the Google Groups "[email protected]" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/CAL2j%2B_Mx9yaZF%2BMVxTyAzmufGDu1ZS-phjxqwpm8x0XPKsNgow%40mail.gmail.com.
