A few weeks ago I looked at a bunch of minidumps of JIT crashes (we save some code around the instruction pointer and Ted wrote a tool [0] to extract it from the dump and pass it to objdump).
Most of these crashes were either obvious memory corruption (for instance, a typical instruction sequence with 4 corrupt bytes in the middle), invalid instruction pointer or valid JIT code but accessing a bogus (object) pointer, suggesting a problem elsewhere. Downloading minidumps and disassembling them takes a lot of time though; it'd be really great if we could integrate this into crash-stats, so that we can analyze dumps much faster. I also found bug 627111 [1] to automatically categorize JIT crashes. If we could run that script every day and post the results as a graph somewhere it may be easier to detect new crashes. Jan [0] http://hg.mozilla.org/users/tmielczarek_mozilla.com/get-minidump-instructions/ [1] https://bugzilla.mozilla.org/show_bug.cgi?id=627111 On Thu, Feb 27, 2014 at 5:57 PM, Bill McCloskey <[email protected]> wrote: > We poison the GC heap with 0xda on nightly and aurora [1]. Lots of our > processMarkStackTop are at this address [2]. It looks like we only poison Ion > code in debug builds [3]. That should be easy to fix by removing the #ifdef > DEBUG. > > I don't think we've found GC heap poisoning to be all that helpful for > debugging in crashstats. The stacks are almost always really generic. All you > know is that some code didn't root something properly, but it could be > anywhere in Gecko. Poisoning has been very helpful when we have an STR, since > it improves reproducibility. However, we rarely get an STR for these crashes. > > One thing I've noticed in the past is that certain add-ons seem to increase > the frequency of GC crashes (not necessarily though any misbehavior on their > part). The correlations that show up in crashstats aren't very helpful here > since so many different crashes are lumped together. I wonder if we could do > a more sophisticated analysis to try to find the problematic add-ons. Then we > could try to do some manual (or automatic) testing with those add-ons and > hopefully get some full crashdumps, which would be more useful than what we > have now. > > -Bill > > [1] http://mxr.mozilla.org/mozilla-central/source/js/src/jsgc.cpp#477 > [2] > https://crash-stats.mozilla.com/report/index/50bb404b-5b0b-458a-b3b6-5a2ba2140226 > [3] http://mxr.mozilla.org/mozilla-central/source/js/src/jit/Ion.cpp#681 > > ----- Original Message ----- >> From: "Benjamin Smedberg" <[email protected]> >> To: "JS Internals list" <[email protected]> >> Sent: Thursday, February 27, 2014 6:23:07 AM >> Subject: [JS-internals] Memory poisoning and other ways to make stability >> issues more reproducible >> >> [tl;dr: please poison GC and JIT memory when it is freed] >> >> I mentioned this at the engineering meeting last Tuesday but I don't >> think I hit the right audience so I thought I'd repost it here. >> >> We currently have some important crashes that are currently not >> actionable based on crash stats. If you take a look at the current 28 >> beta topcrashes, the #3 topcrash (js::GCMarker::processMarkStackTop) is >> a GC crash probably due to a corrupt GC heap/dangling pointers. >> >> The #5 crash (EnterBaseline) is a generic signature for any crash that >> happens in the JIT. >> >> There are a few others such as >> JSScript::markChildren/ScanRope/FinalizeArenas. >> >> A lot of our current efforts have revolved around using fuzzers to >> detect these kinds of situations. But we really need to try and use our >> release population more effectively to discover why these issues are >> happening and fix them. >> >> We recently landed memory poisoning on free() in the Firefox codebase >> (bug 860254). This provides some defense in depth for individual users, >> but its primary purpose is to make use-after-free crashes happen more >> quickly and with stack traces that are more useful. For the most part >> this change does not appear to have affected performance (except for >> tsvg tests on macOS which have strange symptoms). >> >> I'd like the JS team to consider doing the same kind of memory poisoning >> in release builds for the various JS heaps which aren't the jemalloc heap: >> >> * The GC >> * The various JIT code buffers >> * Any other relevant caches or specialized allocators where normal >> free() poisoning doesn't work >> >> Thoughts? I don't know enough about this to file specific bugs, so I'm >> looking to you on the JS team to identify what things are candidates and >> filing the bugs. Especially if there are mentors for the bugs, I have >> some great volunteers who have been fixing OOM issues recently who might >> love to help take some of these bugs and fix them. >> >> --BDS >> >> _______________________________________________ >> dev-tech-js-engine-internals mailing list >> [email protected] >> https://lists.mozilla.org/listinfo/dev-tech-js-engine-internals >> > _______________________________________________ > dev-tech-js-engine-internals mailing list > [email protected] > https://lists.mozilla.org/listinfo/dev-tech-js-engine-internals _______________________________________________ dev-tech-js-engine-internals mailing list [email protected] https://lists.mozilla.org/listinfo/dev-tech-js-engine-internals
