Can you elaborate on which Gecko components you're hoping to fuzz separately? A lot of the core is pretty heavily-intertwined, so I'm pretty skeptical that we'd ever be able to separate out DOM, style, and layout from each other (for example). There are basically two barriers:
(1) These components are enormous, and were not built to be very modular. Any such efforts would require a huge amount of engineering resources, which we would probably not spend just to make the component fuzzable. (2) The performance cost of adding an abstraction layer between tightly-coupled components in C++11 would probably be prohibitive (the situation is different for Rust/Servo because of Traits - we could potentially do this with C++ Concepts/Modules in around a decade). This is all to say that I think a general call to "modularize Gecko" isn't really helpful. But if there are specific leaf-y components that you want to fuzz separately but can't, that might be a good starting point. On Wed, Mar 9, 2016 at 9:54 AM, <twsm...@mozilla.com> wrote: > On Wednesday, March 9, 2016 at 9:38:55 AM UTC-8, Nicolas B. Pierron wrote: > > This discussion is a follow-up discussion to some emails sent privately > by > > accident. > > > > If you have not followed, I will quote David Bryant: > > > Improving release quality is one of the three fundamental goals > Platform > > > Engineering committed to this year. To this end, lmandel built a > Bugzilla > > > dashboard that allows us to track regressions found in any given > release > > > cycle. This dashboard [...] can > > > also be found at: http://mozilla.github.io/releasehealth/ > > > > To David's email, I answered the following: > > > > ---------- > > tl;dr: If we want to improve the quality of our products we should > > split Gecko in standalone programs which are fuzzing-friendly. > > > > One thing which strikes me, is the ratio of regressions per component > > that we have for each versions, and more over who are the persons > > opening these bugs: > > - Release: > > > https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox44&f2=OP&f3=cf_status_firefox43&f4=cf_status_firefox43&f5=cf_status_firefox43&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898533&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on= > > - Beta: > > > https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox45&f2=OP&f3=cf_status_firefox44&f4=cf_status_firefox44&f5=cf_status_firefox44&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898534&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on= > > - Aurora: > > > https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox46&f2=OP&f3=cf_status_firefox45&f4=cf_status_firefox45&f5=cf_status_firefox45&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898536&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on= > > - Nightly: > > > https://bugzilla.mozilla.org/buglist.cgi?columnlist=product%2Ccomponent%2Creporter&f1=cf_status_firefox47&f2=OP&f3=cf_status_firefox46&f4=cf_status_firefox46&f5=cf_status_firefox46&f6=CP&include_fields=id&j2=OR&keywords=regression%2C&keywords_type=allwords&list_id=12898528&o1=equals&o3=equals&o4=equals&o5=equals&query_format=advanced&resolution=---&v1=affected&v3=unaffected&v4=%3F&v5=---&query_based_on= > > > > To be more precise: > > - The small number of regression we have in the JS engine on the > > release channel, versus the Extremely Huge number of regressions we > > have on nightly. > > - And the fact that (almost) all the bugs opened against the JS > > engine are opened by our fuzzing team. > > > > What I want to remark is the fact that our automated fuzzing is better > > at finding recently introduced regressions. And as far as I know, > > Alice is not a bot. > > > > From what I know, the reason fuzzing team is so efficient on the JS > > engine is because we have a *standalone* JS shell. > > The *standalone* JS shell is also the reason why our build time is > > below 2 minutes as opposed to 18 minutes. > > > > So, I think that if we want to improve our quality we should focus on > > making fuzzing-friendly standalone programs for the different > > components of the platform. > > Thus reducing, the compilation time, reducing the test suite time, and > > improving the ability of the fuzzing team to find recently added > > regressions. > > > > Maybe I am wrong, in which case the other alternative might be to > > staff the JS Team to get rid of all these nightly issues before they > > ride the train to release. > > ---------- > > > > To which I got the following replies: > > > > On Wed, Mar 9, 2016 at 3:05 PM, Kyle Huey wrote: > > > The ratio of engineers to code in the js engine is so much higher than > > > the rest of the product that I'm not sure this is a sensible > comparison. > > > The js engine also doesn't depend on things like 3rd party gfx > drivers ... > > > > On Wed, Mar 9, 2016 at 3:05 PM, Olli Pettay wrote: > > > Fuzzing captures only a fraction of issues. > > > > On Wed, Mar 9, 2016 at 3:42 PM, Chris Hofmann wrote: > > > On Wed, Mar 9, 2016 at 7:05 AM, Kyle Huey wrote: > > >> > > >> The ratio of engineers to code in the js engine is so much higher > than the > > >> rest of the product that I'm not sure this is a sensible > comparison. The js > > >> engine also doesn't depend on things like 3rd party gfx drivers ... > > > > > > This is probably not the only step that we need to take to > substantially > > > improve quality so setting up a place to have those discussions is > good. It > > > really is worth some time and effort to brainstorm about all the > things we > > > might do to raise the bar, poke some holes in those ideas, then > decide on > > > and push forward on a few more in the next few quarters. > > > > On Wed, Mar 9, 2016 at 5:23 PM, Al Billings wrote: > > > On 3/9/16 6:58 AM, Nicolas B. Pierron wrote: > > >> So, I think that if we want to improve our quality we should focus on > > >> making fuzzing-friendly standalone programs for the different > > >> components of the platform. > > >> Thus reducing, the compilation time, reducing the test suite time, > and > > >> improving the ability of the fuzzing team to find recently added > > >> regressions. > > > The fuzzing team has asked for this from different teams for various > > > components, with different degrees of support and resistance, > depending > > > on the component and team. > > > > > > It is work to do this but it changes the fuzzing efficiency and > ability > > > to find focused issues when we have this level of support, such as we > > > have with the JS shell. It allows the team to fuzz directly on a > > > component and not have to find a way to reach it through Firefox and > all > > > the additional noise/assertions/etc that it churns up. > > > > -- > > Nicolas B. Pierron > > I think Nicolas is right on the mark! JS shell is a good example of using > a shell for fuzzing and I think media and graphics could also get a good > deal out of a shell. > > To get a bit more specific breaking things in to smaller pieces for > fuzzing give us (the fuzzing team) a major advantage. In addition to what > Nicolas mentioned, it makes it much easier to use different sanitizers > (Address, Memory, Undefined behavior, etc...) It also makes it possible to > use coverage guided fuzzing which is a major advancement in fuzzing. I have > been doing a lot of work fuzzing 3rd party libraries that we use in Firefox > and there have been many times I wished I could have use the same tools as > easily on our own code. That said the fuzzing on the browser as a whole it > also important as well. > > More dev support for the fuzzing means more bugs found and sooner! > > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
