Hi Everyone! I wanted to propose an idea for how we can make the RIL builds more relevant for and accessible to developers. Let me begin by saying that I truly appreciate the work that QA does and that I do personally read through all of the smoketest build results to see whether there are any regressions my work has introduced. I think it's a great resource! This is simply one idea for how we could make it an even better one.
I believe (and I think others would agree) that tests are most useful when a patch is submitted for review. When I do a code review, I like to see the linter and test results alongside the patch to get a full perspective. Travis is actually *much* more useful to me than TBPL because Travis tells me how patches affect things when it matters most: when I'm reading code and deciding whether a patch is ready to land. Imagine how many things we wouldn't have broken if reviewers knew (when they accepted patches) whether or not they were breaking the smoketests! So, without further adieu, I propose that we do just that! Let's run the smoketests on pull requests! There's a catch though -- we have a very small QA team and there are *lots* of pull requests. It's totally reasonable to question whether this is possible. It's my opinion that with some clever automation we could make the volume of QA work totally manageable. In broad strokes... 1. GitHub sends a notification to our service-x to tell us that a pull request has been opened against mozilla-b2g/gaia. 2. service-x parses from the pull request information which bugzilla issue the pull request is trying to fix. 3. service-x parses from the pull request which smoketests would be affected by the patch. 4. service-x fetches the bug from bugzilla and parses out the environment (ie gaia version and gecko version) that the patch is intended to be applied to. 5. service-x cherry picks the patch onto the appropriate gaia branch and builds b2g. 6. service-x sends a notification (maybe an email?) to qa with a link to download the build, the smoketests that need to be run, and a link to the pull request to comment on with the result of the smoketests. I think if we built a service-x, it would make the amount of work that qa needed to do in order to provide smoketest results to all of the pull requests manageable. The manual work would be minimized to flashing a build to a device, running only the affected smoketests, and commenting on a GitHub issue. Any thoughts :) ? -- Best, Gareth _______________________________________________ dev-b2g mailing list dev-b2g@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-b2g
