This is very exciting on a number of fronts: reduced CI spend, detecting
more defects, and reduced cognitive overhead for developers. Thanks to
Andrew and everyone else involved for getting us here!

On Mon, Jun 8, 2020 at 9:14 AM Andrew Halberstadt <a...@mozilla.com> wrote:

> *tl;dr**: Your |mach try auto| pushes can now precisely schedule relevant
> manifests, catching more regressions with fewer resources.*
>
> I'm happy to announce an important new capability that has been added to
> our CI infrastructure: scheduling specific test manifests.
>
> We've been building a machine learning algorithm to determine which tests
> are most relevant to a given push. This is now running on autoland, as well
> as when you push to try with |mach try auto|. Previously, the biggest
> limitation of this system was that test manifests were already assigned to
> tasks by the time we could run the ML algorithm. So we'd have a list of
> important manifests, then run the chunks that contain those manifests. The
> drawback was we'd also run all the other unimportant manifests that
> happened to be in the same chunk as the important one.
>
> As of late last week, |mach try auto| is now running in "manifest
> scheduling" mode. This means the ML algorithm is running earlier in the
> process and is actually creating the test chunks based on how many
> important manifests there are. Therefore we are only running manifests that
> are relevant to your push and are avoiding the collateral damage caused by
> all the "unimportant" ones. As part of this change I've tweaked the minimum
> importance threshold downward, meaning the algorithm classifies a greater
> number of manifests as "important". So |mach try auto| pushes will schedule
> a greater number of relevant manifests, while simultaneously scheduling
> fewer manifests over all.
>
> One thing to note is that the chunks in your |mach try auto| pushes will
> not bear any resemblance to the chunks that are created on autoland. In
> fact, they may bear no resemblance to the previous |mach try auto| push you
> made, even with similar changes. If you need to figure out where a given
> test ran you can use Treeherder's test path filter:
>
>     1. Click the filter icon near the top right in Treeherder (next to
> coloured circles)
>     2. Select "test path" from the drop down
>     3. Type out the test path you are looking for
>     4. Click "Add"
>
> Please note that reftest isn't supported in this filter yet, and WPT
> requires test identifiers (e.g `/foo` instead of
> `testing/web-platform/tests/foo`).
>
> *What's Next?*
>
> There is a lot of work still coming down the pipeline. First and foremost,
> we want to enable "manifest scheduling" on autoland in addition to |mach
> try auto|. There are certain problems that exist on autoland (like sheriff
> backfilling) that we didn't need to worry about on try. Next we'll support
> manifest scheduling on all test suites (the major ones that are missing
> include reftest and derivatives, WPT reftests and mochitest-webgl). Last
> but not least, we're going to continue working through all of the feedback
> we've been gathering over the past weeks. Thanks to everyone who has left
> feedback around |mach try auto|, it's been invaluable!
>
> In case this post prompts you to try it out (or to revisit it again), then
> please file bugs and suggestions under `Firefox Build System :: Try`, or
> ping us in #test-selection on Matrix.
>
> Cheers!
> _______________________________________________
> firefox-dev mailing list
> firefox-...@mozilla.org
> https://mail.mozilla.org/listinfo/firefox-dev
>
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Reply via email to