+1. The goal of shippable builds is twofold:
1. to make sure opt builds+tests, or similar (artifact builds?) answer the question "is my commit good?" as fast as possible, and 2. to make sure shippable builds+tests answer the question "are these binaries correct and ready to ship, if we decide to ship this revision?" I agree that we should run a full suite of tests against shippable builds, which probably includes things like performance testing. We still need some class of builds+tests that answer the question "is my commit good?" quickly. If debug builds are sufficient for the most part, and opt builds+tests on try fill in the gaps, then yes. (That appears to be what this thread is largely about.) If not, I could see us having at least some subset of tests running against opt or artifact builds. If we switch talos to PGO now, we'll probably switch them to shippable builds at some point in the near future. On Thu, Jan 3, 2019 at 8:45 AM Andrew Halberstadt <a...@mozilla.com> wrote: > CC Callek > > How will this interact with the "shippable builds" project that Callek > posted > about awhile back? My understanding is there's a high probability PGO is > going away. Would it make sense to wait for that to project to wrap up? > > -Andrew > > On Thu, Jan 3, 2019 at 11:20 AM jmaher <joel.ma...@gmail.com> wrote: > > > I would like to propose that we do not run tests on linux64-opt, > > windows7-opt, and windows10-opt. > > > > Why am I proposing this: > > 1) All test regressions that were found on trunk are mostly on debug, and > > in fewer cases on PGO. There are no unique regressions found in the > last 6 > > months (all the data I looked at) that are exclusive to OPT builds. > > 2) On mozilla-beta, mozilla-release, and ESR, we only build/test PGO > > builds, we do not run tests on plan OPT builds > > 3) This will reduce the jobs (about 16%) we run which in turn reduces, > cpu > > time, money spent, turnaround time, intermittents, complexity of the > > taskgraph. > > 4) PGO builds are very similar to OPT builds, but we add flags to > generate > > profile data and small adjustments to build scripts behind MOZ_PGO flag > > in-tree, then we launch the browser, collect data, and repack our > binaries > > for faster performance. > > 5) We ship PGO builds, not OPT builds > > > > What are the risks associated with this? > > 1) try server build times will increase as we will be testing on PGO > > instead of OPT > > 2) we could miss a regression that only shows up on OPT, but if we only > > ship PGO and once we leave central we do not build OPT, this is a very > low > > risk. > > > > I would like to hear any concerns you might have on this or other areas > > which I have overlooked. Assuming there are no risks which block this, I > > would like to have a decision by January 11th, and make the adjustments > on > > January 28th when Firefox 67 is on trunk. > > _______________________________________________ > > dev-platform mailing list > > dev-platform@lists.mozilla.org > > https://lists.mozilla.org/listinfo/dev-platform > > > _______________________________________________ > dev-platform mailing list > dev-platform@lists.mozilla.org > https://lists.mozilla.org/listinfo/dev-platform > _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
