On Mon, Aug 8, 2016 at 11:16 AM, Olaf Buddenhagen <olafbuddenha...@gmx.net> wrote: > >> This seems to trade time for machines by optimistically assuming all >> previous PRs passed tests. > > Right. > > As long as intermittent failures are rare, I think it's a fair > assumption. (Actual bugs showing up in pre-commit testing should be > fairly rare, as most are already found by automated PR testing and > optional manual try runs...) > > If we *do* have to deal with frequent intermittent failures, the many > speculative runs can actuall come in handy, as I explained in the other > followup.
I think part of the problem is that intermittents will continue to be an ongoing problem - I've never seen UI-based testing that got intermittents to zero, and not for lack of engineering effort. I'd like to get the numbers down, but I'm really worried about the experience for a new contributor submitting a PR. Already, there were the possible failure modes of "is this my failure" vs. "is this an intermittent," the latter of which the fine treeherder folk are working to build tools that auto-detect. But in this model, there's also "is this a failure from one of the N PRs before me?" which isn't a type of failure that is common (AFAIK) when contributing to OSS projects. Intermittents, especially when they lead to PR bitrot, already chase off users in Servo. I'm worried this extra failure mode will compound that problem. Autorollup, bisection, etc. is also a LOT of extra code for the autolander. As we've seen with homu (and the gecko autolander for that matter), these don't seem to be projects that attract huge contributor communities, despite the fact that lots of people rely on them. Further, they're already incredibly fragile/sensitive to the Rube Goldbergian setup of various build machines, platforms, networks to various bits of cloud infra, etc. Putting on my manager hat, building and rolling out all of this extra auto-retry/rollup stuff kinda scares me, and I worry for the sanity of whomever is on call to keep such a system chugging along. That said, the world I wish we could be in is pre-commit testing, as Servo has today. The autorollup approach does seem like one way to scale things up a bit, and I'm not sure of any better solution (other than possibly test selection/reduction) if you want to stick with pre-commit testing. - Lars _______________________________________________ dev-servo mailing list dev-servo@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-servo
