On Wednesday 2014-04-09 11:00 -0700, Gregory Szorc wrote: > The simple solution is to have a separate in-tree manifest > annotation for intermittents. Put another way, we can describe > exactly why we are not running a test. This is kinda/sorta the realm > of bug 922581. > > The harder solution is to have some service (like orange factor) > keep track of the state of every test. We can have a feedback loop > whereby test automation queries that service to see what tests > should run and what the expected result is. Of course, we will want > that integration to work locally so we have consistent test > execution between automation and developer machines.
I think both of these are bad. It should be visible near the tests whether they are running or not, rather than out of band, so that module owners and those working on the code are aware of the testing coverage. Annotating something as intermittent is halfway to disabling, and should thus be reviewed by test authors / module owners just like disabling should be; it sounds like you're proposing changing that as well. The latter solution also breaks being able to describe a test run with reference to a revision in the VCS repository. -David -- 𝄞 L. David Baron http://dbaron.org/ 𝄂 𝄢 Mozilla https://www.mozilla.org/ 𝄂 Before I built a wall I'd ask to know What I was walling in or walling out, And to whom I was like to give offense. - Robert Frost, Mending Wall (1914)
signature.asc
Description: Digital signature
_______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
