On 04/04/14 03:44 PM, Ehsan Akhgari wrote:
On 2014-04-04, 3:12 PM, L. David Baron wrote:
Are you talking about newly-added tests, or tests that have been
passing for a long time and recently started failing?
In the latter case, the burden should fall on the regressing patch,
and the regressing patch should get backed out instead of disabling
the test.
We have no good way of identifying the regressing patch in many cases.
In cases where the regression isn't caught right away, orangefactor is
currently the only tool at our disposal. While it is great for
identifying priorities to fix, it is less great at drilling down through
raw data to determine regression ranges. It is also very noisy which can
make it hard to interpret the data.
I think we need to spend some time:
A) Exposing more raw data for tools like orangefactor to consume (e.g
resource usage)
B) Figuring out how to separate the signal from the noise
C) Making it super easy to drill through this data to identify a
regression range.
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
