This is a great point, and I still have no idea what caused the linux 32/64
machines to change on July 30th. It appeared to be a gradual rollout
(indicates a machine issue which was picked up on reboot or something
similar). For running talos tests we pin to a specific revision in the
talos
On Tue, Aug 18, 2015 at 3:43 PM, Chris Pearce cpea...@mozilla.com wrote:
We recently had a false positive Talos regression on our team, which
turned out to be caused by a change to the test machine coinciding with our
push. This took up a bunch of energy and time away from our team, which we
We recently had a false positive Talos regression on our team, which
turned out to be caused by a change to the test machine coinciding with
our push. This took up a bunch of energy and time away from our team,
which we really can't afford.
So to mitigate that I propose that *before* the
On Wednesday 2015-08-19 10:43 +1200, Chris Pearce wrote:
We recently had a false positive Talos regression on our team, which turned
out to be caused by a change to the test machine coinciding with our push.
This took up a bunch of energy and time away from our team, which we really
can't
I did see the ts, paint regression. This happened on 4 different platforms and
was backed out for telemetry issues about 5 pushes later:
http://hg.mozilla.org/integration/mozilla-inbound/rev/1190bc7b862d
and the backout:
http://hg.mozilla.org/integration/mozilla-inbound/rev/59ad2812d3c7
By the
kmoir filed https://bugzil.la/1192994 this week to hook SETA up to talos. SETA
seems like the best tool we have right now to both reduce the overall test
burden and increase the reliability of the tests we do run.
___
dev-platform mailing list
On Fri, Aug 14, 2015 at 2:56 PM, Vladan D vdje...@mozilla.com wrote:
Tests are reliable if they detect regressions, aren't very noisy, and if they
measure things that have a real impact on actual Firefox user experience.
Do we track false positives on these? I say that because I got a mail
I don't think anyone systematically logs the false-positives.
Is this the ts_paint regression you're referring to?
https://groups.google.com/forum/#!searchin/mozilla.dev.tree-alerts/ts_paint/mozilla.dev.tree-alerts/FArVsa8guXg/FfY91JK7AAAJ
I don't think the perf sheriffs filed a bug for that one,
In general I'm in favour of this proposal, although it will probably
come back to haunt me in the not-too-distant future. That being said I
would like to know what criteria you used to distinguish reliable
talos tests from the rest.
kats
On Fri, Aug 14, 2015 at 5:26 PM, Vladan Djeric
Tests are reliable if they detect regressions, aren't very noisy, and if they
measure things that have a real impact on actual Firefox user experience.
We're using past experience with the tests to determine which ones meet this
criteria.
On Friday, August 14, 2015 at 5:45:35 PM UTC-4,
There are known issues with the test infrastructure (e.g. differences in
weekend vs weekday results) and those known issues are currently being
masked with human judgement.
A-Team has investigated these issues, and fixed some of them, but fixing
the rest will take a non-trivial amount of effort as
On Fri, Aug 14, 2015 at 3:44 PM, Vladan Djeric vdje...@mozilla.com wrote:
Is this the ts_paint regression you're referring to?
https://groups.google.com/forum/#!searchin/mozilla.dev.tree-alerts/ts_paint/mozilla.dev.tree-alerts/FArVsa8guXg/FfY91JK7AAAJ
Yeah. I only ask because in exercising
12 matches
Mail list logo
