On 4/13/2013 1:28 AM, Mike Hommey wrote:
The first thing to notice in here is the 13 spikes down. The last one is
bug 860371. I wasn't aware of any of the 12 others. It might be worth
looking into them to understand why they happen. Interestingly, my
dev-tree-management archive doesn't show any notification for these
(except for the last one), nor for any of the progressive "regressions".
analyze_talos.py intentionally ignores transient changes. Until
recently it required 5 new test runs to show a statistically significant
difference from the preceding 30 runs. (Bug 858877 changed this to 12
new runs and 12 preceding runs, but the concept is still the same.) So
it's expected that the one-time spikes were ignored.
analyze_talos.py does identify several real regressions in that dataset
-- but it suppresses emails for them because each one is an increase of
less than 2% (see bug 822249). Here are regressions identified by the
latest version of analyze.py:
changeset mem (KB) t-test % change
----------------------------------------
c1ee454506f6 3296860000 139.519 1.05%
0acac77dd920 3330290000 11.809 0.99%
e6ca584f4fe7 3358010000 9.753 0.86%
80d52655c8b8 3398300000 74.742 0.45%
d27445d1eac5 3450400000 26.214 0.99%
eaff15332579 3468970000 70.525 0.52%
0ff1755d6359 2799760000 263.117 -21.58%
1c690be31939 3586590000 11.241 28.06%
Perhaps we should modify bug 822249 to ignore the 2% threshold for
specific tests where we *do* care about small changes. Or for any
change with a very high t-test score. I filed this bug:
https://bugzilla.mozilla.org/show_bug.cgi?id=863061
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
