> On Jun 12, 2017, at 2:55 PM, Michael Gottesman <mgottes...@apple.com> wrote: > >> Current approach to detecting performance changes is fragile for tests that >> have very low absolute runtime, as they are easily over the 5% >> improvement/regression threshold when the test machine gets a little bit >> noisy. For example in benchmark on PR #9806 >> <https://github.com/apple/swift/pull/9806#issuecomment-303370149>: >> >> BitCount 12 14 +16.7% 0.86x >> SuffixCountableRange 10 11 +10.0% 0.91x >> MapReduce 303 331 +9.2% 0.92x >> These are all false changes (and there are quite a few more there). > > The current design assumes that in such cases, the workload will be increased > so that is not an issue.
That is also a valid fix for the problem, which I forgot to mention. -Andy
_______________________________________________ swift-dev mailing list swift-dev@swift.org https://lists.swift.org/mailman/listinfo/swift-dev
