> On Jun 12, 2017, at 4:45 PM, Pavol Vaskovic <p...@pali.sk> wrote:
>
> I have sketched an algorithm for getting more consistent test results, so far
> its in Numbers. I have ran the whole test suite for 100 samples and observed
> the varying distribution of test results. The first result is quite often an
> outlier, with subsequent results being quicker. Depending on the "weather" on
> the test machine, you sometimes measure anomalies. So I'm tracking the
> coefficient of variance from the sample population and purging anomalous
> results when it exceeds 5%. This results in solid sample population where
> standard deviation is a meaningful value, that can be use in judging the
> significance of change between master and branch.
That’s a reasonable approach for running 100 samples. I’m not sure how it fits
with the goal of minimizing turnaround time. Typically you don’t need more than
3 samples (keeping in mind were usually averaging over thousands of iterations
per sample).
-Andy
_______________________________________________
swift-dev mailing list
swift-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-dev
