On Tue, Aug 25, 2015 at 7:13 PM, Chris Lamb <la...@debian.org> wrote: >> > Sure. Are you able to modify the test before running it on the relevant >> > system >> > and find a timing that works reliably? >> >> lamby, do I have access to the system on which the tests don’t pass? > > I fear gaining access to this machine would serve no real purpose; the > solution here is not to bump the values so that the test is "less" flaky > - the test would remain non-determistic and thus this bug would remain > unresolved IMHO, even though it might be harder to trigger. > > As a concept, I have no problem with automated solutions to point out > potential performance regressions, but having a testsuite that fails
I don’t think the intention of the test in question is to point out performance regressions, so while I agree with your general statement about flakyness in general, I’m not convinced it applies here. > non-determinstically is generally perceived to be a Bad Idea in software > engineering. Perhaps some sort of switch or environment variable can be > introduced to enable them so that they do not get in the way of the > regular build. > > > Regards, > > -- > ,''`. > : :' : Chris Lamb > `. `'` la...@debian.org / chris-lamb.co.uk > `- -- Best regards, Michael
