Re: Flaky tests due to timing issues

2017-02-21 Thread Thomas Mueller
Hi, >No I actually meant getting individual time-out values (or a scaling >factor for time-outs) from CIHelper. That class already provides the >means to skip tests based on where they are running. So it should be >relatively straight forward to have it supply scaling factors for >time-outs in a s

Re: Flaky tests due to timing issues

2017-02-21 Thread Michael Dürig
On 21.02.17 15:32, Alex Parvulescu wrote: Hi, If in this context b) actually means 'fix the tests to be more lenient' I agree this should be the way to go. Yes and this is not necessarily bad as I presume that some of the time-outs might well be overly tight (at least in some environments).

Re: Flaky tests due to timing issues

2017-02-21 Thread Michael Dürig
On 21.02.17 14:09, Thomas Mueller wrote: Hi, I assume with (b) you mean: change tests to use loops, combined with very high timeouts. Example: No I actually meant getting individual time-out values (or a scaling factor for time-outs) from CIHelper. That class already provides the means to

Re: Flaky tests due to timing issues

2017-02-21 Thread Alex Parvulescu
Hi, If in this context b) actually means 'fix the tests to be more lenient' I agree this should be the way to go. However I think the failing tests are not being given enough priority currently and if people aren't able to carve out the time for investigation, then it means we're stuck with very

Re: Flaky tests due to timing issues

2017-02-21 Thread Thomas Mueller
Hi, I assume with (b) you mean: change tests to use loops, combined with very high timeouts. Example: Before: save(); Thread.sleep(1000); assertTrue(abc()); After: save(); for(int i=0; !abc() && i<600; i++) { Thread.sleep(100); } assertTrue(abc()); The ad

Flaky tests due to timing issues

2017-02-21 Thread Michael Dürig
Hi, I assume that at least some of the tests that sporadically fail on the Apache Jenkins fail because of timing issues. To address this we could either a) skip these tests on Jenkins, b) increase the time-out, c) apply platform dependent time-outs. I would prefer b). I presume that there