I'm not sure it makes sense to have one single duration that's acceptable for all tests. This ignores the value for each test in how likely it is to catch bugs which is certainly different for each test. If we are going to pick a single duration, then I think it should be much higher than 5 seconds. Of course, we also need to define what environment these runtimes are measured in. We could also put some slower tests behind a define since just because a test is slow of course doesn't mean it's not useful.
-- Michael Mior mm...@apache.org Le lun. 10 sept. 2018 à 03:53, Vladimir Sitnikov < sitnikov.vladi...@gmail.com> a écrit : > Hi all, > > I wonder what is a maximum duration for newly added test case that will > still be "reasonable". > I mean a test case that is executed in Travis to validate pull requests, > and a test case that is executed as a part of regular `mvn test` / `mvn > install`. > > Could you please pick maximum test duration that would still be reasonable > from your point of view? > I would be glad to hear explanations as well. > > 1sec > 5sec <-- I think this is the maximum reasonable test duration (for a single > test) > 10sec > 30sec > 60sec (1m) > 120sec (2m) > 240sec (4m) > 480sec (8m) > 960sec (16m) > 1920sec (32m) > > Note: the duration might depend on CPU performance, however could we just > ignore that? > > For instance: suppose I implement a fix for "CALCITE-2506 RexSimplify: > coalesce(unaryPlus(nullInt), unaryPlus(vInt())) results in AssertionError: > result mismatch", > and I add a test case that takes 60 seconds to execute. Is that reasonable? > Then I fix "CALCITE-2527 Simplify OR with comparisons leads to > AssertionError: result mismatch" and add a test case that takes yet another > 60 seconds. > Then I fix "CALCITE-2526 Simplify OR with comparisons leads to NPE" and add > a test that takes yet another 60 seconds. > > As for me, 60 seconds for a single test is unreasonable since it would > easily bump test execution times to heaven, especially in Travis that uses > one thread only. > > Note: there are cases like "it takes 20-30 seconds to start Cassandra", > however I assume Cassandra is started only once, then all the relevant > tests are run. > Then "restart Cassandra for each test method" is unreasonable, while > starting it once is fine. > > Vladimir >
