For spurious failures, the problematic tests are predominantly dunit tests which are actually end-to-end tests involving 6 jvms. End-to-end tests are notorious for having reliability issues. This is magnified by reusing jvms from one test case to the next. Eventually, one of the 6 jvms hits a problem: 1) a race condition (writting a good distributed test is tricky) or 2) a long stop-the-world GC pause which causes a test to fail with a timeout or pause long enough to hit a more rare race condition. Since the jvms are reused, a test failure is also very likely to pollute the jvm which can in turn cause subsequent test failure(s).
The best overall way to improve the reliability of these dunit tests is to fork brand-new jvms for every dunit test case which would force each test case to run in isolation. I think changing build.gradle distributedTest task from "forkEvery 30" to "forkEvery 1" would help greatly. I'll file a JIRA ticket for forking the dunit jvms every test and propose it on dev. Updating dunit to use junit 4 syntax would help a little by allowing us to use some newer features and rules that are suitable to integration or end-to-end tests. This is already under way with GEODE-217. End-to-end tests are important but have some issues: 1) spurious failures and reliability problems are common 2) excessive redundant code coverage -- they tend to execute the same 10% of the code base 90% of the time 3) long execution time I would be remiss if I didn't use this opportunity to encourage the Geode dev community to refocus on writing true unit tests with Mockito or JMock while "moderating" the creation of new dunit (end-to-end) tests. We have our automated developer tests categorized as UnitTest, IntegrationTest and DistributedTest. True unit tests written using Mockito or JMock do result in meaningful, worthwhile tests both as "tests" and as design tools even for a code base such as Geode. These unit tests no less important than integration tests or end-to-end tests (both are still important and all three provide different types of complementary test coverage). Lastly a list of my favorite books (so far) on unit testing: Working Effectively with Legacy Code by Michael Feathers Practical Unit Testing with JUnit and Mockito by Tomek Kaczanowski Effective Unit Testing by Lasse Koskela Growing Object-Oriented Software, Guided by Tests by Steve Freeman and Nat Pryce Working Effectively with Legacy Code in particular covers best practices for breaking dependencies so you can isolate a class for testing it. -Kirk On Thu, Aug 13, 2015 at 12:55 PM, Roman Shaposhnik <ro...@shaposhnik.org> wrote: > Hi! > > it seems that we're running into various spurious test > failures. What's the plan to get it back to green again? > > Thanks, > Roman. >
