I've created GEODE-218 "Change distributedTest task to fork every 1 test case" for review and group discussion.
-Kirk On Thu, Aug 13, 2015 at 5:29 PM, Kirk Lund <kirk.l...@gmail.com> wrote: > For spurious failures, the problematic tests are predominantly dunit tests > which are actually end-to-end tests involving 6 jvms. End-to-end tests are > notorious for having reliability issues. This is magnified by reusing jvms > from one test case to the next. Eventually, one of the 6 jvms hits a > problem: 1) a race condition (writting a good distributed test is tricky) > or 2) a long stop-the-world GC pause which causes a test to fail with a > timeout or pause long enough to hit a more rare race condition. Since the > jvms are reused, a test failure is also very likely to pollute the jvm > which can in turn cause subsequent test failure(s). > > The best overall way to improve the reliability of these dunit tests is to > fork brand-new jvms for every dunit test case which would force each test > case to run in isolation. I think changing build.gradle distributedTest > task from "forkEvery 30" to "forkEvery 1" would help greatly. I'll file a > JIRA ticket for forking the dunit jvms every test and propose it on dev. > > Updating dunit to use junit 4 syntax would help a little by allowing us to > use some newer features and rules that are suitable to integration or > end-to-end tests. This is already under way with GEODE-217. > > End-to-end tests are important but have some issues: > 1) spurious failures and reliability problems are common > 2) excessive redundant code coverage -- they tend to execute the same 10% > of the code base 90% of the time > 3) long execution time > > I would be remiss if I didn't use this opportunity to encourage the Geode > dev community to refocus on writing true unit tests with Mockito or JMock > while "moderating" the creation of new dunit (end-to-end) tests. > > We have our automated developer tests categorized as UnitTest, > IntegrationTest and DistributedTest. True unit tests written using Mockito > or JMock do result in meaningful, worthwhile tests both as "tests" and as > design tools even for a code base such as Geode. These unit tests no less > important than integration tests or end-to-end tests (both are still > important and all three provide different types of complementary test > coverage). > > Lastly a list of my favorite books (so far) on unit testing: > > Working Effectively with Legacy Code by Michael Feathers > Practical Unit Testing with JUnit and Mockito by Tomek Kaczanowski > Effective Unit Testing by Lasse Koskela > Growing Object-Oriented Software, Guided by Tests by Steve Freeman and Nat > Pryce > > Working Effectively with Legacy Code in particular covers best practices > for breaking dependencies so you can isolate a class for testing it. > > -Kirk > > On Thu, Aug 13, 2015 at 12:55 PM, Roman Shaposhnik <ro...@shaposhnik.org> > wrote: > >> Hi! >> >> it seems that we're running into various spurious test >> failures. What's the plan to get it back to green again? >> >> Thanks, >> Roman. >> > >
