I typically use a smaller subset of tests while iterating on a change. Once I believe it is solid I’ll run the full set before I commit.
Anthony > On Jun 15, 2015, at 5:28 PM, Dan Smith <[email protected]> wrote: > > I think getting reasonable coverage from the pure unit tests is good goal. > I know Mark has done some work in the past to get code coverage reports for > these different categories working with jenkins, maybe we can get that > integrated with the geode CI. I think a combination of these unit tests and > module specific categories could help some get the fast feedback as Ashvin > suggested. > > I also agree with Ashvin that the full test suite should be required for > commits. One of the problems with commit-then-test is that it becomes hard > to track down who broke the tests if there are a lot of changes going in. > > -Dan > > On Mon, Jun 15, 2015 at 5:05 PM, Ashvin A <[email protected]> wrote: > >> Hi, >> >> We need a fast verification method for testing any changes, I agree. I >> think test categories/groups is a step in the correct direction. Then as a >> developer, I can quickly verify my changes at the module level. As Kirk >> said, we will invest in writing new tests and also identifying and fixing >> tests taking very long. >> >> However I think it is important to execute all tests before committing any >> changes. New developers will prefer shortcuts, but it is very difficult to >> weed out bugs later. I would not encourage reduced test set as a commit >> criteria. I fear that tests will be ignored once a reduced set is created. >> >> $.01 >> >> Thanks, >> Ashvin >> >> On Mon, Jun 15, 2015 at 4:39 PM, Kirk Lund <[email protected]> wrote: >> >>> Hi Jun, >>> >>> Here's some info on Geode tests. >>> >>> 1) junit >>> >>> These are unit tests involving mocks as well as end-to-end functional >>> tests. These are currently marked with @Category(UnitTest.class) or >>> @Category(IntegrationTest.class). >>> >>> We need to encourage Geode developers to write a lot of more of the >>> UnitTest category tests. There's really not enough of these at this point >>> in time. There needs to be a strong commitment to writing these unit >> tests >>> and refactoring older code to make it easier or even possible for certain >>> classes. >>> >>> 2) dunit >>> >>> The majority of developer tests are "dunit" tests which extend >>> DistributedTestCase. These create and manipulate 6 JVMs. Think of these >> as >>> end-to-end functional tests for a cluster. Unfortunately it take hours to >>> execute them all and there are some reliability issues in these tests. >> I'm >>> working on replacing DistributedTestCase with a custom junit runner so >> that >>> the syntax of these tests can be updated to using JUnit 4 annotations and >>> rules. >>> >>> 3) hydra >>> >>> These are bigger QA-developed tests that run longer and may use many more >>> JVMs than a dunit. Moving hydra and these tests into Geode is further >> down >>> the road after moving the rest of the dunit tests. >>> >>> >>> On Mon, Jun 15, 2015 at 4:00 PM, jun aoki <[email protected]> wrote: >>> >>>> Hi Dan, >>>> >>>> Unit testing is super important for open source projects, so I'm glad >>> some >>>> high quality testing is coming in to the project! >>>> >>>> Along with Roman's comment, it will be nice those tests ; >>>> 1. are unit level test where mocks are appropriately used and no >> running >>>> instances are required. (IMO, unit test should be lightweight, and more >>>> heavy integration tests should be done separately) >>>> 2. completes up to 30mins so that contributors can get a quicker >>> feedback. >>>> (we are hoping to get hundreds of contributors so be aware of this! :) >> ) >>>> >>>> As Roman suggested, should the unit test of GEODE-6 have basic >> acceptance >>>> tests (can be subset of the several-hour test set) and the open source >> CI >>>> should only tests the subset coverage? >>>> >>>> >>>> On Mon, Jun 15, 2015 at 10:07 AM, Roman Shaposhnik < >> [email protected] >>>> >>>> wrote: >>>> >>>>> On Mon, Jun 15, 2015 at 9:42 AM, Dan Smith <[email protected]> >> wrote: >>>>>> Welcome Jun! >>>>>> >>>>>> I like all of your suggestions! One thing to be aware of regarding >>> the >>>> CI >>>>>> jobs is that the test time is going to go up a lot once GEODE-6 is >>>>>> resolved. We'll be adding several hours of tests. That will make >>>> having a >>>>>> test patch CI job all that more valuable, but it also affects how >>> much >>>>>> resources need to be devoted to that. >>>>> >>>>> Does it just mean that we'd have to have a 'smoke test' testsuite? >>>>> >>>>> Thanks, >>>>> Roman. >>>>> >>>> >>>> >>>> >>>> -- >>>> -jun >>>> >>> >>
