The two main ways to organize tests are hierarchies (such as packages or file 
folders) and tagging (such as tagging).

Each supports different purposes for selecting tests to run.

Hierarchies are good for stable, mutually exclusive purposes. Unit, 
Distributed, Integration, and Acceptance seem like pretty stable purposes for 
selecting tests. Another probably stable purpose is functional area, like when 
you want to run all of the tests for a particular area.

Those two stable purposes are orthogonal, which leads to somewhat replicated 
hierarchies: Each functional area has its own unit, distributed, integration, 
and acceptance tests, or vice versa. As long as the purposes really are stable, 
this replicated structure works just fine. If they’re not stable, it’s better 
(in the absence of Gradle doing unfortunate things when it finds and filters 
tests) to use tagging for the less-stable idea.

Tagging is good when you want to select tests *across* those more stable 
purposes, or *within* them. I don’t know enough about GemFire/Geode to know 
what things might be best represented by tags.

The key, I think, is to figure out which purposes are more stable, and use 
hierarchies to organize tests for those purposes, then use tagging for 
crosscutting or ad hoc selection.

—
Dale Emery
dem...@pivotal.io

> On Jun 26, 2018, at 4:34 PM, Patrick Rhomberg <prhomb...@pivotal.io> wrote:
> 
> I like the idea of good structure around our test-complexity @Category
> layout.
> 
> @Alexander, not to speak for Bruce, but to my mind things like SecurityTest
> / WanTest / etc are orthogonal to the UnitTest / IntegrationTest /
> DistributedTest / AcceptanceTest classification.  I like adding better
> runtime and structure on the latter, but there's no real reason to
> sacrifice the former labeling.
> 
> @Anthony, I'm pretty confident we'll need a commonTest.  Off the top of my
> head, it'll need to host the startup rules, GfshRule, and the
> SimpleSecurityManager.
> 
> On Tue, Jun 26, 2018 at 4:22 PM, Anthony Baker <aba...@pivotal.io> wrote:
> 
>> Sounds good, though I’m not entirely sure we need ‘commonTest/‘.  I guess
>> we’ll find out :-)
>> 
>> Anthony
>> 
>> 
>>> On Jun 26, 2018, at 4:19 PM, Dan Smith <dsm...@pivotal.io> wrote:
>>> 
>>> +1 for the suggested structure.
>>> 
>>> -Dan
>>> 
>>> On Tue, Jun 26, 2018 at 3:36 PM, Jacob Barrett <jbarr...@pivotal.io>
>> wrote:
>>> 
>>>> I'd like to suggest that we refactor our current test source set, which
>>>> contains both unit, integration and distributed tests, into distinct
>> source
>>>> sets, test, integrationTest, distributedTest. These source sets would
>>>> replace the use of the primary categories UnitTest, IntegrationTest and
>>>> DistributedTest.
>>>> 
>>>> The catalyst for this change is an issue that Gradle's test runner
>> doesn't
>>>> pre-filter categories when launching tests, so if the tests are
>> launched in
>>>> separate JVMs or Docker containers, like :distributeTest task, the cost
>> of
>>>> spinning up those resources is realized only to immediately exit without
>>>> running any test for all test classes in the module. Switching to
>> separate
>>>> source sets for each category would remove the need to filter on
>> category
>>>> and only tests in the corresponding source set would get executed in
>> their
>>>> external JVM or Docker container. An example of this issue is
>>>> geode-junit:distributedTest task, which forks all test classes in
>> separate
>>>> JVMs but never actually runs any tests since there are no
>> DistributedTest
>>>> tagged tests.
>>>> 
>>>> The secondary effect is a way too isolate dependencies in each of the
>>>> source sets. Unit tests in the test set would not have dependencies need
>>>> for integration tests or distributed test so that if you accidentally
>> tried
>>>> to import classes from those frameworks you would get a compiler
>> failure.
>>>> Likewise, integration tests would not include distributed test framework
>>>> dependencies. Any shared test classes like mock, dummies, fakes, etc.
>> could
>>>> be shared in a commonTest source set, but would not contain any tests
>>>> itself.
>>>> 
>>>> The proposed structure would look like this:
>>>> 
>>>> test/ - only contains unit tests.
>>>> integrationTest/ - only contains integration style tests.
>>>> distributedTest/ - only includes DUnit based tests.
>>>> commonTest/ - includes commonly shared classes between each test
>> category.
>>>> Does not contain any classes.
>>>> 
>>>> Thoughts?
>>>> 
>>>> -Jake
>>>> 
>> 
>> 

Reply via email to