On 27/08/2011, at 7:54 PM, Luke Daley wrote: > > On 27/08/2011, at 12:10 AM, Adam Murdoch wrote: > >> >> On 26/08/2011, at 6:07 PM, Luke Daley wrote: >> >>> >>> On 26/08/2011, at 6:45 AM, Adam Murdoch wrote: >>> >>>> Hi, >>>> >>>> I'm not sure I like moving (some of) core/src/test out into a separate >>>> project. As a general pattern, I think I would much rather that the test >>>> fixtures exported by a project live in that project, rather than in a >>>> separate project, and are modelled as a separate publication. >>>> >>>> That is, right now, we have the following dependencies: >>>> >>>> core test -> internalTesting main -> core main >>>> >>>> I'd much rather: >>>> >>>> core test -> core testFixtures -> core main. >>> >>> core-test is going to have a compile dependency on core-main as well >>> though, so it's not quite that linear (unsure if that's relevant here). >>> >>>> where 'core testFixtures' are the core-specific test fixtures, such as >>>> classes like HelperUtil, AbstractTaskTest, AbstractTaskSpec. This doesn't >>>> include int testing fixtures such as GradleExecutor, and probably doesn't >>>> include general things such as TemporaryFolder, TestFile, and so on. >>> >>> Makes sense. If you are compiling against core, then you'll likely want to >>> compile against coreTestFixtures. >>> >>> However, at some point we'll probably want to publish a core-fixtures jar >>> of public utilities for this and we may want to actually have tests for >>> these test fixtures. This could be the same for any plugin, e.g. we might >>> want to publish a maven-fixtures jar for unit test fixtures for developers >>> who integrate with/use types from the maven plugin(s). >>> >>> The same could be said for integration tests. There may be a >>> maven-integ-fixtures that provides an in memory http server for setting up >>> repos for integration tests. >>> >>> I think this _potentially_ leaves us with the following groups of source >>> for a project >>> >>> main >>> test >>> testFixtures >>> integTest >>> integTestFixtures >>> >>> If we do want to have public test fixtures that might mean another four >>> (names aside): >>> >>> publicTestFixtures >>> publicTestFixturesTest >>> publicIntegTestFixtures >>> publicIntegTestFixturesTest >> >> I think there's no question all these categories of code will exist. >> However, I'm not sure we necessarily want to split each category out into >> its own source set. We do have some other dimensions we can use to >> categorise code, and we could use a mix of these: >> >> * Split into projects >> * Split into source sets >> * Split into package hierarchies >> * probably some others too >> >> I'm not sure what a good scheme would be, yet. I think we could simplify >> this a bit for now, let it soak for a while and see what we come up with. >> >> I'd suggest something like this, for now: >> >> * main source set -> any public code, regardless of type. >> * testFixtures source set -> internal test fixtures, regardless of type. >> * test source set -> unit tests for any code used outside the project (ie >> main and test fixtures) >> * integTest source set -> int tests for any code used outside the project > > That's what I did yesterday and it seems to work well. > >> One benefit of splitting out the tests and test fixtures is that a change >> to, say, a unit test in core won't go invalidating the unit and integration >> tests in all the other projects. >> >> Another benefit of splitting out test fixtures is that we can reuse these >> internal test fixtures in our other Gradleware work. So, 'internal' above >> means the same thing as 'internal' for APIs: it is primarily intended to be >> used inside Gradle, but you can use it if you like. Just keep in mind it's >> unstable and may change at any time. >> >> We'd keep the internalTesting project, it would contain only testFixtures. > > Putting the internalTesting main classes into testFixtures doesn't feel right > to me. They should be in main as that's the main point of this project. > That's the way this is currently setup. This makes more sense conceptually to > me.
The problem with this approach is that the semantics of the main source set changes based on the project it happens to live in. This means that our build infrastructure needs to know the purpose of the project in order to decide which rules to apply to which code. For example, the code in internalTesting main is semantically the same as the code in core testFixtures, and so should have the same rules applied to it. The build needs to know the internalTesting's purpose is to provide test fixtures, in order to do this. A developer must also have this mapping in their heads when deciding which source directory to add a source file. I think things would be much simpler if the meaning of a source set were the fixed and independent the purpose of the project. Perhaps the problem here is more with the partitioning of the projects. You can think of the core project as having 2 broad categories of production code: * Domain independent infrastructure and services. * Domain-specific APIs, DSLs and services. This is a very similar to the distinction between internalTesting main and core testFixtures. So, some options would be: * Move internalTesting main into core testFixtures and axe internalTesting. * Split core into domain-independent and domain-specified pieces. What is currently in internalTesting main becomes the testFixtures of the domain-independent project. We need to do this at some point, anyway, as tooling API uses some generic infrastructure from core, but must not drag in all the unnecessary runtime dependencies of the domain specific parts of core. For the internalIntegTest project, its purpose is to provide the public integration testing infrastructure; it just happens to have an empty main source set at the moment. It should be called 'integTest' and its main source set should be called testFixtures. > >> Not sure what to do with the integ test fixtures. The question is whether >> integration testing is conceptually part of core or not. It feels to me like >> it is a separate enough concern to warrant its own project. Given this, we >> wouldn't necessarily need an integTestFixtures source set, as the integTest >> project would have: >> >> * main -> public integ test fixtures >> * testFixtures -> internal integ test fixtures >> * test -> unit tests for the fixtures >> * integTest -> integ tests for the fixtures > > I created internalIntegTesting yesterday that contains > GradleDistributionExecuter and friends, but the source lives in main for the > same argument I gave above for internalTesting. > > Let's park the discussion about public versions of this stuff for the time > being until that firms up a little. > >>> This is starting to look like too much, but all of these (apart from main) >>> would be optional. I wouldn't expect many plugins to expose public >>> utilities for integration testing but it could definitely happen (and >>> should in some cases). >>> >>> Also, I'd rather have this source set explosion and very carefully control >>> what's public than try and jam them together (but I guess having so many >>> projects may be burdensome in the IDE). >>> >>>> One problem - apart from the conceptual awkwardness - with having a >>>> project's test fixtures in a separate project is that it blows our 'must >>>> be able to import into eclipse' story right out of the water, as the >>>> eclipse plugin will map these to: >>>> >>>> core -> internalTesting -> core >>>> >>>> which, of course, eclipse can't handle. >>> >>> If we move the integration testing harness out into its own project (as I >>> think you said we should), won't we have the same problem? >> >> We will. This is why my three options below all had a project's integTest >> source set mapped to a separate eclipse project. >> >> I think, if we tried, we could end up with something like: >> >> compile: >> integTest main/fixtures >> -> toolingApi main >> -> wrapper main >> (currently toolingAPI drags in core, but shouldn't) >> >> int test compile (using scala as an example): >> scala int test -> >> -> scala main >> -> plugins main >> -> core main >> -> integTest main/fixtures >> >> int test runtime (forking mode): >> scala int test >> -> integTest main/fixtures >> -> toolingApi main >> -> wrapper main >> -> gradle dist (which happens to contain scala main) >> >> int test runtime (embedded mode): >> scala int test >> -> scala main >> -> plugins main >> -> core main >> -> integTest main/fixtures >> -> toolingAPI main >> -> wrapper main >> -> launcher main >> -> core main >> -> coreImpl main >> -> ui main >> >> The basic idea is that integTest fixtures drag in only the tooling API and >> launcher at runtime. These would then make use of whatever plugins they can >> find on the classpath at runtime. Currently, the integ test fixtures drag in >> all projects at runtime, which means a huge cycle. >> >> A project would automatically pick up any plugins which it has a >> compile/runtime dependency on, and the project can add in more by adding an >> integTest compile/runtime dependency. >> >> A benefit of this approach is that a change to, say, the announce plugin, >> does not invalidate the integration tests for, say, c++. And that to run the >> int tests for the ide plugin, I don't need to build the docs. > > I want to do something like this soon as it would be a productivity > improvement for me if I needed to build less to run integration tests. > > Basically, I want to rework the intTestImage stuff to build an image per > integration test run (i.e. each project) so only what is strictly needed is > built. I have some ideas on how to do this. In embedded mode (the default), almost none of the int tests need anything from the distribution. There're a handful in integTest, but none in the other projects. So, an alternative approach would be to short circuit intTestImage if the tests are running in embedded mode. Or, change the ciBuild task to depend on forkingIntegTest, and change integTest so that it always uses embedded mode. This way, we can just use normal task dependencies to drag in the stuff that's needed. For most projects, integTest just needs to depend on integTestClasses, and forkingIntegTest can depend on integTestImage. For the integTest project, *integTest can depend on the int test image. And for the docs project, *integTest can depend on samples and docs. Which means that intTestImage can then depend on the 'binary' image, rather than the 'all' image. I think this might be a better option than a per-project int test image. > >> I think this works, except if any of integTest's runtime dependencies need >> to have integration tests (eg core, tooling API, or ui). > > Hmm, that's an interesting one. I can see that core and tooling api will. I'm not too concerned about the int tests for core/tooling api/launcher living in a separate project. The reality is that the int tests for these project are, in fact, cross-cutting and touch all these projects. We also need a home for those int tests that cover multiple plugins. For example, we have some int tests to check that the java, groovy and scala plugins all work happily together. -- Adam Murdoch Gradle Co-founder http://www.gradle.org VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting http://www.gradleware.com
