Hi,
Something we want to do soon is to replace the buildSrc project with a regular
project. There are a few motivations for this:
* To improve the user experience for those builds that need dedicated build
logic. For example, currently the buildSrc project's 'build' target is used.
But this runs all the tests and checks, whereas for 95% of the time, the user
is only interested in compiling the classes. Or, currently we need to clean the
buildSrc project when the Gradle version changes, whereas for regular projects
we don't need to do this. Or, currently the buildSrc project does not end up in
the IDE model, but would be included if it were a regular project.
* To allow build logic to both be published and used in the same build (but not
in the same project, for now). This will mean that you can use your enterprise
plugins in the same build that produces them. For example, you can use your
custom release plugin to release your custom release plugin. We may use this in
Gradle, too, when we add a plugin dev plugin.
* To detangle project configuration from the project hierarchy. In particular,
this required for parallel execution, so that projects can be configured in an
arbitrary order, and across multiple JVMs and/or threads.
DSL-wise, there are 3 main use cases:
1. Declare that a given script depends on the build logic from some a project.
2. Declare that every script depends on the build logic from some project. Or
there might be a convention for this, so that you give a project a particular
name or put it in a particular directory, and it is automatically picked up as
a build logic project.
3. Inject configuration to all projects, including those projects that are
built during configuration time.
Use case 1
-------------
I think this is as simple as being able to add project dependencies to the
build script's classpath configuration:
buildscript {
dependencies { classpath project(':buildLogic') }
}
When we simplify the DSL for applying plugins, this might become something like:
apply project: ':buildLogic', plugin: 'my-custom-plugin'
Implementation-wise, the configuration phase would look something like this:
1. Queue up the configuration of each project, in parent-first order (like we
do now).
2. For each project, if not already configured, then execute the project's
build script.
3. For each script that is executed:
* Execute the buildscript { } section of the build script.
* For each project dependency in the build script classpath, recursively
configure and build the target project. Fail if the target project is currently
being configured.
* Resolve the build script classpath and execute the script.
* For each call to evaluationDependsOn(), recursively configure the target
project. Fail if the target project is currently being configured.
4. For each project that is built during configuration:
* Configure the project as above
* For each project dependency required to build the project, recursively
configure the target project. Fail if the target project is currently being
configured.
* Add the tasks that build the runtime class path for the project to the
DAG.
* Execute the tasks.
I think this boils down to some changes to dependency resolution:
During the configuration of a project:
1. When a Configuration is resolved, for each project dependency we trigger
configuration of the target project and building of its artefacts.
2. When a Configuration's buildDependencies are queried, for each project
dependency we trigger configuration of the target project.
At other times (e.g. task execution):
1. When a Configuration is resolved, for each project dependency assert that
the target project has been configured and the artefacts built. It's an error
if not.
2. When a Configuration's buildDependencies are queried, for each project
dependency assert that the target project has been configured. It's an error if
not.
And the same kind of thing for task dependencies:
* When a task's dependencies are resolved during configuration, trigger the
configuration of the target project.
* When a task's dependencies are resolved at other times, assert that the
target project has been configured.
Some open issues:
* Currently, the buildSrc classes are available in the settings script. This
would not be the case if a regular project is used. Some possible solutions:
- Use an external script for any shared logic.
- Allow the settings script to add projects in it's settingsscript { }
section, and resolve configurations as above.
- Move the logic to an external project, and allow plugins to be applied to
the Settings object.
- Allow build scripts to add projects.
- Chop your settings script into 2: one which defines the build logic
projects, and a second one that declares a dependency on that project and uses
it to define the remaining projects.
* Tasks can be executed before the DAG is fully populated, and before the 'DAG
ready' event has been fired. This means that some conditional configuration may
not have been executed when these tasks are executed. Introducing build types
might be an option here, so that the conditional stuff is applied much earlier
in the configuration phase.
* Projects can be configured and tasks executed before the parent project has
had a chance to do configuration injection. More on this below.
Use case 2
------------
I like the idea behind the buildSrc project: you just put your build logic in a
certain place, and it is just made available. It would be a shame to lose this.
I wonder, however, if we really need this, assuming we can reduce the
boilerplate for adding a project dependency to a build script classpath down to
a single statement. We might also tackle this by making script 'plugins' work
more like plugins, so that something like:
apply plugin: 'my-plugin'
might come from a compiled class from another project, or might apply
$rootDir/gradle/my-plugin.gradle (or whatever).
This way, plugins are provided by the environment and the consuming script
doesn't care where they come from. What is currently in buildSrc would turn
into one of the following:
* A regular project in some external build, with plugins published to a
repository.
* A regular project in the same build, with plugins built locally.
* A script in some conventional (or declared) location.
Use case 3
------------
The current approach of using allprojects {} and friends for configuration
injection isn't going to work, as the build logic project will potentially have
been configured and built before the injecting script has a chance to execute.
There are a couple of existing approaches that would work (but are a bit
awkward):
* Move the shared logic to a script, and apply it from various locations
* Move the shared logic to a plugin in a second build logic project, and depend
on it from various locations.
The existing configuration injection methods have some other problems. First,
these methods guarantee that the code is called for every project, and that
every project is configured. However, this stops us doing some things:
* Skip the configuration of projects that aren't relevant to the current build.
Eg in the Gradle build, don't configure all the plugin projects if I'm running
the unit tests for core.
* Short-circuit the configuration of projects whose outputs are up to date. Eg
in the Gradle build, when I'm working on the c++ plugin, don't configure all
the core projects when none of their source or configuration has changed.
* Use compatible pre-built artefacts from a binary repository, rather than
configuring the projects and building their artefacts. Eg in the Gradle build,
when I'm working on the c++ plugin, just get the rest of the binaries from the
CI server (not a great example, but you get the idea).
Second, these methods guarantee that the code is always called in the same
context. This stops us doing some of these things:
* Building separate chunks of the model concurrently.
* Building the model across multiple JVMs or machines.
So, I think we need a new DSL here. Some options:
1. Just change the injection methods, so that they drop these guarantees.
2. Change the injection methods so that they have 2 modes. Allow a build script
to declare which mode it needs.
3. Add new injection methods, with different names to the existing methods.
4. Use scripts in conventional locations. So, perhaps
$rootDir/gradle/allprojects.gradle is applied to each project before it is
configured.
5. Allow configuration to be injected from the settings script (with the new
semantics).
6. Add a new type of build script, with injection methods that have the same
names as the existing ones, but with the new semantics.
Option 1) is not really an option. Options 2), 3) and 6) don't solve the build
logic project problem. Personally, I like 5), because it detangles the build
configuration from the root project. What is interesting about this option is
that it allows you to have a single .gradle file for an entire multi-project
build, that both defines the projects and injects configuration into them.
An open issue is exactly what the semantics of the injection methods would be.
They're going to have to deal with the fact that the configuration code may end
up running in various different JVMs. This has some implications as to how
values are shared across projects, e.g. a calculated version.
Migration
----------
I think eventually we want to get rid of buildSrc altogether. The plan would be
to implement the above use cases as experimental features, leaving buildSrc
alone. Then, we should shake out the new configuration mechanism further with
some of the parallel execution and partial configuration features. Once we're
fairly happy with how this looks, we would deprecate the buildSrc project, and
later remove it.
--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com
