On 30/04/2012, at 6:06 PM, Adam Murdoch wrote:
> Hi,
>
> Something we want to do soon is to replace the buildSrc project with a
> regular project. There are a few motivations for this:
>
> * To improve the user experience for those builds that need dedicated build
> logic. For example, currently the buildSrc project's 'build' target is used.
> But this runs all the tests and checks, whereas for 95% of the time, the user
> is only interested in compiling the classes. Or, currently we need to clean
> the buildSrc project when the Gradle version changes, whereas for regular
> projects we don't need to do this. Or, currently the buildSrc project does
> not end up in the IDE model, but would be included if it were a regular
> project.
>
> * To allow build logic to both be published and used in the same build (but
> not in the same project, for now). This will mean that you can use your
> enterprise plugins in the same build that produces them. For example, you can
> use your custom release plugin to release your custom release plugin. We may
> use this in Gradle, too, when we add a plugin dev plugin.
>
> * To detangle project configuration from the project hierarchy. In
> particular, this required for parallel execution, so that projects can be
> configured in an arbitrary order, and across multiple JVMs and/or threads.
There's another kind of related case that I know of. Wanting to use an artifact
produced by a module (not necessarily “build logic”) in another build. Not sure
it's particularly relevant to the buildSrc discussion, but it's in the same
kind of space.
The case I had was this
I built a small library for reflectively invoking Grails applications in a
version agnostic manner, called grails-launcher. One of the things that this
library provided was some information on what dependencies were necessary for a
given Grails version and some quirks. I wanted to add a module that was
effectively a test suite for launching a bunch of different Grails versions. I
needed to use classes from grails-launcher in the build script to tweak the
setup for each version. There's no way to do this right now with Gradle. I had
to make the test project a different build.
What I would have liked to happen is that Gradle would have been able to
realise that before it could configure the test project it would have to build
the “main” project fully first, and make it available to the test project.
Quite challenging.
The general pattern here is just wanting to use application code within the
build, which I think is commonly desirable.
> DSL-wise, there are 3 main use cases:
> 1. Declare that a given script depends on the build logic from some a project.
> 2. Declare that every script depends on the build logic from some project. Or
> there might be a convention for this, so that you give a project a particular
> name or put it in a particular directory, and it is automatically picked up
> as a build logic project.
> 3. Inject configuration to all projects, including those projects that are
> built during configuration time.
>
> Use case 1
> -------------
>
> I think this is as simple as being able to add project dependencies to the
> build script's classpath configuration:
>
> buildscript {
> dependencies { classpath project(':buildLogic') }
> }
>
> When we simplify the DSL for applying plugins, this might become something
> like:
>
> apply project: ':buildLogic', plugin: 'my-custom-plugin'
>
> Implementation-wise, the configuration phase would look something like this:
>
> 1. Queue up the configuration of each project, in parent-first order (like we
> do now).
> 2. For each project, if not already configured, then execute the project's
> build script.
> 3. For each script that is executed:
> * Execute the buildscript { } section of the build script.
> * For each project dependency in the build script classpath, recursively
> configure and build the target project. Fail if the target project is
> currently being configured.
> * Resolve the build script classpath and execute the script.
> * For each call to evaluationDependsOn(), recursively configure the
> target project. Fail if the target project is currently being configured.
> 4. For each project that is built during configuration:
> * Configure the project as above
> * For each project dependency required to build the project, recursively
> configure the target project. Fail if the target project is currently being
> configured.
> * Add the tasks that build the runtime class path for the project to the
> DAG.
> * Execute the tasks.
>
> I think this boils down to some changes to dependency resolution:
>
> During the configuration of a project:
> 1. When a Configuration is resolved, for each project dependency we trigger
> configuration of the target project and building of its artefacts.
> 2. When a Configuration's buildDependencies are queried, for each project
> dependency we trigger configuration of the target project.
>
> At other times (e.g. task execution):
> 1. When a Configuration is resolved, for each project dependency assert that
> the target project has been configured and the artefacts built. It's an error
> if not.
> 2. When a Configuration's buildDependencies are queried, for each project
> dependency assert that the target project has been configured. It's an error
> if not.
>
> And the same kind of thing for task dependencies:
>
> * When a task's dependencies are resolved during configuration, trigger the
> configuration of the target project.
> * When a task's dependencies are resolved at other times, assert that the
> target project has been configured.
>
>
> Some open issues:
>
> * Currently, the buildSrc classes are available in the settings script. This
> would not be the case if a regular project is used. Some possible solutions:
> - Use an external script for any shared logic.
> - Allow the settings script to add projects in it's settingsscript { }
> section, and resolve configurations as above.
> - Move the logic to an external project, and allow plugins to be applied to
> the Settings object.
> - Allow build scripts to add projects.
> - Chop your settings script into 2: one which defines the build logic
> projects, and a second one that declares a dependency on that project and
> uses it to define the remaining projects.
>
> * Tasks can be executed before the DAG is fully populated, and before the
> 'DAG ready' event has been fired. This means that some conditional
> configuration may not have been executed when these tasks are executed.
> Introducing build types might be an option here, so that the conditional
> stuff is applied much earlier in the configuration phase.
>
> * Projects can be configured and tasks executed before the parent project has
> had a chance to do configuration injection. More on this below.
>
> Use case 2
> ------------
>
> I like the idea behind the buildSrc project: you just put your build logic in
> a certain place, and it is just made available. It would be a shame to lose
> this. I wonder, however, if we really need this, assuming we can reduce the
> boilerplate for adding a project dependency to a build script classpath down
> to a single statement. We might also tackle this by making script 'plugins'
> work more like plugins, so that something like:
>
> apply plugin: 'my-plugin'
>
> might come from a compiled class from another project, or might apply
> $rootDir/gradle/my-plugin.gradle (or whatever).
>
> This way, plugins are provided by the environment and the consuming script
> doesn't care where they come from. What is currently in buildSrc would turn
> into one of the following:
> * A regular project in some external build, with plugins published to a
> repository.
> * A regular project in the same build, with plugins built locally.
> * A script in some conventional (or declared) location.
>
>
> Use case 3
> ------------
>
> The current approach of using allprojects {} and friends for configuration
> injection isn't going to work, as the build logic project will potentially
> have been configured and built before the injecting script has a chance to
> execute.
>
> There are a couple of existing approaches that would work (but are a bit
> awkward):
> * Move the shared logic to a script, and apply it from various locations
> * Move the shared logic to a plugin in a second build logic project, and
> depend on it from various locations.
>
> The existing configuration injection methods have some other problems. First,
> these methods guarantee that the code is called for every project, and that
> every project is configured. However, this stops us doing some things:
> * Skip the configuration of projects that aren't relevant to the current
> build. Eg in the Gradle build, don't configure all the plugin projects if I'm
> running the unit tests for core.
> * Short-circuit the configuration of projects whose outputs are up to date.
> Eg in the Gradle build, when I'm working on the c++ plugin, don't configure
> all the core projects when none of their source or configuration has changed.
> * Use compatible pre-built artefacts from a binary repository, rather than
> configuring the projects and building their artefacts. Eg in the Gradle
> build, when I'm working on the c++ plugin, just get the rest of the binaries
> from the CI server (not a great example, but you get the idea).
>
> Second, these methods guarantee that the code is always called in the same
> context. This stops us doing some of these things:
> * Building separate chunks of the model concurrently.
> * Building the model across multiple JVMs or machines.
>
> So, I think we need a new DSL here. Some options:
>
> 1. Just change the injection methods, so that they drop these guarantees.
> 2. Change the injection methods so that they have 2 modes. Allow a build
> script to declare which mode it needs.
> 3. Add new injection methods, with different names to the existing methods.
> 4. Use scripts in conventional locations. So, perhaps
> $rootDir/gradle/allprojects.gradle is applied to each project before it is
> configured.
> 5. Allow configuration to be injected from the settings script (with the new
> semantics).
> 6. Add a new type of build script, with injection methods that have the same
> names as the existing ones, but with the new semantics.
>
> Option 1) is not really an option. Options 2), 3) and 6) don't solve the
> build logic project problem. Personally, I like 5), because it detangles the
> build configuration from the root project. What is interesting about this
> option is that it allows you to have a single .gradle file for an entire
> multi-project build, that both defines the projects and injects configuration
> into them.
>
> An open issue is exactly what the semantics of the injection methods would
> be. They're going to have to deal with the fact that the configuration code
> may end up running in various different JVMs. This has some implications as
> to how values are shared across projects, e.g. a calculated version.
>
>
> Migration
> ----------
>
> I think eventually we want to get rid of buildSrc altogether. The plan would
> be to implement the above use cases as experimental features, leaving
> buildSrc alone. Then, we should shake out the new configuration mechanism
> further with some of the parallel execution and partial configuration
> features. Once we're fairly happy with how this looks, we would deprecate the
> buildSrc project, and later remove it.
I'm not sure this is a good next direction. I think we should be focussing on
“the new configuration mechanism” first, and then leverage it to meet these use
cases. The buildSrc project as it stands has some shortcomings for sure, but
they aren't killing us.
A lot of the issues you raise above are really more about “the new
configuration mechanism” than the buildSrc issue to my eyes, and mixing the two
will make things more complicated. If we can independently configure and build
projects that are part of a multi module build then we will already have a lot
of what we would need to use other projects in the classpath of other projects.
I'd much rather focus on getting this right, focussing on the more common case
of traditional “project dependencies”, than solve the buildSrc problem. I think
we really need to look at publications and project dependencies (as they are
implemented now) before we can go down “the new configuration mechanism” route.
--
Luke Daley
Principal Engineer, Gradleware
http://gradleware.com
