On 06/05/2012, at 3:13 PM, Adam Murdoch wrote:
> On 06/05/2012, at 8:06 AM, Luke Daley wrote:
>
>>
>> On 30/04/2012, at 6:06 PM, Adam Murdoch wrote:
>>
>>> Hi,
>>>
>>> Something we want to do soon is to replace the buildSrc project with a
>>> regular project. There are a few motivations for this:
>>>
>>> * To improve the user experience for those builds that need dedicated build
>>> logic. For example, currently the buildSrc project's 'build' target is
>>> used. But this runs all the tests and checks, whereas for 95% of the time,
>>> the user is only interested in compiling the classes. Or, currently we need
>>> to clean the buildSrc project when the Gradle version changes, whereas for
>>> regular projects we don't need to do this. Or, currently the buildSrc
>>> project does not end up in the IDE model, but would be included if it were
>>> a regular project.
>>>
>>> * To allow build logic to both be published and used in the same build (but
>>> not in the same project, for now). This will mean that you can use your
>>> enterprise plugins in the same build that produces them. For example, you
>>> can use your custom release plugin to release your custom release plugin.
>>> We may use this in Gradle, too, when we add a plugin dev plugin.
>>>
>>> * To detangle project configuration from the project hierarchy. In
>>> particular, this required for parallel execution, so that projects can be
>>> configured in an arbitrary order, and across multiple JVMs and/or threads.
>>
>> There's another kind of related case that I know of. Wanting to use an
>> artifact produced by a module (not necessarily “build logic”) in another
>> build. Not sure it's particularly relevant to the buildSrc discussion, but
>> it's in the same kind of space.
>>
>> The case I had was this
>>
>> I built a small library for reflectively invoking Grails applications in a
>> version agnostic manner, called grails-launcher. One of the things that this
>> library provided was some information on what dependencies were necessary
>> for a given Grails version and some quirks. I wanted to add a module that
>> was effectively a test suite for launching a bunch of different Grails
>> versions. I needed to use classes from grails-launcher in the build script
>> to tweak the setup for each version. There's no way to do this right now
>> with Gradle. I had to make the test project a different build.
>>
>> What I would have liked to happen is that Gradle would have been able to
>> realise that before it could configure the test project it would have to
>> build the “main” project fully first, and make it available to the test
>> project. Quite challenging.
>>
>> The general pattern here is just wanting to use application code within the
>> build, which I think is commonly desirable.
>>
>>
>>> DSL-wise, there are 3 main use cases:
>>> 1. Declare that a given script depends on the build logic from some a
>>> project.
>>> 2. Declare that every script depends on the build logic from some project.
>>> Or there might be a convention for this, so that you give a project a
>>> particular name or put it in a particular directory, and it is
>>> automatically picked up as a build logic project.
>>> 3. Inject configuration to all projects, including those projects that are
>>> built during configuration time.
>>>
>>> Use case 1
>>> -------------
>>>
>>> I think this is as simple as being able to add project dependencies to the
>>> build script's classpath configuration:
>>>
>>> buildscript {
>>> dependencies { classpath project(':buildLogic') }
>>> }
>>>
>>> When we simplify the DSL for applying plugins, this might become something
>>> like:
>>>
>>> apply project: ':buildLogic', plugin: 'my-custom-plugin'
>>>
>>> Implementation-wise, the configuration phase would look something like this:
>>>
>>> 1. Queue up the configuration of each project, in parent-first order (like
>>> we do now).
>>> 2. For each project, if not already configured, then execute the project's
>>> build script.
>>> 3. For each script that is executed:
>>> * Execute the buildscript { } section of the build script.
>>> * For each project dependency in the build script classpath,
>>> recursively configure and build the target project. Fail if the target
>>> project is currently being configured.
>>> * Resolve the build script classpath and execute the script.
>>> * For each call to evaluationDependsOn(), recursively configure the
>>> target project. Fail if the target project is currently being configured.
>>> 4. For each project that is built during configuration:
>>> * Configure the project as above
>>> * For each project dependency required to build the project,
>>> recursively configure the target project. Fail if the target project is
>>> currently being configured.
>>> * Add the tasks that build the runtime class path for the project to
>>> the DAG.
>>> * Execute the tasks.
>>>
>>> I think this boils down to some changes to dependency resolution:
>>>
>>> During the configuration of a project:
>>> 1. When a Configuration is resolved, for each project dependency we trigger
>>> configuration of the target project and building of its artefacts.
>>> 2. When a Configuration's buildDependencies are queried, for each project
>>> dependency we trigger configuration of the target project.
>>>
>>> At other times (e.g. task execution):
>>> 1. When a Configuration is resolved, for each project dependency assert
>>> that the target project has been configured and the artefacts built. It's
>>> an error if not.
>>> 2. When a Configuration's buildDependencies are queried, for each project
>>> dependency assert that the target project has been configured. It's an
>>> error if not.
>>>
>>> And the same kind of thing for task dependencies:
>>>
>>> * When a task's dependencies are resolved during configuration, trigger the
>>> configuration of the target project.
>>> * When a task's dependencies are resolved at other times, assert that the
>>> target project has been configured.
>>>
>>>
>>> Some open issues:
>>>
>>> * Currently, the buildSrc classes are available in the settings script.
>>> This would not be the case if a regular project is used. Some possible
>>> solutions:
>>> - Use an external script for any shared logic.
>>> - Allow the settings script to add projects in it's settingsscript { }
>>> section, and resolve configurations as above.
>>> - Move the logic to an external project, and allow plugins to be applied
>>> to the Settings object.
>>> - Allow build scripts to add projects.
>>> - Chop your settings script into 2: one which defines the build logic
>>> projects, and a second one that declares a dependency on that project and
>>> uses it to define the remaining projects.
>>>
>>> * Tasks can be executed before the DAG is fully populated, and before the
>>> 'DAG ready' event has been fired. This means that some conditional
>>> configuration may not have been executed when these tasks are executed.
>>> Introducing build types might be an option here, so that the conditional
>>> stuff is applied much earlier in the configuration phase.
>>>
>>> * Projects can be configured and tasks executed before the parent project
>>> has had a chance to do configuration injection. More on this below.
>>>
>>> Use case 2
>>> ------------
>>>
>>> I like the idea behind the buildSrc project: you just put your build logic
>>> in a certain place, and it is just made available. It would be a shame to
>>> lose this. I wonder, however, if we really need this, assuming we can
>>> reduce the boilerplate for adding a project dependency to a build script
>>> classpath down to a single statement. We might also tackle this by making
>>> script 'plugins' work more like plugins, so that something like:
>>>
>>> apply plugin: 'my-plugin'
>>>
>>> might come from a compiled class from another project, or might apply
>>> $rootDir/gradle/my-plugin.gradle (or whatever).
>>>
>>> This way, plugins are provided by the environment and the consuming script
>>> doesn't care where they come from. What is currently in buildSrc would turn
>>> into one of the following:
>>> * A regular project in some external build, with plugins published to a
>>> repository.
>>> * A regular project in the same build, with plugins built locally.
>>> * A script in some conventional (or declared) location.
>>>
>>>
>>> Use case 3
>>> ------------
>>>
>>> The current approach of using allprojects {} and friends for configuration
>>> injection isn't going to work, as the build logic project will potentially
>>> have been configured and built before the injecting script has a chance to
>>> execute.
>>>
>>> There are a couple of existing approaches that would work (but are a bit
>>> awkward):
>>> * Move the shared logic to a script, and apply it from various locations
>>> * Move the shared logic to a plugin in a second build logic project, and
>>> depend on it from various locations.
>>>
>>> The existing configuration injection methods have some other problems.
>>> First, these methods guarantee that the code is called for every project,
>>> and that every project is configured. However, this stops us doing some
>>> things:
>>> * Skip the configuration of projects that aren't relevant to the current
>>> build. Eg in the Gradle build, don't configure all the plugin projects if
>>> I'm running the unit tests for core.
>>> * Short-circuit the configuration of projects whose outputs are up to date.
>>> Eg in the Gradle build, when I'm working on the c++ plugin, don't configure
>>> all the core projects when none of their source or configuration has
>>> changed.
>>> * Use compatible pre-built artefacts from a binary repository, rather than
>>> configuring the projects and building their artefacts. Eg in the Gradle
>>> build, when I'm working on the c++ plugin, just get the rest of the
>>> binaries from the CI server (not a great example, but you get the idea).
>>>
>>> Second, these methods guarantee that the code is always called in the same
>>> context. This stops us doing some of these things:
>>> * Building separate chunks of the model concurrently.
>>> * Building the model across multiple JVMs or machines.
>>>
>>> So, I think we need a new DSL here. Some options:
>>>
>>> 1. Just change the injection methods, so that they drop these guarantees.
>>> 2. Change the injection methods so that they have 2 modes. Allow a build
>>> script to declare which mode it needs.
>>> 3. Add new injection methods, with different names to the existing methods.
>>> 4. Use scripts in conventional locations. So, perhaps
>>> $rootDir/gradle/allprojects.gradle is applied to each project before it is
>>> configured.
>>> 5. Allow configuration to be injected from the settings script (with the
>>> new semantics).
>>> 6. Add a new type of build script, with injection methods that have the
>>> same names as the existing ones, but with the new semantics.
>>>
>>> Option 1) is not really an option. Options 2), 3) and 6) don't solve the
>>> build logic project problem. Personally, I like 5), because it detangles
>>> the build configuration from the root project. What is interesting about
>>> this option is that it allows you to have a single .gradle file for an
>>> entire multi-project build, that both defines the projects and injects
>>> configuration into them.
>>>
>>> An open issue is exactly what the semantics of the injection methods would
>>> be. They're going to have to deal with the fact that the configuration code
>>> may end up running in various different JVMs. This has some implications as
>>> to how values are shared across projects, e.g. a calculated version.
>>>
>>>
>>> Migration
>>> ----------
>>>
>>> I think eventually we want to get rid of buildSrc altogether. The plan
>>> would be to implement the above use cases as experimental features, leaving
>>> buildSrc alone. Then, we should shake out the new configuration mechanism
>>> further with some of the parallel execution and partial configuration
>>> features. Once we're fairly happy with how this looks, we would deprecate
>>> the buildSrc project, and later remove it.
>>
>> I'm not sure this is a good next direction. I think we should be focussing
>> on “the new configuration mechanism” first, and then leverage it to meet
>> these use cases. The buildSrc project as it stands has some shortcomings for
>> sure, but they aren't killing us.
>>
>> A lot of the issues you raise above are really more about “the new
>> configuration mechanism” than the buildSrc issue to my eyes, and mixing the
>> two will make things more complicated. If we can independently configure and
>> build projects that are part of a multi module build then we will already
>> have a lot of what we would need to use other projects in the classpath of
>> other projects. I'd much rather focus on getting this right, focussing on
>> the more common case of traditional “project dependencies”, than solve the
>> buildSrc problem.
>
> Perhaps I'm misunderstanding what you mean, but to me these are the same
> thing. The idea is that when you resolve a project dependency, Gradle makes
> sure that the target project has been configured and the necessary artefacts
> built, regardless of where or when you need the artefacts, be that to compile
> your build script, configure a project, or execute a task. The buildSrc
> project is just a concrete use case that we can use to drive this, and to fix
> some bugs and usability issues along the way. But it also happens to solve
> the more general problem of using project artefacts at configuration time.
>
> The buildSrc project is also a concrete use case that drives a nice subset of
> the more general configuration problem: we need to deal with the fact that
> configuration ordering will change based on project dependencies, but we
> don't yet have to deal with things like concurrent execution, logging,
> isolation, configuration inputs and outputs, caching the model, and so on.
> These will happen in later increments (but we do need to consider them when
> we design the solution).
>
> The main problem with allowing the configuration order to change is that
> existing configuration mechanisms like allprojects { } stop working, so we
> need to address configuration to some degree, whether that's a warning when
> you attempt to configure a project that has already been configured and
> built, or a new DSL, or whatever.
I think my reluctance stems from using a secondary use case to drive the work,
potentially leading to a short lived solution. This is a rather intangible
argument though, so I'll withdraw it.
>
> Given that using project artefacts at configuration time is a use case that
> hasn't been supported so far, so we have the opportunity to evolve an
> experimental DSL for a while, before we make it the preferred configuration
> DSL and start encouraging people to use it for all configuration. So, the
> sequence would be something like:
> 1. Add support for using project dependencies at configuration time,
> documented as an experimental feature [1].
> 2. Remove the experimental tag. Update samples and user guide to prefer using
> project dependencies over buildSrc.
> 3. Deprecate buildSrc.
> 4. Remove buildSrc.
>
> Of course, each step will be separated by a decent period of time (e.g. 4.
> cannot happen until Gradle 3.0 at the earliest).
>
> [1] 'experimental' isn't the right term here, but we don't have a good term
> for this yet.
>
>
>> I think we really need to look at publications and project dependencies (as
>> they are implemented now) before we can go down “the new configuration
>> mechanism” route.
>
> This is happening too. I do think it's somewhat orthogonal to the
> configuration model, in that we can tackle one without tackling the other.
> And we can certainly go parallel with the existing dependency model, but we
> definitely cannot go parallel with the existing configuration model.
>
> We'll be using use cases from c++, and the IDE world, and maybe OSGi, to
> flesh out a dependency model that allows us to solve things like partial
> configuration, short-circuiting project configuration, build aggregation, and
> so on.
And not putting runtime dependencies of project dependencies on the compile
classpath too.
--
Luke Daley
Principal Engineer, Gradleware
http://gradleware.com
