On 07/05/2012, at 5:54 PM, Luke Daley wrote:
>
> On 06/05/2012, at 3:13 PM, Adam Murdoch wrote:
>
>> On 06/05/2012, at 8:06 AM, Luke Daley wrote:
>>
>>>
>>> On 30/04/2012, at 6:06 PM, Adam Murdoch wrote:
>>>
>>>> Hi,
>>>>
>>>> Something we want to do soon is to replace the buildSrc project with a
>>>> regular project. There are a few motivations for this:
>>>>
>>>> * To improve the user experience for those builds that need dedicated
>>>> build logic. For example, currently the buildSrc project's 'build' target
>>>> is used. But this runs all the tests and checks, whereas for 95% of the
>>>> time, the user is only interested in compiling the classes. Or, currently
>>>> we need to clean the buildSrc project when the Gradle version changes,
>>>> whereas for regular projects we don't need to do this. Or, currently the
>>>> buildSrc project does not end up in the IDE model, but would be included
>>>> if it were a regular project.
>>>>
>>>> * To allow build logic to both be published and used in the same build
>>>> (but not in the same project, for now). This will mean that you can use
>>>> your enterprise plugins in the same build that produces them. For example,
>>>> you can use your custom release plugin to release your custom release
>>>> plugin. We may use this in Gradle, too, when we add a plugin dev plugin.
>>>>
>>>> * To detangle project configuration from the project hierarchy. In
>>>> particular, this required for parallel execution, so that projects can be
>>>> configured in an arbitrary order, and across multiple JVMs and/or threads.
>>>
>>> There's another kind of related case that I know of. Wanting to use an
>>> artifact produced by a module (not necessarily “build logic”) in another
>>> build. Not sure it's particularly relevant to the buildSrc discussion, but
>>> it's in the same kind of space.
>>>
>>> The case I had was this
>>>
>>> I built a small library for reflectively invoking Grails applications in a
>>> version agnostic manner, called grails-launcher. One of the things that
>>> this library provided was some information on what dependencies were
>>> necessary for a given Grails version and some quirks. I wanted to add a
>>> module that was effectively a test suite for launching a bunch of different
>>> Grails versions. I needed to use classes from grails-launcher in the build
>>> script to tweak the setup for each version. There's no way to do this right
>>> now with Gradle. I had to make the test project a different build.
>>>
>>> What I would have liked to happen is that Gradle would have been able to
>>> realise that before it could configure the test project it would have to
>>> build the “main” project fully first, and make it available to the test
>>> project. Quite challenging.
>>>
>>> The general pattern here is just wanting to use application code within the
>>> build, which I think is commonly desirable.
>>>
>>>
>>>> DSL-wise, there are 3 main use cases:
>>>> 1. Declare that a given script depends on the build logic from some a
>>>> project.
>>>> 2. Declare that every script depends on the build logic from some project.
>>>> Or there might be a convention for this, so that you give a project a
>>>> particular name or put it in a particular directory, and it is
>>>> automatically picked up as a build logic project.
>>>> 3. Inject configuration to all projects, including those projects that are
>>>> built during configuration time.
>>>>
>>>> Use case 1
>>>> -------------
>>>>
>>>> I think this is as simple as being able to add project dependencies to the
>>>> build script's classpath configuration:
>>>>
>>>> buildscript {
>>>> dependencies { classpath project(':buildLogic') }
>>>> }
>>>>
>>>> When we simplify the DSL for applying plugins, this might become something
>>>> like:
>>>>
>>>> apply project: ':buildLogic', plugin: 'my-custom-plugin'
>>>>
>>>> Implementation-wise, the configuration phase would look something like
>>>> this:
>>>>
>>>> 1. Queue up the configuration of each project, in parent-first order (like
>>>> we do now).
>>>> 2. For each project, if not already configured, then execute the project's
>>>> build script.
>>>> 3. For each script that is executed:
>>>> * Execute the buildscript { } section of the build script.
>>>> * For each project dependency in the build script classpath,
>>>> recursively configure and build the target project. Fail if the target
>>>> project is currently being configured.
>>>> * Resolve the build script classpath and execute the script.
>>>> * For each call to evaluationDependsOn(), recursively configure the
>>>> target project. Fail if the target project is currently being configured.
>>>> 4. For each project that is built during configuration:
>>>> * Configure the project as above
>>>> * For each project dependency required to build the project,
>>>> recursively configure the target project. Fail if the target project is
>>>> currently being configured.
>>>> * Add the tasks that build the runtime class path for the project to
>>>> the DAG.
>>>> * Execute the tasks.
>>>>
>>>> I think this boils down to some changes to dependency resolution:
>>>>
>>>> During the configuration of a project:
>>>> 1. When a Configuration is resolved, for each project dependency we
>>>> trigger configuration of the target project and building of its artefacts.
>>>> 2. When a Configuration's buildDependencies are queried, for each project
>>>> dependency we trigger configuration of the target project.
>>>>
>>>> At other times (e.g. task execution):
>>>> 1. When a Configuration is resolved, for each project dependency assert
>>>> that the target project has been configured and the artefacts built. It's
>>>> an error if not.
>>>> 2. When a Configuration's buildDependencies are queried, for each project
>>>> dependency assert that the target project has been configured. It's an
>>>> error if not.
>>>>
>>>> And the same kind of thing for task dependencies:
>>>>
>>>> * When a task's dependencies are resolved during configuration, trigger
>>>> the configuration of the target project.
>>>> * When a task's dependencies are resolved at other times, assert that the
>>>> target project has been configured.
>>>>
>>>>
>>>> Some open issues:
>>>>
>>>> * Currently, the buildSrc classes are available in the settings script.
>>>> This would not be the case if a regular project is used. Some possible
>>>> solutions:
>>>> - Use an external script for any shared logic.
>>>> - Allow the settings script to add projects in it's settingsscript { }
>>>> section, and resolve configurations as above.
>>>> - Move the logic to an external project, and allow plugins to be applied
>>>> to the Settings object.
>>>> - Allow build scripts to add projects.
>>>> - Chop your settings script into 2: one which defines the build logic
>>>> projects, and a second one that declares a dependency on that project and
>>>> uses it to define the remaining projects.
>>>>
>>>> * Tasks can be executed before the DAG is fully populated, and before the
>>>> 'DAG ready' event has been fired. This means that some conditional
>>>> configuration may not have been executed when these tasks are executed.
>>>> Introducing build types might be an option here, so that the conditional
>>>> stuff is applied much earlier in the configuration phase.
>>>>
>>>> * Projects can be configured and tasks executed before the parent project
>>>> has had a chance to do configuration injection. More on this below.
>>>>
>>>> Use case 2
>>>> ------------
>>>>
>>>> I like the idea behind the buildSrc project: you just put your build logic
>>>> in a certain place, and it is just made available. It would be a shame to
>>>> lose this. I wonder, however, if we really need this, assuming we can
>>>> reduce the boilerplate for adding a project dependency to a build script
>>>> classpath down to a single statement. We might also tackle this by making
>>>> script 'plugins' work more like plugins, so that something like:
>>>>
>>>> apply plugin: 'my-plugin'
>>>>
>>>> might come from a compiled class from another project, or might apply
>>>> $rootDir/gradle/my-plugin.gradle (or whatever).
>>>>
>>>> This way, plugins are provided by the environment and the consuming script
>>>> doesn't care where they come from. What is currently in buildSrc would
>>>> turn into one of the following:
>>>> * A regular project in some external build, with plugins published to a
>>>> repository.
>>>> * A regular project in the same build, with plugins built locally.
>>>> * A script in some conventional (or declared) location.
>>>>
>>>>
>>>> Use case 3
>>>> ------------
>>>>
>>>> The current approach of using allprojects {} and friends for configuration
>>>> injection isn't going to work, as the build logic project will potentially
>>>> have been configured and built before the injecting script has a chance to
>>>> execute.
>>>>
>>>> There are a couple of existing approaches that would work (but are a bit
>>>> awkward):
>>>> * Move the shared logic to a script, and apply it from various locations
>>>> * Move the shared logic to a plugin in a second build logic project, and
>>>> depend on it from various locations.
>>>>
>>>> The existing configuration injection methods have some other problems.
>>>> First, these methods guarantee that the code is called for every project,
>>>> and that every project is configured. However, this stops us doing some
>>>> things:
>>>> * Skip the configuration of projects that aren't relevant to the current
>>>> build. Eg in the Gradle build, don't configure all the plugin projects if
>>>> I'm running the unit tests for core.
>>>> * Short-circuit the configuration of projects whose outputs are up to
>>>> date. Eg in the Gradle build, when I'm working on the c++ plugin, don't
>>>> configure all the core projects when none of their source or configuration
>>>> has changed.
>>>> * Use compatible pre-built artefacts from a binary repository, rather than
>>>> configuring the projects and building their artefacts. Eg in the Gradle
>>>> build, when I'm working on the c++ plugin, just get the rest of the
>>>> binaries from the CI server (not a great example, but you get the idea).
>>>>
>>>> Second, these methods guarantee that the code is always called in the same
>>>> context. This stops us doing some of these things:
>>>> * Building separate chunks of the model concurrently.
>>>> * Building the model across multiple JVMs or machines.
>>>>
>>>> So, I think we need a new DSL here. Some options:
>>>>
>>>> 1. Just change the injection methods, so that they drop these guarantees.
>>>> 2. Change the injection methods so that they have 2 modes. Allow a build
>>>> script to declare which mode it needs.
>>>> 3. Add new injection methods, with different names to the existing methods.
>>>> 4. Use scripts in conventional locations. So, perhaps
>>>> $rootDir/gradle/allprojects.gradle is applied to each project before it is
>>>> configured.
>>>> 5. Allow configuration to be injected from the settings script (with the
>>>> new semantics).
>>>> 6. Add a new type of build script, with injection methods that have the
>>>> same names as the existing ones, but with the new semantics.
>>>>
>>>> Option 1) is not really an option. Options 2), 3) and 6) don't solve the
>>>> build logic project problem. Personally, I like 5), because it detangles
>>>> the build configuration from the root project. What is interesting about
>>>> this option is that it allows you to have a single .gradle file for an
>>>> entire multi-project build, that both defines the projects and injects
>>>> configuration into them.
>>>>
>>>> An open issue is exactly what the semantics of the injection methods would
>>>> be. They're going to have to deal with the fact that the configuration
>>>> code may end up running in various different JVMs. This has some
>>>> implications as to how values are shared across projects, e.g. a
>>>> calculated version.
>>>>
>>>>
>>>> Migration
>>>> ----------
>>>>
>>>> I think eventually we want to get rid of buildSrc altogether. The plan
>>>> would be to implement the above use cases as experimental features,
>>>> leaving buildSrc alone. Then, we should shake out the new configuration
>>>> mechanism further with some of the parallel execution and partial
>>>> configuration features. Once we're fairly happy with how this looks, we
>>>> would deprecate the buildSrc project, and later remove it.
>>>
>>> I'm not sure this is a good next direction. I think we should be focussing
>>> on “the new configuration mechanism” first, and then leverage it to meet
>>> these use cases. The buildSrc project as it stands has some shortcomings
>>> for sure, but they aren't killing us.
>>>
>>> A lot of the issues you raise above are really more about “the new
>>> configuration mechanism” than the buildSrc issue to my eyes, and mixing the
>>> two will make things more complicated. If we can independently configure
>>> and build projects that are part of a multi module build then we will
>>> already have a lot of what we would need to use other projects in the
>>> classpath of other projects. I'd much rather focus on getting this right,
>>> focussing on the more common case of traditional “project dependencies”,
>>> than solve the buildSrc problem.
>>
>> Perhaps I'm misunderstanding what you mean, but to me these are the same
>> thing. The idea is that when you resolve a project dependency, Gradle makes
>> sure that the target project has been configured and the necessary artefacts
>> built, regardless of where or when you need the artefacts, be that to
>> compile your build script, configure a project, or execute a task. The
>> buildSrc project is just a concrete use case that we can use to drive this,
>> and to fix some bugs and usability issues along the way. But it also happens
>> to solve the more general problem of using project artefacts at
>> configuration time.
>>
>> The buildSrc project is also a concrete use case that drives a nice subset
>> of the more general configuration problem: we need to deal with the fact
>> that configuration ordering will change based on project dependencies, but
>> we don't yet have to deal with things like concurrent execution, logging,
>> isolation, configuration inputs and outputs, caching the model, and so on.
>> These will happen in later increments (but we do need to consider them when
>> we design the solution).
>>
>> The main problem with allowing the configuration order to change is that
>> existing configuration mechanisms like allprojects { } stop working, so we
>> need to address configuration to some degree, whether that's a warning when
>> you attempt to configure a project that has already been configured and
>> built, or a new DSL, or whatever.
>
> I think my reluctance stems from using a secondary use case to drive the
> work, potentially leading to a short lived solution. This is a rather
> intangible argument though, so I'll withdraw it.
This is a reasonable concern. However, this isn't the only use case we'll be
using to drive the configuration model. It just happens to be the first one I
wrote up. We'll be considering a few others before we start this work. And the
order that we tackle stuff might change based on this.
As far as preventing short lived solutions go, I think this is a potential
problem with any use case that we solve, where we later discover other use
cases that are awkward or don't work with the solution. Some things I think we
want to do a little different, to help with this:
First, I think we want to do a little more design up front, to consider the use
cases we already know about. Up to now, we've been doing design on demand as
we're writing the code, and sometimes I think we make mistakes that a little
more thinking time might have prevented. I'm trying to do this with the
configuration model and dependency model changes we're planning (hence these
emails; the results are captured in subprojects/docs/src/design-docs).
However, the reality is that we will make design mistakes, and we need a way to
deal with this. Part of this is our add-deprecate-remove approach. To improve
this, I think we want a way to mark DSL elements as experimental and subject to
change without migration (or with limited migration). Currently, we're either
marking a lot of stuff as internal that should really be marked as
experimental, which prevents people from using it, or marking it as public when
it's not quite ready, so preventing us from fixing mistakes. Instead, we'd have
3 classes of code (from a user's point of view):
* Public - documented, and once included in an rc release, is stable until the
next major version. May be deprecated, with a replacement, before the next
major version. Generally, we do not remove support for any use cases, but we
may change how they are solved.
* Experimental - documented, but marked as experimental in the documentation.
Once included in an rc release, is stable only for that minor release (i.e. if
included in 1.1-rc-1, will be stable for 1.1-rc-2 and 1.1, but not 1.2-rc-1).
May change between releases, sometimes with a short deprecation period. As for
public features, we generally do not remove support for any use case, but may
change how they are solved.
* Internal - not documented. May change or disappear at any time, including
between rc releases (unlikely, but possible).
btw, does anyone have suggestions for a better term than 'experimental'?
>
>>
>> Given that using project artefacts at configuration time is a use case that
>> hasn't been supported so far, so we have the opportunity to evolve an
>> experimental DSL for a while, before we make it the preferred configuration
>> DSL and start encouraging people to use it for all configuration. So, the
>> sequence would be something like:
>> 1. Add support for using project dependencies at configuration time,
>> documented as an experimental feature [1].
>> 2. Remove the experimental tag. Update samples and user guide to prefer
>> using project dependencies over buildSrc.
>> 3. Deprecate buildSrc.
>> 4. Remove buildSrc.
>>
>> Of course, each step will be separated by a decent period of time (e.g. 4.
>> cannot happen until Gradle 3.0 at the earliest).
>>
>> [1] 'experimental' isn't the right term here, but we don't have a good term
>> for this yet.
>>
>>
>>> I think we really need to look at publications and project dependencies
>>> (as they are implemented now) before we can go down “the new configuration
>>> mechanism” route.
>>
>> This is happening too. I do think it's somewhat orthogonal to the
>> configuration model, in that we can tackle one without tackling the other.
>> And we can certainly go parallel with the existing dependency model, but we
>> definitely cannot go parallel with the existing configuration model.
>>
>> We'll be using use cases from c++, and the IDE world, and maybe OSGi, to
>> flesh out a dependency model that allows us to solve things like partial
>> configuration, short-circuiting project configuration, build aggregation,
>> and so on.
>
> And not putting runtime dependencies of project dependencies on the compile
> classpath too.
Definitely that one too.
--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com
