On 29/11/2012, at 8:14 AM, Xavier Ducrohet wrote:
> Hello all,
>
> as it’s my first post on this list, some context: I’m the lead for the
> Android dev tools at Google and I’m working on our new build system based on
> Gradle. I’ve worked with Adam and Hans in the last few months to get the
> project started.
>
> We have a concept of build variants in our plugin which allow a project to
> build different versions of the same app (either different flavor of it, or
> different builds of it -- debug vs release for instance). Each variants has
> its own source sets, it own dependencies and its own output. One additional
> complication: some of the source sets could be used by multiple variants.
>
> IDE integration is critical to us and we need to make sure that the Tooling
> APIs can surface this to the IDEs. The idea is that the developer would
> choose which variant to work on, thus enabling a specific set of source sets
> in the IDE, setting up the current classpath, and selecting the proper output
> to run when requested.
>
> Additionally, we’d like to have other plugins be able to act on Android
> projects. For instance the FindBugs plugin would have to be able to know what
> sourcesets to use together, where the compilation output is, etc...
>
> I’d like to propose that Gradle adds native support for build variants, and
> that plugins should be able to “register” variants (our variants are
> automatically created from items defined in the build.gradle file), and the
> API should let other plugins (and the IDE) query for variants and their
> configuration (dependencies, source sets, output, etc...)
>
> I think the biggest challenge is handling sourcesets. The current Java
> SourceSet is very specific to the Java plugin and includes not only the
> source but also the output. Also, creating a SourceSet automatically creates
> a set of tasks to compile this SourceSet, which was not good for us.
> Unfortunately, this is what most other plugins rely on, querying Java
> sourcesets and their output (for instance the STS plugin does this, I’m sure
> the FindBug plugin does this as well).
This is all good stuff to do. Plus there are other nice things Gradle can do in
the Java/JVM space once we detangle our plugins from the idea that there is
exactly one output from a jvm-based project.
I'd see a model something like this:
* A project can produce one or more "jvm components" - things that can run on a
JVM. Some examples:
* A library
* A command-line application
* A web application
* An Android application
* An Android library
* Each jvm component can be "packaged" in one or more ways, depending on its
type:
* A directory (or directories) containing classes and resources.
* A jar.
* A dex file.
* An apk.
* An Android library.
* A native shared library, executable, or .net assembly, etc.
* Every packaging has some associated meta-data, such as runtime dependencies,
byte code version, platform api version, etc.
* Not every packaging makes sense for every component, but every jvm component
can be packaged as a directory containing classes (and almost always, the final
packaging is some transformation of this packaging). A given packaging may not
necessarily be usable.
* Not every packaging represents a component. For example, the unit tests for a
component can be packaged in one or more ways as well.
* Every packaging is built from some source files.
* Source files are grouped into source sets.
* Source sets contain source files and can contain other source sets. They have
some meta-data.
* Some source sets are language specific, containing only source files of a
certain type plus meta-data specific to that type of source.
* Language plugins add support for taking a language specific source set and
transforming it in some way (e.g. compiling or copying or generating resources
or whatever) into the appropriate packaging.
* Some source sets represent a logical grouping, such as main or unit test
source. Exactly which logical groupings make sense for a given component
depends on its type. Usually a logical source set will contain source from
several different languages.
I've avoided using the term 'variant', as we're still sorting out exactly what
'variant' means. Quite possibly the same thing as 'packaging' above. Certainly
each variant would be represented by a packaging, but possibly not every
packaging is a variant.
I don't think we need to care too much about components or variants initially.
We could probably start with something like:
Assuming that:
* The Java/Scala/Groovy plugins add a 'main' and a 'test' logical source set.
* The Android plugins add a 'main' and a 'test' logical source set, plus one
for each product flavor and build type.
Then, we can:
* Add some container of class directory packagings.
* The Java/Scala/Groovy plugins would add 1 class directory packaging for each
logical source set. The Android plugins would add 1 packaging per variant.
* Add some way to attach source sets to class directory packagings. The base
language plugins would know how to take a java source set and compile it into
the classes directory, or take a resource source set and copy it to the classes
directory, and so on.
* The Java/Scala/Groovy plugins would attach each logical source set to its
corresponding packaging. The Android plugins would attach the source sets for a
variant (main or test, product flavor and build type, etc) to the corresponding
packaging.
* Add some way to create a source set that contains generated source.
* The Android plugin would create a source set for the source it generates for
each variant, and attach this source set to the appropriate packaging.
* The Antlr plugin would create a source set for the Java source it generates,
and attach this to the appropriate logical source set (which would mean it
would work with the Android plugins).
* The code quality plugins would continue to inspect each logical source set.
We'd need to do something to choose the appropriate classes to pass to those
tools that work off the byte code.
* Add some way to attach source sets to the various IDE source paths and class
paths.
* The Java/Scala/Groovy plugins would add each logical source set to the
Eclipse source and class paths, and would add the 'main' and 'test' logical
source sets to the IDE source and class paths.
* The Android plugins would add the source sets for the selected variant to the
Eclipse and IDEA source and class paths.
* It might make sense to add a logical source set for each Android variant,
that is a composite made up of the various source sets for the variant.
Then later, we can build more stuff on top of this, introducing components and
variants and so on.
>
> In our case we have our plugin do things a bit differently. We don’t use the
> normal SourceSet and have rolled up our own to decouple the output from the
> source and not create tasks we don’t need. Instead we create our own
> JavaCompile tasks and wire the input and output manually based on the current
> variant and its sourcesets.
> We’d like to see if we can separate the notion of source folders from the
> output (and the creation of the tasks).
>
> On top of Java we also have JNI code as well as other languages (that are
> specific to Android). I’m not sure if we are going to use the CppSourceSet
> just yet as we are likely to simply call out to our make-based build system
> for those for now.
> In general though, we’d like to make sure the tooling API is flexible and
> that the concept of SourceSet is abstract enough so that we can use our own,
> while still benefiting from the ecosystem of Gradle plugins
> (Disclaimer: I’m not super familiar with the Tooling API yet)
I would see the JNI stuff working the same way as the Java stuff: A JNI library
can be packaged as a shared library or a set of object files, a packaging is
built from source files grouped into source sets, some of the source is
generated, base plugins know how to take a C or C++ source set and compile it
to object files, and how to take object files and link them into shared
libraries, and so on.
>
> We are starting to look at IDE integration, and there are a lot of changes to
> make there, but we’re going to need these changes in Gradle to be able to
> make progress on the IDE side.
>
> I realize this is a lot of work there. Adam and I already talked about it a
> bit. We're happy to help but you guys should probably figure out the internal
> design first :)
I'll start putting a spec together for this.
--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com
