On 02/08/2011, at 5:54 PM, Russel Winder wrote:
> Adam,
>
> On Tue, 2011-08-02 at 10:14 +1000, Adam Murdoch wrote:
>
> I may be stating the obvious and known . . .
We're not C/C++ experts, and the domain is much more varied than the jvm
domain, so stating the obvious is a good thing for this discussion.
>
> Immediate reaction to this is that SCons and Waf have already solved
> most, if not all, of the algorithmic issues relating to the header
> transitivity involved here. Whilst it may be that in the Gradle context
> new "from scratch" algorithms are needed, certainly implementations will
> be, there is no point in ignoring the work already done.
Right. The problem is more about how to we fit these algorithms into the Gradle
approaches (or change the Gradle approach to fit the algorithms). We want there
to be a consistent feel about the native stuff and the jvm stuff, rather than
just bolting some stuff on the side to (poorly) handle C/C++. And I want to
break some of the stranglehold that the jvm languages have on Gradle, in its
model and the assumptions it makes.
>
> It may also be worth looking at the Go build scenario. With goinstall,
> gb, gofr, the Go people have taken convention over configuration to the
> extreme in that build requires no configuration files at all. They also
> have transitive import/include issues, and separate compile and link
> issues.
>
> Regarding versioning of libraries, there are POSIX conventions for
> versioning of dynamic libraries (sonames) which have to be taken into
> account.
Right. We will need to make sure the dependency resolution stuff understands
about sonames, so we can find the candidates that we can link against and pass
them through the standard conflict resolution stuff. And that we can bake the
correct soname into the binaries we produce. And that we name the output binary
for version 1.2 of library foo libfoo.so.1.2 rather than libfoo-1.2.so, and so
on.
> Windows DLLs are a whole other hell that I have no knowledge
> about.
>
> With systems such as Debian, Ubuntu, Fedora, etc. there are packaged
> versions of everything and these should be used unless cross-compiling.
>
>> One of the goals for our C++ support is to offer dependency management
>> for binary artifacts built from C/C++.
>>
>>
>> Ideally, we want to reuse the existing dependency dsl, rather than add
>> a new native-specific dsl. So, you'd do something like:
>>
>>
>> apply plugin: 'cpp'
>> apply plugin: 'native-executable'
>>
>>
>> dependencies {
>> compile 'someGroup:someProject:1.2'
>> }
>
> Where is this dependency going to be resolved? Unlike JVM/Maven there
> is no central repository of versioned libraries.
No, but the concept of repositories is still useful, I think:
For example, an enterprise might have a repository manager which they publish
their native artifacts into for use within that enterprise.
Or, a project might publish their binaries to github (say), so that you can use
the github repository type to use them in your build.
We would also have a localLibs repository type, which resolves dependencies by
looking in the standard locations on the local machine. We could take this even
further, where the repository implementation can invoke the native package
manager to install missing dependencies.
Repositories would allow publication, too, just like in the jvm model.
>
> Also is this a dynamic or a static dependency?
Good question. I think there would be some convention, and some way to override
the convention, possibly on a per dependency basis.
Every output binary would have static and dynamic variants defined, so you can
easily build and publish dynamically or statically linked variants of your
binary (or both). When building a dynamically linked variant, dependencies
would resolve to shared libraries, and when building a statically linked
variant, dependencies would resolve to static libraries.
>
> The Go folk are ignoring dynamic binding, and build only static
> executables. C, C++, Fortran, D on the other hand prefer dynamic
> binding over static binding.
>
>>
>> This would imply that:
>> * The headers for the library are downloaded, and the appropriate -I
>> flag add to the compiler invocation.
>
> Downloaded from where? What about stuff already installed? Surely
> search /usr/include first before downloading anything.
I think it depends on the project, and which variant is being built. This a
good reason for modelling repositories, so that a project can define where and
how to find its dependencies, with maybe an implicit localLibs repository at
the end of the chain.
>
> Header files are rarely versioned for C, C++, Fortran, D. Go handles
> things by nigh on requiring everything to be in Git, Mercurial or
> Bazaar. They then provide all the versioning.
>
>> * The library binary for the variant we are building is downloaded,
>> and added to the linker invocation.
>
> From where?
>
>> * The dependency meta-data is included in the published dependency
>> descriptor for the binary we are building.
>
> What meta-data? Is the intention to create the equivalent of Mave but
> for C, C++, Fortran, D
I think there are a few places where the runtime dependency meta-data ends up.
We need to manage each of these:
* When publishing to a repository, the meta-data is included in a dependency
descriptor (eg an ivy.xml) so that consuming projects know what the binary
requires at runtime. The consuming project might be linking against it, in the
case of a library, or installing it.
* The dependency meta-data ends up in native packaging files. We need to
translate these to the coordinates of the native packaging manager.
* The dependency meta-data is baked into the binary at link time. We need to
translate these into the platform's dependency coordinates.
>
>> * Depending on its type, the library binary is made available at
>> runtime when running the tests, or building a distribution, or
>> installing the binary.
>
> Which platform? Dynamic binding or static binding? What about
> different versions of different platforms?
These are what we've been calling a 'variant'. At runtime and link time, a
binary needs to be provided with compatible variants of its dependencies.
Exactly which variants are compatible with other variants is a function of the
implementation language, the target platform, and sometimes, the project.
>
>> There is also some transitive dependency management needed:
>> * Library headers may require headers from some of its dependencies to
>> be available when compiling.
>> * Library binaries require its runtime dependencies to be available at
>> runtime.
>
> See above :-(
>
>> There are several aspects where dependency management for native
>> binaries is different to what we current offer for the jvm:
>>
>>
>> * Different types of artifacts are required at different points in the
>> build: headers at compile time vs binaries at link time. It would be
>> nice to also solve this for building jvm based projects, so that a
>> project can, for example, publish an api jar for use when compiling,
>> and an impl jar for use at runtime.
>>
>>
>> * Related to this, different sets of transitive dependencies are
>> required at different points in the build: header dependencies at
>> compile time, nothing at link time, and full transitive closure of
>> runtime dependencies at runtime. But this is also true for jvm
>> projects, we just ignore the problem at the moment.
>>
>>
>> * There are multiple variants of a given artifact, and this is a
>> multi-dimensional: os, arch, debug, compiler, and so on. Again, it
>> would be nice to also solve this for jvm projects, so that a project
>> can, for example, publish groovy 1.7 and groovy 1.8 variants, or scala
>> 2.7 and scala 2.8 variants. Or a java project with some jni native
>> pieces can publish variants for the different platforms it supports.
>> Or a web app project can publish fat and skinny variants. Or dev, qa
>> and prod variants.
>
> When it comes to native executables you have to depend on a specific
> version of a specific platform. So for example Ubuntu 11.04 is
> incompatible with Ubuntu 11.11, which is very incompatible with Fedora
> 15 or Windows 2000, XP, Vista, 7, etc. Then there are the continuously
> updated platforms such as MacPorts (source based) and Debian Testing
> (except during a freeze)
Continuously updated platforms are something I hadn't considered.
I think the concept of source dependencies would help here. We've talked about
these on and off for a while now, where you can declare a dependency on
something which needs to be built before it can be used by the build. For
source based distributions, this might involve delegating to the native package
manager to find and build the dependencies. Or a repository which can download
a tarball and build the dependency.
>
> Static compilation aids somewhat, hence Gos position on linking
>
>> This also implies that for a given project, some artifacts will be
>> variant-specific (binaries, debug symbol tables), and some artifacts
>> will be variant independent (headers, source files, documentation).
>
> Which is why MacPorts and the whole of Gentoo is source based -- you
> never ship executables you only ship source and compile for your system.
> Fine for owners of one computer something of a disaster for people
> managing hundreds or even thousands.
>
>>
>> * For a given dependency, there are potentially multiple compatible
>> artifacts. For example, on linux, we can link against either a debug
>> or non-debug variant. So, if we're building a debug variant,
>> dependency resolution should prefer the debug variant of a dependency,
>> but can fall back to a non-debug variant if that's the only one
>> available. But, on windows, for example, we can't do this
>> substitution.
>>
>>
>> * Some tools are sensitive to the naming scheme we use for artifacts.
>> For the java tools, we pass the path to the artifact in the cache
>> directly to the tool, and this works fine pretty much regardless of
>> the artifact name. However, for example, we must not include the
>> version number in the header file names, and probably should not
>> include it in shared library names when linking or installing.
>
> Except that you have to bind in the soname to the dynamic libraries
> exactly so that dynamic linking can ensure version compatibility.
>
>>
>> * Different variants may be built and published together, or may be
>> built and published at different times. For example, I might build and
>> publish the linux variants in one gradle invocation, and then later
>> build and publish the windows variants.
>
> Published where? If the intention is to start a repository then this
> all might work. Without a repository, it is not clear that dependency
> resolution is a tractable problem.
>
> The whole of Autoconf (and hence equivalent features in SCons and Waf)
> is about determining whether currently installed dependencies are
> sufficient to the task of compiling
For now, we want to tackle projects which need to publish binaries to a
repository manager, and which resolve dependencies from both a repository
manager, and the local filesystem. And which publish binaries for a
(relatively) small and fixed set of platforms.
At this stage, we're not really interested in building on arbitrary platforms
or probing the local machine for the things we need in order to build. Of
course, we want to get to that point, but it's not a priority for us yet.
That is, for now, we want to tackle enterprise builds, where things are
(sometimes) relatively constrained, over open source builds, where there is a
huge variation in what's available, and where to find it.
>
>> So, that's the problem we want to solve. Of course, I'm sure we can
>> chop this into smaller pieces and solve the problem incrementally.
>>
>>
>> Any thoughts you have on either the description of the problem, or
>> possible solutions are most welcome. I'll post my thoughts about how
>> we solve the problem in some later emails.
>
> Hopefully the above help and are not too obviously already known.
Very helpful. Thank you.
--
Adam Murdoch
Gradle Co-founder
http://www.gradle.org
VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting
http://www.gradleware.com
