On 02/08/2011, at 8:16 PM, Adam Murdoch wrote: >> Immediate reaction to this is that SCons and Waf have already solved >> most, if not all, of the algorithmic issues relating to the header >> transitivity involved here. Whilst it may be that in the Gradle context >> new "from scratch" algorithms are needed, certainly implementations will >> be, there is no point in ignoring the work already done. > > Right. The problem is more about how to we fit these algorithms into the > Gradle approaches (or change the Gradle approach to fit the algorithms). We > want there to be a consistent feel about the native stuff and the jvm stuff, > rather than just bolting some stuff on the side to (poorly) handle C/C++. And > I want to break some of the stranglehold that the jvm languages have on > Gradle, in its model and the assumptions it makes.
I don't think I follow what header transitivity means here. Is this determining all of the include file dependencies, including transitives, of a particular source file? If so, I think it's probably best to separate that from the discussion right now and focus on “dependency management” as we know it from the jvm space (i.e. resolving physical artifacts from named representations). Apologies if I am misunderstanding here. >> It may also be worth looking at the Go build scenario. With goinstall, >> gb, gofr, the Go people have taken convention over configuration to the >> extreme in that build requires no configuration files at all. They also >> have transitive import/include issues, and separate compile and link >> issues. I will do, thanks for the tip. It's unlikely that we can be so restrictive though, for better or worse. >> Regarding versioning of libraries, there are POSIX conventions for >> versioning of dynamic libraries (sonames) which have to be taken into >> account. > > Right. We will need to make sure the dependency resolution stuff understands > about sonames, so we can find the candidates that we can link against and > pass them through the standard conflict resolution stuff. And that we can > bake the correct soname into the binaries we produce. And that we name the > output binary for version 1.2 of library foo libfoo.so.1.2 rather than > libfoo-1.2.so, and so on. Does this affect publishing? i.e. do we need to publish artifacts with this convention? >>> One of the goals for our C++ support is to offer dependency management >>> for binary artifacts built from C/C++. <snip> >> Where is this dependency going to be resolved? Unlike JVM/Maven there >> is no central repository of versioned libraries. > > No, but the concept of repositories is still useful, I think: > > For example, an enterprise might have a repository manager which they publish > their native artifacts into for use within that enterprise. At least initially, I'd say this is the compelling case for c++ with Gradle. That's our (main) potential strength over other c++ tool chains for my money. > We would also have a localLibs repository type, which resolves dependencies > by looking in the standard locations on the local machine. We could take this > even further, where the repository implementation can invoke the native > package manager to install missing dependencies. *that* is an interesting idea. Seems entirely possible, but will be a challenge to make performant. >> Also is this a dynamic or a static dependency? > > Good question. I think there would be some convention, and some way to > override the convention, possibly on a per dependency basis. > > Every output binary would have static and dynamic variants defined, so you > can easily build and publish dynamically or statically linked variants of > your binary (or both). When building a dynamically linked variant, > dependencies would resolve to shared libraries, and when building a > statically linked variant, dependencies would resolve to static libraries. My current mental model is that we have a very abstract representation of the inputs to a compile operation (let's call this a compilespec); source, exported headers, and dependencies. At this level, dependencies are logical units and represent all of the flavours and artifacts of a dependency, which is close to the concept of an ivy configuration. The compile spec then inspects the logical dependency looking for the most suitable artifacts that it needs for the right time. So, inputs are abstract and compilespec implementations are responsible for finding the right artifacts within the logical unit. For example, grab the headers when compiling, grab the right platform binary when linking on the same platform etc. However, I don't think this is practical in reality. I suspect we are going to have to use a more rigid conventional approach with the compile spec implementations advertising what they need and some middle man making the dependency description more concrete before the resolve. I think that's doable, but I worry about its scalability to variant dimensions we don't know about. >> The Go folk are ignoring dynamic binding, and build only static >> executables. C, C++, Fortran, D on the other hand prefer dynamic >> binding over static binding. >> >>> This would imply that: >>> * The headers for the library are downloaded, and the appropriate -I >>> flag add to the compiler invocation. >> >> Downloaded from where? What about stuff already installed? Surely >> search /usr/include first before downloading anything. > > I think it depends on the project, and which variant is being built. This a > good reason for modelling repositories, so that a project can define where > and how to find its dependencies, with maybe an implicit localLibs repository > at the end of the chain. Some interesting problems here. We need to be clear on what Gradle can bring to this space and make sure we hit that. I think that may mean breaking from some traditional practices with dependencies, or maybe not, we'll see. I think we should aim to model all of the inputs to a compile operation. >> Header files are rarely versioned for C, C++, Fortran, D. Go handles >> things by nigh on requiring everything to be in Git, Mercurial or >> Bazaar. They then provide all the versioning. >> >>> * The library binary for the variant we are building is downloaded, >>> and added to the linker invocation. >> >> From where? Potentially anywhere. <snip> >>> * Depending on its type, the library binary is made available at >>> runtime when running the tests, or building a distribution, or >>> installing the binary. >> >> Which platform? Dynamic binding or static binding? What about >> different versions of different platforms? > > These are what we've been calling a 'variant'. At runtime and link time, a > binary needs to be provided with compatible variants of its dependencies. > Exactly which variants are compatible with other variants is a function of > the implementation language, the target platform, and sometimes, the project. Conceptually, I think it's the responsibility of the compile operation (aka compilespec). I am talking about quite concrete concepts in our model now. >>> There is also some transitive dependency management needed: >>> * Library headers may require headers from some of its dependencies to >>> be available when compiling. >>> * Library binaries require its runtime dependencies to be available at >>> runtime. Challenging… very challenging. Not impossible though by a long shot. >>> There are several aspects where dependency management for native >>> binaries is different to what we current offer for the jvm: >>> >>> >>> * Different types of artifacts are required at different points in the >>> build: headers at compile time vs binaries at link time. It would be >>> nice to also solve this for building jvm based projects, so that a >>> project can, for example, publish an api jar for use when compiling, >>> and an impl jar for use at runtime. >>> >>> >>> * Related to this, different sets of transitive dependencies are >>> required at different points in the build: header dependencies at >>> compile time, nothing at link time, and full transitive closure of >>> runtime dependencies at runtime. But this is also true for jvm >>> projects, we just ignore the problem at the moment. The key thing I see here is that it becomes difficult to declare up from what physical artifact files are going to be needed, and then give them to something to use. Instead, we will need to declare what is to be used up front in some abstract sense that encompasses the many different physical forms it can take (which may or may not be the same thing). So up front we declare that we depend on libfoo, then all consumers of libfoo then need to decide what bits of libfoo they want. Notice that the consumer decides, not that someone decides for the consumer. <snip> >> The whole of Autoconf (and hence equivalent features in SCons and Waf) >> is about determining whether currently installed dependencies are >> sufficient to the task of compiling > > For now, we want to tackle projects which need to publish binaries to a > repository manager, and which resolve dependencies from both a repository > manager, and the local filesystem. And which publish binaries for a > (relatively) small and fixed set of platforms. There are two separate parts to this problem, but they do intersect. One is seeing whether the required dependencies are available on the platform, the other describing the characteristics of the platform (e.g. size of an int). Where the two meet is how the nature of the available dependencies becomes a platform characteristic (i.e. how it influences the actual source). In terms of mechanics, I don't see this being too difficult to solve and there are many patterns from auto tools that we could lift. What will be tedious is supporting all of the config probes that auto tools does (if we decide we want to do that). -- Luke Daley Principal Engineer, Gradleware http://gradleware.com
