On 18/07/2012, at 8:40 PM, Luke Daley wrote: > > On 03/07/2012, at 10:45 PM, Adam Murdoch wrote: > >> >> On 28/06/2012, at 11:40 AM, Daz DeBoer wrote: >> >>> For java code, it seems like there are at least 2 axes that could >>> contribute to the artifact 'type': >>> 1) client vs api vs test vs core >>> 2) code vs source vs javadoc >>> >>> Would these all be wrapped up under the single concept of 'type'? It seems >>> like they are both valuable things to model explicitly. >> >> Right. We should treat them separately. The idea is that an artefact's type >> defines the format of the artefact, and how the artefact can be used. A jar, >> for example, can really only be used in a classpath. A set of c++ headers >> can really only be used for c++ compilation. > > I don't think this is going to work, “type” is just not rich enough.
This is type in the object sense, so it's not a single valued attribute. Instead, a type implies a bunch of other attributes. So, an artefact whose type is a native binary will have attributes like os, architecture, compiler, and so on associated with it. Text artefacts will have an encoding associated with them. Bytecode artefacts would have a target jvm version associated with them. And so on. And there's some polymorphism going on here: a war is-a jar is-a zip is-an archive. Or a native shared library is-a native binary. > >> The artefact type doesn't say anything about the purpose of the artefact, >> just its format. It doesn't answer: what's the meaning of the stuff inside >> the artefact? ie. which class paths does this particular jar belong in? >> >> My thought was to model this at the component level, rather than the >> artefact level. A component type would have a set of 'usages' associated >> with it. A usage would imply a set of zero or more of the component's >> artefacts plus a set of zero or more dependencies on other components. > > I think a component is a graph. A “usage” is just a particular query of the > graph. Can you flesh this out a bit more? What would a java library look like as such a graph? What about a c++ library with a bunch of variants? Or a javascript library? > >> We could potentially model this at the artefact level as well, as the >> artefact's 'purpose'. For a given artefact, there is a many-to-many >> relationship with the places it needs to be used. For example, if I package >> my component in a single jar, then that jar must be included in the >> consumer's compile classpath, runtime classpath, test compilation classpath, >> and so on. > > Having the thing specify how it should be used is not right I think. Instead, > it should just clearly express what it is (via advertised facts) and the > consumer can query appropriately. Right. That's what we're talking about. A "usage" is a contract between the producer and the consumer that they agree on in advance. The producer's meta-data states: if you want to use this component in this particular way, then you need these artefacts and these dependencies and this usage-specific meta-data. So, for example, let's say the producer and consumer have agreed that a java library has 2 usages: compile, and runtime. Most likely they've done this indirectly, by using our spec for what a java library is (which is built into Gradle), by using the Java plugin. The publisher produces some meta-data that states: * This is a java library * In order to compile against it, you need my foo-api.jar plus the servlet api version >= 2.4 and java >= 1.5 * In order to use it at runtime, you need my foo-api.jar plus my foo-impl.jar plus the servlet api version >= 2.4 plus slf4j 1.6.6 and java >= 1.5 Every component has a type, and the type implies a bunch of usages and other meta-data. The meta-data for a component states the type, and fills in the blanks that are appropriate for that type. What the consumer actually does with that meta-data is the consumer's business. > >> If I package my component as an api jar + impl jar, then the api jar must be >> included in the consumer's compile classpath, both jars in the runtime >> classpath, and so on. > > The issue is that this is going to be domain specific. Resolving jars will be > different to js libraries etc. I don't think we can try and abstract over all > of this successfully. We're going to go domain specific, no question. And that's the plan. However, there are definitely abstractions that sit underneath all this. So, what we're doing here is to: 1. Model the concrete domain-specific strongly-typed things that we actually deal with (like Java library, native library, web application), instead of abstract generic weakly typed things (like Configuration or Module). 2. Map these concrete things to an abstract model (like Component, Usage, and Artifact). This is described here: https://github.com/gradle/gradle/blob/master/design-docs/dependency-model.md > > I think we need to consider radical changes to succeed here. For example, > forget abstraction and embrace dependency sets and adapting. I don't think > this is the right time to go into this, but I just want to make the point > that I think what we currently do (which is what we inherited) is so deeply > flawed that gradual evolution may not be appropriate. I think we need a > ground up rethink, then work out how to get there. Right. That's exactly what we're doing here: where to we want to end up, and given that, how to we get there? > >> The question is whether a source or javadoc artefact is a 'format' or a >> 'purpose'. I'd say it's both: a source zip has a particular format (i.e. >> it's a zip), and a particular purpose (i.e. it contains source). You could >> package up the source in any archive format and still have a source artefact >> (e.g. as a tgz in for a c++ based component). > > Its relationship to something else is what makes it a “source” artefact. It > then has further characteristics such as whether it's a zip, jar etc. I may > be able to handle either. > > The relationships are completely missing from our current data structures, > and they are crucially important. It's the same for variants. Absolutely. > >> >> >>> Daz >>> >>> On 19 June 2012 16:59, Adam Murdoch <[email protected]> wrote: >>> Hi, >>> >>> There are a couple of dependency management issues we want to fix by >>> introducing the concept of an artefact type. >>> >>> Currently, artefacts are opaque generic things that we know nothing about. >>> Which means we can't do anything particularly useful other than copy them >>> around or download and plonk them in a cache directory. There are some >>> problems with this: >>> * Native executables need to follow a particular platform-specific naming >>> scheme (or at least, should) and need to have the execute bit set. >>> * Native libraries need to follow a particular platform-specific naming >>> scheme, and the name should honour the soname/install_path baked into the >>> binary. >>> * Some types of artefacts, such as c++ headers or javascript source files >>> or api documentation, need to be arranged in a particular hierarchy >>> relative to each other (or, some artefacts represent a file tree, rather >>> than a file). >>> * There's no well-defined way to find the source and documentation of a >>> component that we're using. >>> * There's no good identifier for an artefact. Currently, you can use any >>> attribute from the union of those supported by ivy and maven. Which means >>> that you have to couple your dependency declarations to how the dependency >>> happens to be published. >>> >>> Instead, I think we want to change things so that we explicitly model >>> artefact types and define a set of well-known types. >>> >>> 1. An artifact will have a name and a type. >>> 2. The artefact (name, type) must be unique for a given module. >>> 3. The artefact extension and classifier attributes will be deprecated and >>> removed from our DSL. >>> 4. When publishing to an Ivy repository, the type will be used to map the >>> name to an Ivy artefact with (name, extension, type, m:classifier), which >>> then feeds into the artefact pattern to end up with a destination location >>> for the artefact. >>> 5. When publishing to a Maven repository, the type will be used to map to >>> name to a Maven artefact with (artifactId, extension, classifier), which >>> then is used to determine the destination location for the artefact. >>> >>> Things get more interesting when resolving. >>> >>> A Maven dependency declaration has a 'type' attribute associated with it, >>> which defaults to 'jar'. Maven maps this type to an (extension, classifier) >>> that it uses to look for the target artefact. One question is how we >>> interpret the type attribute. Do we: >>> * Map (group, artifactId, version, type) to Gradle module (group, >>> artifactId, version) and artifact (name, type), then do an exact match on >>> the name + type from the artefacts of the target module? >>> * Map (group, artifactId, version, type) to Gradle module (group, >>> artifactId, version) and artefact type, then select the artefacts of the >>> target module that have the specified type? >>> >>> The result will be the same when the dependency resolves to a module in a >>> maven repository. It may be different when the dependency resolves to a >>> module in an ivy repository. I think the second option works better, as it >>> gives the consumer the opportunity to state what type of things it is >>> after, and the producer the opportunity to decide how the artefacts are >>> structured. >>> >>> Another question is how we interpret a missing type attribute in a maven >>> dependency declaration. Do we: >>> * Always look for artefacts of type 'jar'? >>> * Look for artefacts of a type that is specific to the resolve we're doing? >>> That is, look for jars when resolving the Java compile classpath, look for >>> c++ headers when resolving the C++ compile paths, and so on. >>> >>> I like the second option, but it's probably not the right way to go here. >>> It might be useful, in that it would allow us to publish single variant >>> non-jar projects to an existing maven repository and consume these >>> Gradle-produced artefacts from another Gradle build via the Maven >>> repository. >>> >>> An ivy dependency declaration allows you to request one or more artefacts >>> identified by (name, type, extension). We would map the (type, extension) >>> to a Gradle artefact type, and then back to the repository specific type >>> when looking for the artefact in a particular repository. >>> >>> An ivy dependency also allows you to filter the artefacts by name, type >>> and/or extension. We would map the repository specific type to an ivy >>> artefact type and apply the filters. >>> >>> What this means is that, except for some adapters in the resolver, our >>> resolve and publish code would deal with the Gradle concept of an artefact >>> rather than the merged Ivy + maven concept that we use now. We would also >>> expose the Gradle artefact in the resolve results. >>> >>> >>> -- >>> Adam Murdoch >>> Gradle Co-founder >>> http://www.gradle.org >>> VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting >>> http://www.gradleware.com >>> >>> >>> >>> >>> -- >>> Darrell (Daz) DeBoer >>> Principal Engineer, Gradleware >>> http://www.gradleware.com >>> >> >> >> -- >> Adam Murdoch >> Gradle Co-founder >> http://www.gradle.org >> VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting >> http://www.gradleware.com >> > > -- > Luke Daley > Principal Engineer, Gradleware > http://gradleware.com > -- Adam Murdoch Gradle Co-founder http://www.gradle.org VP of Engineering, Gradleware Inc. - Gradle Training, Support, Consulting http://www.gradleware.com
