Ah, the woes of fragmented jars. That's the bane of EE dependency management. Figuring out the superset/subset stuff is indeed terrible. Good job, Stephen, to propose a solution for it.
Cheers, Paul On Fri, Jun 20, 2014 at 10:03 AM, Stephen Connolly < [email protected]> wrote: > Well this started out from a need for a more effective <excludes>. > > The issue with <excludes> is that you have to apply it to each and every > dependency... when what you really want to say is "see this dependency > here? well that is the same as X, Y and Z so don't pull them in from the > transitive dependencies" > > If you like that is the second use case in my proposal. > > The first use case is less relevant from the "better excludes" point of > view, but I believe it to be more useful. > > The third use case is for when you don't trust upstream to get things > right. Keep in mind that if, as a general principle, you don't trust > upstream to express things mostly correct then what the hell are you doing > trusting them to express their transitive dependencies correctly? > > Tooling would allow us to validate a pom's claims of equivalence. But > ultimately I think we need this concept to deal with te fragmentation of > API jars and implementations of APIs that we see taking place in central. > > > On 20 June 2014 15:27, Paul Benedict <[email protected]> wrote: > > > I am having trouble understanding how grouping together artifacts that > are > > "equivalent" gain me anything in my project building. I am already doing > > this in my POM today by excluding what's equivalent. Is this proposal > > simply about adding semantic information to the POM so it's more apparent > > what the intent is? > > > > Furthermore, whoever defines the equivalence needs to be extremely > careful. > > Not all "equivalent" jars are actually equivalent. Most people have > > learned, for example, that the javaee-api:6.0 jar in Maven is all stubs > and > > can't be used for unit testing; so people go to find EE jars from JBoss > or > > GlassFish, to get the full functionality. So I am not sure I'd ever want > > someone telling me what's equivalent during consuming. > > > > > > Cheers, > > Paul > > > > > > On Fri, Jun 20, 2014 at 7:51 AM, Stephen Connolly < > > [email protected]> wrote: > > > > > "supplies" concept proposal > > > =========================== > > > > > > Introduction > > > ------------ > > > > > > The following is a proposal for Maven in a post-modelVersion-4.0.0 era. > > The > > > aim of this proposal is to simplify the management of dependency trees > in > > > the decentralised era of artifact production that we find ourselves in. > > > > > > The core issue is that different organisations can produce artifacts > that > > > may overlap. The easiest example is the servlet-api. If we restrict > > > ourselves to version 2.5 of the servlet specification there are quite a > > few > > > artifacts that all deliver the exact same content: > > > > > > * `jetty:servlet-api:2.5-6.0.2` > > > * `org.daisy.libs:servlet-api:2.5.0` > > > * `org.mortbay.jetty:servlet-api-2.5:6.1.14` > > > * `org.jboss.spec.javax.servlet:jboss-servlet-api_2.5_spec:1.0.1.Final` > > > * etc > > > > > > **Note:** this is a generic problem that is not restricted to the > > > servlet-api, the servlet-api just provides the example that will be > most > > > familiar to everyone. > > > > > > So where these multiple artifacts supplying the equivalent content > > becomes > > > a problem is when the dependency tree is being calculated. If you have > > two > > > dependencies each declaring transitive dependencies on different > > artifacts > > > that supply equivalent content, then you end up with two copies of the > > same > > > JAR file in your classpath. > > > > > > In the case of the servlet-api, the hack most people use is to declare > > the > > > servlet-api with scope `provided` thus preventing it from being > > transitive. > > > This is, however, a hack. In a more ideal world it would be better to > let > > > the servlet-api be transitive and only when we get to the WAR module > > would > > > we declare that a specific servlet-api is to be provided in the > > containers > > > that the WAR is targets for deployment into. > > > > > > We can take a second example that does not have the luxury of a *de > > facto* > > > hack. > > > > > > * `javax.faces:jsf-api:2.1` > > > * `org.jboss.spec.javax.faces:jboss-jsf-api_2.1_spec:2.1.28.Final` > > > * `org.apache.myfaces.core:myfaces-api:2.1.13` > > > > > > Now in the case of the JSF API, you are supposed to bundle the JSF API > in > > > your WAR file. So if I use three JSF component libraries, I could very > > well > > > end up with three different but equivalent JSF API jars in my WAR file. > > > > > > Ideally what we want is some way of telling Maven that these artifacts > > are > > > equivalent. > > > > > > Proposal > > > -------- > > > > > > Introduce the concept of "supplies" to the project model. The concept > > needs > > > three changes to the project model: > > > > > > 1. An explicit top level construct for a project to explicitly declare > > > up-front artifacts that it knows - at the time the project is being > > > authored - to contain equivalent content to at least a subset of the > > > project's content. Declarations could include a claim from: `subset`, > > > `superset`, `disjoint` and `equivalent` with the default being > > `disjoint`. > > > 2. An explicit sub-element of the `dependency` construct to allow > > consumers > > > to *post-facto* declare a specific dependency as supplying equivalent > > > content for other dependencies > > > 3. An extension to the `dependency/excludes/exclude` construct to allow > > > consumers to remove claims a dependency makes with respect to supplying > > > equivalent content > > > > > > By way of illustration, here are some examples of these constructs > mapped > > > to a Model Version 4.0.0 like XML schema. As the > post-modelVersion-4.0.0 > > > schema is not yet known, this represents the best way to illustrate how > > the > > > concept will work, but note that this proposal does not suggest a > schema > > > for this concept. > > > > > > ### Example 1 > > > > > > This illustrates how we would want, say, the `myfaces-api` project > model > > to > > > look. > > > > > > ``` > > > <project> > > > <groupId>org.apache.myfaces.core</groupId> > > > <artifactId>myfaces-api</artifactId> > > > <version>2.1.3</version> > > > ... > > > <supplyManagement> > > > <supplies> > > > <groupId>javax.faces</groupId> > > > <artifactId>jsf-api</artifactId> > > > <version>[2.1,2.2)</version> > > > <claim>superset</claim> > > > <supplies> > > > <supplies> > > > <groupId>org.jboss.spec.javax.faces</groupId> > > > <artifactId>jboss-jsf-api_2.1_spec</artifactId> > > > <claim>equivalent</claim> > > > <supplies> > > > </supplyManagement> > > > ... > > > </project> > > > ``` > > > > > > This indicates that the `myfaces-api` artifact is intended to be > useable > > as > > > a drop-in replacement for either the `javax.faces:jsf-api` artifact > > within > > > a bounded range or for any version of the > > > `org.jboss.spec.javax.faces:jboss-jsf-api_2.1_spec` artifact. If you > get > > a > > > supplier conflict in your classpath, then Maven should fail the build. > > > > > > For example if somebody forked `myfaces-api` but did not list > > `myfaces-api` > > > in the fork's supplyManagement and you end up with both `myfaces-api` > and > > > `myfaces-fork-api` in your classpath. Maven can detect that there are > two > > > dependencies that both claim to supply `javax.faces:jsf-api` and fail > the > > > build, thereby letting the user add either exclusions or additional > > > supplies information to one of the artifacts and preventing duplicate > > > artifacts on the classpath. The build need not be failed if the > supplies > > > claims provide a resolution. e.g. if the claim is `equivalent` then > that > > > implies that there is a 1:1 mapping and hence the artifacts are drop-in > > > replacements. However where the claim is `superset` we cannot know that > > the > > > extra content in our artifact is the same as the extra content in > another > > > artifact which has a superset of `javax.faces:jsf-api`. > > > > > > ### Example 2 > > > > > > This illustrates a JSF component library that is working with the > > existing > > > JSF APIs > > > > > > ``` > > > <project> > > > ... > > > <dependencies> > > > <dependency> > > > <groupId>javax.faces</groupId> > > > <artifactId>jsf-api</artifactId> > > > <version>2.1</version> > > > <supplyManagement> > > > <supplies> > > > <groupId>org.jboss.spec.javax.faces</groupId> > > > <artifactId>jboss-jsf-api_2.1_spec</artifactId> > > > <claim>equivalent</claim> > > > <supplies> > > > <supplies> > > > <groupId>org.apache.myfaces.core</groupId> > > > <artifactId>myfaces-api</artifactId> > > > <version>[2.1,2.2)</version> > > > <claim>equivalent</claim> > > > </supplies> > > > </supplyManagement> > > > <dependency> > > > </dependencies> > > > ... > > > </project> > > > ``` > > > > > > In this case we are publishing a transitive dependency with additional > > > supplyManagement injected. Consumers of this project would thus gain > the > > > benefit of collapsing their transitive dependencies for any of these > > three > > > artifacts. As all artifacts are declared with `equivalent` claim, thus > > the > > > nearest of those three artifacts to the project will win as per the > > > standard dependency resolution rules when dealing with conflicting > > version > > > requirements in the transitive dependency tree. > > > > > > ### Example 3 > > > > > > Finally, there is the case where you need to correct an incorrect claim > > of > > > supply > > > > > > > > > ``` > > > <project> > > > ... > > > <dependencies> > > > <dependency> > > > <groupId>javax.faces</groupId> > > > <artifactId>jsf-api</artifactId> > > > <version>2.1</version> > > > <exclusions> > > > <exclusion> > > > <groupId>org.jboss.spec.javax.faces</groupId> > > > <artifactId>jboss-jsf-api_2.2_spec</artifactId> > > > <scope>supplies</scope> > > > <exclusion> > > > </exclusions> > > > <dependency> > > > </dependencies> > > > ... > > > </project> > > > ``` > > > > > > This would typically be coupled with adding back in a correct supplies > > > definition, but we need to allow for people to correct the graph after > > the > > > fact of their dependencies being deployed to the remote repository. > > > > > > ### Claim conflict resolution > > > > > > The four classes of claim can be resolved using the following matrix > > > > > > ``` > > > +---------------------------------------------------+ > > > | A | > > > +------------+------------+------------+------------+ > > > | subset | equivalent | superset | disjoint | > > > +---+------------+------------+------------+------------+------------+ > > > | | subset | conflict | A wins | A wins | conflict | > > > | +------------+------------+------------+------------+------------+ > > > | | equivalent | B wins | A or B | A wins | conflict | > > > | B +------------+------------+------------+------------+------------+ > > > | | superset | B wins | B wins | conflict | conflict | > > > | +------------+------------+------------+------------+------------+ > > > | | disjoint | conflict | conflict | conflict | conflict | > > > +---+------------+------------+------------+------------+------------+ > > > ``` > > > > > > The default unspecified claim is `disjoint` which indicates that some > of > > > the content is reproduced, but not all and there is additional content > > > added. With such a claim there will always be conflict and the build > > should > > > fail until the Project Model is updated to either remove some of the > > claims > > > or resolve the dependency clash. > > > > > > The ideal claim is `equivalent` which indicates that the two artifacts > > are > > > bi-directionally substitutable. This does not mean that the contents > are > > > identical. It does mean that they both deliver on the same contract in > an > > > equivalent fashion. > > > > > > The `subset` and `superset` claims are for aggregation APIs. So for > > example > > > the Java EE Web Profile API is a superset of the various spec APIs that > > > make up the Java EE Web Profile and a subset of the full Java EE > > > specification. The use of the `subset` claim should be reserved to > those > > > cases that are strict subsets. If anything is added that is not in the > > > supplied artifact then the correct claim is `disjoint`. > > > > > > ### Validation of supplies claims > > > > > > We do not want to introduce Java bias with this feature. As a result > the > > > validation of claims and supplies directives will be left to plugins. > For > > > the Java case we should probably provide either/both an enforcer rule > or > > a > > > maven hosted plugin to assist in checking JAR projects against the > > declared > > > supplies declarations, but Maven core should not be concerned with > > solving > > > the validation problem. > > > > > > Similarly, while there may be advantages in a more fine grained API > > > contract negotiation between dependencies, to bind such a concept into > > the > > > project model would significantly taint the Maven project model with > more > > > Java-centric concepts. Given that current software development > typically > > > uses at least two core languages: Java and JavaScript, we should be > > aiming > > > to reduce Java-centric constructs from Maven rather than increase them. > > > > > > ### Backporting > > > > > > It will not be possible to fully express this concept in a modelVersion > > > 4.0.0 project model. Thus if generating 4.0.0 compatible project > models, > > > the aim should be to fully resolve the dependencies of the project > using > > > all available information and express that as the transitive > > dependencies. > > > > > > Thus we will not expose the "supplies" information to modelVersion > 4.0.0 > > > parsers but we will expose the end results of that and present the > final > > > effective flattened dependency tree. > > > > > > modelVersion 4.0.0 consumers will thus be no worse off than they > already > > > are and those consumers understanding newer modelVersions can get the > > > enhanced tree resolution that they would not get otherwise. > > > > > >
