So it seems we must enhance that read/write logic with these elements to make them usable:
1. Make it extension/Maven-IoC (even without extensions you can have the same issue with just guice components) friendly (code showing the bug is in a previous response) 2. Decide what is the produced pom, either the exact same pom as the source one without the model > 4 changes (= just a model N -> 4 converter) or a pom optimized to be consumed (flattened, without build section etc). Another point here is about comments, it should likely be a config in the pom to know if they must be stripped or not. If I take all projects I worked on, comments are build comments - except the license - so must not end up in the consummed pom if designed to be used by user (even if it does not hurt). Last pom is that it should also be configurable if some part of the pom are kept or not (maybe using plexus attributes) since some tool tend to parse poms at runtime to extract some metadata from there, guess it will slowly move to other descriptors but pom were nice because 75% of the time there so we should make the migration smooth IMHO by enabling to keep these elements. All that can be seen differently, if you take it from the outside, like from a gradle or sbt pespective, you kind of script the mapping memory descriptor -> consumable pom. Maven is not different as soon as it embraces this model so it must be user controlled - no way we do it right without user inputs. Guess a trivial way is to just let the user give a xslt in a maven pom producer plugin which would replace all this core code. If the plugin is not define we can use the default behavior adding it in the deploy phase out of the box. So overall I to join Matthieu on the target since it is what makes sense IMHO and seems to be widely used by people. Side nice note for us: It also opens the door to way faster resolutions if we add a property in these poms (maven.consumable.pom=true? or so) to cut the recursive dependency resolution if already done at deploy time. Can make it hundreds of time faster - used that in some previous work projects. Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book <https://www.packtpub.com/application-development/java-ee-8-high-performance> Le mar. 5 janv. 2021 à 09:09, Matthieu Brouillard <[email protected]> a écrit : > > Can you give an example? > Like Romain has shown: dynamically added dependencies or a version > computation. > > > Most important is the support for CI-friendly versions > But if extensions dynamically compute and set the versions in the POM using > the API, the changes will not be reflected. > That's why today one has to use flatten-maven-plugin or any other > plugin/task modifying/dumping the POM model to a pom.xml file and > setting/attaching it the POM. > > If the produced artifacts (consumer pom) were computed from the POM (object > model), every change done by any extension would be part of it. > > > but this happens AFTER using the pom > Not always from the pom.xml. I thought extensions were allowed to provide > ModelLocator & ModelReader to both decide which file to use for a project > and how to build the in memory POM model. > So the truth should not be considered to be in the pom.xml but in the POM. > > > On Mon, Jan 4, 2021 at 10:00 PM Robert Scholte <[email protected]> > wrote: > > > answers are below. > > > > Robert > > > > On 4-1-2021 16:52:23, Matthieu Brouillard <[email protected]> > wrote: > > @Robert nothing is broken atm, the changes for consumer/build are > currently > > behind your feature flag. > > Robert Scholte: > > It is active by default, so it is actually not hidden. > > > > > > > > But as I feared previously, and as Romain pointed it, by working at XML > > level (and not at POM level) the produced consumer pom does not reflect > > changes from extensions. > > Robert Scholte: > > Can you give an example? > > > > > > I really thought that all the "consumer/build" stuff would make the > > maven-flatten-plugin useless but it looks like it will not be the case if > > working at XML level. > > Robert Scholte: > > Like with most questions: it depends. Most important is the support for > > CI-friendly versions. In this case you won't need the > flatten-maven-plugin > > anymore. > > However, the plugin can rewrite much more, but this happens AFTER using > > the pom. > > That's something I don't like, because this POM was not used to build the > > project, but it was reassembled afterwards. > > My idea is to provide hooks to parts of the pom that might be adjusted, > > but this is something we can work on during the 4.x releases > > > > Did Romain and I miss the whole point of the "consumer/build" > enhancements > > or is it "just" because current implementation has not yet reached the > > targets/outputs? > > > > On Mon, Jan 4, 2021 at 2:56 PM Romain Manni-Bucau > > wrote: > > > > > Hmm, I don't get a few things of this IT: > > > > > > 1. the formatting seems not expected even if valid (the comments are > > > finishing with the first tag for example "--> > > > public class MyListener extends AbstractMavenLifecycleParticipant { > > > > > > @Override > > > public void afterProjectsRead(final MavenSession session) throws > > > MavenExecutionException { > > > if (session.getCurrentProject() == null) { > > > return; > > > } > > > > > > session.getProjects().forEach(p -> { > > > final Dependency dependency = new Dependency(); > > > dependency.setGroupId("junit"); > > > dependency.setArtifactId("junit"); > > > dependency.setVersion("3.8.1"); > > > p.getDependencies().add(dependency); > > > }); > > > } > > > } > > > > > > > > > 2. If you run mvn (4 snapshot) dependency:tree you get this kind of > > output: > > > > > > [INFO] -------------< org.sonatype.mavenbook.multi:simple-webapp > > > >------------- > > > [INFO] Building Multi Chapter Simple Web Application Project > > > 0.9-MNG6957-SNAPSHOT [6/6] > > > [INFO] --------------------------------[ jar > > > ]--------------------------------- > > > [INFO] > > > [INFO] --- maven-dependency-plugin:3.1.2:tree (default-cli) @ > > simple-webapp > > > --- > > > [INFO] > > org.sonatype.mavenbook.multi:simple-webapp:jar:0.9-MNG6957-SNAPSHOT > > > [INFO] +- > > > > > > > > > org.sonatype.mavenbook.multi:simple-weather:jar:0.9-MNG6957-SNAPSHOT:compile > > > [INFO] \- junit:junit:jar:3.8.1:compile > > > <-- THIS IS WHAT WE WANT TO SEE > > > [INFO] > > > > ------------------------------------------------------------------------ > > > > > > 3. run the build to have produced pom and cat the simple-webapp one: > > > > > > > > > 4.0.0 > > > > > > org.sonatype.mavenbook.multi > > > simple-parent > > > 0.9-MNG6957-SNAPSHOT > > > > > > > > > simple-webapp > > > Multi Chapter Simple Web Application Project > > > > > > > > > org.sonatype.mavenbook.multi > > > simple-weather > > > 0.9-MNG6957-SNAPSHOT > > > > > > > > > > > > simple-webapp > > > > > > > > > > > > org.apache.maven.plugins > > > maven-war-plugin > > > 2.6 > > > > > > > > > > > > > > > > > > > > > As you see the dependency is not there. I guess the expected outout is: > > > > > > > > > > > > 4.0.0 > > > simple-webapp > > > Multi Chapter Simple Web Application Project > > > [description, scm, ..., all central requires sections but not build > ones] > > > > > > > > > org.sonatype.mavenbook.multi > > > simple-weather > > > 0.9-MNG6957-SNAPSHOT > > > > > > > > > junit > > > junit > > > 3.8.1 > > > > > > > > > > > > > > > Am I missing something? > > > > > > Romain Manni-Bucau > > > @rmannibucau | Blog > > > | Old Blog > > > | Github < > > > https://github.com/rmannibucau> | > > > LinkedIn | Book > > > < > > > > > > https://www.packtpub.com/application-development/java-ee-8-high-performance > > > > > > > > > > > > > Le lun. 4 janv. 2021 à 13:41, Robert Scholte a > > > écrit : > > > > > > > > > > > > > > > > > https://github.com/apache/maven-integration-testing/tree/master/core-it-suite/src/test/resources/mng-6957-buildconsumer > > > > is the most complete IT > > > > > > > > On 4-1-2021 12:59:51, Romain Manni-Bucau wrote: > > > > Le lun. 4 janv. 2021 à 12:36, Robert Scholte a > > > > écrit : > > > > > > > > > There's just one thing I want to say: > > > > > I'm having trouble with the term "broken". > > > > > > > > > > > > > Well, literally meant broken as decorelated from the user intent and > > > > extension model. > > > > Anyway, didn't intend to blame but more identify the blockers for a > GA > > so > > > > point was really that it seem that on the two sides only the > producing > > > one > > > > is not yet ready since it keeps does not sanitize the model > completely > > > and > > > > keeps build only data like comments, right? Also not yet clear for me > > if > > > we > > > > loose the extension enrichments there. > > > > > > > > > > > > > If a Maven project could be built with M3.6.3, it can still be > built > > > with > > > > > M4. > > > > > If not, it is either regression (MNG-6957, MNG-7063) which must be > > > fixed, > > > > > or it requires changes to a plugin for understandable reasons > > > > > (maven-pgp-plugin) > > > > > AFAIK an interesting extension like the maven-tiles has been tested > > and > > > > > still works. > > > > > > > > > > > > > Do you have this handy, is it in our test suite? I'd like to check > the > > > > produced pom matches the enriched model but happy to start from > > something > > > > already there. > > > > > > > > > > > > > > > > > > Robert > > > > > > > > > > On 3-1-2021 19:35:25, Romain Manni-Bucau wrote: > > > > > Le dim. 3 janv. 2021 à 19:04, Robert Scholte a > > > > > écrit : > > > > > > > > > > > I don't remember all those details anymore, because I hit those > in > > > the > > > > > > beginning. > > > > > > Trying things over and over again I decided that this is probably > > the > > > > > most > > > > > > successful approach. > > > > > > > > > > > > > > > > > > What of the goals was to keep the pom.xml as is as much as > > possible. > > > > > > We can only decide for the specific Maven elements how to handle > > > them, > > > > we > > > > > > should not decide about comments and licenses. > > > > > > BTW, the license issue was hard to solve. You cannot use it from > > the > > > > > pom's > > > > > > , because there might be multiple licenses. > > > > > > > > > > > > > > > > I disagree, it is saner IMO to evolve to support that than doing > > > > > anything else. > > > > > Once again you keep things which don't make sense in a consumed pom > > in > > > > > current impl so i'd say the sucess in a few cases breaks as much > > cases > > > so > > > > > we need to revisit anyway IMHO. > > > > > > > > > > > > > > > > > > > > > > The current implementation is a solid way to ensure we're not > > > breaking > > > > > too > > > > > > much, because Maven controls the XML filters. > > > > > > > > > > > > > > > > Hmm, breaking extensions seems to break too much (I'm not speaking > of > > > > other > > > > > parts which breaks the ecosystem there but just that is sufficient > > IMHO > > > > to > > > > > say we must check back our solution). > > > > > > > > > > > > > > > > > > > > > > Also keep in mind, that I only want Maven to decide which > > > modifications > > > > > > are done. > > > > > > > > > > > > > > > > For the consumed pom I agree but it is consistent with keeping > > > everything > > > > > working instead of breaking too, no? > > > > > > > > > > > > > > > > > > > > > > Current polyglot projects should still work, but they cannot > > benefit > > > > from > > > > > > the build/consumer functions yet. > > > > > > > > > > > > > > > > So pom -> build model is kept, build model -> produced pom is > broken? > > > Is > > > > it > > > > > the complete status? > > > > > Sounds ok for a 4.0 and a 4.1 can fix it if so. > > > > > Just want to ensure first part is not broken at all. > > > > > > > > > > > > > > > > > > > > > > Robert > > > > > > On 3-1-2021 16:38:38, Romain Manni-Bucau wrote: > > > > > > Le dim. 3 janv. 2021 à 16:18, Robert Scholte a > > > > > > écrit : > > > > > > > > > > > > > > So what I was expecting was: raw xml model -> converted to > > > unified > > > > > > > consumed model -> extensions -> model processing. > > > > > > > > > > > > > > This is only the build pom part. You're missing the consume > part, > > > > where > > > > > > > the xml is distributed. > > > > > > > > > > > > > > > > > > > Yes but with previous chain the consume part is "clean/normalize > -> > > > > dump" > > > > > > since we are using consumed model - only standard model - in > memory > > > > > > already. > > > > > > > > > > > > > > > > > > > Build is raw + enrich, consumer is raw + enrich + reduce > > (removing > > > > > > > relativePath and modules are the first examples, but much more > is > > > > > > possible) > > > > > > > > > > > > > > Going for the in memory was also my first thought, but I would > > > loose > > > > > > > information, hence I came up with the current implementation. > > > > > > > > > > > > > > > > > > > I don't see what you loose ot be honest. > > > > > > You mentionned license but this one is in the pom so not a big > > deal, > > > > > > comments which are undesired IMHO as mentionned and element order > > > which > > > > > can > > > > > > really be discussed since we can desire to enforce an order to > > > > normalize > > > > > > consumption + it shouldn't be important since from the project > > point > > > of > > > > > > view your pom is already "broken"/lost (as all your intelligence > is > > > > lost > > > > > by > > > > > > this "not passthrough" process). > > > > > > So overall I don't see what you would loose from the consumer > side > > > but > > > > I > > > > > > see what you lost from maven ecosystem side. > > > > > > > > > > > > > > > > > > > Again, we're at a point where we can have counter solutions, > but > > > > don't > > > > > > > expect me to implement it. > > > > > > > > > > > > > > > > > > > For now I'm just trying to ensure we agree we don't want to break > > > > > existing > > > > > > extensions and the nice ecosystem we built after years. > > > > > > This was really a move forward and it sounds like we broke it at > > > maven > > > > 4 > > > > > > without any user gain which sounds terrible. > > > > > > > > > > > > > > > > > > > > > > > > > > thanks, > > > > > > > Robert > > > > > > > On 3-1-2021 15:25:21, Romain Manni-Bucau wrote: > > > > > > > Hi all, > > > > > > > > > > > > > > I kind of join Matthieu thoughts there, there is no point to > work > > > at > > > > > xml > > > > > > > level to create the consumed pom - comments is not a point > since > > it > > > > can > > > > > > > commonly/easily refer to a dropped part of the pom so they > should > > > be > > > > > > > stripped. > > > > > > > Current extension model got proven adapted and adopted, using a > > > lower > > > > > > level > > > > > > > extension API will not since XML is, even if still mainstream, > > > often > > > > > > > replaced by alternative configurations and to have done the > work > > to > > > > > > inject > > > > > > > XML configuration programmatically compred to current option, > it > > > is a > > > > > > pain. > > > > > > > The in memory model should stick to consumed model IMHO - being > > > > > > > programmatic there is no point to make it easier, worse case we > > can > > > > add > > > > > > > helper beans (injectable) but in terms of model it will not > help. > > > > > > > > > > > > > > So what I was expecting was: > > > > > > > > > > > > > > raw xml model -> converted to unified consumed model -> > > extensions > > > -> > > > > > > model > > > > > > > processing. > > > > > > > > > > > > > > Indeed, real chain adds a small processing over the first arrow > > > > (inject > > > > > > > versions for example) but nothing crazy and breaking this > overall > > > > flow > > > > > > > which stays user friendly. > > > > > > > > > > > > > > Strictly speaking the new model is just a built-in extension > for > > me > > > > > which > > > > > > > is particular because it will enforce IDE to integrate a new > > > format - > > > > > > > wheres polyglot extensions or others don't require static > > analyzis > > > by > > > > > > > themself not being "standard". > > > > > > > > > > > > > > That said, there is nothing crazy with current implementation, > it > > > > just > > > > > > > require to be updated to be able to take extension changes into > > > > > account. > > > > > > > This can be done by making the extension model 'spyable' (ie > if a > > > > > > > dependency/plugin is added it will be reflected in the final > > > written > > > > > > > pom.xml). > > > > > > > This sounds - instrumenting the extension model API or doing a > > diff > > > > > after > > > > > > > extension phase - like a compromise and let people gets the > best > > of > > > > > both > > > > > > > worlds to me. > > > > > > > > > > > > > > Wdyt? > > > > > > > > > > > > > > Romain Manni-Bucau > > > > > > > @rmannibucau | Blog > > > > > > > | Old Blog > > > > > > > | Github | > > > > > > > LinkedIn | Book > > > > > > > > > > > > > > > > > > > > > > > > > > > > Le dim. 3 janv. 2021 à 14:46, Robert Scholte a > > > > > > > écrit : > > > > > > > > > > > > > > > Hi Matthieu, > > > > > > > > > > > > > > > > As you understand, something had to be changed to move Maven > > > > forward. > > > > > > > > I've decided to pick up that challenge and came up with the > > > current > > > > > > > > solution. > > > > > > > > > > > > > > > > My main concerns was that I wanted to keep the fileModel as > > much > > > as > > > > > is. > > > > > > > > That includes the license, comments and element order. > > > > > > > > This information if not available in the memory model, so I > > > needed > > > > > the > > > > > > > > original pom file. > > > > > > > > With that in mind, the usage of XMLFilters looks like the > most > > > > > > > appropriate > > > > > > > > solution. > > > > > > > > > > > > > > > > I am certain that XML is still the most used format, so if we > > can > > > > > have > > > > > > > > improvements for those users, I'm already very happy. > > > > > > > > > > > > > > > > And yes, there are plugins that needs to be updated, but > doing > > > > > nothing > > > > > > is > > > > > > > > not an option anymore. > > > > > > > > > > > > > > > > There are more people that share their concerns, but it took > me > > > > > several > > > > > > > > years to reach this point. > > > > > > > > We now have something that seems to work, anybody who can > > improve > > > > or > > > > > > can > > > > > > > > come up with an alternative implementation can do so. > > > > > > > > > > > > > > > > thanks, > > > > > > > > Robert > > > > > > > > On 3-1-2021 12:55:41, Matthieu Brouillard wrote: > > > > > > > > Thanks Robert for the video link. > > > > > > > > > > > > > > > > I fully understand the rationales behind the separation of > > > > > > > > build/consumer pom and the video provides some insights on it > > and > > > > you > > > > > > > > explain the actual implementation to introduce this change. > > > > > > > > Still I do not fully understand why it was decided to work on > > top > > > > of > > > > > > XML > > > > > > > by > > > > > > > > filtering/enhancing it instead of working at the POM (in > > > > > > > > memory datamodel) level. > > > > > > > > With the current understanding I have, by doing this choice > of > > > > > working > > > > > > at > > > > > > > > XML level, it looks like it was decided to bypass (if not > kill) > > > > core > > > > > > > > extensions that enhance the POM itself and not the pom.xml ; > > > here I > > > > > can > > > > > > > > think of (but probably not limited to): > > > > > > > > - polyglot-maven: do not use XML but other format to describe > > the > > > > POM > > > > > > > > (yaml, json, kotlin, java, other XML formats, ...) > > > > > > > > - jgitver-maven-plugin (or forks like > > > > > maven-git-versioning-extension): > > > > > > > > dynamic computation of projects version based on git history > > > > > > > > > > > > > > > > With the introduction of core extensions, I thought it was a > > move > > > > to > > > > > > open > > > > > > > > the internals and let externals contribute to the > capabilities > > of > > > > > > maven. > > > > > > > > With the move to a XML handling chain, I see it as a > > > > > > restriction/regress > > > > > > > in > > > > > > > > favor of core closed functionalities. An example of that is > > what > > > is > > > > > > > > provided as CIFriendly stuff, IMO it could/should have been > > > > provided > > > > > > by a > > > > > > > > plugin/extension but instead it is hard written in maven core > > and > > > > is > > > > > > not > > > > > > > > opened for external contribution (plugin/extension I mean). > > > > > > > > > > > > > > > > Perhaps I am totally wrong but I think that maven core should > > > > define > > > > > > all > > > > > > > > its expectations at an API level so that extensions/plugins > > could > > > > > hook > > > > > > at > > > > > > > > this API level. The default packaging of maven could/should > > > provide > > > > > > > default > > > > > > > > implementations of those expectations (for example reading a > > > > pom.xml > > > > > > file > > > > > > > > to a POM model, dumping a POM to a pom-4.0.0.xml, > > > > > > transforming/reducing a > > > > > > > > POM to POM-consumer, dumping POM-consumer to > > > > pom-consumer-5.0.0.xml, > > > > > > ...) > > > > > > > > and let extensions/plugins/default implementations work along > > the > > > > > build > > > > > > > > process with the API & POMs to provide different features and > > > > > > > capabilities. > > > > > > > > > > > > > > > > Matthieu > > > > > > > > > > > > > > > > On Thu, Dec 31, 2020 at 7:01 PM Robert Scholte wrote: > > > > > > > > > > > > > > > > > I've made a recording[1] about it, which hopefully answers > > most > > > > > > > > questions. > > > > > > > > > > > > > > > > > > Robert > > > > > > > > > > > > > > > > > > [1] https://youtu.be/KDAmlNKZJto > > > > > > > > > > > > > > > > > > On 31-12-2020 16:18:57, Matthieu Brouillard > > > > > > > > > wrote: > > > > > > > > > > Not exactly sure what work you mean > > > > > > > > > everything related to maven-xml: > > Build/ConsumerPomXMLFilterxxx, > > > > > > > > > Build/ConsumerModelSourcexxxx and the transformer stuff. > > > > > > > > > Especially, when looking at classes like > > CiFriendlyXMLFilter, I > > > > > would > > > > > > > > have > > > > > > > > > thought that such things could have been done elsewhere, > > > working > > > > on > > > > > > the > > > > > > > > > object model (not on the XML stuff) especially for the > > BuildPom > > > > > part. > > > > > > > > > > > > > > > > > > > however specifically the consumer POM integrates with so > > many > > > > > > > external > > > > > > > > > ecosystems > > > > > > > > > We're aligned here, this has to be stable and well defined > > by a > > > > > > schema. > > > > > > > > > > > > > > > > > > Matthieu > > > > > > > > > > > > > > > > > > On Thu, Dec 31, 2020 at 3:59 PM Bernd Eckenfels > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > > > Not exactly sure what work you mean and I fully agree > that > > > > using > > > > > a > > > > > > > core > > > > > > > > > > model should still be the API for plugins and extensions > to > > > > work > > > > > > > with, > > > > > > > > > > however specifically the consumer POM integrates with so > > many > > > > > > > external > > > > > > > > > > ecosystems, I would expect it to be defined in terms of > XML > > > > > Schema > > > > > > > with > > > > > > > > > > explicite semantic (and the inherent compatibility with > > > exiting > > > > > > > POMs). > > > > > > > > > > > > > > > > > > > > Gruss > > > > > > > > > > Bernd > > > > > > > > > > -- > > > > > > > > > > http://bernd.eckenfels.net > > > > > > > > > > ________________________________ > > > > > > > > > > Von: Matthieu BROUILLARD > > > > > > > > > > Gesendet: Thursday, December 31, 2020 3:19:09 PM > > > > > > > > > > An: [email protected] > > > > > > > > > > Betreff: maven 4.0.0 new XML stuff > > > > > > > > > > > > > > > > > > > > Hello all, > > > > > > > > > > > > > > > > > > > > regarding the active work occurring for maven 4.0.0 I > > noticed > > > > the > > > > > > > > > > introduction of a lot of new stuff around SAX parsing & > > > > > filtering. > > > > > > > > > > I am wondering if that means that it was decided that the > > > input > > > > > > > format > > > > > > > > of > > > > > > > > > > maven projects will be XML forever meaning probably, > among > > > > > others, > > > > > > > the > > > > > > > > > end > > > > > > > > > > of polyglot extensions. > > > > > > > > > > Could you explain such a move (or point to > > > rationals/documents) > > > > > and > > > > > > > why > > > > > > > > > you > > > > > > > > > > did not leverage working on the in memory object model > > > allowing > > > > > > > > > > extensions/plugins to contribute/hook in the chain of > > > building > > > > > the > > > > > > > > > BuildPOM > > > > > > > > > > & ConsumePOM? In the past I really thought that this move > > to > > > > > 'Build > > > > > > > vs > > > > > > > > > > Consumer' POM would make clear separations between the > > input > > > > > format > > > > > > > of > > > > > > > > > > descriptors and the core system but I perhaps > misunderstood > > > > > things. > > > > > > > > > > > > > > > > > > > > Also, are there plans regarding the future of core > > > extensions? > > > > > > > > > > With core extensions it was possible to hook into the POM > > > model > > > > > > > loading > > > > > > > > > and > > > > > > > > > > do transformations to do dynamic changes but by working > on > > > the > > > > > XML > > > > > > > > > directly > > > > > > > > > > I see a shift (if not red stop) in this > > > contribution/delegation > > > > > > > > > mechanism. > > > > > > > > > > > > > > > > > > > > Thanks for your time & answers. > > > > > > > > > > > > > > > > > > > > Matthieu Brouillard > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
