Does hadoop-dist package help? It does the packaging stuff for hadoop, IIRC it defines all the projects so that the dist kicks post everything is built [1], it has scripts mentioned in the pom which does the packaging work.
for the protobuf, I think in the yarn modules we don't have the scope defined, so by default it is taking as compile, maybe putting the scope in the parent pom [2] should help, I tried locally & it did, else you need to define the scope thing in every POM which uses protobuf... -Ayush [1] https://github.com/apache/hadoop/blob/trunk/hadoop-dist/pom.xml#L32 [2] https://github.com/apache/hadoop/commit/c373d3fa39013e8a8f6a6122c3ca230b4aa10abe On Tue, 13 Feb 2024 at 23:37, Steve Loughran <ste...@cloudera.com.invalid> wrote: > it does, but i'm not sure if there is a single module where you can ask for > it and get the full list. > > For that verification project I've got I may declare more poms as > dependencies so can do the aggregate scan there. this would also let me run > maven dependency -verbose, save the output to a file and see what is there. > would let us define lists of libraries we don't want in distributions > > On Mon, 12 Feb 2024 at 18:03, Sangjin Lee <sjl...@gmail.com> wrote: > > > Does the maven dependency plugin help? I might try mvn dependency:tree > and > > see if it takes you somewhere. > > > > Sangjin > > > > > > On Mon, Feb 12, 2024 at 9:50 AM Steve Loughran > <ste...@cloudera.com.invalid > > > > > wrote: > > > > > how can we work out the entire DAG of dependencies in a hadoop distro? > > > > > > I'm asking as there are things in 3.4.0 that we shouldn't need > (protobuf > > > 2.5), and when I add the pR to move off log4j 1.17 to reload4j, I still > > > find one in the yarn timeline lib dir > > > https://github.com/apache/hadoop/pull/6547 > > > > > > > > > see HADOOP-19074 for a list of what is in 3.4.0 RC0, which predates the > > new > > > shaded jar. > > > > > >
