Does hadoop-dist package help? It does the packaging stuff for hadoop, IIRC
it defines all the projects so that the dist kicks post everything is built
[1], it has scripts mentioned in the pom which does the packaging work.
for the protobuf, I think in the yarn modules we don't have the scope
I tried running it at the root project (hadoop), and got a meaningful
dependency tree. It does print an exhaustive and transitive tree of
dependencies.
As for log4j with your patch, I see two ways log4j is introduced:
- log4j -> hadoop-common@2.8.5 ->
it does, but i'm not sure if there is a single module where you can ask for
it and get the full list.
For that verification project I've got I may declare more poms as
dependencies so can do the aggregate scan there. this would also let me run
maven dependency -verbose, save the output to a file
Does the maven dependency plugin help? I might try mvn dependency:tree and
see if it takes you somewhere.
Sangjin
On Mon, Feb 12, 2024 at 9:50 AM Steve Loughran
wrote:
> how can we work out the entire DAG of dependencies in a hadoop distro?
>
> I'm asking as there are things in 3.4.0 that we
how can we work out the entire DAG of dependencies in a hadoop distro?
I'm asking as there are things in 3.4.0 that we shouldn't need (protobuf
2.5), and when I add the pR to move off log4j 1.17 to reload4j, I still
find one in the yarn timeline lib dir
https://github.com/apache/hadoop/pull/6547
