JiaLiangC commented on PR #315:
URL: https://github.com/apache/tez/pull/315#issuecomment-1857129442
@abstractdog @ayushtkn
I apologize for not providing a detailed explanation earlier due to being
too busy:
The command mvn clean package -Dtar -Dhadoop.version=${HADOOP_VERSION}
-Phadoop28 -DskipTests compiles without errors. However, parallel compilation
fails, and the failure error is as posted by "ayushtkn" in last comment.
The reason for the failure is that parallel compilation attempts to compile
all modules in a single pass, and the compilation order is determined as a tree
structure based on dependencies.
In the normal sequential compilation, tez-dist is the last module to
compile, so before compiling this module, all other modules and their
dependencies are already compiled. However, in parallel compilation, tez-dist
is compiled concurrently with other modules. Therefore, it is necessary to
ensure that tez-dist includes dependencies from all other modules. This ensures
that tez-dist is compiled only after all other modules have been compiled.
To resolve the issue using the command mvn -T2C clean package -Dtar
-Dhadoop.version=${HADOOP_VERSION} -Phadoop28 -DskipTests, the error reported
is due to tez-dist missing the tez-job-analyzer dependency. Adding this
dependency will prevent errors during parallel compilation.
I also recommend using mvn dependency:tree and maven-to-plantuml to
troubleshoot dependency issues causing the inability to compile in parallel.
To investigate the dependency relationships between project modules, you can
use the following tools:
mvn dependency:tree: This Maven command generates a tree structure of
project dependencies. It can help you visualize the dependencies between
modules.
maven-to-plantuml: This is a project for analyzing and visualizing Maven
project dependencies. You can use it to generate a PlantUML diagram from the
dependency tree generated by mvn dependency:tree. Here are the steps:
Download maven-to-plantuml: wget
https://github.com/phxql/maven-to-plantuml/releases/download/v1.0/maven-to-plantuml-1.0.jar
Generate the dependency tree: mvn dependency:tree > dep.txt
Generate a PlantUML diagram from the dependency tree: java -jar
maven-to-plantuml.jar --input dep.txt --output dep.puml
These tools will help you analyze the dependencies between modules in your
project and identify any issues that may be causing problems during parallel
compilation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
