Bing-ok created HADOOP-18220:
--------------------------------
Summary: Improve build performance of modules
Key: HADOOP-18220
URL: https://issues.apache.org/jira/browse/HADOOP-18220
Project: Hadoop Common
Issue Type: Improvement
Components: build
Reporter: Bing-ok
Attachments: image-2022-05-04-01-02-55-509.png,
image-2022-05-04-01-03-59-169.png
Hi, I'm now using {*}hadoop{*}. I found that the build time of the project is
not very fast when I used *mvn -T 1C install -DskipTests* command to build the
project during Github Actions. So I try to speed up the build performance of
the project. The goal here is to clean up dependencies between maven modules in
the project in order to improve the build performance.
Thread usage before cleaning up module dependencies was this:
!image-2022-05-04-01-03-59-169.png|width=559,height=116!
Using *mvn dependency:analyze* command I managed to have an overview of the
unused dependencies of every module. I focused only on the dependencies between
modules. The following dependencies can be cleaned up:
{noformat}
org.apache.hadoop:hadoop-client-minicluster(hadoop-client-modules/hadoop-client-minicluster)
->
org.apache.hadoop:hadoop-client-runtime(hadoop-client-modules/hadoop-client-runtime)
org.apache.hadoop:hadoop-client-runtime(hadoop-client-modules/hadoop-client-runtime)
-> org.apache.hadoop:hadoop-client-api(hadoop-client-modules/hadoop-client-api)
org.apache.hadoop:hadoop-client-runtime(hadoop-client-modules/hadoop-client-runtime)
-> org.apache.hadoop:hadoop-client(hadoop-client-modules/hadoop-client)
org.apache.hadoop:hadoop-dist(hadoop-dist) ->
org.apache.hadoop:hadoop-client-check-test-invariants(hadoop-client-modules/hadoop-client-check-test-invariants){noformat}
After clean up, the build time has been reduced by *3min* when I use parallel
build during Github Actions.
Thread usage after cleaning up module dependencies was this:
!image-2022-05-04-01-02-55-509.png|width=553,height=102!
Could you help me review this issue? I can submit a PR to improve build
performance of the project.
Thank you very much for your attention.
Best regards.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
