Bing-ok created HADOOP-18220: -------------------------------- Summary: Improve build performance of modules Key: HADOOP-18220 URL: https://issues.apache.org/jira/browse/HADOOP-18220 Project: Hadoop Common Issue Type: Improvement Components: build Reporter: Bing-ok Attachments: image-2022-05-04-01-02-55-509.png, image-2022-05-04-01-03-59-169.png
Hi, I'm now using {*}hadoop{*}. I found that the build time of the project is not very fast when I used *mvn -T 1C install -DskipTests* command to build the project during Github Actions. So I try to speed up the build performance of the project. The goal here is to clean up dependencies between maven modules in the project in order to improve the build performance. Thread usage before cleaning up module dependencies was this: !image-2022-05-04-01-03-59-169.png|width=559,height=116! Using *mvn dependency:analyze* command I managed to have an overview of the unused dependencies of every module. I focused only on the dependencies between modules. The following dependencies can be cleaned up: {noformat} org.apache.hadoop:hadoop-client-minicluster(hadoop-client-modules/hadoop-client-minicluster) -> org.apache.hadoop:hadoop-client-runtime(hadoop-client-modules/hadoop-client-runtime) org.apache.hadoop:hadoop-client-runtime(hadoop-client-modules/hadoop-client-runtime) -> org.apache.hadoop:hadoop-client-api(hadoop-client-modules/hadoop-client-api) org.apache.hadoop:hadoop-client-runtime(hadoop-client-modules/hadoop-client-runtime) -> org.apache.hadoop:hadoop-client(hadoop-client-modules/hadoop-client) org.apache.hadoop:hadoop-dist(hadoop-dist) -> org.apache.hadoop:hadoop-client-check-test-invariants(hadoop-client-modules/hadoop-client-check-test-invariants){noformat} After clean up, the build time has been reduced by *3min* when I use parallel build during Github Actions. Thread usage after cleaning up module dependencies was this: !image-2022-05-04-01-02-55-509.png|width=553,height=102! Could you help me review this issue? I can submit a PR to improve build performance of the project. Thank you very much for your attention. Best regards. -- This message was sent by Atlassian Jira (v8.20.7#820007) --------------------------------------------------------------------- To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org