Subject: Discussion on Introducing Parallel Compilation Support for Bigtop Components
Dear Bigtop Community Members, I hope this message finds you well. I would like to initiate a discussion regarding the introduction of parallel compilation support for Bigtop components. Background: Within the components maintained by Bigtop, a significant portion is built using Java and relies on Maven as the build tool. Rationale: Compiling components that consist of numerous modules can be a time-consuming process. For instance, some components contain hundreds of modules, and compiling them one by one consumes a substantial amount of time. Even when all dependencies are pre-downloaded for a second compilation, the process remains slow due to the sequential nature of compilation. Additionally, compiling all components together still results in sequential compilation, making it challenging to fully leverage CPU resources and reduce compilation time significantly. Consequently, repetitive compilation and testing phases impose prolonged waiting periods. Proposal: I propose the introduction of a new parameter that allows users to toggle parallel compilation for components built using Maven, thus empowering them to align compilation practices with their specific needs. Related Pull Requests (PRs): https://issues.apache.org/jira/browse/BIGTOP-4044 https://github.com/apache/bigtop/pull/1212 This new feture can be divided into two main parts: The first part entails adding parallel compilation functionality and enabling it for components that have undergone testing without encountering additional issues related to parallel compilation. These components may include Hive, HBase, Flink, ZooKeeper, among others. The second part involves enabling parallel compilation for components that face challenges with parallel compilation and necessitate additional patches to address Maven's parallel compilation capabilities. These components may include Ranger, Tez, Hadoop, Spark, and more. I welcome your insights, opinions, and suggestions on this matter. Please feel free to share your thoughts and concerns regarding the introduction of parallel compilation support for Bigtop components. Your contributions to this discussion are highly valued as we work towards enhancing the efficiency and performance of our build processes. Best regards, jiaLiang
