marton-bod opened a new pull request #1478: URL: https://github.com/apache/iceberg/pull/1478
This is a second iteration after an earlier, draft PR: https://github.com/apache/iceberg/pull/1455 The goal of this PR is to enable building the Iceberg repository using Hive3 and Hadoop3. The build can be launched with a property flag: `gradle build -Dhive3` This will pull in Hive3 and Hadoop3 as dependencies and build all modules, except for Spark and Flink: - iceberg-spark - iceberg-spark2 - iceberg-spark-runtime - iceberg-spark3 - iceberg-spark3-runtime - iceberg-flink - iceberg-flink-runtime The advantage of this approach is that we can run the build for the entire repository with the hive3 flag as above, but it introduces some separation in the gradle build file as to which projects should be included (based on the hive3Enabled flag). The alternative approach would be to only allow hive3 builds for certain submodules, e.g. `gradle :iceberg-mr:build :iceberg-hive-metastore:build :iceberg-hive-runtime:build <...others> -Dhive3` but this would mean that the above command (`gradle build -Dhive3`) would fail, and anytime we enable a new module to work with Hive3, the CI commands would have to be updated to test for that as well. I'm interested in your opinion as to which approach you prefer. If the hive3 build flag is not used, everything should stay the same as before, with no changes to the build output (except that there is a new module called iceberg-mr-hive3). (Note 1: Hive2- and Hive3-specific parts have been moved from iceberg-mr into iceberg-mr-hive2/iceberg-mr-hive3) (Note 2: currently the hive3 build works only for Java 8, there are some exceptions on Java11 regarding HiveRunner and metastore initialization. Therefore hive3 builds are disabled on Java11 for now. This should be addressed in a separate patch.) (Note 3: for some reason the build.gradle diff shows a lot of output, but most of the changes in that file are only indentation, since the Spark and Flink subprojects have been wrapped into if blocks to skip building them when Hive3 dependencies are enabled. IntelliJ seems to handle this more intelligently.) Once this has been merged, we should think about integrating the Java8 hive3 build into our CI pipeline. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
