rdsr commented on a change in pull request #1478:
URL: https://github.com/apache/iceberg/pull/1478#discussion_r493661582
##########
File path: settings.gradle
##########
@@ -17,44 +17,64 @@
* under the License.
*/
+gradle.ext.hive3Enabled = false
+if (System.getProperty("hive3") != null) {
+ if (JavaVersion.current() == JavaVersion.VERSION_1_8) {
+ println "*** 'hive3' flag detected - building with Hive3 / Hadoop3 ***"
+ gradle.ext.hive3Enabled = true
+ } else {
+ println "*** 'hive3' flag detected, but with JDK version other than 1.8 -
skipping Hive3 / Hadoop3 build. ***"
+ }
+}
+
rootProject.name = 'iceberg'
include 'api'
include 'common'
include 'core'
include 'data'
-include 'flink'
-include 'flink-runtime'
include 'mr'
include 'hive-runtime'
include 'orc'
include 'arrow'
include 'parquet'
include 'bundled-guava'
-include 'spark'
-include 'spark3'
-include 'spark3-runtime'
include 'pig'
include 'hive-metastore'
+if (gradle.ext.hive3Enabled) {
+ include 'mr-hive3'
+} else {
+ include 'mr-hive2'
Review comment:
Why have we excluded these modules from the build? Seems like if the
hive3 flag is used, none of these modules/artifacts will be published. Also, in
the future, would we want to publish these modules compiled with hadoop3
artifacts if yes then it makes sense to build all modules, I think.
What are your thoughts on the flag name, should it be hadoop3 or hive3. I
feel like this flag is more general and affects Parquet, ORC, MR as you
mentioned before.
##########
File path: build.gradle
##########
@@ -458,14 +471,29 @@ project(':iceberg-mr') {
testCompile("org.apache.calcite:calcite-core")
testCompile("com.esotericsoftware:kryo-shaded:4.0.2")
testCompile("com.fasterxml.jackson.core:jackson-annotations:2.6.5")
- testCompile("com.klarna:hiverunner:5.2.1") {
+ testCompile("com.klarna:hiverunner") {
exclude group: 'javax.jms', module: 'jms'
exclude group: 'org.apache.hive', module: 'hive-exec'
exclude group: 'org.codehaus.jettison', module: 'jettison'
exclude group: 'org.apache.calcite.avatica'
}
}
}
+if (!gradle.ext.hive3Enabled) {
+ project(':iceberg-mr-hive2') {
Review comment:
nit: we can possibly make this a var which can be set when we check
`hive3Enabled` to either `:iceberg-mr-hive2` or `:iceberg-mr-hive3` . This
would the remove if/else branches
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
