rahil-c commented on code in PR #1862:
URL: https://github.com/apache/polaris/pull/1862#discussion_r2181920269
##########
plugins/spark/v3.5/integration/build.gradle.kts:
##########
@@ -60,12 +60,51 @@ dependencies {
exclude("org.apache.logging.log4j", "log4j-core")
exclude("org.slf4j", "jul-to-slf4j")
}
+
+ // Add spark-hive for Hudi integration - provides HiveExternalCatalog that
Hudi needs
+
testImplementation("org.apache.spark:spark-hive_${scalaVersion}:${spark35Version}")
{
+ // exclude log4j dependencies to match spark-sql exclusions
+ exclude("org.apache.logging.log4j", "log4j-slf4j2-impl")
+ exclude("org.apache.logging.log4j", "log4j-1.2-api")
+ exclude("org.apache.logging.log4j", "log4j-core")
+ exclude("org.slf4j", "jul-to-slf4j")
+ // exclude old slf4j 1.x to log4j 2.x bridge that conflicts with slf4j 2.x
bridge
+ exclude("org.apache.logging.log4j", "log4j-slf4j-impl")
+ }
// enforce the usage of log4j 2.24.3. This is for the log4j-api compatibility
// of spark-sql dependency
testRuntimeOnly("org.apache.logging.log4j:log4j-core:2.24.3")
testRuntimeOnly("org.apache.logging.log4j:log4j-slf4j2-impl:2.24.3")
testImplementation("io.delta:delta-spark_${scalaVersion}:3.3.1")
+
testImplementation("org.apache.hudi:hudi-spark3.5-bundle_${scalaVersion}:0.15.0")
{
+ // exclude log4j dependencies to match spark-sql exclusions and prevent
version conflicts
Review Comment:
@RussellSpitzer For delta it seems we have declared the test dep with
testImplementation
https://github.com/apache/polaris/blob/main/plugins/spark/v3.5/integration/build.gradle.kts#L68
So i had done the same for hudi.
I am using one area in the `SparkCatalogTest` using an assertion of a
specific hudi class to see if the table returned was in fact a hudi table.
```
} else if (PolarisCatalogUtils.useHudi(format)) {
assertThat(loadedTable).isInstanceOf(HoodieInternalV2Table.class);
}
```
So I think for now had left this as testImplementation. If this is a concern
though let me know and I can try to see an alternative here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
