rahil-c commented on code in PR #1862:
URL: https://github.com/apache/polaris/pull/1862#discussion_r2180424419
##########
plugins/spark/v3.5/integration/build.gradle.kts:
##########
@@ -60,12 +60,51 @@ dependencies {
exclude("org.apache.logging.log4j", "log4j-core")
exclude("org.slf4j", "jul-to-slf4j")
}
+
+ // Add spark-hive for Hudi integration - provides HiveExternalCatalog that
Hudi needs
+
testImplementation("org.apache.spark:spark-hive_${scalaVersion}:${spark35Version}")
{
+ // exclude log4j dependencies to match spark-sql exclusions
Review Comment:
We will need this dependency as a test dependency, as when hudi insert is
invoked it looks for this `HiveExternalCatalog`, when removing this dependency
as suspected this fails for class not found exception.
```
org/apache/spark/sql/hive/HiveExternalCatalog$
java.lang.NoClassDefFoundError:
org/apache/spark/sql/hive/HiveExternalCatalog$
at
org.apache.spark.sql.hudi.ProvidesHoodieConfig.buildHiveSyncConfig(ProvidesHoodieConfig.scala:496)
at
org.apache.spark.sql.hudi.ProvidesHoodieConfig.buildHiveSyncConfig$(ProvidesHoodieConfig.scala:467)
at
org.apache.spark.sql.hudi.analysis.HoodieSpark35DataSourceV2ToV1Fallback.buildHiveSyncConfig(HoodieSpark35Analysis.scala:39)
at
org.apache.spark.sql.hudi.ProvidesHoodieConfig.buildHoodieConfig(ProvidesHoodieConfig.scala:63)
at
org.apache.spark.sql.hudi.ProvidesHoodieConfig.buildHoodieConfig$(ProvidesHoodieConfig.scala:55)
at
org.apache.spark.sql.hudi.analysis.HoodieSpark35DataSourceV2ToV1Fallback.buildHoodieConfig(HoodieSpark35Analysis.scala:39)
at
org.apache.spark.sql.hudi.analysis.HoodieSpark35DataSourceV2ToV1Fallback.org$apache$spark$sql$hudi$analysis$HoodieSpark35DataSourceV2ToV1Fallback$$convertToV1(HoodieSpark35Analysis.scala:62)
at
org.apache.spark.sql.hudi.analysis.HoodieSpark35DataSourceV2ToV1Fallback.apply(HoodieSpark35Analysis.scala:50)
at
org.apache.spark.sql.hudi.analysis.HoodieSpark35DataSourceV2ToV1Fallback.apply(HoodieSpark35Analysis.scala:39)
at
org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$2(RuleExecutor.scala:222)
```
Have also explained further here in related comment:
https://github.com/apache/polaris/pull/1862#discussion_r2180478143
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
