danny0405 commented on code in PR #19682:
URL: https://github.com/apache/hudi/pull/19682#discussion_r3819412365
##########
pom.xml:
##########
@@ -128,6 +128,9 @@
<hadoop.version>2.10.2</hadoop.version>
<hive.groupid>org.apache.hive</hive.groupid>
<hive.version>2.3.10</hive.version>
+ <!-- The libthrift the Hive client jars above are compiled against (Hive
2.3.10 bumped
+ it for CVE-2020-13949); keep in lockstep with hive.version. See
#19680. -->
+ <hive.libthrift.version>0.14.1</hive.libthrift.version>
Review Comment:
**[P2] Preserve the Spark 4.x Thrift version**
This root-level value is consumed by the direct `hudi-spark` dependency
under every Spark profile. With `-Dspark4.2 -Dscala-2.13`,
`spark-hive_2.13:4.2.0` selects libthrift 0.16.0, but the new direct dependency
makes the final `hudi-spark_2.13` graph resolve 0.14.1 instead. That downgrades
the Thrift runtime Spark 4.x was built and tested against merely to fix the
Spark 3.x 0.12.0 mediation. Please keep 0.14.1 as the Spark 3 minimum while
overriding this property to 0.16.0 for the Spark 4 profiles, or otherwise avoid
lowering the Spark-selected version.
##########
hudi-spark-datasource/hudi-spark/pom.xml:
##########
@@ -332,6 +332,31 @@
</dependency>
<!-- Hive -->
+ <!-- Direct pin: the Hive 2.3.10 client jars are compiled against
libthrift 0.14.1
+ (HiveAuthUtils.getSocketTransport uses TConfiguration), but
dependency mediation
+ otherwise picks 0.12.0 from spark-hive, which lacks that class and
turns every
+ hive-jdbc connect into a NoClassDefFoundError. See #19680. -->
+ <dependency>
+ <groupId>org.apache.thrift</groupId>
+ <artifactId>libthrift</artifactId>
+ <version>${hive.libthrift.version}</version>
Review Comment:
**[P2] Include the matching runtime in the shaded-Hive bundle**
This fixes the raw `hudi-spark` dependency graph, but
`packaging/hudi-spark-bundle` has an explicit shade `artifactSet` allowlist
that does not include `org.apache.thrift:libthrift`. Under
`-Pspark-bundle-shade-hive`, the bundle embeds and relocates the Hive 2.3.10
client classes while still omitting this 0.14.1 runtime; on Spark 3.x those
classes therefore continue to bind to Spark-provided libthrift 0.12.0 and hit
the same missing `TConfiguration`. Please add and relocate the matching Thrift
artifact for that profile, or otherwise ensure the shaded Hive classes cannot
resolve against the incompatible Spark runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
