voonhous opened a new issue, #19296:
URL: https://github.com/apache/hudi/issues/19296
### Describe the problem
Any test in `hudi-spark` that builds a session with `.enableHiveSupport()`
(embedded Derby metastore) aborts on every JDK with:
```
java.lang.NoClassDefFoundError:
org/apache/thrift/transport/layered/TFramedTransport
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getClass(MetaStoreUtils.java:1708)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:131)
at
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3607)
...
```
Verified on current master with JDK 17 (`mvn test -Dspark3.5 -Pjava17
-DwildcardSuites=...` on `hudi-spark-datasource/hudi-spark`). The failure is
JDK-independent: it is a classpath conflict, not a Java 9+ metastore
incompatibility.
### Root cause
The root pom pins `hive.version=2.3.10`, whose metastore client references
the thrift 0.13+ `layered` package, but the `hudi-spark` test classpath
resolves `org.apache.thrift:libthrift:0.12.0` via Spark 3.5's dependencies (the
pom's `thrift.version=0.13.0` property does not win on this path; confirmed
with `mvn dependency:tree`).
### Impact
- Blocks any HMS-backed-catalog test coverage in the Spark modules. This is
the gap behind HUDI-9678: no SQL-write test runs against a hive-backed session
catalog, and PR #13748 could not add one because of this.
- `TestSparkCatalogMetaStoreClient` (added in #18672) aborts with the same
error when executed (verified locally). The breakage is invisible today because
`org.apache.spark.sql.hive` is missing from the scala-suite wildcard filters
(`SCALA_TEST_DML_FILTER` / `SCALA_TEST_OTHERS_FILTER` in
`.github/workflows/bot.yml`, and the job6 suite list in
`azure-pipelines-20230430.yml`), so that suite never runs in CI.
### Fix direction
1. Manage libthrift to a hive-2.3.10-compatible version (>= 0.13) on the
test classpath, or align the hive pin with Spark 3.5's hive 2.3.9.
2. Add `org.apache.spark.sql.hive` to the CI scala-suite filters so
scalatest suites there actually execute (JUnit tests in that package are
unaffected; they are selected via `-Dtest` filters).
3. With 1 and 2 in place, hive-enabled SQL test coverage (the intent of
#13748) can be added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
