morningman opened a new pull request, #65867:
URL: https://github.com/apache/doris/pull/65867
### What problem does this PR solve?
Paimon reads that go through the Paimon **JNI** reader over an HMS Paimon
catalog — branch read, time travel, incremental read, system tables, and
some primary-key/DML/changelog paths — fail on BE with:
```
[JNI_ERROR]RuntimeException: java.lang.NoClassDefFoundError:
org/apache/hadoop/hive/conf/HiveConf
CAUSED BY: ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
doris::JniReader::open(...)
doris::PaimonJniReader::_do_init_reader(...)
```
**Root cause — a regression from #65733.**
`paimon-hive-connector-3.1` marks `hive-exec` as `provided` and only bundles
*relocated* hive classes under `org.apache.paimon.shade.*`. The real
`org.apache.hadoop.hive.conf.HiveConf` — needed by
`org.apache.paimon.hive.SerializableHiveConf` when a HMS-backed Paimon
`Table`
is deserialized on BE, and by `HiveCatalog.createHiveConf` — was therefore
never packaged into `paimon-scanner-jar-with-dependencies.jar`.
It used to be supplied *accidentally* via the shared BE JVM classpath:
`bin/start_be.sh` preloads both `preload-extensions` **and `java-udf`** onto
`-Djava.class.path`, and that system classloader is the parent of every
per-scanner `JniScannerClassLoader`. `java-udf` used to depend on the ~122MB
`hive-catalog-shade` (which contains `HiveConf`), so paimon-scanner inherited
`HiveConf` through parent delegation.
#65733 slimmed `java-udf` by replacing `hive-catalog-shade` with the minimal
`hive-udf-shade` (10 UDF-contract classes), removing `HiveConf` from the
shared classpath and breaking every Paimon Hive JNI read path.
**Fix.** Make `paimon-scanner` self-contained by bundling the Hive classes it
actually needs, instead of relying on another module's classpath:
- `hive-common` — `HiveConf` plus its whole construction closure
(`common/FileUtils`, `common/type/*`, `hive/common/util/*`).
- `hive-shims-common` — `shims/Utils`, referenced only by the lazy
`HiveConf#getUser`.
Both are pinned to `${hive.version}` (3.1.x — matching the Hive the connector
was built against and the previous `hive-catalog-shade` 3.1.1), and their
transitive trees are pruned with a wildcard exclusion. Hadoop is already on
BE's shared classpath, so nothing else is added.
Verified after rebuild:
- All required classes now present in the fat jar (`HiveConf`, `FileUtils`,
`type/Timestamp`, `HiveStringUtils`, `shims/Utils`).
- Net jar growth ~540KB (hive-common + hive-shims-common only); jetty/orc/ant
class counts unchanged (0 bloat from the change).
- Runtime smoke test on a simulated BE classpath passes: `new HiveConf()`, a
`SerializableHiveConf` Java-serialization round-trip (the FE→BE path), and
`HiveConf.getUser()` all succeed.
- Other scanners are unaffected: `hadoop-hudi-scanner` already bundles its
own
`HiveConf`; iceberg/trino/maxcompute scanners do not reference it.
Regression test to re-run:
`io.trino.tests.product.paimon.TestPaimonFlinkCompatibility.testDorisPaimonBranchReadMatchesFlink`
### Release note
Fix `NoClassDefFoundError: org.apache.hadoop.hive.conf.HiveConf` when reading
Paimon tables (branch / time travel / incremental / system table) through the
Paimon JNI reader over an HMS catalog.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
