adp2201 commented on issue #13691:
URL: https://github.com/apache/iceberg/issues/13691#issuecomment-4050240860
I reproduced this locally and validated a candidate fix.
## Repro
1. Build and unzip the runtime connector distribution from current `main`:
- `./gradlew :iceberg-kafka-connect:iceberg-kafka-connect-runtime:distZip
-x test -x integrationTest`
2. Inspect runtime libs in the produced zip.
3. Trigger Hadoop UGI initialization using only bundled plugin jars (same
classloader constraints Kafka Connect plugin loading applies).
Observed failure matches reports in this issue:
- `NoClassDefFoundError: org/apache/commons/configuration2/Configuration`
- If only `commons-configuration2` is added, next failure is:
`NoClassDefFoundError: org/apache/hadoop/util/PlatformName`
## Findings
In `kafka-connect/build.gradle`, `iceberg-kafka-connect-runtime` currently
excludes:
- `org.apache.hadoop:hadoop-auth`
- `org.apache.commons:commons-configuration2`
Those exclusions make Hadoop catalog mode fail at runtime under plugin
classloader isolation.
## Candidate fix (tested locally)
I removed these two excludes, rebuilt the runtime zip, and re-ran the same
UGI classloading repro using only bundled jars.
Result:
- New zip includes both jars (`hadoop-auth-3.4.3`,
`commons-configuration2-2.10.1`)
- UGI loads successfully (no `NoClassDefFoundError`)
## Tradeoffs
Including these jars likely means:
- Slightly larger runtime bundle
- More transitive surface / CVE management
- Potential for dependency/classloader conflicts in some Kafka Connect
deployments
But for Hadoop catalog mode, this appears necessary for correctness.
## Proposed direction
For now, include these dependencies in `iceberg-kafka-connect-runtime`
(remove those two excludes), then validate in CI/integration tests for
regressions.
Would appreciate feedback from maintainers/community:
- Is this the preferred fix direction?
- Or should Hadoop-catalog-specific dependencies be packaged differently
(e.g., dedicated flavor/profile) to reduce conflict risk while preserving
runtime correctness?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
