voonhous commented on PR #18177:
URL: https://github.com/apache/hudi/pull/18177#issuecomment-4955980802
CI failure on the previous push was mine. Root cause + fix:
### Why it broke
`hive-shims-common` (Hive 2.3.10) pulls in `log4j-slf4j-impl:2.17.2`. That
artifact ships the **same classes** as `log4j-slf4j2-impl`:
- `org.apache.logging.slf4j.Log4jLoggerFactory`
- `org.apache.logging.slf4j.Log4jLogger`
- `org.apache.logging.slf4j.Log4jMarkerFactory`
They are **different artifacts, not different versions**, so Maven does no
mediation: both jars sit on the classpath and the shared classes resolve from
whichever comes first.
This was masked before. The root `dependencyManagement` pinned
`log4j-slf4j-impl` to `${log4j2.version}`, so Hive's copy was force-upgraded
and its duplicate `Log4jLoggerFactory` happened to carry the same
`(Log4jMarkerFactory)` constructor -- the shadowing was harmless by luck.
Swapping that entry to `log4j-slf4j2-impl` dropped the pin, Hive's `2.17.2`
copy arrived with only a no-arg constructor, and:
```
NoSuchMethodError: Log4jLoggerFactory.<init>(Log4jMarkerFactory)
at org.apache.logging.slf4j.SLF4JServiceProvider.initialize
-> ExceptionInInitializerError in HoodieConfig.<clinit>
-> NoClassDefFoundError: Could not initialize class
org.apache.hudi.common.config.HoodieConfig
```
Every class with a static logger died in its initializer, which is why the
failures looked unrelated to logging.
### Fix (latest commit)
Exclude `log4j-slf4j-impl` from the four Hive deps that leak it:
`hive-shims`, `hive-jdbc`, `hive-serde`, `hive-metastore`.
`hive-exec` / `hive-common` / `hive-service` already exclude
`org.apache.logging.log4j:*`, which is why only these four were the source.
This mirrors the existing exclusions for `slf4j-log4j12`, Hive's other slf4j
1.x binding.
### Verified
- `dependency:tree` shows zero `log4j-slf4j-impl` across `spark3.5`,
`spark3.4`, `flink1.20` and the default profile.
- The tests that failed (`TestHoodieSparkQuickstart`,
`TestRandomJsonSource`) pass locally on `spark3.5`.
Side note: `hive-shims-common` 4.0.x no longer declares the binding, so this
exclusion becomes moot whenever we move off Hive 2.3.x.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
