rahil-c opened a new pull request, #19425:
URL: https://github.com/apache/hudi/pull/19425
### Change Logs
**Summary.** `hudi-utilities-bundle` doesn't shade the AWS SDK that
`JsonKinesisSource` needs, so the published bundle jar can't actually run
`JsonKinesisSource` on its own.
`hudi-utilities` depends on `software.amazon.awssdk:kinesis` and
`com.amazonaws:amazon-kinesis-deaggregator` for
`JsonKinesisSource`/`KinesisOffsetGen`/`KinesisDeaggregator`, but
`packaging/hudi-utilities-bundle/pom.xml`'s shade `<includes>` never picked
either up. Neither artifact ends up in the shaded jar, so a deployment running
only `hudi-utilities-bundle` hits `NoClassDefFoundError` on any
`software.amazon.awssdk.services.kinesis.*` class the moment
`JsonKinesisSource` is used.
`hudi-aws-bundle` already solves this same problem for its own AWS
dependencies:
```xml
<include>software.amazon.awssdk:*</include>
...
<relocation>
<pattern>software.amazon.awssdk.</pattern>
<shadedPattern>org.apache.hudi.software.amazon.awssdk.</shadedPattern>
</relocation>
```
This applies the identical include + relocation to `hudi-utilities-bundle`,
plus the `com.amazonaws:amazon-kinesis-deaggregator` include for KPL
de-aggregation. Relocating (rather than just including) also avoids a classpath
collision: without it, the bundle would publish the AWS SDK at its original
coordinates, which can collide with whatever SDK version a consuming
application has pinned on its own classpath.
### Impact
`hudi-utilities-bundle` now carries the AWS SDK classes `JsonKinesisSource`
needs, relocated under `org.apache.hudi.software.amazon.awssdk.**`. No behavior
change for anything not using `JsonKinesisSource`.
### Risk level
**Low.** Additive to the bundle's shaded contents; no existing classes move
or change behavior.
The one thing worth double-checking on the built artifact is AWS SDK v2's
`ServiceLoader`-based HTTP client discovery, since
`KinesisOffsetGen.createKinesisClient` builds a `KinesisClient` without an
explicit `.httpClient(...)`. Verified locally that the relocated jar carries
the rewritten service files, e.g.:
```
$ unzip -p hudi-utilities-bundle_2.12-*.jar
META-INF/services/org.apache.hudi.software.amazon.awssdk.http.SdkHttpService
org.apache.hudi.software.amazon.awssdk.http.apache.ApacheSdkHttpService
```
### Documentation Update
None. No configs, public API, or user-facing behavior change.
### Contributor's checklist
- [x] Read through contributor's guide
- [x] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable — packaging-only change, no
unit-test surface.
- [x] `mvn -pl packaging/hudi-utilities-bundle -am package -DskipTests
-Dscala-2.12 -Dspark3.5 -Dflink1.20` succeeds locally; verified the resulting
jar has zero classes left at the original `software/amazon/awssdk/**` paths and
carries the relocated `META-INF/services/*SdkHttpService` entries.
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
