voonhous commented on code in PR #19425:
URL: https://github.com/apache/hudi/pull/19425#discussion_r3803139286
##########
packaging/hudi-utilities-bundle/pom.xml:
##########
@@ -153,6 +153,43 @@
<include>com.101tec:zkclient</include>
<include>org.apache.kafka:kafka-clients</include>
+ <!-- AWS SDK v2 core, shared by the kinesis and sts clients
below; relocated further down. -->
+ <include>software.amazon.awssdk:annotations</include>
+ <include>software.amazon.awssdk:utils</include>
+ <include>software.amazon.awssdk:sdk-core</include>
+ <include>software.amazon.awssdk:aws-core</include>
+ <include>software.amazon.awssdk:auth</include>
+ <include>software.amazon.awssdk:http-client-spi</include>
+ <include>software.amazon.awssdk:regions</include>
+ <include>software.amazon.awssdk:metrics-spi</include>
+ <include>software.amazon.awssdk:json-utils</include>
+ <include>software.amazon.awssdk:endpoints-spi</include>
+ <include>software.amazon.awssdk:retries</include>
+ <include>software.amazon.awssdk:retries-spi</include>
+ <include>software.amazon.awssdk:checksums</include>
+ <include>software.amazon.awssdk:checksums-spi</include>
+ <include>software.amazon.awssdk:identity-spi</include>
+ <include>software.amazon.awssdk:http-auth</include>
+ <include>software.amazon.awssdk:http-auth-spi</include>
+ <include>software.amazon.awssdk:http-auth-aws</include>
+
<include>software.amazon.awssdk:http-auth-aws-eventstream</include>
+ <include>software.amazon.awssdk:protocol-core</include>
+ <include>software.amazon.awssdk:apache-client</include>
+ <include>software.amazon.awssdk:netty-nio-client</include>
+ <include>org.reactivestreams:reactive-streams</include>
+ <!-- Kinesis client for JsonKinesisSource -->
+ <include>software.amazon.awssdk:kinesis</include>
+ <include>software.amazon.awssdk:aws-cbor-protocol</include>
+ <include>software.amazon.awssdk:aws-json-protocol</include>
+
<include>software.amazon.awssdk:third-party-jackson-dataformat-cbor</include>
+
<include>software.amazon.awssdk:third-party-jackson-core</include>
+ <!-- STS: assume-role credentials for cross-account Kinesis
reads (KinesisOffsetGen). -->
+ <include>software.amazon.awssdk:sts</include>
+ <include>software.amazon.awssdk:aws-query-protocol</include>
+ <include>software.amazon.awssdk:profiles</include>
+ <!-- KPL de-aggregation: extracts user records from Kinesis
Producer Library aggregated records -->
+ <include>com.amazonaws:amazon-kinesis-deaggregator</include>
Review Comment:
Adding `amazon-kinesis-client` at the resolved 1.8.8 would make this worse,
not better: both it and `amazon-kinesis-deaggregator:1.0.3` are under the
Amazon Software License (their POMs say so, and the built jar's
`META-INF/DEPENDENCIES` prints `License: Amazon Software License` for the
deaggregator), which ASF legal lists as Category X and may not ship in a
convenience binary. KCL is Apache-2.0 only from 1.12.0. This include is also
dead weight: it adds two `com/amazonaws/kinesis/deagg/*` classes that nothing
in Hudi calls (`KinesisDeaggregator` uses KCL's `UserRecord`, a different
artifact).
Two asks:
1. Drop `<include>com.amazonaws:amazon-kinesis-deaggregator</include>` from
this PR.
2. State in the PR body that the default path still does not run from the
bundle alone: `hoodie.streamer.source.kinesis.enable.deaggregation` defaults to
true and `KinesisDeaggregator.deaggregate` converts every record to SDK v1
first, so bundle-only fails with `NoClassDefFoundError:
com/amazonaws/services/kinesis/model/Record` even for plain records (verified
from the built jar). The minimal closure for that path is
`amazon-kinesis-client` (>= 1.12.0) + `aws-java-sdk-kinesis` +
`aws-java-sdk-core` + `commons-lang`, relocated under `com.amazonaws` so EMR's
`aws-java-sdk-bundle` cannot shadow them; cleaner is to move
`KinesisDeaggregator` off KCL/SDK v1 entirely (the KPL frame is 4 magic bytes +
one protobuf message + MD5, decodable on the protobuf-java this bundle already
ships). Either belongs in a follow-up PR; this one should not claim standalone
Kinesis works until it lands. Separately, `hudi-utilities` has had a compile
dependency on the ASL deaggregator since #18224, which d
eserves its own issue regardless of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
