linliu-code commented on code in PR #19707:
URL: https://github.com/apache/hudi/pull/19707#discussion_r3832489833
##########
pom.xml:
##########
@@ -812,18 +810,6 @@
<dependencyManagement>
<dependencies>
Review Comment:
Since this one is release-blocking, would it be worth adding
`com.amazonaws:amazon-kinesis-deaggregator` (and
`com.amazonaws:amazon-kinesis-client`, which is where the `UserRecord` class
actually came from) to the existing `bannedDependencies` enforcer block further
up this file? That turns the cleanup into an enforced invariant, so the
Category X artifact can't come back transitively without failing the build.
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/KinesisDeaggregator.java:
##########
@@ -44,36 +59,137 @@ public static List<Record> deaggregate(List<Record>
records) {
if (records == null || records.isEmpty()) {
return new ArrayList<>();
}
- List<com.amazonaws.services.kinesis.model.Record> v1Records = new
ArrayList<>(records.size());
- for (Record r : records) {
- v1Records.add(toV1Record(r));
- }
- List<UserRecord> userRecords = UserRecord.deaggregate(v1Records);
- List<Record> result = new ArrayList<>(userRecords.size());
- for (UserRecord ur : userRecords) {
- result.add(toV2Record(ur));
+ List<Record> result = new ArrayList<>(records.size());
+ for (Record record : records) {
+ byte[] data = record.data() == null ? null : record.data().asByteArray();
Review Comment:
Since we only read from this array, and every sub-payload is copied out
again by `readByteArray()`, `asByteArrayUnsafe()` would avoid cloning each
record's payload just to compare the 4 magic bytes
(`BytesWrapper.asByteArray()` is a `clone()`). With deaggregation on by default
and a `GetRecords` page up to 10MB / 10,000 records per shard, that's a full
extra copy of every page on the hot path even when nothing is aggregated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
