linliu-code opened a new pull request, #18689:
URL: https://github.com/apache/hudi/pull/18689
### Describe the issue this Pull Request addresses
Follow-up to #18224 (JsonKinesisSource). Small correctness fixes that
surfaced after the initial merge.
### Summary and Changelog
1. **Numeric sequence-number comparison.** Kinesis sequence numbers are
128-bit integers represented as decimal strings whose lengths can vary, so
`String.compareTo` is not correct for ordering them. Introduce
`KinesisOffsetGen.CheckpointUtils.compareSequenceNumbers(a, b)` using
`BigInteger`, and use it in:
- `KinesisShardRange.hasUnreadRecords` (closed-shard fully-consumed check)
- `KinesisOffsetGen.checkDataLossOnExpiredShards` (also adds an
empty-`lastSeq` guard for `|endSeq` checkpoints — a closed shard that has not
yet had any record consumed is treated as not consumed)
- `KinesisOffsetGen.checkDataLossOnAvailableShards` (trim-horizon check)
2. **Null-safe `millisBehindLatest`.**
`GetRecordsResponse#millisBehindLatest()` is documented as nullable in the AWS
SDK; the previous `response.millisBehindLatest() == 0` would NPE on auto-unbox
if a server returned null. Extract to a `Long` and null-check before comparing.
3. **Integer-overflow guard on per-call limit.** `(int) (maxTotalRecords -
totalConsumed)` could silently truncate if the remaining budget exceeded
`Integer.MAX_VALUE`. Switch to `Math.toIntExact(...)` so it fails loudly
instead.
4. **Explicit getter names on `KinesisReadConfig`.** Replace the
Lombok-generated `isShouldAddMetaFields()` / `isEnableDeaggregation()` with
explicit `shouldAddMetaFields()` / `isDeaggregationEnabled()` (kept under
`@Getter` with `AccessLevel.NONE` on the two boolean fields). Updates the two
call sites in `JsonKinesisSource`.
### Impact
No user-visible behavior change in the common case. The numeric-comparison
fix only matters when comparing two Kinesis sequence numbers of different
decimal lengths; \`Math.toIntExact\` and the null guard are defensive and only
change behavior on edge-case inputs that previously truncated or NPE'd.
### Risk Level
Low
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable — existing Kinesis tests
(TestKinesisCheckpointUtils, TestJsonKinesisSource, TestShardRecordIterator,
TestKinesisShardRange, TestKinesisSourceFiltering, TestKinesisDataLossChecks)
all pass: 80 tests, 0 failures, 0 errors, 2 pre-existing skips
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
