prashantwason opened a new pull request, #20309:
URL: https://github.com/apache/hudi/pull/20309
### Describe the issue this Pull Request addresses
Closes #20308
The native HFile reader stores the cursor's position in the file as an
`int`. For HFiles larger than 2 GB, such as large metadata table record index
file groups, the offset overflows. A valid lookup in `[fake first key, actual
first key)` after `SEEK_TO_BEFORE_BLOCK_FIRST_KEY` is then rejected with
`IllegalStateException: ... backward seekTo, which is not supported`.
`isAtFirstKeyOfBlock()` compares the negative `int` cursor offset with the
correct `long` block offset, so they never match.
### Summary and Changelog
Lookups on HFiles larger than 2 GB now behave the same as on smaller files.
- `HFileCursor`: the offset is now a `long` (`set(long, KeyValue)`,
`setOffset(long)`).
- `HFileDataBlock.seekTo/next`: take a `long blockStartOffsetInFile`.
Offsets relative to a block are still `int`, computed as `(int)
(cursor.getOffset() - blockStartOffsetInFile)`.
- `HFileReaderImpl`: removed the `int` casts on block offsets. `seekTo()`
now positions the cursor using the first data block's offset. Before, it used
`dataBlockIndexEntryMap.firstKey().getOffset()`, which is the offset of the key
inside its own byte array and only worked because the first data block starts
at offset 0.
- `TestHFileReader`: new `testReadHFilePointAndPrefixLookupBeyond2GB`. It
reruns the existing point and prefix lookup cases, including the fake-first-key
cases, through a reader that serves data blocks at offsets shifted so they
straddle 2 GB, lie beyond 2 GB, or lie beyond 4 GB. No large file is created.
- Without the fix, all 24 new cases fail.
- With only the `seekTo()` change, 19 of 24 fail with the exact production
exception.
- With the full fix, all pass.
We also checked the fix against the real 2.37 GB record index HFile from
production that triggered the failure. For each data block we looked up two
missing keys in `[fake first key, actual first key)`, then the actual first key:
- Before: blocks below 2 GB failed 0 of 10,346; blocks at or beyond 2 GB
failed 1,056 of 1,056.
- After: 0 failures in both groups.
### Impact
Fixes record-level index (and other metadata table) lookups on HFiles larger
than 2 GB. There are no public API changes; `HFileCursor` and the
`HFileDataBlock` methods are internal to `hudi-io`. No performance impact.
### Risk Level
low. The change only widens offset types in the native HFile reader. All 150
`hudi-io` tests pass, and the new test covers offsets straddling 2 GB and
beyond 4 GB.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
