linliu-code opened a new pull request, #667:
URL: https://github.com/apache/hudi-rs/pull/667
**Stacked on #639–#665** — review only the last commit.
## What was there
```rust
fn create_corrupted_block_if_needed(&mut self, _pos: u64, _len: Option<u64>)
-> Option<LogBlock> {
// TODO: support creating corrupted block
None
}
```
A corrupt or truncated block failed the entire log file.
`BlockType::Corrupted` existed in the model but nothing could ever produce one.
## What it does now
A well-formed Hudi log block records its total size **twice** — in the
header length field and again in a trailing reverse pointer — and is followed
by either another block or EOF. Three checks:
1. the trailing pointer has to lie inside the file
2. the size it records has to agree with the header
3. what follows the block has to be a magic marker or the end
Every offset is computed with `checked_add`/`checked_sub`, so a garbage
length **reports corruption rather than panicking or allocating against it**.
That is why the check runs before the body is parsed rather than after — a
bogus content length would otherwise reach a `vec![0u8; len]`.
When a block fails, the reader emits a corrupt marker and resumes at the
next magic marker, found by scanning in 1 MB windows that overlap by
`MAGIC.len() - 1` so a marker straddling a boundary is not missed. **One bad
block now costs its own span instead of the rest of the file.**
Adopted from the internal reader, which has carried this and the recovery
tests for some time.
## Tests
Two harness cases un-ignored and passing — a corrupt tail block, and a
delete-ordering fixture that was failing behind the same read.
Three unit tests on the check itself, since the harness cases only exercise
one shape:
- a length disagreeing with the trailing pointer is corrupt, and the real
length is **not** — the negative case matters, since a check that always says
"corrupt" would pass the positive one
- a length past EOF, and `u64::MAX`, are corrupt — decided arithmetically,
never by reading there
- the recovery offset always lands within the file
Full workspace green: 1185 lib + 79 table-read + 39 datafusion + 21 + 12.
Ignored 7 → 5.
## Note
While verifying this I hit 117 unrelated failures that turned out to be a
**stale fixture-extraction cache** under `$TMPDIR/hudi-rs-test-fixtures`, not
the change — they reproduce on a clean tree. Clearing the directory fixes it.
The cache key includes the zip's mtime, so it should self-invalidate; worth a
look separately if it recurs in CI.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
