ranflarion opened a new issue, #10495:
URL: https://github.com/apache/arrow-rs/issues/10495
**Is your feature request related to a problem or challenge?**
The Avro spec places a 16-byte sync marker after every block in an object
container file specifically so corrupt blocks can be detected, and the Java
reference reader rejects a mismatch (`DataFileStream` throws
`IOException("Invalid sync!")`). Both arrow-avro readers parse each block's
trailing marker into `Block.sync` but never compare it against the file
header's marker, so a desynced or corrupted stream can decode silently into
wrong values. This is not hypothetical for compressed files: avro-deflate block
data is raw DEFLATE with no checksum, so a damaged block either errors
structurally or decodes into plausible garbage. We observed transient
object-store read corruption in production (Azure, hadoop-azure readahead
faults) where only structural damage was detectable; sync validation shrinks
the silent-corruption window and matches the reference implementation.
This continues the recent hardening of the container decode path (#10237
block size/count bounds, #10407 varint bounds).
**Describe the solution you'd like**
After `BlockDecoder::flush()` returns a block (one site in the sync
`Reader`, one in the async reader), error if `block.sync != header.sync()`. Any
file the Java reader accepts still reads fine — appended files reuse the file's
marker, so only spec-violating or damaged data fails.
Depends on the sync marker assembly fix (#10494): with the current assembly
bug, validation false-positives whenever a block's marker straddles a chunk
boundary. Happy to submit a PR covering both readers plus tests
(corrupted-marker file errors in both readers; existing suites stay green).
**Are there any user-facing changes?**
Corrupt files that previously decoded silently now error, matching the Java
reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
