ranflarion opened a new issue, #10494:
URL: https://github.com/apache/arrow-rs/issues/10494
**Describe the bug**
Both `BlockDecoder` and `HeaderDecoder` accumulate the 16-byte sync marker
with
```rust
let write = &mut self.in_progress.sync[16 - to_decode..];
write[..to_decode].copy_from_slice(&buf[..to_decode]);
```
(arrow-avro/src/reader/block.rs, arrow-avro/src/reader/header.rs). The
offset is derived from the size of the incoming fragment, not from how many
marker bytes have been consumed so far, so it only works when all 16 bytes
arrive in a single `decode()` call. When the marker straddles a chunk boundary
— say 10 bytes then 6 — the first fragment is written to `sync[6..16]` and the
second to `sync[10..16]`, overwriting part of the first and leaving
`sync[0..6]` stale. Both decoders document support for incremental decoding
("can be called multiple times with consecutive chunks of data").
**Impact**
For `HeaderDecoder` the bug is live: `read_header` fetches the header in
bounded chunks (16 KB hint in the async path), and the async reader locates the
first block by scanning the stream for the header's sync marker. A header
larger than one fetch whose trailing marker straddles the boundary yields a
garbled `Header::sync()`, the scan never matches, and the reader silently
returns an empty stream ("file has no blocks after the header") for a valid
file.
For `BlockDecoder` the mis-assembled `Block.sync` is currently latent
(nothing reads it), but it blocks per-block sync marker validation (separate
issue).
**To Reproduce**
This test fails on main:
```rust
#[test]
fn test_sync_marker_split_across_decode_calls() {
// count=1 (zig-zag 0x02), size=1 (0x02), one data byte, 16-byte sync
marker
let sync: [u8; 16] = core::array::from_fn(|i| i as u8);
let mut block_bytes = vec![0x02, 0x02, 0xAA];
block_bytes.extend_from_slice(&sync);
for chunk_size in 1..block_bytes.len() {
let mut decoder = BlockDecoder::default();
for chunk in block_bytes.chunks(chunk_size) {
decoder.decode(chunk).unwrap();
}
let block = decoder.flush().expect("complete block");
assert_eq!(block.sync, sync, "chunk_size {chunk_size}");
}
}
```
**Expected behavior**
Fill from the front using the consumed count: `let offset = 16 -
self.bytes_remaining;` then write to `sync[offset..offset + to_decode]`. Happy
to submit a PR (two-line fix plus the test above).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
