This is an automated email from the ASF dual-hosted git repository.
Jefffrey pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/main by this push:
new 300c77250e fix(arrow-avro): reset union offsets after flush (#10971)
300c77250e is described below
commit 300c77250ef2623f59d5d0fad4cf60810a86e51d
Author: hylin <[email protected]>
AuthorDate: Thu Sep 3 20:00:39 2026 +0800
fix(arrow-avro): reset union offsets after flush (#10971)
# Which issue does this PR close?
- Closes #10876.
# Rationale for this change
`UnionDecoderBranches::counts` tracks the next dense-union offset for
each branch. `flush` drained the type IDs, offsets, and child decoders
but left these counters at the previous batch totals, so the next batch
referenced positions beyond its newly emptied children.
# What changes are included in this PR?
- Reset all per-branch union counters after `UnionArray::try_new`
succeeds.
- Add a regression test that decodes and flushes two consecutive batches
through the same dense union decoder and verifies each batch starts at
offset zero.
The reset intentionally happens only after successful array
construction, so a failed flush does not additionally mutate the
counters.
# Are these changes tested?
Yes. TDD evidence on current `main`:
- Before the production change, the second flush failed with `Offsets
must be non-negative and within the length of the Array`.
- After the change, the focused regression passes.
- `cargo test -p arrow-avro --all-features`: 487 passed.
- Doc tests: 27 passed, 1 repository-marked ignored.
- `cargo fmt --all -- --check`: passed.
- `cargo clippy -p arrow-avro --all-targets --all-features -- -D
warnings`: passed.
- `git diff --check`: passed.
The first all-features run had 399 passes and 88 failures because the
official `testing` submodule was not initialized; every failure
referenced missing `testing/data/avro` fixtures. After `git submodule
update --init --depth 1 testing`, the complete suite passed as reported
above.
# Are there any user-facing changes?
Yes. A decoder containing a multi-branch Avro union can now emit
multiple batches without producing stale dense-union offsets. There are
no API changes.
# AI assistance
I used AI assistance to investigate the decoder state, implement the
focused regression and fix, and prepare this description. I reviewed the
complete diff and verified all reported commands and outputs.
---
arrow-avro/src/reader/record.rs | 21 +++++++++++++++++++++
1 file changed, 21 insertions(+)
diff --git a/arrow-avro/src/reader/record.rs b/arrow-avro/src/reader/record.rs
index 63590ab695..5b8f4e607f 100644
--- a/arrow-avro/src/reader/record.rs
+++ b/arrow-avro/src/reader/record.rs
@@ -2284,6 +2284,7 @@ impl UnionDecoder {
children,
)
.map_err(|e| AvroError::ParseError(e.to_string()))?;
+ self.branches.counts.fill(0);
Ok(Arc::new(arr))
}
}
@@ -4639,6 +4640,26 @@ mod tests {
assert_eq!(str_child.value(0), "abc");
}
+ #[test]
+ fn test_union_dense_offsets_reset_after_flush() {
+ let union_dt = make_dense_union_avro(
+ vec![
+ (Codec::Null, "n", DataType::Null),
+ (Codec::Utf8, "s", DataType::Utf8),
+ ],
+ vec![0, 1],
+ );
+ let mut dec = Decoder::try_new(&union_dt).unwrap();
+
+ for _ in 0..2 {
+ dec.decode(&mut AvroCursor::new(&encode_avro_long(0)))
+ .unwrap();
+ let array = dec.flush(None).unwrap();
+ let union = array.as_union();
+ assert_eq!(union.value_offset(0), 0);
+ }
+ }
+
#[test]
fn test_union_decode_negative_branch_index_errors() {
let union_dt = make_dense_union_avro(
