jordepic opened a new pull request, #10713: URL: https://github.com/apache/arrow-rs/pull/10713
# Which issue does this PR close? - Closes #10712. # Rationale for this change Kafka and similar transports already provide an external boundary for each Avro datum. The existing `Decoder::decode` path requires per-record framing, which forces bare-Avro consumers to prepend and copy a synthetic prefix or scan the schema merely to find the end of an already bounded datum. Sparse nullable records also currently materialize placeholder values recursively for every null row. For union-like schemas with several nullable nested records, this repeats the same child traversal for long null runs. # What changes are included in this PR? - Add `Decoder::decode_datum` to decode exactly one unframed datum using the active writer schema and return the number of bytes consumed. - Defer trailing nullable child placeholders and materialize them in bulk before the next non-null value or batch flush. - Add a Criterion case modeling three sparse nullable nested event records with 100-row runs. The deferred state is internal to the record decoder. Output remains ordinary Arrow arrays and child lengths are aligned before flush. # Are these changes tested? Yes. - `cargo test -p arrow-avro --all-features`: 475 unit tests passed; 26 doc tests passed; 1 doc test ignored. - `cargo clippy -p arrow-avro --all-targets --all-features -- -D warnings` - `cargo fmt --all -- --check` Criterion on an Apple M1 Max, comparing this branch to the saved `main` baseline for a complete 10,000-row batch: | Benchmark | main median | this PR median | Change | |---|---:|---:|---:| | Sparse nullable nested records | 820.16 us | 581.44 us | -29.2% time / +41.3% throughput | Command: ```shell cargo bench -p arrow-avro --bench decoder -- 'SparseNested\(Struct\)/10000' --baseline main ``` The motivating downstream isolated benchmark decodes 8,192 Kafka-bounded Nexmark messages into Arrow, with polling, query execution, and sink work excluded. It measured: - Direct one-datum decoding alone: 2.336M to 2.749M rows/s (+17.7%). - Direct datum decoding plus deferred null runs: 2.336M to 4.712M rows/s (2.02x). - A separate best-of-three final run: 4.768M rows/s (209.7 ns/row), 2.69x the Java RowData result. The harness, methodology, profiles, and results are documented in the [StreamFusion optimization note](https://github.com/datafusion-contrib/StreamFusion/blob/5ed8a29/docs/optimizations/avro-one-datum-null-runs.md), with the integration in [StreamFusion commit 5ed8a29](https://github.com/datafusion-contrib/StreamFusion/commit/5ed8a2960d28318361f8fed474b279cb72f11f0b). # Are there any user-facing changes? Yes. `Decoder` gains the non-breaking public `decode_datum` method for externally framed, single-datum payloads. Existing framed decoding behavior is unchanged. AI assistance disclosure: Codex was used to help port the implementation from StreamFusion, draft tests and the benchmark, and prepare the issue and PR text. The resulting code and all reported outputs were reviewed before submission, and I take responsibility for the contribution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
