jordepic opened a new issue, #10712: URL: https://github.com/apache/arrow-rs/issues/10712
arrow-avro's streaming `Decoder` is optimized for SOE/Confluent framing. Transports such as Kafka already provide a message boundary, but callers that need Avro's "decode one datum and ignore trailing bytes" behavior cannot directly invoke the existing one-record decoder primitive. They must add synthetic framing or pre-scan the datum. Sparse nullable records also recursively materialize a placeholder into every descendant array for every null row. Workloads with long null runs (for example, event unions represented as nullable record fields) spend substantial CPU in `Decoder::append_null`. Proposed changes: - Expose a decoder method for exactly one unframed datum using the active writer schema, returning its consumed length. - Defer nullable child placeholders and materialize consecutive nulls through a bulk append path before the next value or flush. - Add focused correctness tests and Criterion coverage. External evidence from an 8,192-row Kafka/Nexmark decode workload using Arrow arrays: - Baseline native decode: 2.336M rows/s. - Direct one-datum decode: 2.749M rows/s (+17.7%). - Direct datum plus deferred null runs: 4.712M rows/s (2.02x baseline). I will submit the implementation and in-repository benchmark results in a follow-up PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
