grundprinzip opened a new pull request, #52131: URL: https://github.com/apache/spark/pull/52131
### What changes were proposed in this pull request? Cherry-Pick: https://github.com/apache/spark/pull/52090 This PR adds a new method in ArrowConverters that allows properly decoding an Arrow IPC stream, which can contain multiple record batches. All of the other methods can only deal with message streams that contain exactly one record batch. ### Why are the changes needed? Previously, when an Arrow IPC stream contained multiple record batches, only the first batch would be processed and the remaining batches would be ignored. This resulted in data loss and incorrect results when working with Arrow data that was serialized as a single stream with multiple batches. ### Does this PR introduce _any_ user-facing change? Yes. This fixes a data correctness issue where users would lose data when processing Arrow streams with multiple batches. The behavior change is that all batches in a stream are now correctly processed instead of only the first one. ### How was this patch tested? Added comprehensive test cases. ### Was this patch authored or co-authored using generative AI tooling? Tests Generated-by: Claude Code 🤖 Generated with [Claude Code](https://claude.ai/code) Closes #52090 from grundprinzip/SPARK-53342. Authored-by: Martin Grund <[email protected]> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
