grundprinzip opened a new pull request, #52090: URL: https://github.com/apache/spark/pull/52090
### What changes were proposed in this pull request? This PR fixes a bug in ArrowConverters where only the first record batch was being read from a single IPC stream containing multiple batches. The fix modifies both InternalRowIteratorWithSchema and InternalRowIteratorWithoutSchema classes to properly iterate through all batches within a single Arrow IPC stream. ### Why are the changes needed? Previously, when an Arrow IPC stream contained multiple record batches, only the first batch would be processed and the remaining batches would be ignored. This resulted in data loss and incorrect results when working with Arrow data that was serialized as a single stream with multiple batches. ### Does this PR introduce _any_ user-facing change? Yes. This fixes a data correctness issue where users would lose data when processing Arrow streams with multiple batches. The behavior change is that all batches in a stream are now correctly processed instead of only the first one. ### How was this patch tested? Added comprehensive test cases: - `multiple record batches in single IPC stream` - tests the schema-aware iterator - `multiple record batches in single stream without schema` - tests the schema-less iterator Both tests verify that all batches (6 rows across 3 batches) are correctly read from a single IPC stream. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code 🤖 Generated with [Claude Code](https://claude.ai/code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
