Alb3e3 opened a new pull request, #50927: URL: https://github.com/apache/arrow/pull/50927
### Rationale for this change IPC schema serialization already treats extension arrays as their storage type. Record batch serialization still handled extension arrays according to the logical `extension` type when deciding whether to emit a validity buffer. That mismatches storage types whose IPC layout differs from the default, including unions. For extension-wrapped dense and sparse unions, the writer emitted a spurious top-level validity-buffer slot. The resulting IPC payload dropped the union buffers from the positions the reader expects, producing invalid data on read. ### What changes are included in this PR? - teach `RecordBatchSerializer::VisitArray()` to use the physical storage array when deciding common IPC buffer handling for extension arrays - keep field-node accounting on the logical extension array while delegating layout-sensitive buffer decisions to the storage array - add dense and sparse union extension roundtrip regression tests for both file and stream IPC writers This follows the same storage-type rule already used by IPC schema serialization. AI-assisted contribution note: this patch was prepared with AI assistance under Arrow's AI-generated code guidance. I reproduced the bug locally, validated the fix, and reviewed the final diff myself. ### Are these changes tested? Yes. Locally verified with: - `arrow-ipc-read-write-test --gtest_filter='TestFileFormat.DenseUnionExtensionRoundTrip:TestFileFormat.SparseUnionExtensionRoundTrip:TestStreamFormat.DenseUnionExtensionRoundTrip:TestStreamFormat.SparseUnionExtensionRoundTrip'` - `arrow-ipc-read-write-test --gtest_filter='-TestSchemaMetadata.MetadataVersionForwardCompatibility'` - `arrow-extension-type-test` The excluded `TestSchemaMetadata.MetadataVersionForwardCompatibility` requires external test data and is unrelated to this patch. ### Are there any user-facing changes? No API changes. This PR contains a "Critical Fix" in Arrow's template sense: it fixes IPC output that could otherwise contain invalid union payloads when an extension type wraps union storage, which can fail validation and crash on readback. * GitHub Issue: #50623 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
