Jefffrey opened a new issue, #11284: URL: https://github.com/apache/arrow-rs/issues/11284
### Describe the bug There are some places in the codebase where we are inconsistent with how we treat nullability for unions. Unions have children with fields which define their nullability, and since unions themselves dont have a null buffer they must encode any nulls into a child. However they should pick a child that has nullability set to true, if possible, instead of encoding into any child. Some related issues: - https://github.com/apache/arrow-rs/issues/10990 - https://github.com/apache/arrow-rs/issues/10992 - https://github.com/apache/arrow-rs/issues/1637 ### To Reproduce One such issue is in ArrayData: https://github.com/apache/arrow-rs/blob/7d6a86492009ee3a27eac98b251c846790225e16/arrow-data/src/data.rs#L804-L826 Here we choose the first child to encode nulls ### Expected behavior Fix the above case to try choose a child that has correct nullability to encode the nulls. Consider if we should panic if we cant find a suitable child. Also to close out this issue, would be good to do a comprehensive analysis of this throughout the codebase to see if there are any other spots we may be missing (other than the listed issues above) ### Additional context Would be good to also double check the union spec, as well as other implementations to see how they deal with this (e.g. C++) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
