nevi-me commented on pull request #9331: URL: https://github.com/apache/arrow/pull/9331#issuecomment-775467510
I think I've found the issue, it's with the way we handle statistics. We create a null array whenever 0 results are returned from predicate filtering. https://github.com/apache/arrow/blob/master/rust/datafusion/src/physical_plan/parquet.rs#L681 Not all data types are supported, which would explain the int96 being converted to a `NullArray`. The solution is to extend the check for `ParquetStatistics::ByteArray` beyond just the `Utf8` Arrow type (https://github.com/apache/arrow/blob/master/rust/datafusion/src/physical_plan/parquet.rs#L676). It's probably safe to create an empty array of whatever Arrow type, because casting to a null array might be unsound. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
