yeya24 commented on PR #8315: URL: https://github.com/apache/arrow-rs/pull/8315#issuecomment-3282706933
Thanks @alamb, I have addressed your comments by exposing predicates via methods. For our usecase, we basically need a `ParquetRecordBatchStreamReader` like reader. Instead of reading and materializing the final results, it only filters the rows based on Row Filter and Row selection and returns the final rows that match the predicates in the Parquet file. The final result can be either `RowSelection` or a boolean array. Then our own customized reader can read and materialize the rows based on the filter results. If we are open to adding a reader for this usecase, then we are happy. If not, then we have to reimplement a reader which requires a lot of other structs and methods to be exposed as public. I can list some: - [ReadPlanBuilder](https://github.com/apache/arrow-rs/blob/main/parquet/src/arrow/arrow_reader/read_plan.rs#L32) - [InMemoryRowGroup](https://github.com/apache/arrow-rs/blob/aa626e12de8bc0d0f56b5349239cae1be8d1a195/parquet/src/arrow/async_reader/mod.rs#L967) and relevant fields like [ColumnChunkData](https://github.com/apache/arrow-rs/blob/aa626e12de8bc0d0f56b5349239cae1be8d1a195/parquet/src/arrow/async_reader/mod.rs#L1133) Would you be open to expose those specific fields or we are open to add a new reader that only filters and returns a boolean array? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
