hhhizzz commented on PR #10136: URL: https://github.com/apache/arrow-rs/pull/10136#issuecomment-5161496965
Hi @devanbenz I am investigating a related path in the Parquet reader for flat optional columns. Given a logical row selection and a presence bitmap, the reader needs to derive two mappings: ```text physical_selection = compress(selected, present) output_validity = compress(present, selected) ``` I noticed that this PR already proposes moving the generic compress primitive to arrow-buffer, so I would like to avoid introducing a second PEXT/helper implementation. Are you still planning to continue this PR? If so, would you prefer the Parquet work to depend on the shared primitive proposed here, or should the generic primitive and the arrow-select consumer be split so other consumers can build on it independently? My Parquet-specific work is primarily about reusable mapper workspace, lazy output-validity materialization, and a portable low-null specialization. CPU-specific BMI2 dispatch would remain a separate optional backend rather than deciding higher-level reader strategy. I would be happy to coordinate the API boundary or help with the shared primitive. cc @alamb for reviewers thoughts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
