neilconway opened a new issue, #11200: URL: https://github.com/apache/arrow-rs/issues/11200
### Is your feature request related to a problem or challenge? When a filter selects only non-null values, the result needs no validity bitmap. However, [FilterPredicate::filter_nulls](https://github.com/apache/arrow-rs/blob/b89e3020d9ad6c5fc5c9e9acafc83fb21866f536/arrow-select/src/filter.rs#L512-L532) currently does the following when the input contains nulls: 1. Filters the input validity bitmap into a temporary buffer. 2. Counts the resulting valid bits to determine the output null count. 3. Discards the buffer if the output contains no nulls. Constructing and discarding the validity bitmap is wasted work that could potentially be avoided. One possible optimization is to check whether the predicate selects any invalid positions before constructing the output bitmap. If `keep & !validity` contains no set bits, `filter_nulls` can return `None` immediately. Ideally, this check would operate on bitmap words without allocating an intermediate buffer. Another possibility is an API that lets callers express this guarantee when they already know it; the design would need to preserve correctness without requiring an invalid intermediate array. Removing the input validity bitmap before calling filter is not a general workaround. For example, a null dictionary slot may contain an out-of-range key that becomes invalid once its validity bit is removed. Struct and `FixedSizeList` validity can likewise mask nulls in fields declared non-nullable. The optimization should preserve the original input and avoid unnecessary output validity work. The performance benefit needs measurement. An additional check could slow filters that retain nulls, particularly when the predicate selects very few positions. Suggested benchmarks should vary array size, selectivity, null density, and random versus clustered nulls, covering both filters that remove every null and filters that retain some. ### Describe the solution you'd like _No response_ ### Describe alternatives you've considered _No response_ ### Additional context _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
