jiangxt2 opened a new pull request, #57336:
URL: https://github.com/apache/spark/pull/57336
### What changes were proposed in this pull request?
- **Add `bitmap_contains(bitmap, bit_position)` scalar function**: returns
true
if the bit at the given position is set in the bitmap, false otherwise.
- **Out-of-range returns false**: Positions less than 0 or beyond the bitmap
range return false rather than raising an error, following the principle
that
invalid lookups in a query filter should not fail the entire query.
### Why are the changes needed?
Spark's flat-bitmap expression family has construction
(`bitmap_construct_agg`),
aggregation (`bitmap_or_agg`, `bitmap_and_agg`), and counting
(`bitmap_count`),
but lacks a membership-test predicate for filtering rows.
`bitmap_contains` enables bitmap columns to be used in `WHERE` filters,
`CASE WHEN` classification, and `JOIN` conditions -- the primary use case of
bitmap indexes in analytical databases. This function is also available in
other bitmap-using engines such as ClickHouse `bitmapContains`, Doris
`bitmap_contains`, and StarRocks `bitmap_contains`.
Example:
```sql
-- Check if a bit is set
SELECT bitmap_contains(X'01', 0); -- true (bit 0 set)
SELECT bitmap_contains(X'01', 1); -- false (bit 1 not set)
-- Filter rows with bitmap column
SELECT count(*)
FROM (SELECT bitmap_construct_agg(bitmap_bit_position(val)) AS bm
FROM VALUES (1), (3), (5) AS tab(val))
WHERE bitmap_contains(bm, bitmap_bit_position(3));
-- 1
```
### Does this PR introduce _any_ user-facing change?
Yes. A new `bitmap_contains` function is available in SQL, the Scala
DataFrame API,
and PySpark (classic and Connect).
### How was this patch tested?
- **`BitmapExpressionUtilsSuite`**: 9 unit tests covering bit-set, bit-unset,
boundary positions, out-of-range, short bitmap, zero-length bitmap, and
early-return guard for extreme position values.
- **`BitmapExpressionsQuerySuite`**: 10 SQL-level tests covering basic usage,
`WHERE` filter, `CASE WHEN`, `NULL` propagation, type errors for both
arguments, and combination with `bitmap_and_agg`.
- **`PlanGenerationTestSuite`**: Spark Connect plan generation test with
generated golden files.
- **`sql-expression-schema.md`**: Regenerated golden file.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code with Claude Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
