jiangxt2 opened a new pull request, #58117:
URL: https://github.com/apache/spark/pull/58117
### What changes were proposed in this pull request?
This PR adds `bitmap_contains(bitmap, bit_position)`, a scalar predicate for
Spark's existing
flat `BinaryType` bitmap function family. It is available through SQL, the
Scala DataFrame API,
PySpark classic, and PySpark Connect.
The function:
- checks a bucket-local bit position in constant time;
- accepts a binary bitmap and a numeric position that is cast to `BIGINT`
using Spark's existing
cast semantics;
- propagates SQL `NULL` for null bitmap or position inputs;
- returns `false` for negative positions, positions at or above 32768, and
positions beyond the
actual bitmap byte length.
For an original value, callers use `bitmap_bucket_number(value)` to select
the bitmap bucket and
`bitmap_bit_position(value)` as the position passed to `bitmap_contains`.
This is a clean resubmission of #57336 against the current Spark codebase.
It keeps Spark's flat
bitmap representation and does not introduce another bitmap type,
serialization format, or
external dependency.
### Why are the changes needed?
Spark can construct, combine, and count flat bitmaps, but it has no built-in
predicate for testing
whether one position is present. Without this function, users must
materialize another collection
representation or use a UDF, which adds work and loses the consistent SQL,
classic, and Connect API
surface.
`bitmap_contains` completes the existing workflow and allows precomputed
bitmaps to be used in
filters, `CASE WHEN` expressions, and bucket-aware join conditions.
### Does this PR introduce _any_ user-facing change?
Yes. It adds the following function in SQL, Scala, PySpark classic, and
PySpark Connect:
```sql
bitmap_contains(bitmap BINARY, bit_position BIGINT) -> BOOLEAN
```
The position is local to one bitmap bucket. A query using original values
must match the bucket as
well as checking `bitmap_bit_position(value)`.
### How was this patch tested?
The implementation includes coverage for helper-level bit lookup, short and
empty bitmaps,
fixed-range and `Long` boundaries, SQL and Scala API behavior, null
propagation, numeric casts in
ANSI and non-ANSI modes, analysis errors, predicate and bucket-aware join
usage, PySpark classic and
Connect parity, and Connect plan generation. SQL expression schema and
Connect query-test golden
files were regenerated.
Previously completed targeted validation for the production implementation:
- `BitmapExpressionUtilsSuite`: 9 tests passed.
- `BitmapExpressionsQuerySuite`: 19 tests passed before the latest
bucket-aware join test rewrite.
- `PlanGenerationTestSuite`: 745 tests passed, with 2 ignored.
- `ProtoToParsedPlanTestSuite`: 750 tests passed.
- PySpark classic functions: 175 tests passed, with 6 dependency-based skips.
- PySpark Connect parity: 123 tests passed, with 4 expected skips.
- Classic and Connect function doctests passed.
Current cumulative diff static validation:
- `git diff --check`: passed.
- Spark Scalafmt validation: passed.
- Spark Scalastyle for main and test sources: passed.
- Ruff on the changed Python files: passed.
The updated `BitmapExpressionsQuerySuite` has not been rerun after rewriting
the join case.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex and Claude AI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
