jiangxt2 opened a new pull request, #57295:
URL: https://github.com/apache/spark/pull/57295
Currently, Spark has three bitmap aggregation functions:
`bitmap_construct_agg`,
`bitmap_or_agg` (since 3.5.0), and `bitmap_and_agg` (since 4.1.0) for
constructing,
union, and intersection of sets represented by bitmaps. However, the
symmetric
difference operation (bitwise XOR) is missing, which is needed for use cases
such
as finding elements belonging to exactly one of two groups and detecting
changes
between snapshots.
### What changes were proposed in this pull request?
- **Implemented `bitmap_xor_agg` expression**: New aggregation function that
performs
bitwise XOR operations on binary column inputs, following the same
`ImperativeAggregate` pattern as `BitmapAndAgg`.
#### Design Decisions
- **Result on empty input is the XOR identity element (all zeros)**: Empty
input
groups return all-zeros bitmaps, since X ^ 0 = X.
- **Missing bytes handling**: For XOR operations, bytes beyond the shorter
input's
length are left unchanged, which is equivalent to XOR with 0 (absent bits).
This is consistent with the OR operation and requires no explicit
trailing-byte
zeroing unlike AND.
### Why are the changes needed?
Symmetric difference is a fundamental set operation alongside union and
intersection.
Other analytical engines provide this functionality. Adding `bitmap_xor_agg`
completes
the bitmap aggregation function family.
Example:
```sql
-- Symmetric difference: 0x10 ^ 0x30 ^ 0x40 = 0x60
SELECT substring(hex(bitmap_xor_agg(col)), 0, 6)
FROM VALUES (X'10'), (X'30'), (X'40') AS tab(col);
-- 600000
-- Same values cancel out: 0x10 ^ 0x10 = 0x00
SELECT substring(hex(bitmap_xor_agg(col)), 0, 6)
FROM VALUES (X'10'), (X'10') AS tab(col);
-- 000000
-- Count elements in exactly one group: 0x10 ^ 0x30 ^ 0x40 = 3
SELECT bitmap_count(bitmap_xor_agg(col)) AS symmetric_diff_count
FROM VALUES (X'10'), (X'30'), (X'40') AS tab(col);
-- 3
```
### Does this PR introduce _any_ user-facing change?
Yes. A new `bitmap_xor_agg` function is available in SQL, the Scala
DataFrame API,
and PySpark (classic and Connect).
### How was this patch tested?
- **`BitmapExpressionUtilsSuite`**: 5 new unit tests covering equal-length
XOR,
different-length XOR, all-zeros identity, self-XOR equals zero, and
sign-bit bytes.
- **`BitmapExpressionsQuerySuite`**: 3 new integration tests covering basic
XOR,
same-value cancellation, zero identity, different-length bitmaps, empty
input,
FILTER clause, GROUP BY, complex `bitmap_construct_agg` composition, and
type
mismatch error handling.
- **`PlanGenerationTestSuite`**: Spark Connect plan generation test with
generated
golden files.
- **`ExpressionsSchemaSuite`** / **`ExpressionInfoSuite`**: Regenerated
golden files
and verified example outputs.
### Was this patch authored or co-authored using generative AI tooling?
Generative AI tooling (Claude Code) was used as an assistive tool for
implementation
guidance and code review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
