jiangxt2 opened a new pull request, #58025: URL: https://github.com/apache/spark/pull/58025
### What changes were proposed in this pull request? Currently, Spark has three bitmap aggregation functions: `bitmap_construct_agg`, `bitmap_or_agg` (since 3.5.0), and `bitmap_and_agg` (since 4.1.0) for constructing, union, and intersection of sets represented by bitmaps. However, the symmetric difference operation (bitwise XOR) is missing, which is needed for use cases such as finding elements belonging to exactly one of two groups and detecting changes between snapshots. This PR adds a new aggregation function `bitmap_xor_agg` that performs bitwise XOR on binary bitmap inputs, following the same `ImperativeAggregate` pattern as `BitmapAndAgg`. #### Design Decisions - **Result on empty input is the XOR identity element (all zeros)**: An ungrouped empty input, or an existing group with no matching rows after FILTER, returns the all-zero bitmap as the XOR identity element (X ^ 0 = X). - **Missing bytes handling**: For XOR operations, bytes beyond the shorter input's length are left unchanged, which is equivalent to XOR with 0 (absent bits). This is consistent with the OR operation and requires no explicit trailing-byte zeroing unlike AND. - **Input validation**: Only BINARY input is accepted, with the same `UNEXPECTED_INPUT_TYPE` error class as the other bitmap aggregates. ### Why are the changes needed? Symmetric difference is a fundamental set operation alongside union and intersection. Other analytical engines provide this functionality. Adding `bitmap_xor_agg` completes the bitmap aggregation function family. Example: ```sql -- Symmetric difference: 0x10 ^ 0x30 ^ 0x40 = 0x60 SELECT substring(hex(bitmap_xor_agg(col)), 0, 6) FROM VALUES (X'10'), (X'30'), (X'40') AS tab(col); -- 600000 -- Same values cancel out: 0x10 ^ 0x10 = 0x00 SELECT substring(hex(bitmap_xor_agg(col)), 0, 6) FROM VALUES (X'10'), (X'10') AS tab(col); -- 000000 -- Count set bits in the XOR result: 0x10 ^ 0x30 ^ 0x40 = 0x60 SELECT bitmap_count(bitmap_xor_agg(col)) AS symmetric_diff_count FROM VALUES (X'10'), (X'30'), (X'40') AS tab(col); -- 2 ``` ### Does this PR introduce _any_ user-facing change? Yes. A new `bitmap_xor_agg` function is available in SQL, the Scala DataFrame API, and PySpark (classic and Connect). ### How was this patch tested? - **`BitmapExpressionUtilsSuite`**: 5 new unit tests covering equal-length XOR, different-length XOR, all-zeros identity, self-XOR equals zero, and sign-bit bytes. - **`BitmapExpressionsQuerySuite`**: 3 new tests covering basic XOR, same-value cancellation, zero identity, different-length bitmaps, empty input, FILTER clause, GROUP BY, complex `bitmap_construct_agg` composition, and type mismatch error handling. - **`PlanGenerationTestSuite`**: Spark Connect plan generation test with regenerated golden files. - **`ProtoToParsedPlanTestSuite`**: Parsed-plan golden test validating the Connect explain artifact. - **`ExpressionsSchemaSuite`**: Regenerated the SQL expression schema golden file. - **`ExpressionInfoSuite`**: Verified the new expression examples. - **PySpark doctest smoke**: Targeted doctests for the `bitmap_xor_agg` wrapper in classic PySpark. - **`spark-pr-precheck.py`**: Pre-push static checks (format, license, golden, version consistency) all passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex and Claude AI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
