HwangDongJun opened a new pull request, #13621: URL: https://github.com/apache/gluten/pull/13621
## What changes are proposed in this pull request? `bit_count` returns a different result from Spark for a negative `TINYINT`, `SMALLINT` or `INT` argument, with no error and no fallback. Spark counts the set bits of the value sign-extended to a long, so `bit_count(cast(-1 as tinyint))` is 64. Velox's `BitCountFunction` counts only as many bits as the input type is wide (8, 16 or 32). `BIGINT`, `BOOLEAN`, non-negative values and `NULL` already match Spark. See #13619. This PR widens byte, short and int arguments of `BitwiseCount` to `BIGINT` in `VeloxSparkPlanExecApi.extraExpressionConverter`, the same hook that already rewrites `RoundCeil` / `RoundFloor` on decimals. The cast is a widening one and cannot fail. The rewritten expression is passed as `original`, so the function signature is built from `BIGINT`. Only the Velox backend is touched. The alternative mentioned in the issue, counting 64 bits in the Spark flavor of the function in Velox, would make this workaround unnecessary. | `bit_count(x)` as | Spark | Gluten before | |---|---|---| | `x = -1`: tinyint / smallint / int | 64 / 64 / 64 | 8 / 16 / 32 | | `x = -7`: tinyint / smallint / int | 62 / 62 / 62 | 6 / 14 / 30 | Fixes #13619. ## How was this patch tested? Added `bit_count on negative tinyint, smallint and int` to `ScalarFunctionsValidateSuite`: the boundary values of every integral type (`-1`, `-7`, MIN, MAX, `0`, `7`) and `NULL` are written to Parquet and `bit_count` is run on all four columns with `runQueryAndCompare` (compared with vanilla Spark) and `checkGlutenPlan[ProjectExecTransformer]` (the projection stays offloaded). The `BIGINT` column was already correct and guards against the cast changing it. CI on a PR in my fork (same base as this one) passed (one `tpc-test-ubuntu` job failed once while downloading a Maven plugin and passed on re-run), and the new test is listed as passed in the `spark-test-spark35`, `spark-test-spark40` and `spark-test-spark41` logs. CI only runs the code after the fix, so for the before/after I built the Spark 4.1 bundle from this branch with the native libraries of the 2026-11-11 nightly (`ca9b6dadf`, two commits behind this base, the only native change in between is `SparkExprToSubfieldFilterParser.cc`) and ran the same data in local mode: before the fix `-1` gives 8 / 16 / 32 and `Int.MinValue` gives 1 for every type; after it all 7 rows equal vanilla Spark (`-1`: 64, `Int.MinValue`: 57 / 49 / 33 / 1) and the plan still contains `ProjectExecTransformer`. Not verified: the Velox branch Gluten builds from (I read `BitCountFunction` and its test in `facebookincubator/velox` main), ANSI mode, `bit_count` nested in larger expressions or aggregates, and the Spark 3.4 and 4.2 test logs (those jobs compiled and passed). I did not extend the SQL golden test `bitwise.sql`, which has four copies and only positive tinyint/smallint/int inputs. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: claude-sonnet-5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
