HwangDongJun opened a new issue, #13619:
URL: https://github.com/apache/gluten/issues/13619

   ### Backend
   
   VL (Velox)
   
   ### Bug description
   
   `bit_count` is offloaded to Velox and returns a wrong result, without any 
error or fallback message, when the argument is a **negative** `TINYINT`, 
`SMALLINT` or `INT`. `BIGINT`, `BOOLEAN`, non-negative values and `NULL` are 
correct. The function support table lists `bit_count` as supported (`S`) with 
no limitation.
   
   Spark counts the set bits of the value **sign-extended to a 64-bit long** 
(the function is documented as returning "the number of bits that are set in 
the argument expr as an unsigned 64-bit integer"). Velox's `BitCountFunction` 
counts only as many bits as the input type is wide.
   
   **Reproduce** (plain SQL, default optimizer settings, Parquet table):
   
   ```sql
   CREATE TABLE t_bitcount (c_tinyint TINYINT, c_smallint SMALLINT, c_int INT, 
c_bigint BIGINT) USING parquet;
   INSERT INTO t_bitcount VALUES (-1, -1, -1, -1), (-7, -7, -7, -7), (7, 7, 7, 
7), (0, 0, 0, 0);
   
   SELECT c_int,
          bit_count(c_tinyint)  AS bc_tinyint,
          bit_count(c_smallint) AS bc_smallint,
          bit_count(c_int)      AS bc_int,
          bit_count(c_bigint)   AS bc_bigint
   FROM t_bitcount ORDER BY c_int;
   ```
   
   Spark (`spark.gluten.enabled=false`):
   
   | c_int | bc_tinyint | bc_smallint | bc_int | bc_bigint |
   |---|---|---|---|---|
   | -7 | 62 | 62 | 62 | 62 |
   | -1 | 64 | 64 | 64 | 64 |
   | 0 | 0 | 0 | 0 | 0 |
   | 7 | 3 | 3 | 3 | 3 |
   
   Gluten (the plan contains `ProjectExecTransformer`, so the expression is 
executed natively):
   
   | c_int | bc_tinyint | bc_smallint | bc_int | bc_bigint |
   |---|---|---|---|---|
   | -7 | **6** | **14** | **30** | 62 |
   | -1 | **8** | **16** | **32** | 64 |
   | 0 | 0 | 0 | 0 | 0 |
   | 7 | 3 | 3 | 3 | 3 |
   
   Boundary values, same behavior (Spark vs Gluten): `TINYINT -128` 57 vs 1, 
`SMALLINT -32768` 49 vs 1, `INT -2147483648` 33 vs 1.
   
   I compared about 30 other functions (bitwise ops, shifts, arithmetic, 
rounding, hashing, casts, aggregates) over TINYINT/SMALLINT/INT/BIGINT boundary 
values and found no other difference of this kind, so this looks specific to 
`bit_count`.
   
   **Analysis**
   
   - Spark (`BitwiseCount.nullSafeEval`) calls 
`java.lang.Long.bitCount(input.asInstanceOf[Byte|Short|Int])`, i.e. the narrow 
value is widened to a signed long first, so every negative value has the upper 
bits set.
   - Velox (`velox/functions/sparksql/Bitwise.h`):
   
     ```cpp
     struct BitCountFunction {
       template <typename TInput>
       FOLLY_ALWAYS_INLINE void call(int32_t& result, TInput num) {
         constexpr int kMaxBits = sizeof(TInput) * CHAR_BIT;
         auto value = static_cast<uint64_t>(num);
         result = bits::countBits(&value, 0, kMaxBits);
       }
     };
     ```
   
     The value is sign-extended into `uint64_t`, but only `sizeof(TInput) * 8` 
bits are countedtwiseTest.bitCount`) pins this behavior, e.g. 
`bitCount<int32_t>(kMin32) == 1`. I read thisfrom `facebookincubator/velox` 
main; I could not check the exact branch Gluten builds from.
   - Gluten maps `BitwiseCount` to `bit_count` unchanged 
(`ExpressionMappings`), without casting the argument.
   - The existing SQL tests (`bitwise.sql` for the Velox backend, Spark 3.5 to 
4.2) and `ScalarFunctionsValidateSuite` only use positive values for 
TINYINT/SMALLINT/INT (the only negative inputs are BIGINT), which is why this 
is not caught.
   
   **Possible fixes** (maintainers' choice):
   
   1. Count 64 bits in the Spark flavor of the function in Velox (and update 
its unit test), or
   2. In Gluten, cast the argument of `BitwiseCount` to `BIGINT` for 
byte/short/int inputs before it is passed to Velox (BIGINT already matches 
Spark).
   
   I am happy to prepare a PR for the Gluten-side change plus a SQL test with 
negative narrow inputs, if that direction is acceptable.
   
   Reproduced on: the Gluten 1.7.0 release bundle and the 1.9.0-SNAPSHOT 
nightly built from `main` (commit `ca9b6dadf`, 2026-11-11), with the same 
result in both. Spark 4.1.1.
   
   ### Gluten version
   
   main branch
   
   ### Spark version
   
   Spark-4.1.x
   
   ### Spark configurations
   
   spark.plugins=org.apache.gluten.GlutenPlugin
   spark.memory.offHeap.enabled=true
   spark.memory.offHeap.size=1g
   spark.sql.ansi.enabled=false
   
   ### System information
   
   Linux x86_64 (CentOS 8 based build), JDK 17
   
   ### Relevant logs
   
   ```bash
   No error or warning is logged; the query succeeds with the wrong values.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to