brijrajk opened a new issue, #12613: URL: https://github.com/apache/gluten/issues/12613
### Backend VL (Velox) ### Bug description `VeloxBloomFilterAggregate` (the JVM-side expression used for both user-facing and runtime-filter bloom filters) and the native `bloom_filter_agg` Velox function size their internal bit array differently, even when given the identical `estimatedNumItems`/`numBits` arguments: - JVM `VeloxBloomFilterAggregate.createAggregationBuffer()` calls `VeloxBloomFilter.empty(estimatedNumItems)`, sizing the buffer purely from the raw item count and ignoring `numBits` entirely. - Native `BloomFilterAggAggregate.cpp` computes `capacity_ = min(numBits, maxNumBits) / 16` and ignores the raw item count once `numBits` is known. For example, given `estimatedNumItems=1000000, numBits=8388608` (Spark's plain defaults), the JVM side allocates a 16,777,216-bit buffer while the native side allocates an 8,388,608-bit buffer -- exactly 2x different, deterministically, for any query using this code path. ### Why this matters Two-phase aggregation runs the partial and final stages as separate physical operators, which can independently land on the JVM or on native Velox (e.g. via Gluten's whole-stage fallback policy, or via `spark.gluten.sql.columnar.hashagg.enabled=false`). When the partial aggregate runs on one engine and the final aggregate (which merges partial buffers) runs on the other, the merge combines two differently-sized bit arrays. Velox's `BloomFilter::merge` (`velox/common/base/BloomFilter.h`) guards the size match with `VELOX_DCHECK_EQ`, which is compiled out in release builds, so the mismatched merge proceeds silently instead of throwing. The result is silent data corruption: bits inserted relative to one array size are later queried relative to a different array size, causing bloom filter false negatives (values that were definitely inserted are reported as absent). ### Reproduction Verified in a local build (Spark 4.0, Velox backend, release build): forcing a native-partial + JVM-final split on a runtime-filter query (`GlutenConfig.COLUMNAR_WHOLESTAGE_FALLBACK_THRESHOLD=1`, no other operator forced) produces a query that silently drops rows it should have returned (6 of 10 expected rows in a minimal repro), with no exception thrown. The reverse case, both partial and final on the same engine (either both native or both JVM, e.g. via whole-stage reversion of the entire subquery), is unaffected because both stages agree on capacity in that case. ### Proposed fix Make `VeloxBloomFilterAggregate`'s JVM-side buffer sizing use the same formula as the native side (derive capacity from `numBits`, not from raw `estimatedNumItems`), so both engines agree on capacity for the same input arguments regardless of which engine executes which stage. ### Context Found while investigating apache/gluten#12151 (GLUTEN-12013 bloom-filter byte-format fix), which needs this capacity alignment as a prerequisite before it can safely make its runtime-filter rewrite reversion-safe via a `final`-phase rule registration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
