rdtr commented on PR #13531:
URL: https://github.com/apache/gluten/pull/13531#issuecomment-6874795025

   Thanks for working on this, @surnaik. I had a fix for #13489 on a branch 
(https://github.com/rdtr/incubator-gluten/commit/67ed8efa8d05df569194a2a2e9b9b4ff644f3d82)
 but never opened a PR, so I compared the two and ran my tests against this 
PR's head (5a21ff0).
   
   The approach is the same: a pre-CBO rule that offloads only verified 
`might_contain(ScalarSubquery(Aggregate(bloom_filter_agg)))` pairs, a cast to 
bigint for int/short/byte, `IS NOT NULL` for nullable input, and the physical 
rule kept for `InjectRuntimeFilter`. My branch marks the vanilla side with 
separate expression classes instead of a tag. With the XxHash64 check you added 
in the physical rule, a lost tag can only affect a user-written call over 
`xxhash64(...)`, so I'm fine with the tag.
   
   I ran `GlutenBloomFilterFallbackSuite` and 12 extra cases on Spark 4.0 and 
3.5 (backends-velox, `SPARK_ANSI_SQL_MODE=false`). 
`GlutenBloomFilterFallbackSuite` passes. 11 of the 12 extra cases pass:
   
   - int/short/byte input, nullable input, string input
   - build-only `bloom_filter_agg` returns Spark-readable bytes
   - Spark-format constant, stored filter read by a subquery
   - three pairs in one query, each decided separately
   - a pair inside an `IN` subquery
   - a pair whose subquery has `HAVING count(DISTINCT ...)`
   - the same subquery both returned to the caller and consumed by 
`might_contain`
   - all-null input with `spark.gluten.sql.columnar.hashagg.enabled=false` 
(@philo-he's case). My branch fails this one, so the `eval` change here covers 
something mine missed.
   
   The case that fails is a Velox-format constant, see the inline comment.
   
   One more thing, outside the diff: `Test bloom_filter_agg produces 
identically-sized bytes on native and JVM` in gluten-ut/spark40 
`GlutenBloomFilterAggregateQuerySuite` no longer runs anything natively. Its 
query is a build-only `SELECT bloom_filter_agg(col, ...)`, which this PR keeps 
on vanilla Spark. Both runs now produce Spark bytes, so the size check passes 
without comparing the native and JVM Velox buffers. In my branch I registered 
`VeloxBloomFilterAggregate` as `velox_bloom_filter_agg` in that test and assert 
that the native run has a `HashAggregateExecBaseTransformer`: 
https://github.com/rdtr/incubator-gluten/blob/67ed8efa8d05df569194a2a2e9b9b4ff644f3d82/gluten-ut/spark40/src/test/scala/org/apache/spark/sql/GlutenBloomFilterAggregateQuerySuite.scala#L125-L171.
 Feel free to take it.
   
   My test cases are in 
https://github.com/rdtr/incubator-gluten/blob/67ed8efa8d05df569194a2a2e9b9b4ff644f3d82/backends-velox/src/test/scala/org/apache/gluten/sql/GlutenBloomFilterFallbackSuite.scala#L505-L756
 if any are useful.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to