rdtr commented on PR #13531: URL: https://github.com/apache/gluten/pull/13531#issuecomment-6874795025
Thanks for working on this, @surnaik. I had a fix for #13489 on a branch (https://github.com/rdtr/incubator-gluten/commit/67ed8efa8d05df569194a2a2e9b9b4ff644f3d82) but never opened a PR, so I compared the two and ran my tests against this PR's head (5a21ff0). The approach is the same: a pre-CBO rule that offloads only verified `might_contain(ScalarSubquery(Aggregate(bloom_filter_agg)))` pairs, a cast to bigint for int/short/byte, `IS NOT NULL` for nullable input, and the physical rule kept for `InjectRuntimeFilter`. My branch marks the vanilla side with separate expression classes instead of a tag. With the XxHash64 check you added in the physical rule, a lost tag can only affect a user-written call over `xxhash64(...)`, so I'm fine with the tag. I ran `GlutenBloomFilterFallbackSuite` and 12 extra cases on Spark 4.0 and 3.5 (backends-velox, `SPARK_ANSI_SQL_MODE=false`). `GlutenBloomFilterFallbackSuite` passes. 11 of the 12 extra cases pass: - int/short/byte input, nullable input, string input - build-only `bloom_filter_agg` returns Spark-readable bytes - Spark-format constant, stored filter read by a subquery - three pairs in one query, each decided separately - a pair inside an `IN` subquery - a pair whose subquery has `HAVING count(DISTINCT ...)` - the same subquery both returned to the caller and consumed by `might_contain` - all-null input with `spark.gluten.sql.columnar.hashagg.enabled=false` (@philo-he's case). My branch fails this one, so the `eval` change here covers something mine missed. The case that fails is a Velox-format constant, see the inline comment. One more thing, outside the diff: `Test bloom_filter_agg produces identically-sized bytes on native and JVM` in gluten-ut/spark40 `GlutenBloomFilterAggregateQuerySuite` no longer runs anything natively. Its query is a build-only `SELECT bloom_filter_agg(col, ...)`, which this PR keeps on vanilla Spark. Both runs now produce Spark bytes, so the size check passes without comparing the native and JVM Velox buffers. In my branch I registered `VeloxBloomFilterAggregate` as `velox_bloom_filter_agg` in that test and assert that the native run has a `HashAggregateExecBaseTransformer`: https://github.com/rdtr/incubator-gluten/blob/67ed8efa8d05df569194a2a2e9b9b4ff644f3d82/gluten-ut/spark40/src/test/scala/org/apache/spark/sql/GlutenBloomFilterAggregateQuerySuite.scala#L125-L171. Feel free to take it. My test cases are in https://github.com/rdtr/incubator-gluten/blob/67ed8efa8d05df569194a2a2e9b9b4ff644f3d82/backends-velox/src/test/scala/org/apache/gluten/sql/GlutenBloomFilterFallbackSuite.scala#L505-L756 if any are useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
