rdtr opened a new pull request, #13587: URL: https://github.com/apache/gluten/pull/13587
Depends on #13520. Only the last commit belongs to this PR: https://github.com/rdtr/incubator-gluten/commit/a3d1a94b579ff97604d1151ad3348d2a670abf99 ## What changes are proposed in this pull request? Spark 4.3 (SPARK-58272) lets `InjectRuntimeFilter` build a Bloom filter from a fully materialized, disk-backed cache. The cache qualifies only when `InMemoryRelation.hasRepeatablePhysicalPlan` accepts its physical plan, and that check is a private allowlist of Spark operators. A cache built from Gluten transformers never matches it, so `statsAvailable` stays false and the filter isn't injected. For `spark.range(...).where("id = 8").selectExpr(...).persist(MEMORY_AND_DISK)` the cached plan has `WholeStageTransformer`, `ProjectExecTransformer`, `FilterExecTransformer` and `ColumnarRangeExec`. The join still returns the right rows, it just loses this optimization. 9 of the SPARK-58272 tests in `GlutenInjectRuntimeFilterSuite` assert those stats, so they stay excluded and the exclude now gives the reason. This adds one Gluten test where the optimization does apply: the cache is built with `spark.gluten.enabled=false`, then the query runs with Gluten on. It checks the Bloom filter is injected, the join and the `might_contain` filter are native, the Bloom filter aggregate is native and reads the cache, and the result matches the query without the filter. SPARK-58310 also stays excluded, with a corrected reason. It fails with `Cannot retain a released vector` in `ColumnarArrowEvalPythonExec`, which isn't in the runtime filter path. I'll look at it separately. ## How was this patch tested? `GlutenInjectRuntimeFilterSuite` on Spark 4.3: 17 passed, 0 failed, 11 ignored (the 10 excludes above and the existing "Merge runtime bloom filters" exclude). The new test is "SPARK-58272: native join with a bloom filter from a cache built without Gluten". With `spark.gluten.enabled` left on while building the cache, it fails with `relation.statsAvailable was false`. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
