rdtr opened a new pull request, #13587:
URL: https://github.com/apache/gluten/pull/13587

   Depends on #13520. Only the last commit belongs to this PR: 
https://github.com/rdtr/incubator-gluten/commit/a3d1a94b579ff97604d1151ad3348d2a670abf99
   
   ## What changes are proposed in this pull request?
   
   Spark 4.3 (SPARK-58272) lets `InjectRuntimeFilter` build a Bloom filter from 
a fully materialized, disk-backed cache. The cache qualifies only when 
`InMemoryRelation.hasRepeatablePhysicalPlan` accepts its physical plan, and 
that check is a private allowlist of Spark operators. A cache built from Gluten 
transformers never matches it, so `statsAvailable` stays false and the filter 
isn't injected. For `spark.range(...).where("id = 
8").selectExpr(...).persist(MEMORY_AND_DISK)` the cached plan has 
`WholeStageTransformer`, `ProjectExecTransformer`, `FilterExecTransformer` and 
`ColumnarRangeExec`. The join still returns the right rows, it just loses this 
optimization.
   
   9 of the SPARK-58272 tests in `GlutenInjectRuntimeFilterSuite` assert those 
stats, so they stay excluded and the exclude now gives the reason. This adds 
one Gluten test where the optimization does apply: the cache is built with 
`spark.gluten.enabled=false`, then the query runs with Gluten on. It checks the 
Bloom filter is injected, the join and the `might_contain` filter are native, 
the Bloom filter aggregate is native and reads the cache, and the result 
matches the query without the filter.
   
   SPARK-58310 also stays excluded, with a corrected reason. It fails with 
`Cannot retain a released vector` in `ColumnarArrowEvalPythonExec`, which isn't 
in the runtime filter path. I'll look at it separately.
   
   ## How was this patch tested?
   
   `GlutenInjectRuntimeFilterSuite` on Spark 4.3: 17 passed, 0 failed, 11 
ignored (the 10 excludes above and the existing "Merge runtime bloom filters" 
exclude). The new test is "SPARK-58272: native join with a bloom filter from a 
cache built without Gluten". With `spark.gluten.enabled` left on while building 
the cache, it fails with `relation.statsAvailable was false`.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (Claude Opus 5.5)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

  • [PR] [GLUTEN-13517][VL] Test ... rdtr (via GitHub) via commits

Reply via email to