andygrove commented on PR #5394: URL: https://github.com/apache/datafusion-comet/pull/5394#issuecomment-5816836458
@sunchao all three addressed in 31230882a. 1. Plan-only reports now render through `ExtendedExplainInfo.generateVerboseInfo`, which always emits the annotated tree and coverage summary. The `verbose` format uses the same method, so its output is unchanged. There's a test with the format set to `fallback`. 2. The `maxRows == 0` guess is gone. A prep-rule pass counts as an AQE re-plan only when it runs inside `AdaptiveSparkPlanExec.reOptimize`, so initial preparation is never mistaken for one. `SELECT id FROM range(0) DISTRIBUTE BY id` now gets one report with AQE on. The old guess also missed a re-plan that isn't itself empty: `count(*)` over a join that becomes empty at runtime was reported twice. Both shapes are in the empty-plan matrix under AQE on/off and `collect`/`toRdd.count`, and both fail against the previous commit. 3. Subquery dedupe no longer uses the execution ID. Spark prepares a query's subqueries on the same thread just before the query itself, so the canonical hashes are kept in a thread-local record. It is cleared when the query's own plan arrives, and again once execution passes begin. A subquery preparation is recognised by `QueryExecution.prepareExecutedPlan` (non-AQE and DPP) or `InsertAdaptiveSparkPlan.compileSubquery` (AQE) on the call stack. Your nested-subquery query now gives 3 reports under AQE on/off with both `collect()` and `queryExecution.toRdd.count()`, and that matrix is now tested. These checks read Spark method names off the stack. I checked that `reOptimize`, `prepareExecutedPlan` and `compileSubquery` exist unchanged from 3.4 through 4.2. If one were ever renamed, the result would be duplicate reports; execution isn't affected. The plan-only tests pass on Spark 3.4 and 4.1, and `CometExecRuleSuite`, `RevertNativeForTransitionHeavyStagesSuite`, `CometScanRuleSuite` and `CometIcebergWriteDetectionSuite` pass on 4.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
