andygrove opened a new pull request, #4772: URL: https://github.com/apache/datafusion-comet/pull/4772
## Which issue does this PR close? Closes #1197. ## Rationale for this change `SparkSessionExtensionSuite` exercises Spark's `injectColumnar` extension point with a user-supplied `MyColumnarRule` / `PreRuleReplaceAddWithBrokenVersion`. Issue #1197 reported that four of its tests (`inject columnar AQE on`, `inject columnar AQE off`, `SPARK-39991: AQE should retain column statistics from completed query stages`, and `reset column vectors`) failed with `BoundReference cannot be cast to ColumnarExpression` when Comet was enabled, because Comet's own injected columnar rules collided with the test's rule. The diffs worked around this by forcing `spark.comet.enabled=false` for those tests. That conflict no longer reproduces: the suite passes with Comet genuinely injected, so the workaround is now stale and hides real coverage of Comet coexisting with a user-injected columnar rule. ## What changes are included in this PR? Remove the three `spark.comet.enabled=false` overrides (and their `// https://github.com/apache/datafusion-comet/issues/1197` comments) from `SparkSessionExtensionSuite` in the Spark 3.4.3, 3.5.8, 4.0.2, and 4.1.2 diffs. Each diff was regenerated from a clean checkout of the corresponding Spark tag, so the only change is the removal of that one file hunk. ## How are these changes tested? The existing `SparkSessionExtensionSuite` now runs with Comet enabled (`ENABLE_COMET=true`) across all four Spark versions in CI. Locally, the full suite (27 tests, including the four previously worked around) passed on Spark 3.5.8 with a control assertion confirming Comet's columnar rule was actually injected into the test session, ruling out a vacuous pass. CI exercises the remaining versions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
