andygrove commented on code in PR #6852:
URL: https://github.com/apache/datafusion-comet/pull/6852#discussion_r4402241920
##########
spark/src/test/scala/org/apache/comet/rules/RevertNativeForTransitionHeavyStagesSuite.scala:
##########
@@ -803,6 +842,104 @@ class RevertNativeForTransitionHeavyStagesSuite extends
CometTestBase {
}
}
+ // https://github.com/apache/datafusion-comet/issues/6687
+ test("transition-heavy revert restores columnar output for a cached AQE
result stage") {
Review Comment:
You are right, the branch does not depend on the Spark version, so the 3.4
and 3.5 profiles should run it. 5fc0358206 adds a rule-level test, "keeps the
output format of the result root", that applies the rule to
`CometSparkToColumnarExec(CometNativeColumnarToRowExec(cometFilter(leaf)))`
over a row-only leaf and asserts a `RowToColumnarExec` over the Spark
`FilterExec` with no Comet execs left. It then wraps that root in a
`ColumnarToRowExec` and asserts the reverted stage still outputs rows, with no
`RowToColumnarExec`. Both run with AQE on and off. I ran the suite on Spark 4.1
and 3.5, and with the branch changed to `outputColumnar = false` both new tests
fail on 3.5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
