dwsmith1983 commented on code in PR #6785:
URL: https://github.com/apache/datafusion-comet/pull/6785#discussion_r4233318734
##########
spark/src/test/scala/org/apache/comet/exec/CometJoinSuite.scala:
##########
@@ -1219,6 +1251,237 @@ class CometJoinSuite extends CometTestBase {
}
}
+ // Spark removes a sort above a sort-merge join whose output ordering
satisfies it, so the
+ // forced hash join must be chosen before that happens or the sort is lost
(#6770).
+ private def withSortLossConf(adaptive: Boolean)(f: => Unit): Unit =
withSQLConf(
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> adaptive.toString,
+ SQLConf.ADAPTIVE_AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1",
+ SQLConf.SHUFFLE_PARTITIONS.key -> "2",
+ CometConf.COMET_FORCE_SHJ.key -> "true") {
+ withParquetTable((0 until 10000).map(i => (i % 100, i)), "big") {
+ withParquetTable((0 until 10).map(i => (i * 10, i)), "small") {
Review Comment:
> could the config block become one class-level helper that takes `adaptive`
and serves the semi join, window and sort tests?
Done. The semi join, window and sort tests now share one helper that takes
`adaptive`, sets the five configs plus `maxBuildSize=-1`, and builds the tables
with `withChainedJoinTables(midRows = 3000, midKeys = 100)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
