mbutrovich commented on code in PR #2258:
URL: https://github.com/apache/datafusion-comet/pull/2258#discussion_r2376408516
##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -325,14 +325,11 @@ object CometConf extends ShimCometConf {
.booleanConf
.createWithDefault(true)
- // RangePartitioning contains bugs
https://github.com/apache/datafusion-comet/issues/1906
val COMET_EXEC_SHUFFLE_WITH_RANGE_PARTITIONING_ENABLED: ConfigEntry[Boolean]
=
conf("spark.comet.native.shuffle.partitioning.range.enabled")
- .doc("Experimental feature to enable range partitioning for Comet native
shuffle. " +
- "This feature is experimental while we investigate scenarios that
don't partition data " +
- "correctly.")
+ .doc("Whether to enable range partitioning for Comet native shuffle.")
.booleanConf
- .createWithDefault(false)
+ .createWithDefault(true)
Review Comment:
I discussed with @andygrove and we were comfortable merging with `true` back
in June. I think if you're opting into native shuffle we should try to
accelerate all partitioning schemes, and if we discover issues it can be
toggled off.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
