brijrajk commented on code in PR #12360:
URL: https://github.com/apache/gluten/pull/12360#discussion_r3479874203
##########
gluten-substrait/src/main/scala/org/apache/spark/shuffle/GlutenShuffleUtils.scala:
##########
@@ -76,6 +76,13 @@ object GlutenShuffleUtils {
.toLowerCase(Locale.ROOT)
val supportedCodecs =
BackendsApiManager.getSettings.shuffleSupportedCodec()
if (!supportedCodecs.contains(codec)) {
+ if (codec == "none") {
Review Comment:
The throw is intentional — this was discussed in #12333 with @FelixYBW and
@marin-ma. Gluten's native shuffle writer doesn't have a "no codec" code path;
the correct knob to disable compression is `spark.shuffle.compress=false`
(already handled in `ColumnarShuffleWriter` and `ColumnarBatchSerializer`).
This PR improves the `none` case specifically by pointing users directly to
that config, rather than the misleading
`spark.gluten.sql.columnar.shuffle.codec` message from #12333.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
