[ https://issues.apache.org/jira/browse/SPARK-20943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034296#comment-16034296 ]
Saisai Shao commented on SPARK-20943: ------------------------------------- I think the original purpose of comment is to say {{Aggregator}} and {{Ordering}} is not used in map side shuffle write, those {{Aggregator}} {{Ordering}} set in ShuffleRDD will only be used in shuffle reader side. > Correct BypassMergeSortShuffleWriter's comment > ---------------------------------------------- > > Key: SPARK-20943 > URL: https://issues.apache.org/jira/browse/SPARK-20943 > Project: Spark > Issue Type: Improvement > Components: Documentation, Shuffle > Affects Versions: 2.1.1 > Reporter: CanBin Zheng > Priority: Trivial > Labels: starter > > There are some comments written in BypassMergeSortShuffleWriter.java about > when to select this write path, the three required conditions are described > as follows: > 1. no Ordering is specified, and > 2. no Aggregator is specified, and > 3. the number of partitions is less than > spark.shuffle.sort.bypassMergeThreshold > Obviously, the conditions written are partially wrong and misleading, the > right conditions should be: > 1. map-side combine is false, and > 2. the number of partitions is less than > spark.shuffle.sort.bypassMergeThreshold -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org