[
https://issues.apache.org/jira/browse/SPARK-26297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16715119#comment-16715119
]
ASF GitHub Bot commented on SPARK-26297:
----------------------------------------
hvanhovell commented on a change in pull request #23249: [SPARK-26297][SQL]
improve the doc of Distribution/Partitioning
URL: https://github.com/apache/spark/pull/23249#discussion_r240299365
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/physical/partitioning.scala
##########
@@ -118,10 +115,12 @@ case class HashClusteredDistribution(
/**
* Represents data where tuples have been ordered according to the `ordering`
- * [[Expression Expressions]]. This is a strictly stronger guarantee than
- * [[ClusteredDistribution]] as an ordering will ensure that tuples that share
the
- * same value for the ordering expressions are contiguous and will never be
split across
- * partitions.
+ * [[Expression Expressions]]. Its requirement is defined as the following:
+ * - Given any 2 adjacent partitions, all the rows of the second partition
must be larger than or
+ * equal to any row in the first partition, according to the `ordering`
expressions.
Review comment:
Global sort (actually the `RangePartitioner`) currently guarantees that all
rows in partition `p + 1` are larger than the rows in partition `p`. I don't
think we should relax this, besides collect limit there aren't any use cases I
can think of that could work with this relaxed requirement.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> improve the doc of Distribution/Partitioning
> --------------------------------------------
>
> Key: SPARK-26297
> URL: https://issues.apache.org/jira/browse/SPARK-26297
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Wenchen Fan
> Assignee: Wenchen Fan
> Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
