[ https://issues.apache.org/jira/browse/SPARK-38591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Enrico Minack updated SPARK-38591: ---------------------------------- Summary: Add sortWithinGroups to KeyValueGroupedDataset (was: Add flatMapSortedGroups and cogroupSorted to KeyValueGroupedDataset) > Add sortWithinGroups to KeyValueGroupedDataset > ---------------------------------------------- > > Key: SPARK-38591 > URL: https://issues.apache.org/jira/browse/SPARK-38591 > Project: Spark > Issue Type: New Feature > Components: SQL > Affects Versions: 3.4.0 > Reporter: Enrico Minack > Assignee: Apache Spark > Priority: Major > > The existing methods {{KeyValueGroupedDataset.flatMapGroups}} and > {{KeyValueGroupedDataset.cogroup}} provide an iterator of rows for each group > key. If user code requires those rows in a particular order, that iterator > would have to be sorted first, which is against the idea of an iterator in > the first place. Methods {{flatMapGroups}} and {{cogroup}} have the great > advantage that they work with groups that are too large to fit into memory of > one executor. Sorting them in the user function breaks this property. > [org.apache.spark.sql.KeyValueGroupedDataset|https://github.com/apache/spark/blob/47485a3c2df3201c838b939e82d5b26332e2d858/sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala#L134-L137]: > {noformat} > Internally, the implementation will spill to disk if any given group is too > large to fit into > memory. However, users must take care to avoid materializing the whole > iterator for a group > (for example, by calling `toList`) unless they are sure that this is possible > given the memory > constraints of their cluster. > {noformat} > The implementations of {{KeyValueGroupedDataset.flatMapGroups}} and > {{KeyValueGroupedDataset.cogroup}} already sort each partition according to > the group key. By additionally sorting by some data columns, the iterator can > be guaranteed to provide some order. > New method {{KeyValueGroupedDataset.sortWithinGroups}} (similar to > {{{}Dataset.sortWithinPartitions){}}}would allow to define order within the > groups. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org