[
https://issues.apache.org/jira/browse/SPARK-61279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh Kumar updated SPARK-61279:
---------------------------------
Description:
{{GroupedPartitionCoalescer}}, which {{GroupPartitionsExec}} uses to coalesce
partitions for storage-partitioned operations, keeps its groups in a field that
is serialized into every task binary, because {{CoalescedRDD}}, and
{{SortedMergeCoalescedRDD}} for a k-way merge, keep the coalescer. Only
{{getPartitions}} on the driver reads the groups, so large groupings add task
serialization cost for nothing.
Its location choice can also be improved. It reads only the parent's own
{{preferredLocations}} and names the most common entry for every group, the
first in a hash map's order on a tie. Over a scan, {{GroupPartitionsExec}}
coalesces the scan's mapped RDD, which reports no locations of its own, so no
group names one, and the scheduler offers the locations of each group's first
input that has any.
Proposed:
* A group of several partitions prefers a host only when more of its partitions
prefer it than any other host. It finds each partition's locations through the
parent's lineage with {{SparkContext.getPreferredLocs}}, as
{{DefaultPartitionCoalescer}} does. When the only location that the group's
partitions name on that host is an executor or an HDFS cache, the group keeps
that location.
* Every other group names no location, so the scheduler finds the locations
itself when it submits each stage, as it does today over a scan: a group of one
partition, a group with no host ahead, groups that share partitions, as the
slots of a key under {{REPLICATED_FOR_JOIN}} do in a partially clustered join,
and groups whose parent's narrow lineage is persisted, since only the scheduler
sees the cached blocks.
In a join that zips the grouped RDDs directly, a side whose group names no
location adds none when the other side's group names one, so the task prefers
the other side's location.
Split out of the review of SPARK-60265
(https://github.com/apache/spark/pull/59522), which reuses this coalescer for
V2 scan partition packing.
was:
{{GroupedPartitionCoalescer}}, which {{GroupPartitionsExec}} uses to coalesce
partitions for storage-partitioned operations, keeps its groups in a field that
is serialized into every task binary, because {{CoalescedRDD}} keeps the
coalescer. Only {{getPartitions}} on the driver reads the groups, so large
groupings add task serialization cost for nothing.
Its location choice can also be improved:
* A group of one partition names one of that partition's locations, so the
scheduler sees one host instead of all of them. It can name no location
instead, so the scheduler uses the partition's own locations.
* For larger groups, ties between equally preferred locations all go to the
same location, the first in a hash map's order, so several groups can prefer
one host. Ties can be spread across groups instead.
Over a scan, {{GroupPartitionsExec}} coalesces the scan's mapped RDD, which
reports no locations, so the location changes have no effect there: the
scheduler still gives each group the locations of its first input that has any.
Split out of the review of SPARK-60265
(https://github.com/apache/spark/pull/59522), which reuses this coalescer for
V2 scan partition packing.
> Keep GroupedPartitionCoalescer's groups out of tasks and improve its location
> choice
> ------------------------------------------------------------------------------------
>
> Key: SPARK-61279
> URL: https://issues.apache.org/jira/browse/SPARK-61279
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Rakesh Kumar
> Priority: Major
> Labels: pull-request-available
>
> {{GroupedPartitionCoalescer}}, which {{GroupPartitionsExec}} uses to coalesce
> partitions for storage-partitioned operations, keeps its groups in a field
> that is serialized into every task binary, because {{CoalescedRDD}}, and
> {{SortedMergeCoalescedRDD}} for a k-way merge, keep the coalescer. Only
> {{getPartitions}} on the driver reads the groups, so large groupings add task
> serialization cost for nothing.
> Its location choice can also be improved. It reads only the parent's own
> {{preferredLocations}} and names the most common entry for every group, the
> first in a hash map's order on a tie. Over a scan, {{GroupPartitionsExec}}
> coalesces the scan's mapped RDD, which reports no locations of its own, so no
> group names one, and the scheduler offers the locations of each group's first
> input that has any.
> Proposed:
> * A group of several partitions prefers a host only when more of its
> partitions prefer it than any other host. It finds each partition's locations
> through the parent's lineage with {{SparkContext.getPreferredLocs}}, as
> {{DefaultPartitionCoalescer}} does. When the only location that the group's
> partitions name on that host is an executor or an HDFS cache, the group keeps
> that location.
> * Every other group names no location, so the scheduler finds the locations
> itself when it submits each stage, as it does today over a scan: a group of
> one partition, a group with no host ahead, groups that share partitions, as
> the slots of a key under {{REPLICATED_FOR_JOIN}} do in a partially clustered
> join, and groups whose parent's narrow lineage is persisted, since only the
> scheduler sees the cached blocks.
> In a join that zips the grouped RDDs directly, a side whose group names no
> location adds none when the other side's group names one, so the task prefers
> the other side's location.
> Split out of the review of SPARK-60265
> (https://github.com/apache/spark/pull/59522), which reuses this coalescer for
> V2 scan partition packing.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]