[
https://issues.apache.org/jira/browse/SPARK-31555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Holden Karau updated SPARK-31555:
---------------------------------
Description:
We should explore the following improvements to cache block migration:
1) Peer selection (right now may overbalance on certain peers)
2) Do we need to configure the number of blocks to be migrated at the same time
3) Are there any blocks we don't need to replicate (e.g. they are already
stored on the desired number of executors even once we remove the executors
slated for decommissioning).
4) Do we want to prioritize migrating blocks with no replicas
was:
We should explore the following improvements to cache block migration:
1) Peer selection (right now may overbalance on certain peers)
2) Do we need to configure the number of blocks to be migrated at the same time
3) Do we want to prioritize migrating blocks with no replicas
> Improve cache block migration
> -----------------------------
>
> Key: SPARK-31555
> URL: https://issues.apache.org/jira/browse/SPARK-31555
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
> Affects Versions: 3.1.0
> Reporter: Holden Karau
> Priority: Major
>
> We should explore the following improvements to cache block migration:
> 1) Peer selection (right now may overbalance on certain peers)
> 2) Do we need to configure the number of blocks to be migrated at the same
> time
> 3) Are there any blocks we don't need to replicate (e.g. they are already
> stored on the desired number of executors even once we remove the executors
> slated for decommissioning).
> 4) Do we want to prioritize migrating blocks with no replicas
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
