[
https://issues.apache.org/jira/browse/SPARK-26288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
weixiuli updated SPARK-26288:
-----------------------------
Description:
As we all know that spark on Yarn uses DB to record RegisteredExecutors
information which can be reloaded and used again when the
ExternalShuffleService is restarted .
The RegisteredExecutors information can't be recorded both in the mode of
spark's standalone and spark on k8s , which will cause the RegisteredExecutors
information to be lost ,when the ExternalShuffleService is restarted.
To solve the problem above, a method is proposed and is committed .
was:
As we all know that spark on Yarn uses DB to record RegisteredExecutors
information, when the ExternalShuffleService restart and it can be reloaded,
which will be used as well .
While neither spark's standalone nor spark on k8s can record it's
RegisteredExecutors information by db or others ,so when ExternalShuffleService
restart ,which RegisteredExecutors information will be lost,it is't what we
looking forward to .
This commit add initRegisteredExecutorsDB which can be used either spark
standalone or spark on k8s to record RegisteredExecutors information , when the
ExternalShuffleService restart and it can be reloaded, which will be used as
well .
> add initRegisteredExecutorsDB in ExternalShuffleService
> -------------------------------------------------------
>
> Key: SPARK-26288
> URL: https://issues.apache.org/jira/browse/SPARK-26288
> Project: Spark
> Issue Type: New Feature
> Components: Kubernetes, Shuffle
> Affects Versions: 2.4.0
> Reporter: weixiuli
> Priority: Major
> Fix For: 2.4.0
>
>
> As we all know that spark on Yarn uses DB to record RegisteredExecutors
> information which can be reloaded and used again when the
> ExternalShuffleService is restarted .
> The RegisteredExecutors information can't be recorded both in the mode of
> spark's standalone and spark on k8s , which will cause the
> RegisteredExecutors information to be lost ,when the ExternalShuffleService
> is restarted.
> To solve the problem above, a method is proposed and is committed .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
