weixiuli created SPARK-26288:
--------------------------------
Summary: add initRegisteredExecutorsDB in ExternalShuffleService
Key: SPARK-26288
URL: https://issues.apache.org/jira/browse/SPARK-26288
Project: Spark
Issue Type: New Feature
Components: Kubernetes, Shuffle
Affects Versions: 2.4.0
Reporter: weixiuli
Fix For: 2.4.0
As we all know that spark on Yarn uses DB to record RegisteredExecutors
information, when the ExternalShuffleService restart and it can be reloaded,
which will be used as well .
While neither spark's standalone nor spark on k8s can record it's
RegisteredExecutors information by db or others ,so when ExternalShuffleService
restart ,which RegisteredExecutors information will be lost,it is't what we
looking forward to .
This commit add initRegisteredExecutorsDB which can be used either spark
standalone or spark on k8s to record RegisteredExecutors information , when the
ExternalShuffleService restart and it can be reloaded, which will be used as
well .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
