[
https://issues.apache.org/jira/browse/SPARK-58322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuming Wang updated SPARK-58322:
--------------------------------
Description:
h2. Problem
In YARN cluster mode, when a driver terminates and its RPC port is reused by a
new driver from a {*}different application{*}, executors from the old
application can connect to and register with the new driver. This leads to
silent data loss.
This was observed in production with the following scenario:
* Old app (user A, long-running ThriftServer) had its driver at host:30202
* Old driver crashed, port 30202 became free
* New app (user B, short ETL job) started on the same host, bound to port
30202 (via spark.port.maxRetries)
* Old AM (still alive) launched 278 executors with --driver-url pointing to
host:30202. These were fresh executor processes in old app YARN containers, but
they connected to the *new* driver
* New driver accepted all 278 executors without any application identity
verification
* Tasks were scheduled to these wrong-app executors, writing data to incorrect
staging directories
* New driver committed the job, deleted staging directories. Data was lost.
h2. Root Causes
# spark.authenticate defaults to false: No RPC-level identity check between
executor and driver.
# RegisterExecutor message does not include appId: The driver has no way to
verify which application an executor belongs to.
# RetrieveSparkAppConfig does not verify appId: An executor can fetch config
from any driver that happens to be at the configured address.
# CoarseGrainedExecutorBackend.run() overwrites spark.app.id: The line
cfg.sparkProperties ++ Seq((spark.app.id, arguments.appId)) unconditionally
overwrites the driver's correct spark.app.id with the executor's CLI argument
(from --app-id). This was originally a fallback (SPARK-3377, 2014) when
sparkProperties was eagerly computed before spark.app.id was set, but became a
latent bug when sparkProperties was changed to a lazy val.
was:
h2. Problem
In YARN cluster mode, when a driver terminates and its RPC port is reused by a
new driver from a *different application*, executors from the old application
can connect to and register with the new driver. This leads to silent data loss.
This was observed in production with the following scenario:
* Old app (user A, long-running ThriftServer) had its driver at host:30202
* Old driver crashed, port 30202 became free
* New app (user B, short ETL job) started on the same host, bound to port 30202
(via spark.port.maxRetries)
* Old AM (still alive) launched 278 executors with --driver-url pointing to
host:30202. These were fresh executor processes in old app YARN containers, but
they connected to the *new* driver
* New driver accepted all 278 executors without any application identity
verification
* Tasks were scheduled to these wrong-app executors, writing data to incorrect
staging directories
* New driver committed the job, deleted staging directories. Data was lost.
h2. Root Causes
# spark.authenticate defaults to false: No RPC-level identity check between
executor and driver.
# RegisterExecutor message does not include appId: The driver has no way to
verify which application an executor belongs to.
# RetrieveSparkAppConfig does not verify appId: An executor can fetch config
from any driver that happens to be at the configured address.
# CoarseGrainedExecutorBackend.run() overwrites spark.app.id: The line
cfg.sparkProperties ++ Seq((spark.app.id, arguments.appId)) unconditionally
overwrites the driver's correct spark.app.id with the executor's CLI argument
(from --app-id). This was originally a fallback (SPARK-3377, 2014) when
sparkProperties was eagerly computed before spark.app.id was set, but became a
latent bug when sparkProperties was changed to a lazy val.
h2. Proposed Fix
Add appId verification in CoarseGrainedExecutorBackend.run(): after
RetrieveSparkAppConfig returns, compare arguments.appId (from --app-id CLI arg)
with cfg.sparkProperties[spark.app.id] (from driver's SparkConf). If they
mismatch, throw SparkException to prevent the executor from registering to the
wrong driver.
Also fix the ++ overwrite to only fall back to the CLI argument when the driver
does not provide spark.app.id (race condition during SparkContext
initialization).
Add config spark.executor.appIdVerification.enabled (default true) to allow
disabling.
The same fix is applied to KubernetesExecutorBackend.
> Executors can register to wrong driver due to port reuse, causing silent data
> loss
> ----------------------------------------------------------------------------------
>
> Key: SPARK-58322
> URL: https://issues.apache.org/jira/browse/SPARK-58322
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.2.0
> Reporter: Yuming Wang
> Priority: Major
> Labels: data-loss, executor, pull-request-available, yarn
>
> h2. Problem
> In YARN cluster mode, when a driver terminates and its RPC port is reused by
> a new driver from a {*}different application{*}, executors from the old
> application can connect to and register with the new driver. This leads to
> silent data loss.
> This was observed in production with the following scenario:
> * Old app (user A, long-running ThriftServer) had its driver at host:30202
> * Old driver crashed, port 30202 became free
> * New app (user B, short ETL job) started on the same host, bound to port
> 30202 (via spark.port.maxRetries)
> * Old AM (still alive) launched 278 executors with --driver-url pointing to
> host:30202. These were fresh executor processes in old app YARN containers,
> but they connected to the *new* driver
> * New driver accepted all 278 executors without any application identity
> verification
> * Tasks were scheduled to these wrong-app executors, writing data to
> incorrect staging directories
> * New driver committed the job, deleted staging directories. Data was lost.
> h2. Root Causes
> # spark.authenticate defaults to false: No RPC-level identity check between
> executor and driver.
> # RegisterExecutor message does not include appId: The driver has no way to
> verify which application an executor belongs to.
> # RetrieveSparkAppConfig does not verify appId: An executor can fetch config
> from any driver that happens to be at the configured address.
> # CoarseGrainedExecutorBackend.run() overwrites spark.app.id: The line
> cfg.sparkProperties ++ Seq((spark.app.id, arguments.appId)) unconditionally
> overwrites the driver's correct spark.app.id with the executor's CLI argument
> (from --app-id). This was originally a fallback (SPARK-3377, 2014) when
> sparkProperties was eagerly computed before spark.app.id was set, but became
> a latent bug when sparkProperties was changed to a lazy val.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
