[
https://issues.apache.org/jira/browse/SPARK-58322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuming Wang updated SPARK-58322:
--------------------------------
Affects Version/s: 4.2.0
(was: 5.0.0)
> Executors can register to wrong driver due to port reuse, causing silent data
> loss
> ----------------------------------------------------------------------------------
>
> Key: SPARK-58322
> URL: https://issues.apache.org/jira/browse/SPARK-58322
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.2.0
> Reporter: Yuming Wang
> Priority: Major
> Labels: data-loss, executor, yarn
>
> h2. Problem
> In YARN cluster mode, when a driver terminates and its RPC port is reused by
> a new driver from a *different application*, executors from the old
> application can connect to and register with the new driver. This leads to
> silent data loss.
> This was observed in production with the following scenario:
> * Old app (user A, long-running ThriftServer) had its driver at host:30202
> * Old driver crashed, port 30202 became free
> * New app (user B, short ETL job) started on the same host, bound to port
> 30202 (via spark.port.maxRetries)
> * Old AM (still alive) launched 278 executors with --driver-url pointing to
> host:30202. These were fresh executor processes in old app YARN containers,
> but they connected to the *new* driver
> * New driver accepted all 278 executors without any application identity
> verification
> * Tasks were scheduled to these wrong-app executors, writing data to
> incorrect staging directories
> * New driver committed the job, deleted staging directories. Data was lost.
> h2. Root Causes
> # spark.authenticate defaults to false: No RPC-level identity check between
> executor and driver.
> # RegisterExecutor message does not include appId: The driver has no way to
> verify which application an executor belongs to.
> # RetrieveSparkAppConfig does not verify appId: An executor can fetch config
> from any driver that happens to be at the configured address.
> # CoarseGrainedExecutorBackend.run() overwrites spark.app.id: The line
> cfg.sparkProperties ++ Seq((spark.app.id, arguments.appId)) unconditionally
> overwrites the driver's correct spark.app.id with the executor's CLI argument
> (from --app-id). This was originally a fallback (SPARK-3377, 2014) when
> sparkProperties was eagerly computed before spark.app.id was set, but became
> a latent bug when sparkProperties was changed to a lazy val.
> h2. Proposed Fix
> Add appId verification in CoarseGrainedExecutorBackend.run(): after
> RetrieveSparkAppConfig returns, compare arguments.appId (from --app-id CLI
> arg) with cfg.sparkProperties[spark.app.id] (from driver's SparkConf). If
> they mismatch, throw SparkException to prevent the executor from registering
> to the wrong driver.
> Also fix the ++ overwrite to only fall back to the CLI argument when the
> driver does not provide spark.app.id (race condition during SparkContext
> initialization).
> Add config spark.executor.appIdVerification.enabled (default true) to allow
> disabling.
> The same fix is applied to KubernetesExecutorBackend.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
