Tian Gao created SPARK-59912:
--------------------------------
Summary: Fix flaky WorkerReuseTest and WorkerPoolCrashTest by
pinning spark.python.use.daemon
Key: SPARK-59912
URL: https://issues.apache.org/jira/browse/SPARK-59912
Project: Spark
Issue Type: Test
Components: PySpark
Affects Versions: 5.0.0
Reporter: Tian Gao
WorkerReuseTest.test_reuse_worker_of_parallelize_range and
WorkerPoolCrashTest.test_worker_crash assert that Python worker PIDs are reused
across consecutive jobs. Worker reuse (stable PIDs) only happens on the daemon
path; the simple-worker path (spark.python.use.daemon=false) spawns a fresh
process per task. Sibling suites in the same file (SimpleWorkerTests,
WorkerSegfaultNonDaemonTest) set spark.python.use.daemon=false, and because all
suites in a test module share one gateway JVM, whichever SparkContext launches
it first leaks its conf into the JVM system properties (same mechanism as
SPARK-59885). When a non-daemon suite launches first, the reuse tests inherit
use.daemon=false, run in simple-worker mode, and the PID-reuse assertions fail.
Because test ordering is not deterministic, the failure is flaky. Fix: pin
spark.python.use.daemon=true in the two reuse suites so the assertions no
longer depend on the ambient JVM default or test ordering.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
