L. C. Hsieh created SPARK-61775:
-----------------------------------
Summary: Do not wait forever for the read to end after the idle
timeout killed the Python worker
Key: SPARK-61775
URL: https://issues.apache.org/jira/browse/SPARK-61775
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
When the Python worker idle timeout kills a worker
(spark.sql.execution.pyspark.udf.idleTimeoutSeconds with
spark.sql.execution.pyspark.udf.killOnIdleTimeout),
BasePythonRunner.IdleTimeoutHandler keeps waiting for the read from the worker
to end: the read has to return the end of the stream or fail (SPARK-61765)
before the idle timeout error is thrown. Every further idle timeout after the
kill only logs "Waiting for Python worker process to terminate after idle
timeout" and tries the read again.
If the connection to the killed worker is never closed, the read never ends and
the task hangs, although the worker has already been killed. This can happen,
for example, when another process still holds the worker's end of the socket
open.
The handler should bound this wait and fail the task with the idle timeout
error once the killed worker is known to be gone, or after some time, instead
of waiting forever.
One thing to check first: for a worker forked by the Python daemon, whether
ProcessHandle.isAlive() is reliable between the kill and the daemon reaping the
process, since a check like "timed out again after the kill and the process is
no longer alive" depends on it.
This was raised in the review of SPARK-61765
(https://github.com/apache/spark/pull/60540).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]