L. C. Hsieh created SPARK-61775:
-----------------------------------

             Summary: Do not wait forever for the read to end after the idle 
timeout killed the Python worker
                 Key: SPARK-61775
                 URL: https://issues.apache.org/jira/browse/SPARK-61775
             Project: Spark
          Issue Type: Improvement
          Components: PySpark
    Affects Versions: 5.0.0
            Reporter: L. C. Hsieh


When the Python worker idle timeout kills a worker 
(spark.sql.execution.pyspark.udf.idleTimeoutSeconds with 
spark.sql.execution.pyspark.udf.killOnIdleTimeout), 
BasePythonRunner.IdleTimeoutHandler keeps waiting for the read from the worker 
to end: the read has to return the end of the stream or fail (SPARK-61765) 
before the idle timeout error is thrown. Every further idle timeout after the 
kill only logs "Waiting for Python worker process to terminate after idle 
timeout" and tries the read again.

If the connection to the killed worker is never closed, the read never ends and 
the task hangs, although the worker has already been killed. This can happen, 
for example, when another process still holds the worker's end of the socket 
open.

The handler should bound this wait and fail the task with the idle timeout 
error once the killed worker is known to be gone, or after some time, instead 
of waiting forever.

One thing to check first: for a worker forked by the Python daemon, whether 
ProcessHandle.isAlive() is reliable between the kill and the daemon reaping the 
process, since a check like "timed out again after the kill and the process is 
no longer alive" depends on it.

This was raised in the review of SPARK-61765 
(https://github.com/apache/spark/pull/60540).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to