[ 
https://issues.apache.org/jira/browse/SPARK-61775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

L. C. Hsieh updated SPARK-61775:
--------------------------------
    Affects Version/s: 4.4.0
                           (was: 5.0.0)

> Do not wait forever for the read to end after the idle timeout killed the 
> Python worker
> ---------------------------------------------------------------------------------------
>
>                 Key: SPARK-61775
>                 URL: https://issues.apache.org/jira/browse/SPARK-61775
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark
>    Affects Versions: 4.4.0
>            Reporter: L. C. Hsieh
>            Priority: Major
>
> When the Python worker idle timeout kills a worker 
> (spark.sql.execution.pyspark.udf.idleTimeoutSeconds with 
> spark.sql.execution.pyspark.udf.killOnIdleTimeout), 
> BasePythonRunner.IdleTimeoutHandler keeps waiting for the read from the 
> worker to end: the read has to return the end of the stream or fail 
> (SPARK-61765) before the idle timeout error is thrown. Every further idle 
> timeout after the kill only logs "Waiting for Python worker process to 
> terminate after idle timeout" and tries the read again.
> If the connection to the killed worker is never closed, the read never ends 
> and the task hangs, although the worker has already been killed. This can 
> happen, for example, when another process still holds the worker's end of the 
> socket open.
> The handler should bound this wait and fail the task with the idle timeout 
> error once the killed worker is known to be gone, or after some time, instead 
> of waiting forever.
> One thing to check first: for a worker forked by the Python daemon, whether 
> ProcessHandle.isAlive() is reliable between the kill and the daemon reaping 
> the process, since a check like "timed out again after the kill and the 
> process is no longer alive" depends on it.
> This was raised in the review of SPARK-61765 
> (https://github.com/apache/spark/pull/60540).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to