[ 
https://issues.apache.org/jira/browse/SPARK-61195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gengliang Wang updated SPARK-61195:
-----------------------------------
    Description: 
Some operators read the input of a task on a thread of their own, which the 
task does not wait for: the writer thread of script transformation 
({{TRANSFORM}}), the stdin writer thread of {{PipedRDD}}, and the writer thread 
of the R runner ({{BaseRRunner}}). Such a thread can still read a columnar 
batch after its task completed, while the completion listener of 
{{DataSourceRDD}} closes the reader that returned the batch, for example under 
a {{LIMIT}} or when the script exits early. The writer of pipelined Python UDFs 
({{spark.python.udf.pipelined.enabled}}) does not: its pre-completion listener 
interrupts it and waits for it to exit before any completion listener runs 
(SPARK-33277, {{PythonRunner.startPipelinedWriter}}).

The readers of native data sources read the buffers of a batch at their 
addresses (SPARK-60753). So a reader that is closed before the end of its 
stream cannot free the buffers of its last batch at once: it leaves them to a 
Cleaner, which frees them once the reader and the batch are unreachable, at a 
garbage collection that this off-heap memory does not trigger. With 
{{spark.sql.columnVector.offheap.enabled}}, the vectorized Parquet reader 
instead frees its {{OffHeapColumnVector}}s at close, so such a thread can read 
freed memory there.

SPARK-60753 makes script transformation stop its script once its task completes 
before it read the whole output of the script: a completion listener sends the 
script SIGTERM, closes its output and interrupts the writer thread, but does 
not wait for the thread. The listener of {{PipedRDD}} likewise destroys its 
process and interrupts its threads without waiting for them. The writer thread 
of {{PipedRDD}} creates the iterator of its input itself, so {{DataSourceRDD}} 
usually registers its completion listener after the one of {{PipedRDD}}, and 
closes the reader first.

Proposal: have these operators stop their thread and wait for it to exit from a 
pre-completion listener, as the pipelined Python writer does, so that no thread 
reads the input of a task once its completion listeners run. A writer blocked 
in a write to a script that ignores SIGTERM needs care, e.g. a forcible kill 
after a grace period. {{NativePartitionReader.close()}} can then free the 
buffers of its last batch at once, without the Cleaner.

Discussion: https://github.com/apache/spark/pull/59961#discussion_r4488238438, 
https://github.com/apache/spark/pull/59961#discussion_r4500198812 and 
https://github.com/apache/spark/pull/59961#discussion_r4518855898

  was:
Some operators read their input on a thread of their own, which the task does 
not wait for: the writer thread of script transformation ({{TRANSFORM}}), the 
stdin writer thread of {{PipedRDD}}, and the writer thread of pipelined Python 
UDFs ({{spark.python.udf.pipelined.enabled}}). Such a thread can still read a 
columnar batch after its task completed, while the completion listener of 
{{DataSourceRDD}} closes the reader that returned the batch, for example under 
a {{LIMIT}} or when the script exits early.

The readers of native data sources read the buffers of a batch at their 
addresses (SPARK-60753). So a reader that is closed before the end of its 
stream cannot free the buffers of its last batch at once: it leaves them to a 
Cleaner, which frees them once the reader and the batch are unreachable, at a 
garbage collection that this off-heap memory does not trigger. With 
{{spark.sql.columnVector.offheap.enabled}}, the vectorized Parquet reader 
instead frees its {{OffHeapColumnVector}}s at close, so such a thread can read 
freed memory there.

Script transformation also does not destroy its process when its task 
completes, unlike {{PipedRDD}}. Under a {{LIMIT}}, a script that keeps writing 
output that nobody reads any more blocks once the pipe is full, and so does the 
writer thread, which then keeps the thread, the process and the batch alive.

Proposal: have these operators stop their thread, and wait for it to end, from 
a task completion listener that they register when they start the thread. 
Listeners run in reverse order of registration, so this one runs before 
{{DataSourceRDD}} closes the readers. Script transformation should destroy its 
process without interrupting its writer thread: the interrupt makes 
{{proc.waitFor()}} in the {{finally}} block of the writer throw 
{{InterruptedException}}, which {{NonFatal}} does not catch, so that 
{{SparkUncaughtExceptionHandler}} exits the executor. The failures that the 
writer gets once its task completed should not be logged as errors. 
{{NativePartitionReader.close()}} can then free the buffers of its last batch 
at once, without the Cleaner.

Discussion: https://github.com/apache/spark/pull/59961#discussion_r4488238438, 
https://github.com/apache/spark/pull/59961#discussion_r4500198812 and 
https://github.com/apache/spark/pull/59961#discussion_r4500198806



> Stop the threads that read a task's input on their own before its readers are 
> closed
> ------------------------------------------------------------------------------------
>
>                 Key: SPARK-61195
>                 URL: https://issues.apache.org/jira/browse/SPARK-61195
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Gengliang Wang
>            Priority: Major
>
> Some operators read the input of a task on a thread of their own, which the 
> task does not wait for: the writer thread of script transformation 
> ({{TRANSFORM}}), the stdin writer thread of {{PipedRDD}}, and the writer 
> thread of the R runner ({{BaseRRunner}}). Such a thread can still read a 
> columnar batch after its task completed, while the completion listener of 
> {{DataSourceRDD}} closes the reader that returned the batch, for example 
> under a {{LIMIT}} or when the script exits early. The writer of pipelined 
> Python UDFs ({{spark.python.udf.pipelined.enabled}}) does not: its 
> pre-completion listener interrupts it and waits for it to exit before any 
> completion listener runs (SPARK-33277, {{PythonRunner.startPipelinedWriter}}).
> The readers of native data sources read the buffers of a batch at their 
> addresses (SPARK-60753). So a reader that is closed before the end of its 
> stream cannot free the buffers of its last batch at once: it leaves them to a 
> Cleaner, which frees them once the reader and the batch are unreachable, at a 
> garbage collection that this off-heap memory does not trigger. With 
> {{spark.sql.columnVector.offheap.enabled}}, the vectorized Parquet reader 
> instead frees its {{OffHeapColumnVector}}s at close, so such a thread can 
> read freed memory there.
> SPARK-60753 makes script transformation stop its script once its task 
> completes before it read the whole output of the script: a completion 
> listener sends the script SIGTERM, closes its output and interrupts the 
> writer thread, but does not wait for the thread. The listener of {{PipedRDD}} 
> likewise destroys its process and interrupts its threads without waiting for 
> them. The writer thread of {{PipedRDD}} creates the iterator of its input 
> itself, so {{DataSourceRDD}} usually registers its completion listener after 
> the one of {{PipedRDD}}, and closes the reader first.
> Proposal: have these operators stop their thread and wait for it to exit from 
> a pre-completion listener, as the pipelined Python writer does, so that no 
> thread reads the input of a task once its completion listeners run. A writer 
> blocked in a write to a script that ignores SIGTERM needs care, e.g. a 
> forcible kill after a grace period. {{NativePartitionReader.close()}} can 
> then free the buffers of its last batch at once, without the Cleaner.
> Discussion: 
> https://github.com/apache/spark/pull/59961#discussion_r4488238438, 
> https://github.com/apache/spark/pull/59961#discussion_r4500198812 and 
> https://github.com/apache/spark/pull/59961#discussion_r4518855898



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to