[ 
https://issues.apache.org/jira/browse/SPARK-61195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18141713#comment-18141713
 ] 

Huang Kuan Hao commented on SPARK-61195:
----------------------------------------

Thanks for filing this. One catch: the listener-order argument works for 
TRANSFORM, but not for {{PipedRDD}}.

{{PipedRDD}} creates its parent iterator inside the stdin writer thread 
({{firstParent[T].iterator(split, context)}}, {{PipedRDD.scala:149}}), so the 
scan registers its close listener on that thread, after the writer thread has 
started. A task completion listener added when the thread starts would 
therefore run after the reader is closed. With a {{mapPartitions}} between the 
scan and {{pipe("cat")}}, the {{mapPartitions}} function runs on the {{stdin 
writer for List(cat)}} thread.

So {{PipedRDD}} probably needs {{addTaskPreCompletionListener}}, like the 
pipelined Python writer, which uses it for the same listener-ordering issue.

The writer also needs a way to stop. It writes through a {{PrintWriter}}, which 
swallows {{IOException}} once the process is gone, so the loop doesn't notice 
and keeps reading the parent iterator. Without a check like 
{{out.checkError()}} or a flag set by the listener, joining the thread could 
mean waiting for the rest of the partition to be read.

I reproduced this on Spark 4.1.3 in local mode with Comet turned off, 
{{spark.memory.offHeap.enabled=true}} and a 4-million-row Parquet table. 
{{spark.sql.columnVector.offheap.enabled}} falls back to that setting.
{{spark.read.parquet(p).rdd.pipe("head -1").count()}} in a loop: SIGSEGV in 
{{OffHeapColumnVector.isNullAt}} on the stdin writer thread.
{{SELECT TRANSFORM(id, s) USING 'cat' AS (a, b) FROM t LIMIT 1}} in a loop: the 
same crash on the TRANSFORM feed thread.

I ran into this while testing DataFusion Comet, whose native plans hit the same 
race. Happy to open a PR for this if that helps.

> Stop the threads that read a task's input on their own before its readers are 
> closed
> ------------------------------------------------------------------------------------
>
>                 Key: SPARK-61195
>                 URL: https://issues.apache.org/jira/browse/SPARK-61195
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Gengliang Wang
>            Priority: Major
>
> Some operators read their input on a thread of their own, which the task does 
> not wait for: the writer thread of script transformation ({{TRANSFORM}}), the 
> stdin writer thread of {{PipedRDD}}, and the writer thread of pipelined 
> Python UDFs ({{spark.python.udf.pipelined.enabled}}). Such a thread can still 
> read a columnar batch after its task completed, while the completion listener 
> of {{DataSourceRDD}} closes the reader that returned the batch, for example 
> under a {{LIMIT}} or when the script exits early.
> The readers of native data sources read the buffers of a batch at their 
> addresses (SPARK-60753). So a reader that is closed before the end of its 
> stream cannot free the buffers of its last batch at once: it leaves them to a 
> Cleaner, which frees them once the reader and the batch are unreachable, at a 
> garbage collection that this off-heap memory does not trigger. With 
> {{spark.sql.columnVector.offheap.enabled}}, the vectorized Parquet reader 
> instead frees its {{OffHeapColumnVector}}s at close, so such a thread can 
> read freed memory there.
> Script transformation also does not destroy its process when its task 
> completes, unlike {{PipedRDD}}. Under a {{LIMIT}}, a script that keeps 
> writing output that nobody reads any more blocks once the pipe is full, and 
> so does the writer thread, which then keeps the thread, the process and the 
> batch alive.
> Proposal: have these operators stop their thread, and wait for it to end, 
> from a task completion listener that they register when they start the 
> thread. Listeners run in reverse order of registration, so this one runs 
> before {{DataSourceRDD}} closes the readers. Script transformation should 
> destroy its process without interrupting its writer thread: the interrupt 
> makes {{proc.waitFor()}} in the {{finally}} block of the writer throw 
> {{InterruptedException}}, which {{NonFatal}} does not catch, so that 
> {{SparkUncaughtExceptionHandler}} exits the executor. The failures that the 
> writer gets once its task completed should not be logged as errors. 
> {{NativePartitionReader.close()}} can then free the buffers of its last batch 
> at once, without the Cleaner.
> Discussion: 
> https://github.com/apache/spark/pull/59961#discussion_r4488238438, 
> https://github.com/apache/spark/pull/59961#discussion_r4500198812 and 
> https://github.com/apache/spark/pull/59961#discussion_r4500198806



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to