[
https://issues.apache.org/jira/browse/SPARK-60733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146915#comment-18146915
]
Gengliang Wang commented on SPARK-60733:
----------------------------------------
SPARK-60726 already tells a library to stop a read once its task was killed,
failed or completed: {{is_interrupted}} of the task context of a read then
returns true.
https://github.com/apache/spark/pull/60640 proposed to also return true once
the task starts to complete, before its pre-completion listeners run, and once
Spark closes the reader while another thread is in {{get_next}}. I closed that
PR: its main use was as the base of https://github.com/apache/spark/pull/60641
(SPARK-61195), which is closed in favor of the smaller SPARK-62031, which does
not need it.
The two remaining gaps are narrow: a thread that a pre-completion listener
waits for while it is in {{get_next}}, which today is only the writer of
pipelined Python UDFs ({{spark.python.udf.pipelined.enabled}}, off by default),
and a reader that is closed outside of a task. The PR can be reopened if they
matter.
> Tell a native data source library to stop a read once Spark closes its reader
> -----------------------------------------------------------------------------
>
> Key: SPARK-60733
> URL: https://issues.apache.org/jira/browse/SPARK-60733
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Kousuke Saruta
> Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]