[ https://issues.apache.org/jira/browse/SPARK-45942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated SPARK-45942: ----------------------------------- Labels: pull-request-available (was: ) > Only do the thread interruption check for putIterator on executors > ------------------------------------------------------------------ > > Key: SPARK-45942 > URL: https://issues.apache.org/jira/browse/SPARK-45942 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 4.0.0 > Reporter: Huanli Wang > Priority: Major > Labels: pull-request-available > > https://issues.apache.org/jira/browse/SPARK-45025 > introduces a peaceful thread interruption handling. However, there is an edge > case: when a streaming query is stopped on the driver, it interrupts the > stream execution thread. If the streaming query is doing memory store > operations on driver and performs {{doPutIterator}} at the same time, the > [unroll process will be > broken|https://github.com/apache/spark/blob/39fc6108bfaaa0ce471f6460880109f948ba5c62/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala#L224] > and [returns used > memory|https://github.com/apache/spark/blob/39fc6108bfaaa0ce471f6460880109f948ba5c62/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala#L245-L247]. > This can result in {{closeChannelException}} as it falls into this [case > clause|https://github.com/apache/spark/blob/aa646d3050028272f7333deaef52f20e6975e0ed/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L1614-L1622] > which opens an I/O channel and persists the data into the disk. However, > because the thread is interrupted, the channel will be closed at the begin: > [https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/java/nio/channels/spi/AbstractInterruptibleChannel.java#L172] > and throws out {{closeChannelException}} > On executors, [the task will be killed if the thread is > interrupted|https://github.com/apache/spark/blob/39fc6108bfaaa0ce471f6460880109f948ba5c62/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala#L374], > however, we don't do it on the driver. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org