[
https://issues.apache.org/jira/browse/SPARK-39847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-39847.
-----------------------------------
Fix Version/s: 3.3.1
3.2.3
3.4.0
Resolution: Fixed
Issue resolved by pull request 37260
[https://github.com/apache/spark/pull/37260]
> Race condition related to interruption of task threads while they are in
> RocksDBLoader.loadLibrary()
> ----------------------------------------------------------------------------------------------------
>
> Key: SPARK-39847
> URL: https://issues.apache.org/jira/browse/SPARK-39847
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 3.2.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
> Priority: Major
> Fix For: 3.3.1, 3.2.3, 3.4.0
>
>
> One of our workloads experienced a rare failure in `RocksDBLoader`
> {code:java}
> Caused by: java.lang.IllegalThreadStateException
> at java.lang.Thread.start(Thread.java:708)
> at
> org.apache.spark.sql.execution.streaming.state.RocksDBLoader$.loadLibrary(RocksDBLoader.scala:51)
> {code}
> After investigation, we determined that was due to task cancellation: if the
> task which starts the RocksDB library loading is interrupted, another thread
> may begin a load and crash with the thread state exception.
> Skimming through the code in
> [RocksDBLoader|https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBLoader.scala],
> I spot a potential race condition:
> * Although the native JNI call is uninterruptible, the thread which calls
> loadLibrary is still interruptible. Let’s call that thread the “task thread”.
> * Say we have two tasks, A and B, which both want to load the JNI library.
> * Say that Task A wins the race to perform the load and enters the
> synchronized block in loadLibrary(), spawns a child thread to perform the
> actual loading, then blocks in the loadLibraryThread.join() call.
> * If Task A is interrupted, an InterruptedException will be thrown and it
> will exit the loadLibrary synchronized block.
> * At this point, Task B enters the synchronized block and sees that
> exception == null because the loading thread is still running, so it calls
> loadLibraryThread.start() and hits the thread state error.
> One way to fix this is to add
> {code:java}
> if (loadLibraryThread.getState == Thread.State.NEW) {
> loadLibraryThread.start()
> }{code}
> to ensure that only one thread starts the loadLibraryThread. If the original
> starter thread is interrupted then a new thread will encounter this block,
> skip the start(), proceed to the join() and block on the original load thread.
> I will submit a PR with this fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
