GitHub user viirya opened a pull request:
https://github.com/apache/spark/pull/23272
[SPARK-26265][Core] Fix deadlock in BytesToBytesMap.MapIterator when
locking both BytesToBytesMap.MapIterator and TaskMemoryManager
## What changes were proposed in this pull request?
In `BytesToBytesMap.MapIterator.advanceToNextPage`, We will first lock this
`MapIterator` and then `TaskMemoryManager` when going to free a memory page by
calling `freePage`. At the same time, it is possibly that another memory
consumer first locks `TaskMemoryManager` and then this `MapIterator` when it
acquires memory and causes spilling on this `MapIterator`.
So it ends with the `MapIterator` object holds lock to the `MapIterator`
object and waits for lock on `TaskMemoryManager`, and the other consumer holds
lock to `TaskMemoryManager` and waits for lock on the `MapIterator` object.
To avoid deadlock here, this patch proposes to keep reference to the page
to free and free it after releasing the lock of `MapIterator`.
## How was this patch tested?
Added test and manually test by running the test 100 times to make sure
there is no deadlock.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/viirya/spark-1 SPARK-26265
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/23272.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #23272
----
commit 25e8e068047b714f27706399e1e6c03c338ac178
Author: Liang-Chi Hsieh <viirya@...>
Date: 2018-12-10T07:59:09Z
Fix deadlock in BytesToBytesMap.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
