[
https://issues.apache.org/jira/browse/SPARK-49133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongze Zhang updated SPARK-49133:
---------------------------------
Summary: Make MemoryConsumer#used atomic (was: Make MemoryConsumer
thread-safe)
> Make MemoryConsumer#used atomic
> -------------------------------
>
> Key: SPARK-49133
> URL: https://issues.apache.org/jira/browse/SPARK-49133
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.5.1
> Reporter: Hongze Zhang
> Priority: Major
>
> TaskMemoryManager is thread-safe however MemoryConsumer is not.
> Thinking of multiple threads are allocating memory in the same task (it's a
> corner case but could be adopted for certain purposes from user side), it
> leads to practice that developer should add a lock in the caller code of
> MemoryConsumer. However since it's common that one memory consumer spills
> another, using lock in user code could easily causes ABBA dead lock, e.g.,
> consumer A locks itself to acquire memory from task memory manager, while
> consumer B locks task memory manager then spills A.
> A solution could be done in the Spark side to ensure basic thread safety of
> the MemoryConsumer, so that users don't have to lock the entire memory
> consumer which could lead to the ABBA dead lock.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
