[ https://issues.apache.org/jira/browse/SPARK-10677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Or resolved SPARK-10677. ------------------------------- Resolution: Won't Fix > UnsafeExternalSorter should atomically release and acquire > ---------------------------------------------------------- > > Key: SPARK-10677 > URL: https://issues.apache.org/jira/browse/SPARK-10677 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.5.0 > Reporter: Andrew Or > Assignee: Andrew Or > Priority: Blocker > > We have code like the following: > {code} > private void acquireNewPage() throws IOException { > final long memoryAcquired = > shuffleMemoryManager.tryToAcquire(pageSizeBytes); > if (memoryAcquired < pageSizeBytes) { > shuffleMemoryManager.release(memoryAcquired); > spill(); > final long memoryAcquiredAfterSpilling = > shuffleMemoryManager.tryToAcquire(pageSizeBytes); > if (memoryAcquiredAfterSpilling != pageSizeBytes) { > shuffleMemoryManager.release(memoryAcquiredAfterSpilling); > throw new IOException("Unable to acquire " + pageSizeBytes + " bytes > of memory"); > } > } > ... > {code} > Context: in this code we're trying to acquire a new page. If the memory > request fails, we spill and try again. If the second memory request still > fails, then we throw an exception. > Problem: When we spill, we release ALL the memory we currently hold onto only > to re-acquire some of it immediately afterwards. This creates the opportunity > for other tasks to jump in and steal our memory allocation, hence starving us. > Solution: Instead, we should make the release and acquire atomic where > possible. If we know we want exactly a page after the spill, then spill > everything minus a page. > I believe this is also the cause of SPARK-10474, where we fail to acquire > memory for the pointer array immediately after spilling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org