[ https://issues.apache.org/jira/browse/SPARK-3018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Reynold Xin updated SPARK-3018: ------------------------------- Description: BlockFetcherIterator retains ManagedBuffers returned by BlockClient.fetchBlocks. Those buffers are released when the iterators are traversed fully. In the case of task failures or completion without exhausting the iterator, this could lead to memory leak. (was: BlockFetcherIterator retains ReferenceCountedBuffers returned by client.fetchBlocks. Those buffers are released when the iterators are traversed fully. In the case of task failures or completion without exhausting the iterator, this could lead to memory leak.) > Release all ManagedBuffers upon task completion/failure > ------------------------------------------------------- > > Key: SPARK-3018 > URL: https://issues.apache.org/jira/browse/SPARK-3018 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core > Reporter: Reynold Xin > Assignee: Reynold Xin > > BlockFetcherIterator retains ManagedBuffers returned by > BlockClient.fetchBlocks. Those buffers are released when the iterators are > traversed fully. In the case of task failures or completion without > exhausting the iterator, this could lead to memory leak. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org