[ https://issues.apache.org/jira/browse/SPARK-15357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Davies Liu reassigned SPARK-15357: ---------------------------------- Assignee: Davies Liu > Cooperative spilling should check consumer memory mode > ------------------------------------------------------ > > Key: SPARK-15357 > URL: https://issues.apache.org/jira/browse/SPARK-15357 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 1.6.0 > Reporter: Andrew Or > Assignee: Davies Liu > > In TaskMemoryManager.java: > {code} > for (MemoryConsumer c: consumers) { > if (c != consumer && c.getUsed() > 0) { > try { > long released = c.spill(required - got, consumer); > if (released > 0 && mode == tungstenMemoryMode) { > got += memoryManager.acquireExecutionMemory(required - got, > taskAttemptId, mode); > if (got >= required) { > break; > } > } > } catch(...) { ... } > } > } > } > {code} > Currently, when non-tungsten consumers acquire execution memory, they may > force other tungsten consumers to spill and then NOT use the freed memory. A > better way to do this is to incorporate the memory mode in the consumer > itself and spill only those with matching memory modes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org