[ https://issues.apache.org/jira/browse/SPARK-12251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15144988#comment-15144988 ]
Josh Rosen commented on SPARK-12251: ------------------------------------ It's off by default; the 1.6.0 documentation was wrong and this has been corrected for 1.61. For "certain operations", this means places in Spark SQL which allocate MemoryBlock / MemoryPage. Grep through the code and look for call sites of TaskMemoryManager.allocatePage(), etc. if you want to get a comprehensive picture of where this is used. > Document Spark 1.6's off-heap memory configurations and add config validation > ----------------------------------------------------------------------------- > > Key: SPARK-12251 > URL: https://issues.apache.org/jira/browse/SPARK-12251 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Reporter: Josh Rosen > Assignee: Josh Rosen > Fix For: 1.6.0 > > > We need to document the new off-heap memory limit configurations which were > added in Spark 1.6, add simple configuration validation (for instance, you > shouldn't be able to enable off-heap execution when the off-heap memory limit > is zero), and alias the old and confusing `spark.unsafe.offHeap` > configuration to something that lives in the `spark.memory` namespace. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org