[ https://issues.apache.org/jira/browse/SPARK-9973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Cheng Lian updated SPARK-9973: ------------------------------ Summary: Wrong initial size of in-memory columnar buffers (was: wrong buffle size) > Wrong initial size of in-memory columnar buffers > ------------------------------------------------ > > Key: SPARK-9973 > URL: https://issues.apache.org/jira/browse/SPARK-9973 > Project: Spark > Issue Type: Bug > Components: SQL > Reporter: xukun > Assignee: xukun > > When cache table in memory in spark sql, we allocate too more memory. > InMemoryColumnarTableScan.class > val initialBufferSize = columnType.defaultSize * batchSize > ColumnBuilder(attribute.dataType, initialBufferSize, attribute.name, > useCompression) > BasicColumnBuilder.class > buffer = ByteBuffer.allocate(4 + size * columnType.defaultSize) > So total allocate size is (4+ size * columnType.defaultSize * > columnType.defaultSize), We change it to 4+ size * columnType.defaultSize. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org