[
https://issues.apache.org/jira/browse/SPARK-9973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheng Lian updated SPARK-9973:
------------------------------
Fix Version/s: 1.5.0
> Wrong initial size of in-memory columnar buffers
> ------------------------------------------------
>
> Key: SPARK-9973
> URL: https://issues.apache.org/jira/browse/SPARK-9973
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.0
> Reporter: xukun
> Assignee: xukun
> Fix For: 1.5.0
>
>
> Two much memory is allocated for in-memory columnar buffers. The
> {{initialSize}} argument in {{ColumnBuilder.initialize}} is the initial
> number of rows rather than bytes, but the value passed in in
> {{InMemoryColumnarTableScan}} is the latter:
> {code}
> // Class InMemoryColumnarTableScan
> val initialBufferSize = columnType.defaultSize * batchSize
> ColumnBuilder(attribute.dataType, initialBufferSize, attribute.name,
> useCompression)
> {code}
> Then it's converted to byte size again by multiplying
> {{columnType.defaultSize}}:
> {code}
> // Class BasicColumnBuilder
> buffer = ByteBuffer.allocate(4 + size * columnType.defaultSize)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
