[
https://issues.apache.org/jira/browse/SPARK-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kazuaki Ishizaki updated SPARK-14098:
-------------------------------------
Description:
[Here|https://docs.google.com/document/d/1-2BnW5ibuHIeQzmHEGIGkEcuMUCTk87pmPis2DKRg-Q/edit?usp=sharing]
is a design document for this change (***TODO: Update the document***).
This JIRA implements a new in-memory cache feature used by DataFrame.cache and
Dataset.cache. The followings are basic design based on discussions with
Sameer, Weichen, Xiao, Herman, and Nong.
* Use ColumnarBatch with ColumnVector that are common data representations for
columnar storage
* Use multiple compression scheme (such as RLE, intdelta, and so on) for each
ColumnVector in ColumnarBatch depends on its data typpe
* Generate code that is simple and specialized for each in-memory cache to
build an in-memory cache
* Generate code that directly reads data from ColumnVector for the in-memory
cache by whole-stage codegen.
* Enhance ColumnVector to keep UnsafeArrayData
* Use primitive-type array for primitive uncompressed data type in ColumnVector
* Use byte[] for UnsafeArrayData and compressed data
Based on this design, this JIRA generates two kinds of Java code for
DataFrame.cache()/Dataset.cache()
* Generate Java code to build CachedColumnarBatch, which keeps data in
ColumnarBatch
* Generate Java code to get a value of each column from ColumnarBatch
** a Get a value directly from from ColumnarBatch in code generated by whole
stage code gen (primary path)
** b Get a value thru an iterator if whole stage code gen is disabled (e.g. #
of columns is more than 100, as backup path)
was:
When DataFrame.cache() is called, data is stored as column-oriented storage in
CachedBatch. The current Catalyst generates Java program to get a value of a
column from an InternalRow that is translated from CachedBatch. This issue
generates Java code to get a value of a column from CachedBatch. While a column
for a cache may be compressed, this issue handles float and double types that
are never compressed.
Other primitive types, whose column may be compressed, will be addressed in
another entry.
> Generate Java code to build CachedColumnarBatch and get values from
> CachedColumnarBatch when DataFrame.cache() is called
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-14098
> URL: https://issues.apache.org/jira/browse/SPARK-14098
> Project: Spark
> Issue Type: Umbrella
> Components: SQL
> Reporter: Kazuaki Ishizaki
>
> [Here|https://docs.google.com/document/d/1-2BnW5ibuHIeQzmHEGIGkEcuMUCTk87pmPis2DKRg-Q/edit?usp=sharing]
> is a design document for this change (***TODO: Update the document***).
> This JIRA implements a new in-memory cache feature used by DataFrame.cache
> and Dataset.cache. The followings are basic design based on discussions with
> Sameer, Weichen, Xiao, Herman, and Nong.
> * Use ColumnarBatch with ColumnVector that are common data representations
> for columnar storage
> * Use multiple compression scheme (such as RLE, intdelta, and so on) for each
> ColumnVector in ColumnarBatch depends on its data typpe
> * Generate code that is simple and specialized for each in-memory cache to
> build an in-memory cache
> * Generate code that directly reads data from ColumnVector for the in-memory
> cache by whole-stage codegen.
> * Enhance ColumnVector to keep UnsafeArrayData
> * Use primitive-type array for primitive uncompressed data type in
> ColumnVector
> * Use byte[] for UnsafeArrayData and compressed data
> Based on this design, this JIRA generates two kinds of Java code for
> DataFrame.cache()/Dataset.cache()
> * Generate Java code to build CachedColumnarBatch, which keeps data in
> ColumnarBatch
> * Generate Java code to get a value of each column from ColumnarBatch
> ** a Get a value directly from from ColumnarBatch in code generated by whole
> stage code gen (primary path)
> ** b Get a value thru an iterator if whole stage code gen is disabled (e.g. #
> of columns is more than 100, as backup path)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
