Repository: incubator-carbondata Updated Branches: refs/heads/master 9f38a3dde -> e441ab0d4
Update file structure info as per V3 format definition Project: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/commit/b128868c Tree: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/tree/b128868c Diff: http://git-wip-us.apache.org/repos/asf/incubator-carbondata/diff/b128868c Branch: refs/heads/master Commit: b128868cd80aab2704b7e9958af6bce67739c92a Parents: 9f38a3d Author: chenliang613 <chenliang...@huawei.com> Authored: Wed Mar 22 16:32:06 2017 +0530 Committer: ravipesala <ravi.pes...@gmail.com> Committed: Thu Mar 23 10:02:32 2017 +0530 ---------------------------------------------------------------------- docs/file-structure-of-carbondata.md | 21 ++++++++++++++------ docs/images/carbon_data_file_structure_new.png | Bin 78374 -> 9477 bytes docs/images/carbon_data_format_new.png | Bin 73708 -> 35510 bytes 3 files changed, 15 insertions(+), 6 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/b128868c/docs/file-structure-of-carbondata.md ---------------------------------------------------------------------- diff --git a/docs/file-structure-of-carbondata.md b/docs/file-structure-of-carbondata.md index 63e34ec..bfbcee4 100644 --- a/docs/file-structure-of-carbondata.md +++ b/docs/file-structure-of-carbondata.md @@ -17,20 +17,29 @@ under the License. --> -# CarbonData File Structure +# CarbonData File Structure -CarbonData files contain groups of data called blocklets, along with all required information like schema, offsets and indices etc, in a file footer, co-located in HDFS. +CarbonData files contain groups of data called blocklets, along with all required information like schema, offsets and indices etc, in a file header and footer, co-located in HDFS. The file footer can be read once to build the indices in memory, which can be utilized for optimizing the scans and processing for all subsequent queries. -Each blocklet in the file is further divided into chunks of data called data chunks. Each data chunk is organized either in columnar format or row format, and stores the data of either a single column or a set of columns. All blocklets in a file contain the same number and type of data chunks. +### Understanding CarbonData File Structure +* Block : It would be as same as HDFS block, CarbonData creates one file for each data block, user can specify TABLE_BLOCKSIZE during creation table. Each file contains File Header, Blocklets and File Footer.  -Each data chunk contains multiple groups of data called as pages. There are three types of pages. +* File Header : It contains CarbonData file version number, list of column schema and schema updation timestamp. +* File Footer : it contains Number of rows, segmentinfo ,all blockletsâ info and index, you can find the detail from the below diagram. +* Blocklet : Rows are grouped to form a blocklet, the size of the blocklet is configurable and default size is 64MB, Blocklet contains Column Page groups for each column. +* Column Page Group : Data of one column and it is further divided to pages, it is guaranteed to be contiguous in file. +* Page : It has the data of one column and the number of row is fixed to 32000 size. -* Data Page: Contains the encoded data of a column/group of columns. + + +### Each page contains three types of data +* Data Page: Contains the encoded data of a column of columns. * Row ID Page (optional): Contains the row ID mappings used when the data page is stored as an inverted index. * RLE Page (optional): Contains additional metadata used when the data page is RLE coded. - + + http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/b128868c/docs/images/carbon_data_file_structure_new.png ---------------------------------------------------------------------- diff --git a/docs/images/carbon_data_file_structure_new.png b/docs/images/carbon_data_file_structure_new.png index 3f9241b..1c6f22b 100644 Binary files a/docs/images/carbon_data_file_structure_new.png and b/docs/images/carbon_data_file_structure_new.png differ http://git-wip-us.apache.org/repos/asf/incubator-carbondata/blob/b128868c/docs/images/carbon_data_format_new.png ---------------------------------------------------------------------- diff --git a/docs/images/carbon_data_format_new.png b/docs/images/carbon_data_format_new.png index 9d0b194..f0fc553 100644 Binary files a/docs/images/carbon_data_format_new.png and b/docs/images/carbon_data_format_new.png differ
