kumar vishal created CARBONDATA-2584:
----------------------------------------
Summary: CarbonData Local Dictionary Support
Key: CARBONDATA-2584
URL: https://issues.apache.org/jira/browse/CARBONDATA-2584
Project: CarbonData
Issue Type: New Feature
Reporter: kumar vishal
Currently CarbonData supports global dictionary or No-Dictionary (Plain-Text
stored in LV format) for storing dimension column data.
*Bottleneck with Global Dictionary*
It’s difficult for user to determine whether the column should be dictionary or
not if number of columns in table is high.
Global dictionary generation generally slows down the load process.
Multiple IO operations are made during load even though dictionary already
exists.
During query, multiple IO operations done for reading dictionary files and
carbondata files.
*Bottleneck with No-Dictionary*
Storage size is high as we store the data in LV format
Query on No-Dictionary column is slower as data read/processed is more
Filtering is slower on No-Dictionary columns as number of comparison is high
Memory footprint is high
*The above bottlenecks can be solved by generating dictionary for low
cardinality columns at each blocklet level, which will help to achieve below
benefits:*
Reduces the extra IO operations read/write on the dictionary files generated in
case of global dictionary.
It will eliminate the problem for user to identify the dictionary columns when
the number of columns are more in a table.
It helps in getting more compression on dimension columns with less cardinality.
Filter queries and full scan queries on No-dictionary columns with local
dictionary will be faster as filter will be done on encoded data.
It will help in reducing the store size and memory footprint as only unique
values will be stored {color:#000000}as {color}part of local dictionary and
corresponding data will be stored as encoded data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
