[ https://issues.apache.org/jira/browse/HIVE-2097?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160726#comment-13160726 ]
alex gemini commented on HIVE-2097: ----------------------------------- another issue is about efficient serialization/deserialization,for the same example above,assume every gender,age,region have 100 message equally store in one dfs block,in gender column we store value like this:{'male'}[1-60k]{'female'}[60k+1 - 120k],age column look like this:{21}[1-3k]{22}[3k+1 - 6k]{23}[6k+1 - 9k],and region column is like:{'LA'}[1-300]{'NY'}[301-600]. When we issue a query on a single table like :select sum(age) from logs where region='LA' and age=30,we count every column represented at 'select,where,group' clause,so we know the last column means lowest selectivity(in this example is region),we find the region value={'LA'}[(1-300),(30k+1 - 30k+300),(60k+1 -60k+300)....] and 'NY' value={'NY'}[(301-600),(30k+301 - 30k+600),(60k+301 -60k+600)....] we just need to deserialization it but don't need to decompression it because we know the lowest selectivity column,then we organize inputSplit's key like {[age='21'][region='LA']} and value is {(1-300),(30k+1 - 30k+300),(60k+1 -60k+300)....},this inputSplit key and value is unique per dfs block because we already sort column by selectivity,the lowest selectivity column presented at (select,where,group) must be unique. > Explore mechanisms for better compression with RC Files > ------------------------------------------------------- > > Key: HIVE-2097 > URL: https://issues.apache.org/jira/browse/HIVE-2097 > Project: Hive > Issue Type: Improvement > Components: Query Processor, Serializers/Deserializers > Reporter: Krishna Kumar > Assignee: Krishna Kumar > Priority: Minor > > Optimization of the compression mechanisms used by RC File to be explored. > Some initial ideas > > 1. More efficient serialization/deserialization based on type-specific and > storage-specific knowledge. > > For instance, storing sorted numeric values efficiently using some delta > coding techniques > 2. More efficient compression based on type-specific and storage-specific > knowledge > Enable compression codecs to be specified based on types or individual > columns > 3. Reordering the on-disk storage for better compression efficiency. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira