[
https://issues.apache.org/jira/browse/HIVE-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556977#comment-13556977
]
Joydeep Sen Sarma commented on HIVE-3874:
-----------------------------------------
couple of observations:
- one use case mentioned is external indices. but in my experience, secondary
index pointers have little correlation with the primary key ordering. If the
use case is to speed up secondary index lookups - then one will be forced to
consider smaller row groups. At that point - this starts breaking down - large
row groups are good for scanning for scanning and compression - but poor for
lookups.
a possible way out is to do a two level structure - stripes or chunks as the
unit of compression (column dictionaries maintained at this level), but a
smaller unit for row-groups (a single 250MB chunk has many smaller row groups
all encoded using a common dictionary). this can give a good balance of
compression and lookup capabilities.
at this point - i believe - we are closer to a HFile data structure - and I
think converging HFile* so it works well for Hive would be a great goal. A lot
of people would benefit from letting HBase do indexing and let Hive/Hadoop
chomp on HBase produced HFiles.
- another use case mentioned is pruning based on column ranges. Once again -
these use cases typically only benefit columns whose values are correlated with
the primary row order. Timestamps and anything correlated with timestamps do
benefit - but others don't. In systems like Netezza - this is used as a
substitute for partitioning.
The issue is that pruning at the block level is not enough - because one has
already generated large number splits for MR to chomp on. And large number
splits make processing really slow - even if everything is pruned out inside
each mapper. Unless that issue is addressed - most users would end up
repartitioning their (using Hive's dynamic partitioning) based on column values
- and the whole column range stuff would largely not come in use.
> Create a new Optimized Row Columnar file format for Hive
> --------------------------------------------------------
>
> Key: HIVE-3874
> URL: https://issues.apache.org/jira/browse/HIVE-3874
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Attachments: OrcFileIntro.pptx
>
>
> There are several limitations of the current RC File format that I'd like to
> address by creating a new format:
> * each column value is stored as a binary blob, which means:
> ** the entire column value must be read, decompressed, and deserialized
> ** the file format can't use smarter type-specific compression
> ** push down filters can't be evaluated
> * the start of each row group needs to be found by scanning
> * user metadata can only be added to the file when the file is created
> * the file doesn't store the number of rows per a file or row group
> * there is no mechanism for seeking to a particular row number, which is
> required for external indexes.
> * there is no mechanism for storing light weight indexes within the file to
> enable push-down filters to skip entire row groups.
> * the type of the rows aren't stored in the file
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
