[
https://issues.apache.org/jira/browse/HIVE-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13583385#comment-13583385
]
Phabricator commented on HIVE-3874:
-----------------------------------
omalley has commented on the revision "HIVE-3874 [jira] Create a new Optimized
Row Columnar file format for Hive".
| Do you want to have a simple test for HIVE-4015 as part of this patch ?
Since the test will fail until I change the grammer, I think it would be
better to wait until they will pass.
| 1. Can you add more comments - specially, in the class/interface
definitions Writer/TreeWriter/StreamFactory to name a few.
Sure.
| 2. Can column statistics be made optional ? (can be a follow-up)
They are very cheap in practice, but it wouldn't be hard to disable them.
| 3. This has a lot of new code - I mean, is it possible to use some of the
constructs which are already there - for eg. RedBlackTrees, RLE etc. Can you
use some existing implementations instead of writing these from scratch ?
I'm a big fan of not writing new code when I can just use someone else's.
That said, it is always a trade off when evaluating when to re-use versus write
new that involves comparing the requirements to what the other code provides.
I'm not aware of any open source java red-black trees that work on primitives
without allocating multiple objects per an entry. Do you have a suggestion?
The RLE is very specific to ORC format and it didn't seem like any of the
implementations available were a good match. I'm also considering how to do a
better delta and small integer encoding, but I'll do that in a follow up jira.
| Right now, the RLE is fixed. Should it be pluggable ? I mean - we can have
a different scheme to store deltas.
I think that pluggable will create compatibility problems, since you won't be
able to read an ORC file that was written by a different plugin.
My preferred direction is to use the ColumnEncoding to allow the Writer to
pick a different encoding based on the observed data. For example, by looking
at the first 100,000 values the writer should be able to decide if a dictionary
or direct encoding is better. We could use the same mechanism to add additional
encodings.
INLINE COMMENTS
ql/src/java/org/apache/hadoop/hive/ql/orc/OutStream.java:136-140 There is an
requirement that the codec's compress method will return false rather than take
more space than the input. Given that, if the compressed is empty, we can't
need the overflow.
ql/src/java/org/apache/hadoop/hive/ql/orc/OrcInputFormat.java:149-151 I've
removed it.
ql/src/java/org/apache/hadoop/hive/ql/orc/OrcStruct.java:307 fixed.
ql/src/java/org/apache/hadoop/hive/ql/orc/WriterImpl.java:561-562 I've added
the size of the dictionary to the estimate of the memory size, which should be
better.
ql/src/java/org/apache/hadoop/hive/ql/orc/BitFieldReader.java:18 i managed to
move the directory in the wrong place. fixed.
REVISION DETAIL
https://reviews.facebook.net/D8529
To: JIRA, omalley
Cc: kevinwilfong, njain
> Create a new Optimized Row Columnar file format for Hive
> --------------------------------------------------------
>
> Key: HIVE-3874
> URL: https://issues.apache.org/jira/browse/HIVE-3874
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Attachments: hive.3874.2.patch, HIVE-3874.D8529.1.patch,
> HIVE-3874.D8529.2.patch, HIVE-3874.D8529.3.patch, OrcFileIntro.pptx, orc.tgz
>
>
> There are several limitations of the current RC File format that I'd like to
> address by creating a new format:
> * each column value is stored as a binary blob, which means:
> ** the entire column value must be read, decompressed, and deserialized
> ** the file format can't use smarter type-specific compression
> ** push down filters can't be evaluated
> * the start of each row group needs to be found by scanning
> * user metadata can only be added to the file when the file is created
> * the file doesn't store the number of rows per a file or row group
> * there is no mechanism for seeking to a particular row number, which is
> required for external indexes.
> * there is no mechanism for storing light weight indexes within the file to
> enable push-down filters to skip entire row groups.
> * the type of the rows aren't stored in the file
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
