[
https://issues.apache.org/jira/browse/HADOOP-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857588#action_12857588
]
Hong Tang commented on HADOOP-6708:
-----------------------------------
bq. Shortcomings of TFile:
bq. • Length fields are encoded as integers, not longs. This does not support
records > 2 GB.
This is an intentional restriction. All integers are in VInt/VLong format which
is fully wire compatible. You can easily make a case to request such limit be
lifted.
bq. • The expected length of the value must be precisely-known or written as
-1. I have records where I have a minimum bound on their size (more precisely:
Ahead of time, I know their length in characters but not in bytes), but not
necessarily a more precise byte count value. A goal of this format is the
ability to efficiently partially-read a character stream-oriented record, then
skip quickly to the next record. The format I propose will allow you to specify
that a record is at least a certain length. That way you can efficiently skip
most of a record and then search for a synchronization boundary after that.
(Since I expect most characters to be encoded in a single byte, but not
necessarily all of them.)
Even if you do not know the length of the record you write (namely specifying
-1 during writing), you can still efficiently skip a record (even after
partially consuming some bytes of the record). Isn't it sufficient for your
case? Searching for a synchronization boundary is very inefficient than
length-prefixed encoding.
> New file format for very large records
> --------------------------------------
>
> Key: HADOOP-6708
> URL: https://issues.apache.org/jira/browse/HADOOP-6708
> Project: Hadoop Common
> Issue Type: New Feature
> Components: io
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: lobfile.pdf
>
>
> A file format that handles multi-gigabyte records efficiently, with lazy disk
> access
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
