[
https://issues.apache.org/jira/browse/HADOOP-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857620#action_12857620
]
Hong Tang commented on HADOOP-6708:
-----------------------------------
bq. So does this mean that the API for TFile could be changed without
complication to accept/return long values?
yes, if by "complication" you mean wire compatibility. You may also need to
remove the checks for length violations in various places. Note that there is
still a key length restriction of 64KB.
bq. How does TFile support length skipping if you don't pre-declare the lengths?
it uses chunk encoding. The whole value stream is encoded with a chain of
chunks. We use an internal buffer to accumulate small writes and once full,
flush out the buffer as one chunk. Each chunk is length prefixed. All but the
terminal chunk have their lengths written out as negative integers. The chunk
size is controlled by parameter "tfile.io.chunk.size" (default to 1MB). When
skipping, it skips chunk by chunk.
> New file format for very large records
> --------------------------------------
>
> Key: HADOOP-6708
> URL: https://issues.apache.org/jira/browse/HADOOP-6708
> Project: Hadoop Common
> Issue Type: New Feature
> Components: io
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: lobfile.pdf
>
>
> A file format that handles multi-gigabyte records efficiently, with lazy disk
> access
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
