[
https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12769378#action_12769378
]
Scott Carey commented on AVRO-160:
----------------------------------
{quote}
Yes, if we kept a global block index, we could avoid this scan. However since
HDFS blocks are ~64MB, and Avro file blocks are ~64k, the scan is less than a
tenth of a percent of the overall map cost, so this is perhaps not a worthwhile
optimization.
{quote}
I had imagined that blocks were closer to 1MB for whatever reason. I had
actually imagined that many writers would want to minimize the metadata blocks
if they were not changing their schema. I had also imagined that there might
be some use case where a single tuple was MBs in size (perhaps a large array,
or a byte[] representing some sort of media, or the text content of a
document). At that point, should there be a metadata block per tuple?
There are a lot of trade-offs with respect to block sizes.
At what size does the metadata block overhead represent larger overhead than
storing extra information per tuple like Thrift/Protobuf? Put another way, if
blocks are too small and schemas large, Avro's size advantage could be lost.
At what size does compression become less effective? Are larger blocks better
for streaming read/write performance?
An index of all of a 1TB file's block locations is fairly small if the blocks
are 4MB, but very large if they are 64KB. So the usefulness and costs of such
an index varies.
If the expectation is smaller blocks (around 64K) then seeking to the next will
be cheap and my main concern is minimizing the metadata block size.
If there is an expectation that large block sizes will be used, i'm concerned
with the time it takes to find the start of a block and the efficiency of file
splitting.
> file format should be friendly to streaming
> -------------------------------------------
>
> Key: AVRO-160
> URL: https://issues.apache.org/jira/browse/AVRO-160
> Project: Avro
> Issue Type: Improvement
> Components: spec
> Reporter: Doug Cutting
>
> It should be possible to stream through an Avro data file without seeking to
> the end.
> Currently the interpretation is that schemas written to the file apply to all
> entries before them. If this were changed so that they instead apply to all
> entries that follow, and the initial schema is written at the start of the
> file, then streaming could be supported.
> Note that the only change permitted to a schema as a file is written is to,
> if it is a union, to add new branches at the end of that union. If it is not
> a union, no changes may be made. So it is still the case that the final
> schema in a file can read every entry in the file and thus may be used to
> randomly access the file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
