[
https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790516#action_12790516
]
Philip Zeyliger commented on AVRO-160:
--------------------------------------
bq. This problem is really an HDFS problem. A future feature for HDFS should
probably make side files cheap rather than complicating issues for file type
implementations (multiple data streams in one file, side files are just
optional extra data streams - crc's, indexes, parity, etc). I should probably
open a ticket for that.
Several applications can work with multiple files. If you want per-block extra
stuff (indexes, bloom-filters, whatever), side files work less well because
they lose being co-located with the original data. And that's fine;
applications that need that will develop more complicated file formats.
> file format should be friendly to streaming
> -------------------------------------------
>
> Key: AVRO-160
> URL: https://issues.apache.org/jira/browse/AVRO-160
> Project: Avro
> Issue Type: Improvement
> Components: spec
> Reporter: Doug Cutting
> Assignee: Doug Cutting
>
> It should be possible to stream through an Avro data file without seeking to
> the end.
> Currently the interpretation is that schemas written to the file apply to all
> entries before them. If this were changed so that they instead apply to all
> entries that follow, and the initial schema is written at the start of the
> file, then streaming could be supported.
> Note that the only change permitted to a schema as a file is written is to,
> if it is a union, to add new branches at the end of that union. If it is not
> a union, no changes may be made. So it is still the case that the final
> schema in a file can read every entry in the file and thus may be used to
> randomly access the file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
