[ https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-160: ------------------------------ Attachment: AVRO-160.patch Here's a patch that implements this for Java and updates the spec. It's slightly different from what's proposed above. A file is a header followed by zero or more blocks. These each contain: - header -- magic -- metadata -- sync - block -- length -- data -- sync Thus every block is both preceded and followed by a sync marker. I also split the Java data file reading code into two classes: a base class that only permits sequential access, and a subclass that supports random access. This will permit us to, e.g., process standard input. > file format should be friendly to streaming > ------------------------------------------- > > Key: AVRO-160 > URL: https://issues.apache.org/jira/browse/AVRO-160 > Project: Avro > Issue Type: Improvement > Components: spec > Reporter: Doug Cutting > Assignee: Doug Cutting > Attachments: AVRO-160.patch > > > It should be possible to stream through an Avro data file without seeking to > the end. > Currently the interpretation is that schemas written to the file apply to all > entries before them. If this were changed so that they instead apply to all > entries that follow, and the initial schema is written at the start of the > file, then streaming could be supported. > Note that the only change permitted to a schema as a file is written is to, > if it is a union, to add new branches at the end of that union. If it is not > a union, no changes may be made. So it is still the case that the final > schema in a file can read every entry in the file and thus may be used to > randomly access the file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.