[
https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12768517#action_12768517
]
Doug Cutting commented on AVRO-160:
-----------------------------------
> I think you'll also have to get rid of getCount(), no? I won't miss it.
It would certainly be unavailable to a streaming reader. But the existing
reader is not streaming, but random access, in order to support splitting and
seeking.
However this raises the question of whether we should continue to also flush
the metadata on close. If so, then we could still keep a count of entries. To
support random access and schema changes, we need to be able to read the final
schema in the file. To make this efficient, we must flush the metadata at
close, otherwise we'll have to scan for the final metadata. So, entries that
appear after the last metadata would not be included in the count, but such
entries should only occur if a program crashes, and might reasonably be
considered to be after the last "commit point" and hence ignored.
So giving up on counts is also giving up on the ability to change the schema.
The motivation for putting metadata at the end rather than in a header was
precisely to permit appends that alter metadata.
So my inclination is to continue to ignore entries after the final metadata,
that it always re-iterates the penultimate metadata. Each time the schema
changes, the metadata must be re-written, to support schema changes for
streaming readers. And when the file is closed, metadata is also dumped.
To be consistent, we might prohibit appends that do not first truncate to the
end of the last metadata. A streaming reader might still read these, however.
Sigh. This is getting messy!
> file format should be friendly to streaming
> -------------------------------------------
>
> Key: AVRO-160
> URL: https://issues.apache.org/jira/browse/AVRO-160
> Project: Avro
> Issue Type: Improvement
> Components: spec
> Reporter: Doug Cutting
>
> It should be possible to stream through an Avro data file without seeking to
> the end.
> Currently the interpretation is that schemas written to the file apply to all
> entries before them. If this were changed so that they instead apply to all
> entries that follow, and the initial schema is written at the start of the
> file, then streaming could be supported.
> Note that the only change permitted to a schema as a file is written is to,
> if it is a union, to add new branches at the end of that union. If it is not
> a union, no changes may be made. So it is still the case that the final
> schema in a file can read every entry in the file and thus may be used to
> randomly access the file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
