[
https://issues.apache.org/jira/browse/AVRO-160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12790429#action_12790429
]
Doug Cutting commented on AVRO-160:
-----------------------------------
Philip> To be clear, you're preserving appendability and splittabability, but
ditching schema-evolution-for-appendability.
The other thing we're ditching is computed metadata. Previously you could
store the total count of items or an index of items in the metadata at end of
file. This is certainly useful, but complicates things. Side files are the
alternative, and the only problem with them is that adding more files is
HDFS-unfriendly.
Philip> How big are blocks (rule of thumb?)?
64kB. Just big enough for compression to work and for per-block overheads to
be insignificant.
Scott> The last thing I want to clarify is the sync marker behavior.
The assumption is that with a 16-byte marker there will never be a collision.
We've never seen a collision yet with this approach in the lifetime of Hadoop.
The odds are in our favor: it would take a file with well over an exabyte (1000
petabytes, 1M terabytes) before this is near likely. I suspect the chances are
significantly greater that all three replicas of a file in HDFS will become
corrupt. The significant advantage of this approach is that no escape
processing is required: bytes can travel to and from the file unexamined in
bulk.
The count of items and bytes per block provide a checksum of sorts: if the next
sync marker is not found where expected then the file is either corrupt or a
collision has occurred. Corruption should be noticed by the filesystem. A
collision does not cause data loss, since the file can still be processed from
its start. So I suppose one could specify that, if the next sync marker is not
found where expected then one should scan back earlier or later in the file (as
appropriate) to find a different sync marker and start processing there.
> file format should be friendly to streaming
> -------------------------------------------
>
> Key: AVRO-160
> URL: https://issues.apache.org/jira/browse/AVRO-160
> Project: Avro
> Issue Type: Improvement
> Components: spec
> Reporter: Doug Cutting
> Assignee: Doug Cutting
>
> It should be possible to stream through an Avro data file without seeking to
> the end.
> Currently the interpretation is that schemas written to the file apply to all
> entries before them. If this were changed so that they instead apply to all
> entries that follow, and the initial schema is written at the start of the
> file, then streaming could be supported.
> Note that the only change permitted to a schema as a file is written is to,
> if it is a union, to add new branches at the end of that union. If it is not
> a union, no changes may be made. So it is still the case that the final
> schema in a file can read every entry in the file and thus may be used to
> randomly access the file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
