[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheolsoo Park updated PIG-3015:
-------------------------------
Attachment: good.avro
bad.avro
Test.java
TestInput.java
Hi Scott,
Thank you very much. That makes sense. After several tries and errors, I
managed to "correctly" corrupt a data block and was able to verify the recovery.
The output from 'java-tool.jar tojson bad.avro' is as follows:
{code}
Caused by: java.io.IOException: Block read partially, the data may be corrupt
at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:194)
... 3 more
{code}
The output from my test program is as follows:
{code}
next(): 685
tell(): 8196
next(): 686
tell(): 8196
hasNext() or next() failed
tell(): 8240
next(): 2656
tell(): 16432
next(): 2657
tell(): 16432
{code}
The data are sequential integers (0 ~ 1M). Here is the number of lost integers
due to a single corrupted data block with different sync intervals:
||Sync interval in bytes||Num. of lost values||
|32|1970|
|16,000|5389|
In summary,
* Avro can recover from a data block corruption but cannot from a sync marker
corruption.
* The amount of data loss depends on the sync interval. By default, it's 16KB,
but it can vary from 32 to 2^30 bytes. The greater the sync interval is, the
more data loss is.
I am attaching my test program and input files if anyone's interested.
Thanks!
> Rewrite of AvroStorage
> ----------------------
>
> Key: PIG-3015
> URL: https://issues.apache.org/jira/browse/PIG-3015
> Project: Pig
> Issue Type: Improvement
> Components: piggybank
> Reporter: Joseph Adler
> Assignee: Joseph Adler
> Attachments: bad.avro, good.avro, PIG-3015-2.patch, PIG-3015-3.patch,
> PIG-3015-4.patch, PIG-3015-5.patch, TestInput.java, Test.java
>
>
> The current AvroStorage implementation has a lot of issues: it requires old
> versions of Avro, it copies data much more than needed, and it's verbose and
> complicated. (One pet peeve of mine is that old versions of Avro don't
> support Snappy compression.)
> I rewrote AvroStorage from scratch to fix these issues. In early tests, the
> new implementation is significantly faster, and the code is a lot simpler.
> Rewriting AvroStorage also enabled me to implement support for Trevni (as
> TrevniStorage).
> I'm opening this ticket to facilitate discussion while I figure out the best
> way to contribute the changes back to Apache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
