[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13544222#comment-13544222
]
Russell Jurney commented on PIG-3015:
-------------------------------------
Joe, some comments on handling errors in PIG-3059:
Regarding Avro, in reading
https://github.com/apache/avro/blob/trunk/lang/java/avro/src/main/java/org/apache/avro/file/DataFileReader.java
- it looks like you can still sync to the next record under most bad reads. We
should do so.
You're right about a bad sync halting things, but in the case of a bad sync -
you might try advancing by some amount using seek() and then sync'ing again? I
think this would work. I could be wrong, but in looking how seeks work - I
think that would be ok. Kinda neat, maybe? Worst case, we would only throw out
inputsplits on a bad sync(), not a bad read(). length() should help, as might
pastSync(), skip() and available()
> Rewrite of AvroStorage
> ----------------------
>
> Key: PIG-3015
> URL: https://issues.apache.org/jira/browse/PIG-3015
> Project: Pig
> Issue Type: Improvement
> Components: piggybank
> Reporter: Joseph Adler
> Assignee: Joseph Adler
> Attachments: PIG-3015-2.patch, PIG-3015-3.patch, PIG-3015-4.patch,
> PIG-3015-5.patch
>
>
> The current AvroStorage implementation has a lot of issues: it requires old
> versions of Avro, it copies data much more than needed, and it's verbose and
> complicated. (One pet peeve of mine is that old versions of Avro don't
> support Snappy compression.)
> I rewrote AvroStorage from scratch to fix these issues. In early tests, the
> new implementation is significantly faster, and the code is a lot simpler.
> Rewriting AvroStorage also enabled me to implement support for Trevni (as
> TrevniStorage).
> I'm opening this ticket to facilitate discussion while I figure out the best
> way to contribute the changes back to Apache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
