[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493550#comment-13493550
]
Joseph Adler commented on PIG-3015:
-----------------------------------
I hate breaking backwards compatibility. (One of the reaons for doing the
rewrite is that Avro broke backwards compatibility.) But I think we have some
good reasons to do so here:
- Options for AvroStorage are very different than options for other storage
functions in Pig. In moving AvroStorage to builtin, it makes sense for
AvroStorage to behave as close as possible to PigStorage, etc.
- The huge number of crazy options make the code slow and complicated.
- There are good workarounds for many changes in the options. For example, all
the weird stuff about selecting a schema using an index could be easily changed
to explicit schema definitions.
- It gets harder to make changes with time. This is probably the best
opportunity to make the options simpler and clearer.
> Rewrite of AvroStorage
> ----------------------
>
> Key: PIG-3015
> URL: https://issues.apache.org/jira/browse/PIG-3015
> Project: Pig
> Issue Type: Improvement
> Components: piggybank
> Reporter: Joseph Adler
> Assignee: Joseph Adler
>
> The current AvroStorage implementation has a lot of issues: it requires old
> versions of Avro, it copies data much more than needed, and it's verbose and
> complicated. (One pet peeve of mine is that old versions of Avro don't
> support Snappy compression.)
> I rewrote AvroStorage from scratch to fix these issues. In early tests, the
> new implementation is significantly faster, and the code is a lot simpler.
> Rewriting AvroStorage also enabled me to implement support for Trevni.
> I'm opening this ticket to facilitate discussion while I figure out the best
> way to contribute the changes back to Apache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
