[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486427#comment-13486427
]
Cheolsoo Park commented on PIG-3015:
------------------------------------
Hi Joseph,
Thank you very much for opening the jira. I have recently worked on AvroStorage
by myself, and I totally agree with you. Since you already have code to
contribute, this is even better. :-)
As part of re-write, I would also like to propose to migrate AvroStorge from
Piggybank to the core Pig. I have 2 reasons for this:
# AvroStorage is widely used, so it makes sense to include it in the core Pig
rather than in Piggybank.
# Until migration is complete, we can maintain both versions (new one in core
Pig and old one in Piggybank) to avoid breaking backward compatibility. Another
motivation for re-write to me is to clean up funny options that the current
AvroStorage has. So I think that it's unavoidable to break backward
compatibility.
I asked this question on the [user mailing
list|http://mail-archives.apache.org/mod_mbox/pig-user/201208.mbox/%3C27EE5059-F811-4E19-B1A3-951B4BB3BDDF%40hortonworks.com%3E]
a while ago, and nobody disagreed. But please let me know if anyone has
objections.
To start with, I am wondering if you can post your code as a patch to this jira
and the review board. Assuming that we're going to move AvroStorage to the core
Pig, you can probably create a new package called
"org.apache.pig.backend.hadoop.avro" and add your code there. If you could
break your patch into smaller pieces and attach them to sub-tasks of this jira,
that would be helpful too.
Please let me know what you think.
Thanks!
> Rewrite of AvroStorage
> ----------------------
>
> Key: PIG-3015
> URL: https://issues.apache.org/jira/browse/PIG-3015
> Project: Pig
> Issue Type: Improvement
> Components: piggybank
> Reporter: Joseph Adler
>
> The current AvroStorage implementation has a lot of issues: it requires old
> versions of Avro, it copies data much more than needed, and it's verbose and
> complicated. (One pet peeve of mine is that old versions of Avro don't
> support Snappy compression.)
> I rewrote AvroStorage from scratch to fix these issues. In early tests, the
> new implementation is significantly faster, and the code is a lot simpler.
> Rewriting AvroStorage also enabled me to implement support for Trevni.
> I'm opening this ticket to facilitate discussion while I figure out the best
> way to contribute the changes back to Apache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
