[ https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13509992#comment-13509992 ]
Joseph Adler commented on PIG-3015: ----------------------------------- I think that approach makes sense; each object in a file should be wrapped in a Tuple. Suppose that a file example.avro contained the data: {[1, 2, 3, 4, 5]} {[6, 7, 8, 9, 10]} and had this schema: {"name" : "IntArray", "type" : "array", "items" : "int"}, and we loaded this as A = LOAD 'example.avro' USING AvroStorage; The bag A would have the Pig schema A:{(IntArray:{(int)})}; it would contain two tuples, which would in turn each contain one bag of integers. Does that sound correct? If so, I'll go implement that. > Rewrite of AvroStorage > ---------------------- > > Key: PIG-3015 > URL: https://issues.apache.org/jira/browse/PIG-3015 > Project: Pig > Issue Type: Improvement > Components: piggybank > Reporter: Joseph Adler > Assignee: Joseph Adler > Attachments: PIG-3015.patch > > > The current AvroStorage implementation has a lot of issues: it requires old > versions of Avro, it copies data much more than needed, and it's verbose and > complicated. (One pet peeve of mine is that old versions of Avro don't > support Snappy compression.) > I rewrote AvroStorage from scratch to fix these issues. In early tests, the > new implementation is significantly faster, and the code is a lot simpler. > Rewriting AvroStorage also enabled me to implement support for Trevni. > I'm opening this ticket to facilitate discussion while I figure out the best > way to contribute the changes back to Apache. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira