[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13509296#comment-13509296
]
Joseph Adler commented on PIG-3015:
-----------------------------------
I made most of the recommended changes (thanks for looking this over), and have
a follow up question:
I have always assumed that AvroStorage was designed to be used with Hadoop
sequence files that contained a series of records, so I implemented AvroStorage
to only work with a file in this format. Are there cases where the highest
level schema for a file will be another type? If so... what does that mean for
pig? Is there one record per file?
Here's a specific example: suppose that we have this schema:
{"name" : "IntArray", "type" : "array", "items" : "int"}
Suppose that we have 3 files to load, each with this schema, each containing an
array of 10 integers. Should we load this into pig as a single bag with 30
integers? A bag containing three bags (each, in turn, containing 10 integers)?
Or reject this file entirely?
> Rewrite of AvroStorage
> ----------------------
>
> Key: PIG-3015
> URL: https://issues.apache.org/jira/browse/PIG-3015
> Project: Pig
> Issue Type: Improvement
> Components: piggybank
> Reporter: Joseph Adler
> Assignee: Joseph Adler
> Attachments: PIG-3015.patch
>
>
> The current AvroStorage implementation has a lot of issues: it requires old
> versions of Avro, it copies data much more than needed, and it's verbose and
> complicated. (One pet peeve of mine is that old versions of Avro don't
> support Snappy compression.)
> I rewrote AvroStorage from scratch to fix these issues. In early tests, the
> new implementation is significantly faster, and the code is a lot simpler.
> Rewriting AvroStorage also enabled me to implement support for Trevni.
> I'm opening this ticket to facilitate discussion while I figure out the best
> way to contribute the changes back to Apache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
