[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13600212#comment-13600212
]
Aaron Klish commented on PIG-3015:
----------------------------------
For Multiple Schema Support, I don't think we should follow the rules in
PIG-2579. While this was a good first attempt
at supporting multiple data files with different schemas, it has problems:
1) The rules only support schema evolution for flat records. Nested records
are not supported.
2) The AVRO specification already defines a set of rules for schema evolution.
In fact, it already has code inside GenericDatumReader that will resolve data
between two different schemas.
We should be using that logic instead of writing logic that only works in very
limited situations.
3) The idea of a merged schema is not as powerful as letting the user define
the 'table' schema they expect the data to conform to. I would recommend
looking at how LinkedIn added support for AVRO to Hive. Every partition can
have a different schema, but there is a single 'table' schema. Hive will map
the file/partition schema(s) to the 'table' schema using the logic already
inside AVRO.
> Rewrite of AvroStorage
> ----------------------
>
> Key: PIG-3015
> URL: https://issues.apache.org/jira/browse/PIG-3015
> Project: Pig
> Issue Type: Improvement
> Components: piggybank
> Reporter: Joseph Adler
> Assignee: Joseph Adler
> Attachments: bad.avro, good.avro, PIG-3015-10.patch,
> PIG-3015-11.patch, PIG-3015-2.patch, PIG-3015-3.patch, PIG-3015-4.patch,
> PIG-3015-5.patch, PIG-3015-6.patch, PIG-3015-7.patch, PIG-3015-9.patch,
> PIG-3015-doc-2.patch, PIG-3015-doc.patch, TestInput.java, Test.java,
> with_dates.pig
>
>
> The current AvroStorage implementation has a lot of issues: it requires old
> versions of Avro, it copies data much more than needed, and it's verbose and
> complicated. (One pet peeve of mine is that old versions of Avro don't
> support Snappy compression.)
> I rewrote AvroStorage from scratch to fix these issues. In early tests, the
> new implementation is significantly faster, and the code is a lot simpler.
> Rewriting AvroStorage also enabled me to implement support for Trevni (as
> TrevniStorage).
> I'm opening this ticket to facilitate discussion while I figure out the best
> way to contribute the changes back to Apache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
