[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13659573#comment-13659573
]
Paul Mazak commented on PIG-3015:
---------------------------------
One simple workaround for us was to override AvroStorage's checkSchema this way.
{code}
/**
* In Pig script do:
* REGISTER 'lib/this.jar'
* DEFINE AvroStorage com.this.JoinableAvroStorage;
*/
public class JoinableAvroStorage extends AvroStorage {
@Override
public void checkSchema(ResourceSchema s) throws IOException {
try {
super.checkSchema(s);
}
catch (SchemaParseException spe) {
ResourceFieldSchema[] pigFields = s.getFields();
for (int i = 0; i < pigFields.length; i++) {
String outname = pigFields[i].getName();
if (outname.contains("::")) {
String newOutname = outname.split("::")[1];
pigFields[i].setName(newOutname);
}
}
super.checkSchema(s);
}
}
}
{code}
> Rewrite of AvroStorage
> ----------------------
>
> Key: PIG-3015
> URL: https://issues.apache.org/jira/browse/PIG-3015
> Project: Pig
> Issue Type: Improvement
> Components: piggybank
> Reporter: Joseph Adler
> Assignee: Joseph Adler
> Attachments: bad.avro, good.avro, PIG-3015-10.patch,
> PIG-3015-11.patch, PIG-3015-2.patch, PIG-3015-3.patch, PIG-3015-4.patch,
> PIG-3015-5.patch, PIG-3015-6.patch, PIG-3015-7.patch, PIG-3015-9.patch,
> PIG-3015-doc-2.patch, PIG-3015-doc.patch, TestInput.java, Test.java,
> with_dates.pig
>
>
> The current AvroStorage implementation has a lot of issues: it requires old
> versions of Avro, it copies data much more than needed, and it's verbose and
> complicated. (One pet peeve of mine is that old versions of Avro don't
> support Snappy compression.)
> I rewrote AvroStorage from scratch to fix these issues. In early tests, the
> new implementation is significantly faster, and the code is a lot simpler.
> Rewriting AvroStorage also enabled me to implement support for Trevni (as
> TrevniStorage).
> I'm opening this ticket to facilitate discussion while I figure out the best
> way to contribute the changes back to Apache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
