[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheolsoo Park updated PIG-3015:
-------------------------------
Attachment: PIG-3015-3.patch
I am attaching a new patch that includes the following fixes:
# {{hadoop-core.jar}} is pulled as it's a dependency of {{trevni-core.jar}}.
This causes compile errors with hadoop-2.0.x. I fixed this as follows:
{code}
+ <dependency org="org.apache.avro" name="trevni-avro" rev="${avro.version}"
+ conf="compile->default;checkstyle->master">
+ <exclude org="org.apache.hadoop" module="hadoop-core"/>
+ </dependency>
{code}
# {{avro-tools.jar}} contains hadoop classes such as Configuration. This causes
compile errors with hadoop-2.0.x. I fixed this as follows:
{code}
+ <dependency org="org.apache.avro" name="avro-tools" rev="${avro.version}"
+ conf="test->default">
+ <artifact name="nodeps" type="jar"/>
+ </dependency>
{code}
Now I can run {{TestAvroStorage}} with {{-Dhadoopversion=23}}, but all test
cases currently fail. I haven't investigated yet.
> Rewrite of AvroStorage
> ----------------------
>
> Key: PIG-3015
> URL: https://issues.apache.org/jira/browse/PIG-3015
> Project: Pig
> Issue Type: Improvement
> Components: piggybank
> Reporter: Joseph Adler
> Assignee: Joseph Adler
> Attachments: PIG-3015-2.patch, PIG-3015-3.patch, PIG-3015.patch
>
>
> The current AvroStorage implementation has a lot of issues: it requires old
> versions of Avro, it copies data much more than needed, and it's verbose and
> complicated. (One pet peeve of mine is that old versions of Avro don't
> support Snappy compression.)
> I rewrote AvroStorage from scratch to fix these issues. In early tests, the
> new implementation is significantly faster, and the code is a lot simpler.
> Rewriting AvroStorage also enabled me to implement support for Trevni (as
> TrevniStorage).
> I'm opening this ticket to facilitate discussion while I figure out the best
> way to contribute the changes back to Apache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
