[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cheolsoo Park updated PIG-3015:
-------------------------------
Attachment: PIG-3015-4.patch
I am uploading a new patch that includes the following changes:
* I forgot to pass the {{--codec}} option to the Avro tool when generating
compressed {{.avro}} files, so I fixed it.
* I found that the unit test fails with MR2 because {{mapred.output.compress}}
is set to {{true}} while {{mapred.output.compression.codec}} is not set.
{code}
ERROR 0: 'mapred.output.compress' is set but no value is specified for
'mapred.output.compression.codec'.
{code}
What's worse is that any test cases that run after
{{testStoreSnappyCompressedRecords}} fails for the same reason. Apparently,
this property remains enabled and affects other test cases. For now, I removed
the following line from {{identity_codec.pig}}:
{code}
SET mapred.output.compress true
{code}
With this change, I can actually run test jobs, but they still fail. The reason
is because output files do not match with expected ones. I didn't investigate
why this happens with MR2.
* Lastly, I also found that the current patch does not implement compression
support. The test cases such as {{testStoreSnappyCompressedRecords}} and
{{testStoreDeflateCompressedRecords}} do not check {{avro.codec}}. Please
correct me if I am wrong.
Thanks!
> Rewrite of AvroStorage
> ----------------------
>
> Key: PIG-3015
> URL: https://issues.apache.org/jira/browse/PIG-3015
> Project: Pig
> Issue Type: Improvement
> Components: piggybank
> Reporter: Joseph Adler
> Assignee: Joseph Adler
> Attachments: PIG-3015-2.patch, PIG-3015-3.patch, PIG-3015-4.patch,
> PIG-3015.patch
>
>
> The current AvroStorage implementation has a lot of issues: it requires old
> versions of Avro, it copies data much more than needed, and it's verbose and
> complicated. (One pet peeve of mine is that old versions of Avro don't
> support Snappy compression.)
> I rewrote AvroStorage from scratch to fix these issues. In early tests, the
> new implementation is significantly faster, and the code is a lot simpler.
> Rewriting AvroStorage also enabled me to implement support for Trevni (as
> TrevniStorage).
> I'm opening this ticket to facilitate discussion while I figure out the best
> way to contribute the changes back to Apache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
