[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13537146#comment-13537146
]
Cheolsoo Park commented on PIG-3015:
------------------------------------
Hi Joe,
I found a few errors with the new patch.
- python createtests.py fails with the following errors:
{code}
creating data/avro/deflate/records.avro
sh: data/avro/deflate/records.avro: No such file or directory
creating data/avro/deflate/recordsAsOutputByPig.avro
sh: data/avro/deflate/recordsAsOutputByPig.avro: No such file or directory
creating data/avro/snappy/records.avro
sh: data/avro/snappy/records.avro: No such file or directory
creating data/avro/snappy/recordsAsOutputByPig.avro
sh: data/avro/snappy/recordsAsOutputByPig.avro: No such file or directory
{code}
These errors are due to a typo at line 85 in {{createtests.py}}:
"{{data/avro/}}" should be "{{data/avro/compressed/}}".
- After fixing this typo, I get the following failures in {{TestAvroStorage}}:
{code}
Testcase: testStoreSnappyCompressedRecords took 4.439 sec
FAILED
Testcase: testLoadDeflateCompressedRecords took 0.007 sec
FAILED
Testcase: testStoreDeflateCompressedRecords took 3.557 sec
FAILED
Testcase: testLoadSnappyCompressedRecords took 0.004 sec
FAILED
{code}
These errors are due to typos in {{TestAvroStorage.java}}:
"{{data/avro/deflate/}}" and "{{data/avro/snappy/}}" should be
"{{data/avro/compressed/deflate/}}" and "{{data/avro/compressed/sanppy/}}"
respectively.
- Lastly, I realized that {{createtest.py}} doesn't generate the input avro
file for {{testPartialLoadGlob}}, and I saw your comment:
{code}
Please copy and run this command manually (doesn't work correctly from python
right now... arg...
{code}
In fact, I haven't been able to fix this myself yet. When I run the command
manually, {{testPartialLoadGlob}} passes.
> Rewrite of AvroStorage
> ----------------------
>
> Key: PIG-3015
> URL: https://issues.apache.org/jira/browse/PIG-3015
> Project: Pig
> Issue Type: Improvement
> Components: piggybank
> Reporter: Joseph Adler
> Assignee: Joseph Adler
> Attachments: PIG-3015.patch
>
>
> The current AvroStorage implementation has a lot of issues: it requires old
> versions of Avro, it copies data much more than needed, and it's verbose and
> complicated. (One pet peeve of mine is that old versions of Avro don't
> support Snappy compression.)
> I rewrote AvroStorage from scratch to fix these issues. In early tests, the
> new implementation is significantly faster, and the code is a lot simpler.
> Rewriting AvroStorage also enabled me to implement support for Trevni (as
> TrevniStorage).
> I'm opening this ticket to facilitate discussion while I figure out the best
> way to contribute the changes back to Apache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
