[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496378#comment-13496378
]
Cheolsoo Park commented on PIG-3015:
------------------------------------
Hi Joseph,
Thanks for the update. I support what you're proposing. I appreciate your
effort to clean up the code. Just to be clear, I have the following questions:
{quote}
Test pig scripts will be kept in discrete files, with parameters as file names.
I'll modify the test runner to set the runtime parameters correctly.
{quote}
Ideally, all Pig unit test should be written this way. Currently, Pig queries
are hard-coded in the test code, which is not very nice. But changing it is
going to be a long-term effort. Your changes for this jira will be isolated in
{{TestAvroStorage}}, won't they? If not, can you please provide more detail? I
am just trying to understand the scope of your proposal.
{quote}
I'm thinking about modifying the build process to compile human readable files
(in JSON format) into avro files before running the tests.
{quote}
This will be fully automated in the current framework (ant + junit), so I can
run {{ant test -Dtestcase=TestAvroStorage}} to run unit test cases, right? One
exception for this might be a test case for corrupted Avro files I guess.
Thanks!
> Rewrite of AvroStorage
> ----------------------
>
> Key: PIG-3015
> URL: https://issues.apache.org/jira/browse/PIG-3015
> Project: Pig
> Issue Type: Improvement
> Components: piggybank
> Reporter: Joseph Adler
> Assignee: Joseph Adler
>
> The current AvroStorage implementation has a lot of issues: it requires old
> versions of Avro, it copies data much more than needed, and it's verbose and
> complicated. (One pet peeve of mine is that old versions of Avro don't
> support Snappy compression.)
> I rewrote AvroStorage from scratch to fix these issues. In early tests, the
> new implementation is significantly faster, and the code is a lot simpler.
> Rewriting AvroStorage also enabled me to implement support for Trevni.
> I'm opening this ticket to facilitate discussion while I figure out the best
> way to contribute the changes back to Apache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
