[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491073#comment-13491073
]
Joseph Adler commented on PIG-3015:
-----------------------------------
Started working on this now. Two questions
(1) I'm a new contributor. What's the best way to organize the code within Pig?
I have a lot of helper classes and methods, and would like to put different
classes in different files to maximize readability. Should I put the helper
classes in an existing package (org.apache.pig.impl.builtin seems like the
closest match, though still not quite right), create a new package for the
helper classes, or do something else? I couldn't find documentation on the best
way to do this.
(2) Here's what I came up with for options: the first argument is either an
explicit schema or specifies the record names if a schema is automatically
generated. The second argument is a list of options (like in PigStorage):
<li><code>-namespace</code> Namespace for an automatically generated output
schema.</li>
<li><code>-ignoreerrors</code> Tells function to ignore errors in input
files.</li>
<li><code>-schemafile</code> Specifies URL for avro schema file from which to
read the input schema (can be local file, hdfs, url, etc).</li>
<li><code>-examplefile</code> Specifies URL for avro data file from which to
copy the input schema (can be local file, hdfs, url, etc).</li>
I considered providing an explicit option to provide a schema with a "-schema"
flag, but would have had to do something much more complicated to correctly
parse the options if an option could include a JSON schema. (Plus, I don't
think the meaning of the argument will be ambiguous: it will either be a valid
JSON object describing a schema or valid name.)
> Rewrite of AvroStorage
> ----------------------
>
> Key: PIG-3015
> URL: https://issues.apache.org/jira/browse/PIG-3015
> Project: Pig
> Issue Type: Improvement
> Components: piggybank
> Reporter: Joseph Adler
> Assignee: Joseph Adler
>
> The current AvroStorage implementation has a lot of issues: it requires old
> versions of Avro, it copies data much more than needed, and it's verbose and
> complicated. (One pet peeve of mine is that old versions of Avro don't
> support Snappy compression.)
> I rewrote AvroStorage from scratch to fix these issues. In early tests, the
> new implementation is significantly faster, and the code is a lot simpler.
> Rewriting AvroStorage also enabled me to implement support for Trevni.
> I'm opening this ticket to facilitate discussion while I figure out the best
> way to contribute the changes back to Apache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
