[
https://issues.apache.org/jira/browse/PIG-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250999#comment-13250999
]
Scott Carey commented on PIG-2632:
----------------------------------
Avro has a pure Java code generator that uses the Velocity template engine.
The bulit-in templates produce SpecificRecord objects with builder-pattern
semantics. Custom templates can be loaded from a classpath, so you can convert
a schema to whatever you wish -- java source, yaml, c#, etc.
The challenge with Pig and Avro for this use case is transforming a
o.a.pig.Schema into an o.a.a.Schema. However, there is source code that does
this at: https://issues.apache.org/jira/browse/AVRO-592
Any future bytecode generation would resemble the template approach -- a schema
would decorate a (source code template / byte code fragments) to produce an
output. Both would call pre-existing methods on a superclass that does most of
the work. Refining a template to be as small and simple as possible
corresponds with refining bytecode generation to do the minimum necessary.
> Create a SchemaTuple which generates efficient Tuples via code gen
> ------------------------------------------------------------------
>
> Key: PIG-2632
> URL: https://issues.apache.org/jira/browse/PIG-2632
> Project: Pig
> Issue Type: Improvement
> Reporter: Jonathan Coveney
> Assignee: Jonathan Coveney
> Fix For: 0.11
>
> Attachments: PIG-2632-0.patch, PIG-2632-1.patch, PIG-2632-3.patch
>
>
> This work builds on Dmitriy's PrimitiveTuple work. The idea is that, knowing
> the Schema on the frontend, we can code generate Tuples which can be used for
> fun and profit. In rudimentary tests, the memory efficiency is 2-4x better,
> and it's ~15% smaller serialized (heavily heavily depends on the data,
> though). Need to do get/set tests, but assuming that it's on par (or even
> faster) than Tuple, the memory gain is huge.
> Need to clean up the code and add tests.
> Right now, it generates a SchemaTuple for every inputSchema and outputSchema
> given to UDF's. The next step is to make a SchemaBag, where I think the
> serialization savings will be really huge.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
