[
https://issues.apache.org/jira/browse/PIG-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Julien Le Dem updated PIG-2318:
-------------------------------
Attachment: PIG-2318_3.patch
PIG-2318_3.patch handles the special case for Storer.
It turns out that the PigOutputFormat did not deserialize the PigContext. Which
explains why the classloader was not initialized.
Now the PigOutputFormat initializes the PigContext first.
> Push extra jars to distributed cache and use the classloader enxtension
> mechanism in PigContext to load them on the backend
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-2318
> URL: https://issues.apache.org/jira/browse/PIG-2318
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: Julien Le Dem
> Assignee: Julien Le Dem
> Attachments: PIG-2318.patch, PIG-2318_2.patch, PIG-2318_3.patch
>
>
> This is related to PIG-2010 with a slightly different approach
> https://issues.apache.org/jira/browse/PIG-2010
> Currently Pig bundles up all dependencies in a single jar which is a lot of
> overhead when there are a lot of dependencies and short lived jobs. This
> patch instead pushes the dependencies to distributed cache and uses the
> PigContext classloading mechanism to make the UDFs available.
> Possible improvements: push jars to HDFS/distributed cache only once per
> script. have a cache on HDFS to avoid repeatedly pushing jars to HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
