[
https://issues.apache.org/jira/browse/PIG-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13254455#comment-13254455
]
Dmitriy V. Ryaboy commented on PIG-2653:
----------------------------------------
Reusing the physical plan is going to be a problem -- as Ashutosh mentioned,
physical plan includes actual load / store locations. Plus things are set on
the physical plan based on examining the actual data (things like parallelism).
Reusing the Logical Plan would be easier, as it does not change between
executions. Anyone know how long the different compilation steps take relative
to each other?
Don't know about metastore as storage for this. Too permanent. Something like
memcache of hbase seems better, as both have built-in expiration mechanisms.
Maybe a pluggable (off by default) plan caching strategy.
Would need a way to force bypassing the cache, or overwriting the cache.
Would also want to make pig build part of the cache key, so that different
versions of pig can be used on the same cluster without setting up different
caches.
> Precompile option in PIG (Ability to store the plan for queries which are run
> multiple times)
> ---------------------------------------------------------------------------------------------
>
> Key: PIG-2653
> URL: https://issues.apache.org/jira/browse/PIG-2653
> Project: Pig
> Issue Type: Improvement
> Components: parser
> Reporter: Rajesh Balamohan
>
> Based on the size of the PIG script, it takes 1 or 2 minutes in certain cases
> for PIG compiler to create the MR plan. If the same script has to be run
> later point in time, it has to go through this process again.
> It would be nice, if PIG can store the result (execution plan) which can be
> reused when the same script is run again.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
