[
https://issues.apache.org/jira/browse/PIG-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13753912#comment-13753912
]
Julien Le Dem commented on PIG-3419:
------------------------------------
[~cheolsoo]: thanks a lot for looking into this.
Here are my thoughts:
1. let's change it back
2. 4. 5. 6. 7. are either internal to Pig or necessary to add the execution
engine abstraction.
3.
JobStats still exists but the MR specific part is split into MRJobStats which
extends JobStats
Same thing for PigStatsUtil and ScriptState. Those classes are not disappearing
but the MR specific part is abstracted out.
HExecutionEngine could be renamed back to what it was but this is again what is
becoming the new abstraction.
Unfortunately tools like Ambrose and Lipstick depend on the MR specific parts
of Pig and look at the internals. This patch is a necessary change so that
those tools can work independently of the execution engine in the future.
The changes to Ambrose and Lipstick should be minimal though with this patch.
But yes they would suffer from some incompatibility, but again there is no way
around it when a tool looks inside the execution engine internals.
I think we should revert 1. and commit the patch.
> Pluggable Execution Engine
> ---------------------------
>
> Key: PIG-3419
> URL: https://issues.apache.org/jira/browse/PIG-3419
> Project: Pig
> Issue Type: New Feature
> Affects Versions: 0.12
> Reporter: Achal Soni
> Assignee: Achal Soni
> Priority: Minor
> Attachments: execengine.patch, mapreduce_execengine.patch,
> stats_scriptstate.patch, test_failures.txt, test_suite.patch,
> updated-8-22-2013-exec-engine.patch, updated-8-23-2013-exec-engine.patch,
> updated-8-27-2013-exec-engine.patch, updated-8-28-2013-exec-engine.patch
>
>
> In an effort to adapt Pig to work using Apache Tez
> (https://issues.apache.org/jira/browse/TEZ), I made some changes to allow for
> a cleaner ExecutionEngine abstraction than existed before. The changes are
> not that major as Pig was already relatively abstracted out between the
> frontend and backend. The changes in the attached commit are essentially the
> barebones changes -- I tried to not change the structure of Pig's different
> components too much. I think it will be interesting to see in the future how
> we can refactor more areas of Pig to really honor this abstraction between
> the frontend and backend.
> Some of the changes was to reinstate an ExecutionEngine interface to tie
> together the front end and backend, and making the changes in Pig to delegate
> to the EE when necessary, and creating an MRExecutionEngine that implements
> this interface. Other work included changing ExecType to cycle through the
> ExecutionEngines on the classpath and select the appropriate one (this is
> done using Java ServiceLoader, exactly how MapReduce does for choosing the
> framework to use between local and distributed mode). Also I tried to make
> ScriptState, JobStats, and PigStats as abstract as possible in its current
> state. I think in the future some work will need to be done here to perhaps
> re-evaluate the usage of ScriptState and the responsibilities of the
> different statistics classes. I haven't touched the PPNL, but I think more
> abstraction is needed here, perhaps in a separate patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
