[
https://issues.apache.org/jira/browse/MAPREDUCE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428272#comment-13428272
]
Arun C Murthy commented on MAPREDUCE-4495:
------------------------------------------
{quote}
So, I see two options:
# Enhance JobControl api to work in AM by making MR-AM, specifially MRAppMaster
thread-safe. This will allow for multiple objects of MRAppMaster to be created.
This means there are no new interfaces to MapReduce.
# Go the full distance, make it generic, import code from oozie, come up with a
new set of interfaces for generic DAG mgmt infrastructure etc. etc. and do it
in a separate Incubator project.
{quote}
I think this is coming to a point where we are arguing too much in the
abstract. Frankly, this is really not how I want to spend my time.
Maybe we can wait for a detailed proposal from Bo or Alejandro and then revisit
this discussion. I believe I have laid my thoughts out clearly with respect to
the options etc. Let's discuss when we actually have something concrete (design
or code).
OTOH, if we can agree on the Incubator proposal I'm happy to do the legwork for
Alejandro right-away. At least that is tractable and not merely abstract.
> Workflow Application Master in YARN
> -----------------------------------
>
> Key: MAPREDUCE-4495
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4495
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Affects Versions: 2.0.0-alpha
> Reporter: Bo Wang
> Assignee: Bo Wang
>
> It is useful to have a workflow application master, which will be capable of
> running a DAG of jobs. The workflow client submits a DAG request to the AM
> and then the AM will manage the life cycle of this application in terms of
> requesting the needed resources from the RM, and starting, monitoring and
> retrying the application's individual tasks.
> Compared to running Oozie with the current MapReduce Application Master,
> these are some of the advantages:
> - Less number of consumed resources, since only one application master will
> be spawned for the whole workflow.
> - Reuse of resources, since the same resources can be used by multiple
> consecutive jobs in the workflow (no need to request/wait for resources for
> every individual job from the central RM).
> - More optimization opportunities in terms of collective resource requests.
> - Optimization opportunities in terms of rewriting and composing jobs in the
> workflow (e.g. pushing down Mappers).
> - This Application Master can be reused/extended by higher systems like Pig
> and hive to provide an optimized way of running their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
