[
https://issues.apache.org/jira/browse/MAPREDUCE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427407#comment-13427407
]
Robert Joseph Evans commented on MAPREDUCE-4495:
------------------------------------------------
Arun pinged me off-line and said he was a little confused by my comments. We
had talked about this before at Hadoop Summit and my last comment came off a
little differently, so I want to clarify a bit. I think whatever we do needs
to be incremental. I don't think pulling in full oozie support is something
that we should do on the first change. I would like to see JobControl moved to
optionally run in its own AM, all it would do is to launch other MR AMs to
start out with. The next step would be to add in recovery so that if the AM
dies we can pick up where we left off. That might simply be a list of jobs
written to HDFS that it ran, so it can query their state when it starts up
again. After that we can look at adding in more pluggability to launch other
AMs, etc. and discuss where the code should reside longterm. I just don't
think until we have an architecture decided on that we should create a new
project. And perhaps at that point MR truly becomes a separate project
too.
> Workflow Application Master in YARN
> -----------------------------------
>
> Key: MAPREDUCE-4495
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4495
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Affects Versions: 2.0.0-alpha
> Reporter: Bo Wang
> Assignee: Bo Wang
>
> It is useful to have a workflow application master, which will be capable of
> running a DAG of jobs. The workflow client submits a DAG request to the AM
> and then the AM will manage the life cycle of this application in terms of
> requesting the needed resources from the RM, and starting, monitoring and
> retrying the application's individual tasks.
> Compared to running Oozie with the current MapReduce Application Master,
> these are some of the advantages:
> - Less number of consumed resources, since only one application master will
> be spawned for the whole workflow.
> - Reuse of resources, since the same resources can be used by multiple
> consecutive jobs in the workflow (no need to request/wait for resources for
> every individual job from the central RM).
> - More optimization opportunities in terms of collective resource requests.
> - Optimization opportunities in terms of rewriting and composing jobs in the
> workflow (e.g. pushing down Mappers).
> - This Application Master can be reused/extended by higher systems like Pig
> and hive to provide an optimized way of running their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
