[
https://issues.apache.org/jira/browse/MAPREDUCE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427412#comment-13427412
]
Josh Wills commented on MAPREDUCE-4495:
---------------------------------------
Two comments, as someone who would be interested in using this kind of workflow
management application from an incubating project I work on, Crunch:
1) Chris, it's the "Going from there" part of the incubator process that
involves the overhead. Setting up a project infrastructure takes time. You need
to file a lot of INFRA tickets, and those typically take about a week to get
resolved. I'm not hating on anybody here, the INFRA guys are doing yeoman's
work and we cannot do enough nice for things for them in return, but let's not
pretend that there isn't overhead involved in having someone setup a repo for
you, create a JIRA project, setup Jenkins, and set up CMS.
This is all anecdotal, of course. But I'm a data scientist, so my anecdotal
experiences come with a sheen of statistical validity. ;)
Git setup for Crunch: https://issues.apache.org/jira/browse/INFRA-4883 (7 days)
JIRA setup for Crunch: https://issues.apache.org/jira/browse/INFRA-4889 (27
days)
CMS for Crunch: https://issues.apache.org/jira/browse/INFRA-5092 (6+ days)
The ASF has good reasons for not creating umbrella projects, as have been
outlined earlier in the thread. But let's not pretend that there isn't overhead
to creating a new project.
2) I think Bobby (Bobby, I don't know that we've ever met, so I hope you don't
mind me calling you Bobby) has the right idea in terms of making this an
incremental change, and perhaps we should create a JIRA that reflects what we
really want, which is to deconstruct MapReduce into its individual components
and allow clients to assemble valid workflows out of those components, whether
they're MapReduce jobs or MapMapMapReduceReduce jobs or whatever. That seems
like it clearly belongs within this project.
> Workflow Application Master in YARN
> -----------------------------------
>
> Key: MAPREDUCE-4495
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4495
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Affects Versions: 2.0.0-alpha
> Reporter: Bo Wang
> Assignee: Bo Wang
>
> It is useful to have a workflow application master, which will be capable of
> running a DAG of jobs. The workflow client submits a DAG request to the AM
> and then the AM will manage the life cycle of this application in terms of
> requesting the needed resources from the RM, and starting, monitoring and
> retrying the application's individual tasks.
> Compared to running Oozie with the current MapReduce Application Master,
> these are some of the advantages:
> - Less number of consumed resources, since only one application master will
> be spawned for the whole workflow.
> - Reuse of resources, since the same resources can be used by multiple
> consecutive jobs in the workflow (no need to request/wait for resources for
> every individual job from the central RM).
> - More optimization opportunities in terms of collective resource requests.
> - Optimization opportunities in terms of rewriting and composing jobs in the
> workflow (e.g. pushing down Mappers).
> - This Application Master can be reused/extended by higher systems like Pig
> and hive to provide an optimized way of running their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
