[
https://issues.apache.org/jira/browse/MAPREDUCE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13427387#comment-13427387
]
Robert Joseph Evans commented on MAPREDUCE-4495:
------------------------------------------------
I am on the side of Alejandor here. I see a lot of potential synergy with MR
on this. Also I respectfully disagree that we cannot make a DAG AM do more then
just run MR jobs, and even if all it does is MR pig, hive and oozie have made
due with only being able to run a single job until now, why is it limiting to
give them an API to run multiple jobs? We already provide such an API although
a limited one, JobControl, that I know pig uses. If all we did in the first
round is move this over so the execution happens on the AM instead of the
client, it would be great for pig. I don't know if Hive uses it, but we could
probably extend the API to support what Oozie needs as well.
Like I said previously I have been thinking about this a lot. I don't have
time right now to write up a formal design, and it is not really on my priority
list right now either, but I want to put up a brief explanation of what I have
been thinking about to get feedback, and perhaps steer the conversation a bit
:).
Map/Reduce is more complicated then just a Map and a Reduce. There are many
different steps and these steps can be in a few different combinations (these
are not all of the steps) setup, read, deserialize, map, combiner, shuffle,
reduce, serialize, write, cleanup. With the addition of a pluggable shuffle
that is in progress all of these steps can be replaced with custom code. The
ordering however, is still rigidly locked down to a few combinations. You can
add in a combiner or not. You can have the map write directly to HDFS and
bypass the reduce side or not. Many people want more flexibility in how these
pieces fit together, i.e Map Reduce Reduce.
Why don't we instead break down map/reduce into smaller completely pluggable
primitive operations that can be stringed together into a DAG, and then have
common, pluggable, optimizations and error checks that run on this DAG. This
allows us to do things that would be totally impossible before. For example,
one optimizer may look at the DAG and see that a specific output simply becomes
the input for another job. It could combine the two together into a
map/reduce/write to HDFS on the side/map(running in the same task as the
reduce)/reduce, or if pig/hive were aware of the DAG primitives, and not just
using the MR APIs, they may completely remove the write to HDFS part, because
they know it is a temporary file that is not going to be consumed by anyone but
the one job.
We could even make the DAG dynamic, so that when oozie decides to run a pig
script the pig compiler would kick in on the AM, produce its DAG, and then
insert it into the already existing DAG. The original DAG AM would not need
any knowledge about pig or oozie. Oozie would just be providing a special
subclass of a LocalDAGStep(or something of the sort) that would ask the pig
complier for a DAG and then insert it into the current DAG.
I have already used up a lot of space, and I know this is a huge amount of work
that would need to formally be broken up into small steps that can
incrementally take us to that point. If this was split off into its own
project, it would really be a fork of Map/Reduce and eventually Map/Reduce
itself would start to look like a small layer on top of it, assuming that we
kept both of them in sync, because many of these things are stuff that we have
discussed about adding to MR in the past.
> Workflow Application Master in YARN
> -----------------------------------
>
> Key: MAPREDUCE-4495
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4495
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Affects Versions: 2.0.0-alpha
> Reporter: Bo Wang
> Assignee: Bo Wang
>
> It is useful to have a workflow application master, which will be capable of
> running a DAG of jobs. The workflow client submits a DAG request to the AM
> and then the AM will manage the life cycle of this application in terms of
> requesting the needed resources from the RM, and starting, monitoring and
> retrying the application's individual tasks.
> Compared to running Oozie with the current MapReduce Application Master,
> these are some of the advantages:
> - Less number of consumed resources, since only one application master will
> be spawned for the whole workflow.
> - Reuse of resources, since the same resources can be used by multiple
> consecutive jobs in the workflow (no need to request/wait for resources for
> every individual job from the central RM).
> - More optimization opportunities in terms of collective resource requests.
> - Optimization opportunities in terms of rewriting and composing jobs in the
> workflow (e.g. pushing down Mappers).
> - This Application Master can be reused/extended by higher systems like Pig
> and hive to provide an optimized way of running their workflows.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
