Yes. The reason I pointed it to him was that it seems like he's trying to do something with Hadoop for which MapReduce may not be right execution model. Yarn/MRv2 gives you the ability to try other execution models. As you pointed out, it may require some extra development, but it is more flexible than straight MapReduce and probably a better option in the long run over submitting jobs from inside of jobs.
-Joey On Tue, Oct 4, 2011 at 12:11 PM, GOEKE, MATTHEW (AG/1000) <matthew.go...@monsanto.com> wrote: > Joey, > > Is yarn just a synonym for MRv2? And if so he would still have to create a > custom application master for his job type right? > > Matt > > -----Original Message----- > From: Joey Echeverria [mailto:j...@cloudera.com] > Sent: Tuesday, October 04, 2011 11:06 AM > To: mapreduce-user@hadoop.apache.org > Subject: Re: Submitting a Hadoop task from withing a reducer > > You may want to check out Yarn, coming in Hadoop 0.23: > > https://issues.apache.org/jira/browse/MAPREDUCE-279 > > -Joey > > On Tue, Oct 4, 2011 at 11:45 AM, Yaron Gonen <yaron.go...@gmail.com> wrote: >> Hi, >> Hadoop tasks are always stacked to form a linear user-managed workflow (a >> reduce step cannot start before all previous mappers have stopped etc). This >> may be problematic in recursive tasks: for example in a BFS we will not get >> any output until the longest branch has been reached. >> In order to solve than, an idea came up of submitting a whole Hadoop task >> from within a reducer. Have anyone tried it? >> Thanks. >> > > > > -- > Joseph Echeverria > Cloudera, Inc. > 443.305.9434 > This e-mail message may contain privileged and/or confidential information, > and is intended to be received only by persons entitled > to receive such information. If you have received this e-mail in error, > please notify the sender immediately. Please delete it and > all attachments from any servers, hard drives or any other media. Other use > of this e-mail by you is strictly prohibited. > > All e-mails and attachments sent and received are subject to monitoring, > reading and archival by Monsanto, including its > subsidiaries. The recipient of this e-mail is solely responsible for checking > for the presence of "Viruses" or other "Malware". > Monsanto, along with its subsidiaries, accepts no liability for any damage > caused by any such code transmitted by or accompanying > this e-mail or any attachment. > > > The information contained in this email may be subject to the export control > laws and regulations of the United States, potentially > including but not limited to the Export Administration Regulations (EAR) and > sanctions regulations issued by the U.S. Department of > Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this > information you are obligated to comply with all > applicable U.S. export laws and regulations. > > -- Joseph Echeverria Cloudera, Inc. 443.305.9434
