Thanks guys. I'll dig a little in Yarn. On Tue, Oct 4, 2011 at 6:28 PM, Joey Echeverria <j...@cloudera.com> wrote:
> Yes. The reason I pointed it to him was that it seems like he's trying > to do something with Hadoop for which MapReduce may not be right > execution model. Yarn/MRv2 gives you the ability to try other > execution models. As you pointed out, it may require some extra > development, but it is more flexible than straight MapReduce and > probably a better option in the long run over submitting jobs from > inside of jobs. > > -Joey > > On Tue, Oct 4, 2011 at 12:11 PM, GOEKE, MATTHEW (AG/1000) > <matthew.go...@monsanto.com> wrote: > > Joey, > > > > Is yarn just a synonym for MRv2? And if so he would still have to create > a custom application master for his job type right? > > > > Matt > > > > -----Original Message----- > > From: Joey Echeverria [mailto:j...@cloudera.com] > > Sent: Tuesday, October 04, 2011 11:06 AM > > To: mapreduce-user@hadoop.apache.org > > Subject: Re: Submitting a Hadoop task from withing a reducer > > > > You may want to check out Yarn, coming in Hadoop 0.23: > > > > https://issues.apache.org/jira/browse/MAPREDUCE-279 > > > > -Joey > > > > On Tue, Oct 4, 2011 at 11:45 AM, Yaron Gonen <yaron.go...@gmail.com> > wrote: > >> Hi, > >> Hadoop tasks are always stacked to form a linear user-managed workflow > (a > >> reduce step cannot start before all previous mappers have stopped etc). > This > >> may be problematic in recursive tasks: for example in a BFS we will not > get > >> any output until the longest branch has been reached. > >> In order to solve than, an idea came up of submitting a whole Hadoop > task > >> from within a reducer. Have anyone tried it? > >> Thanks. > >> > > > > > > > > -- > > Joseph Echeverria > > Cloudera, Inc. > > 443.305.9434 > > This e-mail message may contain privileged and/or confidential > information, and is intended to be received only by persons entitled > > to receive such information. If you have received this e-mail in error, > please notify the sender immediately. Please delete it and > > all attachments from any servers, hard drives or any other media. Other > use of this e-mail by you is strictly prohibited. > > > > All e-mails and attachments sent and received are subject to monitoring, > reading and archival by Monsanto, including its > > subsidiaries. The recipient of this e-mail is solely responsible for > checking for the presence of "Viruses" or other "Malware". > > Monsanto, along with its subsidiaries, accepts no liability for any > damage caused by any such code transmitted by or accompanying > > this e-mail or any attachment. > > > > > > The information contained in this email may be subject to the export > control laws and regulations of the United States, potentially > > including but not limited to the Export Administration Regulations (EAR) > and sanctions regulations issued by the U.S. Department of > > Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of > this information you are obligated to comply with all > > applicable U.S. export laws and regulations. > > > > > > > > -- > Joseph Echeverria > Cloudera, Inc. > 443.305.9434 >
