Job Submission schedule, one file at a time ?

[Daniel Yehdego]

Hi, 
I do have a folder with 50 different files and and I want to submit a Hadoop 
MapReduce job using each file as an input.My Map/Reduce programs basically do 
the same job for each of my files but I want to schedule and submit a job one 
file at a time. Its like submitting a job with one file input, wait until the 
job completes and submit the second job (second file) right after.I want to 
have 50 different Mapreduce outputs for the 50 input files. 
Looking forward for your inputs , Thanks.
Regards, 


  

Re: Job Submission schedule, one file at a time ?

[Michel Segel]

Not sure what you are attempting to do...
If you submit the directory name... You get a single m/r job to process all. 
( but it doesn't sound like that is what you want...)

You could use Oozie, or just a simple shell script that will walk down a list 
of files in the directory and then launch a Hadoop task...

Or did you want something else?


Sent from a remote device. Please excuse any typos...

Mike Segel

On Oct 25, 2011, at 3:53 PM, Daniel Yehdego  wrote:

> 
> Hi, 
> I do have a folder with 50 different files and and I want to submit a Hadoop 
> MapReduce job using each file as an input.My Map/Reduce programs basically do 
> the same job for each of my files but I want to schedule and submit a job one 
> file at a time. Its like submitting a job with one file input, wait until the 
> job completes and submit the second job (second file) right after.I want to 
> have 50 different Mapreduce outputs for the 50 input files. 
> Looking forward for your inputs , Thanks.
> Regards, 
> 
> 
> 

RE: Job Submission schedule, one file at a time ?

[Daniel Yehdego]

Hi Mike, Thanks for your quick response.
What I am looking is, I have m/r job which accepts an input file, process it 
and outputs to a single reducer.But I do have many file ques waiting to be 
processed similarly and I don't want to submit each m/r job foreach file 
manually. Is there a way to submit these files one after another or 
concurrently. All the files are independentto each other. I think its kind of 
job scheduler or something ? I am not sure on how to proceed .

Regards, 


> Subject: Re: Job Submission schedule, one file at a time ?
> From: michael_se...@hotmail.com
> Date: Tue, 25 Oct 2011 16:25:40 -0500
> To: common-user@hadoop.apache.org
> 
> Not sure what you are attempting to do...
> If you submit the directory name... You get a single m/r job to process all. 
> ( but it doesn't sound like that is what you want...)
> 
> You could use Oozie, or just a simple shell script that will walk down a list 
> of files in the directory and then launch a Hadoop task...
> 
> Or did you want something else?
> 
> 
> Sent from a remote device. Please excuse any typos...
> 
> Mike Segel
> 
> On Oct 25, 2011, at 3:53 PM, Daniel Yehdego  wrote:
> 
> > 
> > Hi, 
> > I do have a folder with 50 different files and and I want to submit a 
> > Hadoop MapReduce job using each file as an input.My Map/Reduce programs 
> > basically do the same job for each of my files but I want to schedule and 
> > submit a job one file at a time. Its like submitting a job with one file 
> > input, wait until the job completes and submit the second job (second file) 
> > right after.I want to have 50 different Mapreduce outputs for the 50 input 
> > files. 
> > Looking forward for your inputs , Thanks.
> > Regards, 
> > 
> > 
> >