RE: How to access contents of a Map Reduce job's working directory

[Subroto Sanyal]

Hi Smriti,

 

I would suggest you to have custom OutputCommiter which will be extension of
FileOutputCommiter and will help you achieve your desired functionality.

 

Regards,

 

Regards, 

Computers make very fast and accurate mistakes....... 


This email and its attachments contain confidential information from HUAWEI,
which is intended only for the person or entity whose address is listed
above. 
Any use of the information contained herein in any way (including, but not
limited to, total or partial disclosure, reproduction, or dissemination) by
persons other than the intended recipient(s) is prohibited. 
If you receive this e-mail in error, please notify the sender by phone or
email immediately and delete it! 

  _____  

From: smriti singh [mailto:smritis...@gmail.com] 
Sent: Tuesday, August 02, 2011 3:48 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: How to access contents of a Map Reduce job's working directory

 

Hi Harsh, let me explain this in detail:

 

this is what I am trying to do in my mapper:

File setupFolder = new File(setupFolderName);

setupFolder.mkdirs();



MARD mard = new MARD(setupFolder);

Text valuz = new Text();

IntWritable intval = new IntWritable();

File original = new File("Vca1652.txt");

File mardedxml = new File("Vca1652-mardedxml.txt");

File marded = new File("Vca1652-marded.txt");



mardedxml.createNewFile();

marded.createNewFile();

NormalisationStats stats;

try {

stats = mard.normaliseFile(original,mardedxml,marded,50.0);

//This meathod requires access to the myMardfolder


System.out.println(stats);

} catch (MARDException e) {

// TODO Auto-generated catch block

e.printStackTrace();

}

 

 

Now,
1. This mard.normalise() creates files in the "setup" folder.
2. I have no control on this method as I just a got a jar mard.jar to call
this meathod.
3. Mard.normalise() searches for folder called "foul" in the working
directory. If not found throws on Exception. It is this folder's data that
mard.normalise() method process on to generate files in the "setup"
folder. I passed this folder to the working directory through the
-archives option (**by first compressing it)
3. I am not using the "input path" data in anyway in the mapper.
4. Hence not using the key and values generated.
5. I am using and Identity reducer as there is no need of any reduction .
6. Hence also the output is of no use for me.
7. I need to get the content of "setup folder", but I dont know the method
to do so.


* I might be wrong in the way I am doing it because I had no formal hadoop
training I have just learned it by reading articles on the net.

 

Thanking you in anticipation

 

Regards

 

Smriti

 

 

On Tue, Aug 2, 2011 at 11:33 AM, Harsh J <ha...@cloudera.com> wrote:

Smriti,

By working directory, do you mean the task attempt's working directory
or the global job staging directory?


On Tue, Aug 2, 2011 at 6:22 AM, smriti singh <smritis...@gmail.com> wrote:
> I want to run a MapReduce job in hadoop which needs to create a "setup"
> folder in working directory. During the execution the job will generate
> some additional text files within this "setup" folder. The problem is I
> dont know how to access or move this setup folder content to my local file
> system as at end of the job, the job directory will be cleaned up.
>
> It would be great if you can help.
> Regards
> Smriti




--
Harsh J

 

<<attachment: image001.jpg>>

<<attachment: image002.jpg>>