Pe 30.01.2012 07:47, aliyeh saeedi a scris:
I want to save them with my own names, How NameNode will keep their names?
________________________________
From: Joey Echeverria<j...@cloudera.com>
To: mapreduce-user@hadoop.apache.org; aliyeh saeedi<a1_sae...@yahoo.com>
Sent: Sunday, 29 January 2012, 17:10
Subject: Re: reducers outputs
Reduce output is normally stored in HDFS, just like your other files.
Are you seeing
different behavior?
-Joey
On Sun, Jan 29, 2012 at 1:05 AM, aliyeh saeedi<a1_sae...@yahoo.com> wrote:
Hi
I want to save reducers outputs like other files in Hadoop. Does NameNode
keep any information about them? How can I do this?
Or can I add a new component to Hadoop like NameNode and make JobTracker to
consult with it too (I mean I want to make JobTracker to consult with
NameNode AND myNewComponent both)?
You aren't making a lot of sens, at least to me :). But if tou wish to
save reducer output somehow different you will have to implement your
own class that implements
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/OutputFormat.html.
It's easier to extend
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/FileOutputFormat.html
and override the parts that you need.
The framework will call methods from the above mentioned class to
persist the data from reducers. You instruct the framework to use your
class when you call
job.setOutputFormatClass(SequenceFileOutputFormat.class); (this makes
the output a SequenceFile).
Example to save under a different name:
public static class RenamedSequenceFile extends SequenceFileOutputFormat {
@Override
public Path getDefaultWorkFile(TaskAttemptContext context,
String extension) throws IOException {
FileOutputCommitter committer = (FileOutputCommitter)
getOutputCommitter(context);
return new Path(committer.getWorkPath(), "myBetterName");
}
}
This will output your reducer data into "myBetterName" file as key
values pairs (behaviour inherited from SequanceFileOutputFormat).
I hope this helps,
--
Ioan Eugen Stan
http://ieugen.blogspot.com
