Re: Files written by reducers

2012-01-29 Thread Harsh J
Arun just answered this on your other thread, which I quote here: "The NameNode doesn't care how you wrote the file i.e. either via 'bin/hadoop dfs -put <>' or via a MR job." So the answer is yes. Writes to HDFS are agnostic about what the source is - mapper/reducer/hbase/direct. On Mon, Jan 30,

Files written by reducers

2012-01-29 Thread aliyeh saeedi
Hi Does Hadoop behave with reducer's output like other files in the case of replication and keeping their metadata in NameNode?

Re: reducers outputs

2012-01-29 Thread Arun C Murthy
The NameNode doesn't care how you wrote the file i.e. either via 'bin/hadoop dfs -put <>' or via a MR job. Arun On Jan 29, 2012, at 10:09 PM, aliyeh saeedi wrote: > I studied it, but I could not get the point. I mean if I save reducer's > output with my own selected names, does NameNode behave

Re: Fw: reducers outputs

2012-01-29 Thread Harsh J
Aliyeh, You may be complicating things here. The HDFS and MapReduce are two separate components of Hadoop. HDFS provides a distributed FileSystem, MapReduce provides a distributed processing layer. They aren't glued. A reducer creates an output file on a 'filesystem'. It does not know nor care i

Re: Fw: reducers outputs

2012-01-29 Thread aliyeh saeedi
I studied it, but I could not get the point. I mean if I save reducer's output with my own selected names, does NameNode behave with them like other files? regards. From: Ashwanth Kumar To: mapreduce-user@hadoop.apache.org; aliyeh saeedi Sent: Monday, 30 Jan

Re: Fw: reducers outputs

2012-01-29 Thread Ashwanth Kumar
You should have a look at this - http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/FileOutputFormat.html - Ashwanth Kumar On Mon, Jan 30, 2012 at 11:17 AM, aliyeh saeedi wrote: > > > > I want to save them with my own names, How NameNode will keep their names? > >--

Fw: reducers outputs

2012-01-29 Thread aliyeh saeedi
I want to save them with my own names, How NameNode will keep their names? From: Joey Echeverria To: mapreduce-user@hadoop.apache.org; aliyeh saeedi Sent: Sunday, 29 January 2012, 17:10 Subject: Re: reducers outputs Reduce output is normally stored i

Re: reducers outputs

2012-01-29 Thread Joey Echeverria
Reduce output is normally stored in HDFS, just like your other files. Are you seeing different behavior? -Joey On Sun, Jan 29, 2012 at 1:05 AM, aliyeh saeedi wrote: > Hi > I want to save reducers outputs like other files in Hadoop. Does NameNode > keep any information about them? How can I do th

Re: Cannot open filename - in mapreduce program

2012-01-29 Thread Harsh J
Hi Vamshi, This question does not look specific to HBase. Moving to mapreduce-user@hadoop.apache.org. Please use the appropriate lists for your questions. (Bcc'd u...@hbase.apache.org, cc'd OP). For your problem, I believe your mistake may lie in thinking that the output path of your previous job

anyway to do "local" reduce like the combiner does?

2012-01-29 Thread Jianhui Zhang
I have a problem at hand that seems to need "local" reducing: I have a large data input, in which each line is a data mapping, something like "name : attribute". The attributes for the same name are usually pretty close in the file, so they are very likely to be processed by the same mapper.