RE: More replication of map reduce output
Yes I hav mark it as final. Now one more exception arises, my map reduce program for word count is throwing exception. 09/05/21 11:31:37 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 09/05/21 11:31:37 INFO hdfs.DFSClient: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /hadoop/mapred/system/job_20090527_0003/job.jar could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(F SNamesystem.java:1280) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:3 51) at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:481) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:894) at org.apache.hadoop.ipc.Client.call(Client.java:697) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216) at $Proxy0.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvo cationHandler.java:82) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocation Handler.java:59) at $Proxy0.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DF SClient.java:2814) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(D FSClient.java:2696) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.j ava:1996) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSCli ent.java:2183) 09/05/21 11:31:37 WARN hdfs.DFSClient: NotReplicatedYetException sleeping /hadoop/mapred/system/job_20090527_0003/job.jar retries left 4 -Original Message- From: Michael C. Toren [mailto:m...@toren.net] Sent: Thursday, May 21, 2009 11:27 AM To: core-user@hadoop.apache.org Subject: Re: More replication of map reduce output On Thu, May 21, 2009 at 11:16:59AM +0530, Puri, Aseem wrote: > I mean when my reduce tasks is set 1 part-0 filw shows replication facor as 3. But I set replication factor as 1 in hadoop-site.xml Did you mark the replication factor configuration option as "final"? e.g.: dfs.replication 1 true -mct
Re: More replication of map reduce output
On Thu, May 21, 2009 at 11:16:59AM +0530, Puri, Aseem wrote: > I mean when my reduce tasks is set 1 part-0 filw shows replication facor > as 3. But I set replication factor as 1 in hadoop-site.xml Did you mark the replication factor configuration option as "final"? e.g.: dfs.replication 1 true -mct
RE: More replication of map reduce output
I mean when my reduce tasks is set 1 part-0 filw shows replication facor as 3. But I set replication factor as 1 in hadoop-site.xml -Original Message- From: edw...@udanax.org [mailto:edw...@udanax.org] On Behalf Of Edward J. Yoon Sent: Thursday, May 21, 2009 11:14 AM To: core-user@hadoop.apache.org Subject: Re: More replication of map reduce output Do you mean the three files such as, part-0? If so, you can set the number of reduce tasks as 1. On Thu, May 21, 2009 at 2:39 PM, Puri, Aseem wrote: > Hi > > I have running a map reduce program on two node. My DFS > replication factor is one. Al files for input have one replication but > the output from reduce always have replication 3. Can anyone please tell > why it is so? > > > > Thanks & Regards > > Aseem Puri > > > > -- Best Regards, Edward J. Yoon @ NHN, corp. edwardy...@apache.org http://blog.udanax.org
Re: More replication of map reduce output
Do you mean the three files such as, part-0? If so, you can set the number of reduce tasks as 1. On Thu, May 21, 2009 at 2:39 PM, Puri, Aseem wrote: > Hi > > I have running a map reduce program on two node. My DFS > replication factor is one. Al files for input have one replication but > the output from reduce always have replication 3. Can anyone please tell > why it is so? > > > > Thanks & Regards > > Aseem Puri > > > > -- Best Regards, Edward J. Yoon @ NHN, corp. edwardy...@apache.org http://blog.udanax.org
