Re: Trying to put 16gb file onto hdfs
Check your Datanode logs.. or do "hadoop fsck /" or "hadoop dfsadmin -report" to get more details about your HDFS. Seems like DN is down. Regards, Praveenesh On Tue, Jun 5, 2012 at 12:13 AM, wrote: > Hi Sean, > > It seems your HDFS has not properly started. Go through your HDFS > webconsole top verify if NN and all DN are up. You can access that on > http://:50070 > Also ensure yourself that your NN has left Safe Mode before start moving > data to HDFS. > > > -Original Message- > From: Barry, Sean F [mailto:sean.f.ba...@intel.com] > Sent: lunes, 04 de junio de 2012 19:11 > To: common-user@hadoop.apache.org > Subject: Trying to put 16gb file onto hdfs > > I am trying to put a 16gb file on to hdfs but I was given all of these > messages and I don't know why this is happening. Can someone please shed > some light on this scenario. > Thanks in advance > > hduser@master:~> hadoop fs -put ~/tests/wiki16gb.txt > /user/hduser/wiki/16gb.txt > > 12/06/04 10:52:05 WARN hdfs.DFSClient: DataStreamer Exception: > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File > /user/hduser/wiki/16gb.txt could only be replicated to 0 nodes, instead of > 1 >at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1520) >at > org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:665) >at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) >at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >at java.lang.reflect.Method.invoke(Method.java:601) >at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557) >at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434) >at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430) >at java.security.AccessController.doPrivileged(Native Method) >at javax.security.auth.Subject.doAs(Subject.java:415) >at > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157) >at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428) > >at org.apache.hadoop.ipc.Client.call(Client.java:1107) >at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226) >at $Proxy0.addBlock(Unknown Source) >at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) >at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >at java.lang.reflect.Method.invoke(Method.java:601) >at > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) >at > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) >at $Proxy0.addBlock(Unknown Source) >at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3553) >at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3421) >at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2100(DFSClient.java:2627) >at > > org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2822) > > 12/06/04 10:52:05 WARN hdfs.DFSClient: Error Recovery for block > blk_6880553273766367307_1055 bad datanode[0] nodes == null > 12/06/04 10:52:05 WARN hdfs.DFSClient: Could not get block locations. > Source file "/user/hduser/wiki/16gb.txt" - Aborting... > put: java.io.IOException: File /user/hduser/wiki/16gb.txt could only be > replicated to 0 nodes, instead of 1 > 12/06/04 10:52:05 ERROR hdfs.DFSClient: Exception closing file > /user/hduser/wiki/16gb.txt : org.apache.hadoop.ipc.RemoteException: > java.io.IOException: File /user/hduser/wiki/16gb.txt could only be > replicated to 0 nodes, instead of 1 >at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1520) >at > org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:665) >at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) >at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >at java.lang.reflect.Method.invoke(Method.java:601) >at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557) >at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434) >at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430) >at java.security.AccessController.doPrivileged(Native Method) >at javax.security.auth.Subject.doAs(Subject.java:415) >at > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157) >at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428) > > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File > /user/hduser/wiki/16gb.txt could only be replicated to 0 nodes, instead of > 1 >at > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.
RE: Trying to put 16gb file onto hdfs
Hi Sean, It seems your HDFS has not properly started. Go through your HDFS webconsole top verify if NN and all DN are up. You can access that on http://:50070 Also ensure yourself that your NN has left Safe Mode before start moving data to HDFS. -Original Message- From: Barry, Sean F [mailto:sean.f.ba...@intel.com] Sent: lunes, 04 de junio de 2012 19:11 To: common-user@hadoop.apache.org Subject: Trying to put 16gb file onto hdfs I am trying to put a 16gb file on to hdfs but I was given all of these messages and I don't know why this is happening. Can someone please shed some light on this scenario. Thanks in advance hduser@master:~> hadoop fs -put ~/tests/wiki16gb.txt /user/hduser/wiki/16gb.txt 12/06/04 10:52:05 WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hduser/wiki/16gb.txt could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1520) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:665) at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428) at org.apache.hadoop.ipc.Client.call(Client.java:1107) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226) at $Proxy0.addBlock(Unknown Source) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) at $Proxy0.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3553) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3421) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2100(DFSClient.java:2627) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2822) 12/06/04 10:52:05 WARN hdfs.DFSClient: Error Recovery for block blk_6880553273766367307_1055 bad datanode[0] nodes == null 12/06/04 10:52:05 WARN hdfs.DFSClient: Could not get block locations. Source file "/user/hduser/wiki/16gb.txt" - Aborting... put: java.io.IOException: File /user/hduser/wiki/16gb.txt could only be replicated to 0 nodes, instead of 1 12/06/04 10:52:05 ERROR hdfs.DFSClient: Exception closing file /user/hduser/wiki/16gb.txt : org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hduser/wiki/16gb.txt could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1520) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:665) at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428) org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /user/hduser/wiki/16gb.txt could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1520) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:665) at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601)
