I double checked the cluster, all of the disk have plenty of free space. I could not put any data into the cluster. The cluster summary shows
== Cluster Summary***1987209 files and directories, 1993327 blocks = 3980536 total. Heap Size is 4.2 GB / 5.56 GB (75%) * Configured Capacity : 120.88 TB DFS Used : 49.55 TB Non DFS Used : 6.12 TBDFS Remaining:65.21 TBDFS Used%:40.99 %DFS Remaining%:53.95 % Live Nodes <http://cdn-hadoop-01:50070/dfsnodelist.jsp?whatNodes=LIVE> : 14 Dead Nodes <http://cdn-hadoop-01:50070/dfsnodelist.jsp?whatNodes=DEAD> : 0 == and hadoop fsck / shows good == Status: HEALTHY Total size: 17900052452875 B Total dirs: 35613 Total files: 1951594 (Files currently being written: 1) Total blocks (validated): 1992835 (avg. block size 8982204 B) Minimally replicated blocks: 1992835 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 5 (2.5089885E-4 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 3.0000422 Corrupt blocks: 0 Missing replicas: 21 (3.5125346E-4 %) Number of data-nodes: 14 Number of racks: 1 The filesystem under path '/' is HEALTHY == Any suggestions? Thanks! On Sat, Apr 16, 2011 at 9:48 PM, Sonal Goyal <[email protected]> wrote: > Hi, > > What is the cluster free disk space and current usage? > > Thanks and Regards, > Sonal > <https://github.com/sonalgoyal/hiho>Hadoop ETL and Data > Integration<https://github.com/sonalgoyal/hiho> > Nube Technologies <http://www.nubetech.co> > > <http://in.linkedin.com/in/sonalgoyal> > > > > > > > On Sat, Apr 16, 2011 at 9:18 AM, 茅旭峰 <[email protected]> wrote: > >> Hi Jun Young, >> >> It looks like I'm also running into the same issue you've got. >> Have you found more info? I googled >> 'java.io.IOException: Unable to create new block.', some guys >> suggested that it might be running out of fd, but it's not my case. >> I verified the fds in use by lsof and ls /proc/pid/fd. >> Do you have any hints? >> >> Thanks! >> >> Mao Xu-Feng >> >> On Fri, Feb 18, 2011 at 10:51 AM, Jun Young Kim <[email protected]> >> wrote: >> >> > hi, harsh. >> > you're always giving a response very quickly. ;) >> > >> > I am using a version 0.21.0 now. >> > before asking about this problem, I've checked already file system >> healthy. >> > >> > $> hadoop fsck / >> > ..... >> > ..... >> > Status: HEALTHY >> > Total size: 24231595038 B >> > Total dirs: 43818 >> > Total files: 41193 (Files currently being written: 2178) >> > Total blocks (validated): 40941 (avg. block size 591866 B) (Total >> > open file blocks (not validated): 224) >> > Minimally replicated blocks: 40941 (100.0 %) >> > Over-replicated blocks: 1 (0.0024425392 %) >> > Under-replicated blocks: 2 (0.0048850784 %) >> > Mis-replicated blocks: 0 (0.0 %) >> > Default replication factor: 2 >> > Average block replication: 2.1106226 >> > Corrupt blocks: 0 >> > Missing replicas: 4 (0.00462904 %) >> > Number of data-nodes: 8 >> > Number of racks: 1 >> > >> > The filesystem under path '/' is HEALTHY >> > >> > additionally, I found a little different error. here it is. >> > >> > java.io.IOException: Bad connect ack with firstBadLink as >> > 10.25.241.107:50010 at >> > >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:889) >> > at >> > >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:820) >> > at >> > >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:427) >> > >> > >> > here is my execution environment. >> > >> > average job count : 20 >> > max map capacity : 128 >> > max reduce capacity : 128 >> > avg/slot per node : 32 >> > >> > avg input file size per job : 200M ~ 1G >> > >> > >> > thanks. >> > >> > Junyoung Kim ([email protected]) >> > >> > >> > On 02/18/2011 11:43 AM, Harsh J wrote: >> > >> >> You may want to check your HDFS health stat via 'fsck' >> >> (http://namenode/fsck or `hadoop fsck`). There may be a few corrupt >> >> files or bad DNs. >> >> >> >> Would also be good to know what exact version of Hadoop you're running. >> >> >> >> On Fri, Feb 18, 2011 at 7:59 AM, Jun Young Kim<[email protected]> >> >> wrote: >> >> >> >>> hi, all. >> >>> >> >>> I got errors from hdfs. >> >>> >> >>> 2011-02-18 11:21:29[WARN ][DFSOutputStream.java]run()(519) : >> DataStreamer >> >>> Exception: java.io.IOException: Unable to create new block. >> >>> at >> >>> >> >>> >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:832) >> >>> at >> >>> >> >>> >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:427) >> >>> >> >>> 2011-02-18 11:21:29[WARN >> >>> ][DFSOutputStream.java]setupPipelineForAppendOrRecovery()(730) : Could >> >>> not >> >>> get block locations. Source file >> >>> "/user/test/51/output/ehshop00newsvc-r-00000" - Aborting... >> >>> 2011-02-18 11:21:29[WARN ][Child.java]main()(234) : Exception running >> >>> child >> >>> : java.io.EOFException >> >>> at java.io.DataInputStream.readShort(DataInputStream.java:298) >> >>> at >> >>> >> >>> >> org.apache.hadoop.hdfs.protocol.DataTransferProtocol$Status.read(DataTransferProtocol.java:113) >> >>> at >> >>> >> >>> >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:881) >> >>> at >> >>> >> >>> >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:820) >> >>> at >> >>> >> >>> >> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:427) >> >>> >> >>> 2011-02-18 11:21:29[INFO ][Task.java]taskCleanup()(996) : Runnning >> >>> cleanup >> >>> for the task >> >>> >> >>> >> >>> >> >>> I think this one is also not different error. >> >>> >> >>> org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: >> >>> blk_-2325764274016776017_8292 file=/user/test/51/input/kids.txt >> >>> >> >>> at >> >>> >> >>> >> org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:559) >> >>> >> >>> at >> >>> >> >>> >> org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:367) >> >>> >> >>> at >> >>> org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:514) >> >>> >> >>> at java.io.DataInputStream.read(DataInputStream.java:83) >> >>> >> >>> at >> org.apache.hadoop.util.LineReader.readLine(LineReader.java:138) >> >>> >> >>> at >> >>> >> >>> >> org.apache.hadoop.mapreduce.lib.input.LineRecordReader.nextKeyValue(LineRecordReader.java:149) >> >>> >> >>> at >> >>> >> >>> >> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:465) >> >>> >> >>> at >> >>> >> >>> >> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80) >> >>> >> >>> at >> >>> >> >>> >> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:90) >> >>> >> >>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) >> >>> >> >>> >> >>> --> I've checked the file '/user/test/51/input/kids.txt ', but, there >> is >> >>> not >> >>> strange ones. this file is healthy. >> >>> >> >>> Does anybody know about this error? >> >>> How could I fix this one? >> >>> >> >>> thanks. >> >>> >> >>> -- >> >>> Junyoung Kim ([email protected]) >> >>> >> >>> >> >>> >> >> >> >> >> > >
