When I checked hdfs I found out that in system directory present in HDFS , all
the Under replicated blocks are present :
[root@master ~]# Hadoop/hadoop-0.20.2/bin/hadoop dfs -ls
/root/Hadoop//hadoop-root//mapred/system
Found 6 items
drwxrwxrwx - root supergroup 0 2011-09-27 18:17
/root/Hadoop/hadoop-root/mapred/system/job_201109201112_0001
drwxrwxrwx - root supergroup 0 2011-09-27 18:20
/root/Hadoop/hadoop-root/mapred/system/job_201109201112_0003
drwxrwxrwx - root supergroup 0 2011-09-27 18:24
/root/Hadoop/hadoop-root/mapred/system/job_201109201112_0005
drwxrwxrwx - root supergroup 0 2011-09-27 18:28
/root/Hadoop/hadoop-root/mapred/system/job_201109201112_0006
drwxrwxrwx - root supergroup 0 2011-09-28 10:09
/root/Hadoop/hadoop-root/mapred/system/job_201109201112_0009
-rw------- 2 root supergroup 4 2011-09-20 11:13
/root/Hadoop/hadoop-root/mapred/system/jobtracker.info
Now since , I just have replication set to 2 nodes, can I just fix under
replicated block error ?
-----Original Message-----
From: Stuti Awasthi
Sent: Monday, October 17, 2011 12:02 PM
To: hdfs-user@hadoop.apache.org
Subject: RE: Missing Replicas and Under replicated blocks
After some search I found out few things which I think can be linked in some
way but I am not sure. If anyone can suggest on this :
1. I have set <hadoop.tmp.dir> value in core-site.xml <property>
<name>hadoop.tmp.dir</name>
<value>/root/Hadoop/hadoop-root</value>
<description>A base for other temporary directories.</description>
</property>
2. I have not set <mapred.system.dir> separately in conf files.
3. In servers directory structure ${ hadoop.tmp.dir}/mapred, "system" directory
is not created.
4. "system " is created under /tmp directory :
/tmp/hadoop-root/mapred/system
Can all this be linked with Missing replicas ?
-----Original Message-----
From: Stuti Awasthi
Sent: Monday, October 17, 2011 11:52 AM
To: hdfs-user@hadoop.apache.org
Subject: Missing Replicas and Under replicated blocks
Hi ,
I have a mini Hadoop cluster of 3 nodes with DN and TT running on each node
and replication is set to 3.By mistake I tried to write few files which
requires replication of 10. When I checked the HDFS status with FSCK utility it
shows :
/root/Hadoop/hadoop-root/mapred/system/job_201109201112_0001/job.jar 2352841
bytes, 1 block(s): Under replicated blk_-9220069752024407718_1546. Target
Replicas is 10 but found 3 replica(s).
0. blk_-9220069752024407718_1546 len=2352841 repl=3 [10.33.100.74:50010,
10.33.64.235:50010, 10.33.100.117:50010]
/root/Hadoop/hadoop-root/mapred/system/job_201109201112_0001/libjars <dir>
/root/Hadoop/hadoop-root/mapred/system/job_201109201112_0001/libjars/hadoop-0.20.2-core.jar
2707856 bytes, 1 block(s): Under replicated blk_-4791859336845413240_1544.
Target Replicas is 10 but found 3 replica(s).
0. blk_-4791859336845413240_1544 len=2707856 repl=3 [10.33.100.74:50010,
10.33.64.235:50010, 10.33.100.117:50010]
....................................................
Status: HEALTHY
Total size: 45239704 B
Total dirs: 101
Total files: 88 (Files currently being written: 3)
Total blocks (validated): 88 (avg. block size 514087 B) (Total open file
blocks (not validated): 3)
Minimally replicated blocks: 88 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 20 (22.727272 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.4318182
Corrupt blocks: 0
Missing replicas: 140 (65.42056 %)
Number of data-nodes: 3
Number of racks: 1
I wanted to know that can I fix :
1. Under-replicated
2. Missing replicas
Please Suggest
Thanks
Stuti
::DISCLAIMER::
-----------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended
for the named recipient(s) only.
It shall not attach any liability on the originator or HCL or its affiliates.
Any views or opinions presented in
this email are solely those of the author and may not necessarily reflect the
opinions of HCL or its affiliates.
Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of
this message without the prior written consent of the author of this e-mail is
strictly prohibited. If you have
received this email in error please delete it and notify the sender
immediately. Before opening any mail and
attachments please check them for viruses and defect.
-----------------------------------------------------------------------------------------------------------------------
