Thanks Harsh, I will go through what you have suggested. But I think these options will help to prevent under replicated blocks. What about missing replicas. Is there any way to fix it or to prevent it ?
-----Original Message----- From: Harsh J [mailto:ha...@cloudera.com] Sent: Monday, October 17, 2011 10:58 PM To: hdfs-user@hadoop.apache.org Subject: Re: Missing Replicas and Under replicated blocks Stuti, You'll want to: 1. Set "2" for mapred.submit.replication in mapred-site.xml to prevent the job related staging files to be created with default replication factor of 10. 2. Learn about the "-setrep" option to toggle replication factors of a path. See http://hadoop.apache.org/common/docs/current/file_system_shell.html#setrep HTH. On Mon, Oct 17, 2011 at 11:51 AM, Stuti Awasthi <stutiawas...@hcl.com> wrote: > Hi , > > I have a mini Hadoop cluster of 3 nodes with DN and TT running on each node > and replication is set to 3.By mistake I tried to write few files which > requires replication of 10. When I checked the HDFS status with FSCK utility > it shows : > > /root/Hadoop/hadoop-root/mapred/system/job_201109201112_0001/job.jar 2352841 > bytes, 1 block(s): Under replicated blk_-9220069752024407718_1546. Target > Replicas is 10 but found 3 replica(s). > 0. blk_-9220069752024407718_1546 len=2352841 repl=3 > [10.33.100.74:50010, 10.33.64.235:50010, 10.33.100.117:50010] > > /root/Hadoop/hadoop-root/mapred/system/job_201109201112_0001/libjars > <dir> > /root/Hadoop/hadoop-root/mapred/system/job_201109201112_0001/libjars/hadoop-0.20.2-core.jar > 2707856 bytes, 1 block(s): Under replicated blk_-4791859336845413240_1544. > Target Replicas is 10 but found 3 replica(s). > 0. blk_-4791859336845413240_1544 len=2707856 repl=3 > [10.33.100.74:50010, 10.33.64.235:50010, 10.33.100.117:50010] > .................................................... > Status: HEALTHY > Total size: 45239704 B > Total dirs: 101 > Total files: 88 (Files currently being written: 3) > Total blocks (validated): 88 (avg. block size 514087 B) (Total > open file blocks (not validated): 3) > Minimally replicated blocks: 88 (100.0 %) > Over-replicated blocks: 0 (0.0 %) > Under-replicated blocks: 20 (22.727272 %) > Mis-replicated blocks: 0 (0.0 %) > Default replication factor: 2 > Average block replication: 2.4318182 > Corrupt blocks: 0 > Missing replicas: 140 (65.42056 %) > Number of data-nodes: 3 > Number of racks: 1 > > I wanted to know that can I fix : > 1. Under-replicated > 2. Missing replicas > > Please Suggest > Thanks > Stuti > > ::DISCLAIMER:: > ---------------------------------------------------------------------- > ------------------------------------------------- > > The contents of this e-mail and any attachment(s) are confidential and > intended for the named recipient(s) only. > It shall not attach any liability on the originator or HCL or its > affiliates. Any views or opinions presented in this email are solely those of > the author and may not necessarily reflect the opinions of HCL or its > affiliates. > Any form of reproduction, dissemination, copying, disclosure, > modification, distribution and / or publication of this message > without the prior written consent of the author of this e-mail is > strictly prohibited. If you have received this email in error please delete > it and notify the sender immediately. Before opening any mail and attachments > please check them for viruses and defect. > > ---------------------------------------------------------------------- > ------------------------------------------------- > -- Harsh J
