Have you had any Name Node failures lately? I had them every couple of days and found that there were files being left in hdfs /log/hadoop/tmp/mapred/staging/... when communications with the Name Node was lost. Not sure why they never got replicated correctly (maybe because they are in /log?)
I went in and removed the old files (say 2 days older or older) and saw the # of blocks drop to 0. Hope this helps, Chris On Mon, Feb 20, 2012 at 1:25 AM, praveenesh kumar <praveen...@gmail.com>wrote: > I recently added a new DN/TT to my cluster. Could it be the reason for such > behaviour ? > > Thanks, > Praveenesh > > On Mon, Feb 20, 2012 at 11:51 AM, Harsh J <ha...@cloudera.com> wrote: > > > Hi, > > > > The tool "hadoop fsck" will tell you which files are under replicated > > with a count of what was expected instead, just run it over /. > > > > While it isn't a 'normal' thing to see it come up suddenly it is still > > in the safe zone, and is most likely an indicator that either one of > > your DN or one of its disks has gone bad, or you have a bad > > mapred.submit.replication value for your cluster size (default is 10 > > replicas for all MR job submit data), or bit rot of existing blocks on > > HDDs around the cluster, etc. -- You can mostly spot the pattern of > > files causing it by running the fsck and obtaining the listing. > > > > On Mon, Feb 20, 2012 at 11:43 AM, praveenesh kumar <praveen...@gmail.com > > > > wrote: > > > Hi, > > > > > > I am suddenly seeing some under-replicated blocks on my cluster. > Although > > > its not causing any problems, but It seems like few blocks are not > > > replicated properly. > > > > > > Number of Under-Replicated Blocks : 147 > > > > > > Is it okay behavior on hadoop. If no, How can I know what are the files > > > with under-replicated blocks and how can I configure it properly to > > reduce > > > the number of under-replicated blocks. > > > > > > Thanks, > > > Praveenesh > > > > > > > > -- > > Harsh J > > Customer Ops. Engineer > > Cloudera | http://tiny.cloudera.com/about > > >
