Sorry if I'm asking a silly question... Are you sure your RSs and Datanodes are all up and running? Are you sure they are collocated?
> Datanode on l-hbase[26-31].data.cn8 and regionserver on > l-hbase[25-31].data.cn8, Could be that your only live RS is on l-hbase25.data.cn8, which would cause that behavior... Btw, why 25th is not collocated with datanode? Alex Baranau -- http://cdap.io - open source framework to build and run data applications on Hadoop & HBase On Fri, May 15, 2015 at 8:12 PM, Louis Hust <louis.h...@gmail.com> wrote: > Hi, Esteban, > > Hadoop Version 2.2.0, r1537062. > So i do not know why it always write other datanode instead of local > datanode, > If there is some log for the hdfs write policy? And now the cluster is > working not healthy, > with heavy networking. > > 2015-05-15 1:28 GMT+08:00 Esteban Gutierrez <este...@cloudera.com>: > > > Hi Louis, > > > > Locality 0 is not right for a cluster of that size and having 3 replicas > > per block unless all RS cannot connect to the local DN and somehow the > > local DN to the RS is always excluded from the pipeline. In Hadoop > > 2.0-alpha there was a bug (HDFS-3224) that caused the NN to report a DN > as > > live and dead if the storage ID was changed in a single volume (e.g. > after > > replacing one drive) and that caused fs.getFileBlockLocations() to report > > less blocks for calculating the HDFS locality index. Unless your cluster > is > > using Hadoop 2.0-alpha I won't worry too much about that. > > > > Regarding the logs its odd that the JN is taking about 1.5 seconds just > to > > send less than 200 bytes. Perhaps some IO contention issue is going on in > > your cluster? > > > > thanks, > > esteban. > > > > -- > > Cloudera, Inc. > > > > > > On Thu, May 14, 2015 at 5:48 AM, Louis Hust <louis.h...@gmail.com> > wrote: > > > > > Hi, Esteban > > > > > > Each region server has about 122 regions, data is large. HDFS replica > is > > > defined as default 3, and namenode have some WARN like below. > > > > > > {log} > > > 2015-05-14 20:45:37,463 WARN > > > org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Took > 1503ms > > to > > > send a batch of 3 edits (179 bytes) to remote journal > 192.168.44.29:8485 > > > {/log} > > > > > > Regionserver's log seems normal: > > > > > > {log} > > > 2015-05-14 20:46:59,890 INFO [Thread-15] regionserver.HRegion: > Finished > > > memstore flush of ~44.4 M/46586984, currentsize=0/0 for region > > > > > > > > > qmq_backup,0066485937885860620cb396a3e65c6c9de92cae9aa29,1412429632233.65684ef65f58cb3e27986ca38d397bee. > > > in 3141ms, sequenceid=7493455453, compaction requested=true > > > 2015-05-14 20:46:59,890 INFO > > > [regionserver60020-smallCompactions-1431462564717] > regionserver.HRegion: > > > Starting compaction on m in region > > > > > > > > > qmq_backup,0066485937885860620cb396a3e65c6c9de92cae9aa29,1412429632233.65684ef65f58cb3e27986ca38d397bee. > > > {/log} > > > > > > Any idea? > > > > > > > > > > > > 2015-05-13 1:26 GMT+08:00 Esteban Gutierrez <este...@cloudera.com>: > > > > > > > Hi, > > > > > > > > How many regions you per RS? one possibility is that you have very > > little > > > > data in your cluster and regions have moved around and there are no > > > blocks > > > > in the local DN to the RS. Another possibility is that you have one > > > replica > > > > configured and regions moved too so that makes even harder to have > some > > > > local blocks in the DN to the RS. Lastly it could be some other > problem > > > > where the HDFS pipeline has excluded the local DN. Have you seen any > > > > exception in the RSs or the NameNode that might be interesting? > > > > > > > > thanks, > > > > esteban. > > > > > > > > > > > > > > > > -- > > > > Cloudera, Inc. > > > > > > > > > > > > On Tue, May 12, 2015 at 2:59 AM, 娄帅 <louis.hust...@gmail.com> wrote: > > > > > > > > > Hi, all, > > > > > > > > > > I am maintaining an hbase 0.96.0 cluster, but from the web ui of > > HBase > > > > > regionserver, > > > > > i saw Block locality is 0 for all regionserver. > > > > > > > > > > Datanode on l-hbase[26-31].data.cn8 and regionserver on > > > > > l-hbase[25-31].data.cn8, > > > > > > > > > > Any idea? > > > > > > > > > > > > > > >
