stack created HBASE-15207: ----------------------------- Summary: Stuck balancer; logging ten lines a millisecond Key: HBASE-15207 URL: https://issues.apache.org/jira/browse/HBASE-15207 Project: HBase Issue Type: Bug Components: Balancer Affects Versions: 1.2.0 Reporter: stack
Last night my logs filled with this (10x256MB log files): .... 2016-02-01 11:25:26,958 DEBUG [B.defaultRpcServer.handler=9,queue=0,port=16000] balancer.BaseLoadBalancer: Lowest locality region server with non zero regions is ve0542.halxg.cloudera.com with locality 0.0 2016-02-01 11:25:26,958 DEBUG [B.defaultRpcServer.handler=9,queue=0,port=16000] balancer.BaseLoadBalancer: Lowest locality region index is 0 and its region server contains 1 regions ... Added by this: commit 54028140f4f19a6af81c8c8f29dda0c52491a0c9 Author: tedyu <yuzhih...@gmail.com> Date: Thu Aug 13 09:11:59 2015 -0700 HBASE-13376 Improvements to Stochastic load balancer (Vandana Ayyalasomayajula) Looks like balancer got stuck. Logging at ten lines a millisecond. Here is lead up. Nothing in particular jumps out. Rerun doesn't show this. {code} 2016-01-28 05:56:22,572 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Server ve0526.halxg.cloudera.com had 0 regions. 2016-01-28 05:56:22,572 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Server ve0532.halxg.cloudera.com had 0 regions. 2016-01-28 05:56:22,572 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Server ve0538.halxg.cloudera.com had 0 regions. 2016-01-28 05:56:22,572 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Lowest locality region server with non zero regions is ve0540.halxg.cloudera.com with locality 0.0 2016-01-28 05:56:22,572 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Lowest locality region index is 0 and its region server contains 1 regions 2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Server ve0526.halxg.cloudera.com had 0 regions. 2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Server ve0532.halxg.cloudera.com had 0 regions. 2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Server ve0538.halxg.cloudera.com had 0 regions. 2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Lowest locality region server with non zero regions is ve0540.halxg.cloudera.com with locality 0.0 2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Lowest locality region index is 0 and its region server contains 1 regions 2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Server ve0526.halxg.cloudera.com had 0 regions. 2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Server ve0532.halxg.cloudera.com had 0 regions. 2016-01-28 05:56:22,573 DEBUG [ve0524.halxg.cloudera.com,16000,1453988766013_ChoreService_1] balancer.BaseLoadBalancer: Server ve0538.halxg.cloudera.com had 0 regions. .... {code} Nothing else is happening on this master Happens just after a Master joins cluster after being killed by a monkey. -- This message was sent by Atlassian JIRA (v6.3.4#6332)