shen guanpu created HBASE-6465: ---------------------------------- Summary: Load balancer repeatedly close and open region in the same regionserver. Key: HBASE-6465 URL: https://issues.apache.org/jira/browse/HBASE-6465 Project: HBase Issue Type: Bug Components: master, regionserver Affects Versions: 0.94.0 Reporter: shen guanpu
Through the master and regionserver log,I find load balancer repeatedly close and open region in the same regionserver(period in hbase.balancer.period ). Does this is a bug in load balancer and how can I dig into or avoid this? the hbase and hadoop version is HBase Version0.94.0, r1332822Hadoop Version0.20.2-cdh3u1, rbdafb1dbffd0d5f2fbc6ee022e1c8df6500fd638 the following is a detail log about the same region trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956, and it repeats again and again.: 2012-07-16 00:12:49,843 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956., src=192.168.1.2,60020,1342017399608, dest=192.168.1.2,60020,1342002082592 2012-07-16 00:12:49,843 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956. (offlining) 2012-07-16 00:12:49,843 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x4384d0a47f40068 Creating unassigned node for 93caf5147d40f5dd4625e160e1b7e956 in a CLOSING state 2012-07-16 00:12:49,845 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to 192.168.1.2,60020,1342017399608 for region trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956. 2012-07-16 00:12:50,555 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_CLOSED, server=192.168.1.2,60020,1342017399608, region=93caf5147d40f5dd4625e160e1b7e956 2012-07-16 00:12:50,555 DEBUG org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED event for 93caf5147d40f5dd4625e160e1b7e956 2012-07-16 00:12:50,555 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956. state=CLOSED, ts=1342368770556, server=192.168.1.2,60020,1342017399608 2012-07-16 00:12:50,555 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x4384d0a47f40068 Creating (or updating) unassigned node for 93caf5147d40f5dd4625e160e1b7e956 with OFFLINE state 2012-07-16 00:12:50,558 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE, server=10.75.18.34,60000,1342017369575, region=93caf5147d40f5dd4625e160e1b7e956 2012-07-16 00:12:50,558 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956. destination server is 192.168.1.2,60020,1342002082592 2012-07-16 00:12:50,558 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan for region trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956.; plan=hri=trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956., src=192.168.1.2,60020,1342017399608, dest=192.168.1.2,60020,1342002082592 2012-07-16 00:12:50,558 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956. to 192.168.1.2,60020,1342002082592 2012-07-16 00:12:50,574 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=192.168.1.2,60020,1342017399608, region=93caf5147d40f5dd4625e160e1b7e956 2012-07-16 00:12:50,635 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=192.168.1.2,60020,1342017399608, region=93caf5147d40f5dd4625e160e1b7e956 2012-07-16 00:12:50,639 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=192.168.1.2,60020,1342017399608, region=93caf5147d40f5dd4625e160e1b7e956 2012-07-16 00:12:50,639 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED event for trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956. from 192.168.1.2,60020,1342017399608; deleting unassigned node 2012-07-16 00:12:50,640 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x4384d0a47f40068 Deleting existing unassigned node for 93caf5147d40f5dd4625e160e1b7e956 that is in expected state RS_ZK_REGION_OPENED 2012-07-16 00:12:50,641 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: The znode of region trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956. has been deleted. 2012-07-16 00:12:50,641 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x4384d0a47f40068 Successfully deleted unassigned node for region 93caf5147d40f5dd4625e160e1b7e956 in expected state RS_ZK_REGION_OPENED 2012-07-16 00:12:50,641 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the region trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956. that was online on 192.168.1.2,60020,1342017399608 2012-07-16 00:17:49,870 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956., src=192.168.1.2,60020,1342017399608, dest=192.168.1.2,60020,1342002082592 2012-07-16 00:17:49,870 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Starting unassignment of region trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956. (offlining) 2012-07-16 00:17:49,870 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x4384d0a47f40068 Creating unassigned node for 93caf5147d40f5dd4625e160e1b7e956 in a CLOSING state 2012-07-16 00:17:49,872 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Sent CLOSE to 192.168.1.2,60020,1342017399608 for region trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956. 2012-07-16 00:17:50,464 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_CLOSED, server=192.168.1.2,60020,1342017399608, region=93caf5147d40f5dd4625e160e1b7e956 2012-07-16 00:17:50,464 DEBUG org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED event for 93caf5147d40f5dd4625e160e1b7e956 2012-07-16 00:17:50,464 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956. state=CLOSED, ts=1342369070465, server=192.168.1.2,60020,1342017399608 2012-07-16 00:17:50,464 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x4384d0a47f40068 Creating (or updating) unassigned node for 93caf5147d40f5dd4625e160e1b7e956 with OFFLINE state 2012-07-16 00:17:50,467 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE, server=10.75.18.34,60000,1342017369575, region=93caf5147d40f5dd4625e160e1b7e956 2012-07-16 00:17:50,467 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956. destination server is 192.168.1.2,60020,1342002082592 2012-07-16 00:17:50,467 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan for region trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956.; plan=hri=trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956., src=192.168.1.2,60020,1342017399608, dest=192.168.1.2,60020,1342002082592 2012-07-16 00:17:50,467 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956. to 192.168.1.2,60020,1342002082592 2012-07-16 00:17:50,509 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=192.168.1.2,60020,1342017399608, region=93caf5147d40f5dd4625e160e1b7e956 2012-07-16 00:17:50,761 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=192.168.1.2,60020,1342017399608, region=93caf5147d40f5dd4625e160e1b7e956 2012-07-16 00:17:50,774 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENED, server=192.168.1.2,60020,1342017399608, region=93caf5147d40f5dd4625e160e1b7e956 2012-07-16 00:17:50,774 DEBUG org.apache.hadoop.hbase.master.handler.OpenedRegionHandler: Handling OPENED event for trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956. from 192.168.1.2,60020,1342017399608; deleting unassigned node 2012-07-16 00:17:50,774 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x4384d0a47f40068 Deleting existing unassigned node for 93caf5147d40f5dd4625e160e1b7e956 that is in expected state RS_ZK_REGION_OPENED 2012-07-16 00:17:50,775 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: The znode of region trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956. has been deleted. 2012-07-16 00:17:50,775 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x4384d0a47f40068 Successfully deleted unassigned node for region 93caf5147d40f5dd4625e160e1b7e956 in expected state RS_ZK_REGION_OPENED 2012-07-16 00:17:50,775 INFO org.apache.hadoop.hbase.master.AssignmentManager: The master has opened the region trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956. that was online on 192.168.1.2,60020,1342017399608 2012-07-16 00:22:49,916 INFO org.apache.hadoop.hbase.master.HMaster: balance hri=trackurl_status_list,zO6u4o8,1342291884831.93caf5147d40f5dd4625e160e1b7e956., src=192.168.1.2,60020,1342017399608, dest=192.168.1.2,60020,1342002082592 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira