[ https://issues.apache.org/jira/browse/HBASE-8912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13810240#comment-13810240 ]
Sergey Kirichenko commented on HBASE-8912: ------------------------------------------ May be this helps (HBase from cloudera - 0.94.6-cdh4.4.0): grep by region caused exception on master: {noformat} 2013-10-31 00:07:52,871 WARN org.apache.hadoop.hbase.master.AssignmentManager: Region 3a476d37da81f620a3e53179d7d9192b has null regionLocation. But its table table_x isn't in ENABLING state. 2013-10-31 00:07:53,057 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x242045137a20070 Async create of unassigned node for 3a476d37da81f620a3e53179d7d9192b with OFFLINE state 2013-10-31 00:07:53,467 DEBUG org.apache.hadoop.hbase.master.AssignmentManager$CreateUnassignedAsyncCallback: rs=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. state=OFFLINE, ts=1383163673057, server=null, server=xxx100,60020,1383163665902 2013-10-31 00:07:53,495 DEBUG org.apache.hadoop.hbase.master.AssignmentManager$ExistsUnassignedAsyncCallback: rs=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. state=OFFLINE, ts=1383163673057, server=null 2013-10-31 00:07:54,834 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=xxx100,60020,1383163665902, region=3a476d37da81f620a3e53179d7d9192b 2013-10-31 00:07:56,953 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_FAILED_OPEN, server=xxx100,60020,1383163665902, region=3a476d37da81f620a3e53179d7d9192b 2013-10-31 00:07:56,953 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. destination server is xxx100,60020,1383163665902 2013-10-31 00:07:56,953 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. so generated a random one; hri=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b., src=, dest=xxx108,60020,1383163666006; 9 (online=9, available=8) available servers 2013-10-31 00:07:56,955 DEBUG org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED event for 3a476d37da81f620a3e53179d7d9192b 2013-10-31 00:07:56,956 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. state=CLOSED, ts=1383163675624, server=xxx100,60020,1383163665902 2013-10-31 00:07:56,956 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x242045137a20070 Creating (or updating) unassigned node for 3a476d37da81f620a3e53179d7d9192b with OFFLINE state 2013-10-31 00:07:57,003 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. destination server is xxx108,60020,1383163666006 2013-10-31 00:07:57,003 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan for region table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.; plan=hri=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b., src=, dest=xxx108,60020,1383163666006 2013-10-31 00:07:57,003 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. to xxx108,60020,1383163666006 2013-10-31 00:07:58,545 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_FAILED_OPEN, server=xxx108,60020,1383163666006, region=3a476d37da81f620a3e53179d7d9192b 2013-10-31 00:07:58,545 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. destination server is xxx108,60020,1383163666006 2013-10-31 00:07:58,545 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. so generated a random one; hri=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b., src=, dest=xxx106,60020,1383163666003; 9 (online=9, available=8) available servers 2013-10-31 00:07:58,546 DEBUG org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED event for 3a476d37da81f620a3e53179d7d9192b 2013-10-31 00:07:58,546 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. state=CLOSED, ts=1383163677110, server=xxx108,60020,1383163666006 2013-10-31 00:07:58,546 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x242045137a20070 Creating (or updating) unassigned node for 3a476d37da81f620a3e53179d7d9192b with OFFLINE state 2013-10-31 00:07:58,553 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_FAILED_OPEN, server=xxx108,60020,1383163666006, region=3a476d37da81f620a3e53179d7d9192b 2013-10-31 00:07:58,554 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. destination server is xxx106,60020,1383163666003 2013-10-31 00:07:58,554 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: No previous transition plan was found (or we are ignoring an existing plan) for table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. so generated a random one; hri=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b., src=, dest=xxx104,60020,1383163665976; 9 (online=9, available=8) available servers 2013-10-31 00:07:58,554 DEBUG org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED event for 3a476d37da81f620a3e53179d7d9192b 2013-10-31 00:07:58,554 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. state=CLOSED, ts=1383163677110, server=xxx108,60020,1383163666006 2013-10-31 00:07:58,571 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. destination server is xxx104,60020,1383163665976 2013-10-31 00:07:58,571 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan for region table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.; plan=hri=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b., src=, dest=xxx104,60020,1383163665976 2013-10-31 00:07:58,571 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. to xxx104,60020,1383163665976 2013-10-31 00:07:58,595 FATAL org.apache.hadoop.hbase.master.HMaster: Unexpected state : table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. state=PENDING_OPEN, ts=1383163678594, server=xxx104,60020,1383163665976 .. Cannot transit it to OFFLINE. java.lang.IllegalStateException: Unexpected state : table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. state=PENDING_OPEN, ts=1383163678594, server=xxx104,60020,1383163665976 .. Cannot transit it to OFFLINE. at org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1831) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1661) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1426) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1398) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1393) at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:662) {noformat} grep by region caused exception on xxx100: {noformat} 2013-10-31 00:07:54,000 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. 2013-10-31 00:07:54,000 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x242045137a20071 Attempting to transition node 3a476d37da81f620a3e53179d7d9192b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-10-31 00:07:54,029 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x242045137a20071 Successfully transitioned node 3a476d37da81f620a3e53179d7d9192b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-10-31 00:07:55,439 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: {NAME => 'table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.', STARTKEY => '6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8', ENDKEY => '71c71c71c71c71c71c71c71c71c71c71c71c71c0', ENCODED => 3a476d37da81f620a3e53179d7d9192b,} 2013-10-31 00:07:55,439 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. 2013-10-31 00:07:55,447 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile: Store file hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/A/table_y=abca20169d0fdd22f2a13d6caf41d83d-0729d4dbd652443bbd44381db5b7b26b is a link 2013-10-31 00:07:55,501 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/A/table_y=abca20169d0fdd22f2a13d6caf41d83d-0729d4dbd652443bbd44381db5b7b26b, isReference=false, isBulkLoadResult=false, seqid=46816, majorCompaction=true 2013-10-31 00:07:55,546 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile: Store file hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-8606885898507153833 is a link 2013-10-31 00:07:55,602 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile: Store file hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-b2449476f78e42b7ba0bba2ac69a24b8 is a link 2013-10-31 00:07:55,613 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-b2449476f78e42b7ba0bba2ac69a24b8, isReference=false, isBulkLoadResult=false, seqid=46816, majorCompaction=false 2013-10-31 00:07:55,618 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b., starting to roll back the global memstore size. 2013-10-31 00:07:55,621 INFO org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opening of region {NAME => 'table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.', STARTKEY => '6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8', ENDKEY => '71c71c71c71c71c71c71c71c71c71c71c71c71c0', ENCODED => 3a476d37da81f620a3e53179d7d9192b,} failed, marking as FAILED_OPEN in ZK 2013-10-31 00:07:55,621 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x242045137a20071 Attempting to transition node 3a476d37da81f620a3e53179d7d9192b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN 2013-10-31 00:07:55,630 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x242045137a20071 Successfully transitioned node 3a476d37da81f620a3e53179d7d9192b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN {noformat} grep by region caused exception on xxx108: {noformat} 2013-10-31 00:07:57,003 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. 2013-10-31 00:07:57,010 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x242045137a20074 Attempting to transition node 3a476d37da81f620a3e53179d7d9192b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-10-31 00:07:57,042 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x242045137a20074 Successfully transitioned node 3a476d37da81f620a3e53179d7d9192b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-10-31 00:07:57,043 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: {NAME => 'table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.', STARTKEY => '6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8', ENDKEY => '71c71c71c71c71c71c71c71c71c71c71c71c71c0', ENCODED => 3a476d37da81f620a3e53179d7d9192b,} 2013-10-31 00:07:57,043 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. 2013-10-31 00:07:57,049 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile: Store file hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/A/table_y=abca20169d0fdd22f2a13d6caf41d83d-0729d4dbd652443bbd44381db5b7b26b is a link 2013-10-31 00:07:57,060 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/A/table_y=abca20169d0fdd22f2a13d6caf41d83d-0729d4dbd652443bbd44381db5b7b26b, isReference=false, isBulkLoadResult=false, seqid=46816, majorCompaction=true 2013-10-31 00:07:57,065 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile: Store file hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-8606885898507153833 is a link 2013-10-31 00:07:57,095 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile: Store file hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-b2449476f78e42b7ba0bba2ac69a24b8 is a link 2013-10-31 00:07:57,105 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-b2449476f78e42b7ba0bba2ac69a24b8, isReference=false, isBulkLoadResult=false, seqid=46816, majorCompaction=false 2013-10-31 00:07:57,107 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b., starting to roll back the global memstore size. 2013-10-31 00:07:57,108 INFO org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opening of region {NAME => 'table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.', STARTKEY => '6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8', ENDKEY => '71c71c71c71c71c71c71c71c71c71c71c71c71c0', ENCODED => 3a476d37da81f620a3e53179d7d9192b,} failed, marking as FAILED_OPEN in ZK 2013-10-31 00:07:57,108 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x242045137a20074 Attempting to transition node 3a476d37da81f620a3e53179d7d9192b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN 2013-10-31 00:07:57,125 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x242045137a20074 Successfully transitioned node 3a476d37da81f620a3e53179d7d9192b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN {noformat} grep by region caused exception on xxx104: {noformat} 2013-10-31 00:07:58,581 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Received request to open region: table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. 2013-10-31 00:07:58,587 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x420451326a0070 Attempting to transition node 3a476d37da81f620a3e53179d7d9192b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-10-31 00:07:58,602 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x420451326a0070 Successfully transitioned node 3a476d37da81f620a3e53179d7d9192b from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-10-31 00:07:58,603 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region: {NAME => 'table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.', STARTKEY => '6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8', ENDKEY => '71c71c71c71c71c71c71c71c71c71c71c71c71c0', ENCODED => 3a476d37da81f620a3e53179d7d9192b,} 2013-10-31 00:07:58,604 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Instantiated table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b. 2013-10-31 00:07:58,610 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile: Store file hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/A/table_y=abca20169d0fdd22f2a13d6caf41d83d-0729d4dbd652443bbd44381db5b7b26b is a link 2013-10-31 00:07:58,621 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/A/table_y=abca20169d0fdd22f2a13d6caf41d83d-0729d4dbd652443bbd44381db5b7b26b, isReference=false, isBulkLoadResult=false, seqid=46816, majorCompaction=true 2013-10-31 00:07:58,627 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile: Store file hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-8606885898507153833 is a link 2013-10-31 00:07:58,639 DEBUG org.apache.hadoop.hbase.regionserver.StoreFile: Store file hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-b2449476f78e42b7ba0bba2ac69a24b8 is a link 2013-10-31 00:07:58,650 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded hdfs://xxx-master/hbase/table_x/3a476d37da81f620a3e53179d7d9192b/B/table_y=abca20169d0fdd22f2a13d6caf41d83d-b2449476f78e42b7ba0bba2ac69a24b8, isReference=false, isBulkLoadResult=false, seqid=46816, majorCompaction=false 2013-10-31 00:07:58,652 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b., starting to roll back the global memstore size. 2013-10-31 00:07:58,653 INFO org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opening of region {NAME => 'table_x,6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8,1382557475861.3a476d37da81f620a3e53179d7d9192b.', STARTKEY => '6eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee8', ENDKEY => '71c71c71c71c71c71c71c71c71c71c71c71c71c0', ENCODED => 3a476d37da81f620a3e53179d7d9192b,} failed, marking as FAILED_OPEN in ZK 2013-10-31 00:07:58,653 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x420451326a0070 Attempting to transition node 3a476d37da81f620a3e53179d7d9192b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN 2013-10-31 00:07:58,670 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: regionserver:60020-0x420451326a0070 Successfully transitioned node 3a476d37da81f620a3e53179d7d9192b from RS_ZK_REGION_OPENING to RS_ZK_REGION_FAILED_OPEN {noformat} 1) Initially AM try to assign region with 'bulk assign' on xxx100 (PENDING_OPEN => RS_ZK_REGION_OPENING); but xxx100 failed to open region and AM handles this event (RS_ZK_REGION_FAILED_OPEN => CLOSED => OFFLINE) 2) AM try to assign region in ClosedRegionHandler on xxx108 (there is no RS_ZK_REGION_OPENING event in master's logs, but we see it in regionserver's logs); it fails again 3) AM chose xxx106 for region assignment but receives RS_ZK_REGION_FAILED_OPEN before sending request => CLOSED => ClosedRegionHandler => xxx104 => exception > [0.94] AssignmentManager throws IllegalStateException from PENDING_OPEN to > OFFLINE > ---------------------------------------------------------------------------------- > > Key: HBASE-8912 > URL: https://issues.apache.org/jira/browse/HBASE-8912 > Project: HBase > Issue Type: Bug > Reporter: Enis Soztutar > Fix For: 0.94.14 > > Attachments: HBase-0.94 #1036 test - testRetrying [Jenkins].html > > > AM throws this exception which subsequently causes the master to abort: > {code} > java.lang.IllegalStateException: Unexpected state : > testRetrying,jjj,1372891751115.9b828792311001062a5ff4b1038fe33b. > state=PENDING_OPEN, ts=1372891751912, > server=hemera.apache.org,39064,1372891746132 .. Cannot transit it to OFFLINE. > at > org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:1879) > at > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1688) > at > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1424) > at > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1399) > at > org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1394) > at > org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:105) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:662) > {code} > This exception trace is from the failing test TestMetaReaderEditor which is > failing pretty frequently, but looking at the test code, I think this is not > a test-only issue, but affects the main code path. > https://builds.apache.org/job/HBase-0.94/1036/testReport/junit/org.apache.hadoop.hbase.catalog/TestMetaReaderEditor/testRetrying/ -- This message was sent by Atlassian JIRA (v6.1#6144)