[ https://issues.apache.org/jira/browse/HBASE-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-4511: ------------------------- Attachment: 4511-v2.txt Thank you for the log Jinchao. That helped. So, yes, metaLocation can be null at this point. It won't be set until its been 'verified'. So what we want if metaLocation is null is what is up in -ROOT- (which is what I believe you were saying above in your first feedback). This patch adds a method to CatalogTracker that will return metaLocation if non-null else whatever the current content of -ROOT-. > There is data loss when master failovers > ---------------------------------------- > > Key: HBASE-4511 > URL: https://issues.apache.org/jira/browse/HBASE-4511 > Project: HBase > Issue Type: Bug > Components: master > Affects Versions: 0.92.0 > Reporter: gaojinchao > Assignee: stack > Priority: Minor > Fix For: 0.92.0 > > Attachments: 4511-v2.txt, 4511.txt, > org.apache.hadoop.hbase.master.TestMasterFailover-output.rar, sketch.txt > > > It goes like this: > Master crashed , at the same time RS with meta is crashing, but RS doesn't > eixt. > Master startups again and finds all living RS. > Master verifies the meta failed, because this RS is crashing. > Master reassigns the meta, but it doesn't split the Hlog. > So some meta data is loss. > About the logs of a failover test case fail. > //It said that we want to kill a RS > 2011-09-28 19:54:45,694 INFO [Thread-988] regionserver.HRegionServer(1443): > STOPPED: Killing for unit test > 2011-09-28 19:54:45,694 INFO [Thread-988] master.TestMasterFailover(1007): > RS 192.168.2.102,54385,1317264874629 killed > //Rs didn't crash. > 2011-09-28 19:54:51,763 INFO [Master:0;192.168.2.102,54557,1317264885720] > master.HMaster(458): Registering server found up in zk: > 192.168.2.102,54385,1317264874629 > 2011-09-28 19:54:51,763 INFO [Master:0;192.168.2.102,54557,1317264885720] > master.ServerManager(232): Registering > server=192.168.2.102,54385,1317264874629 > 2011-09-28 19:54:51,770 DEBUG [Master:0;192.168.2.102,54557,1317264885720] > zookeeper.ZKUtil(491): master:54557-0x132b31adbb30005 Unable to get data of > znode /hbase/unassigned/1028785192 because node does not exist (not an error) > 2011-09-28 19:54:51,771 DEBUG [Master:0;192.168.2.102,54557,1317264885720] > zookeeper.ZKUtil(1003): master:54557-0x132b31adbb30005 Retrieved 33 byte(s) > of data from znode /hbase/root-region-server and set watcher; > 192.168.2.102,54383,131726487... > //Meta verification failed and ressigned the meta. So all the regions in the > meta is loss. > 2011-09-28 19:54:51,773 INFO [Master:0;192.168.2.102,54557,1317264885720] > catalog.CatalogTracker(476): Failed verification of .META.,,1 at > address=192.168.2.102,54385,1317264874629; > org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: > org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server > 192.168.2.102,54385,1317264874629 not running, aborting > 2011-09-28 19:54:51,773 DEBUG [Master:0;192.168.2.102,54557,1317264885720] > catalog.CatalogTracker(316): new .META. server: > 192.168.2.102,54385,1317264874629 isn't valid. Cached .META. server: null > 2011-09-28 19:54:52,274 DEBUG [Master:0;192.168.2.102,54557,1317264885720] > zookeeper.ZKUtil(1003): master:54557-0x132b31adbb30005 Retrieved 33 byte(s) > of data from znode /hbase/root-region-server and set watcher; > 192.168.2.102,54383,131726487... > 2011-09-28 19:54:52,277 INFO [Master:0;192.168.2.102,54557,1317264885720] > catalog.CatalogTracker(476): Failed verification of .META.,,1 at > address=192.168.2.102,54385,1317264874629; > org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: > org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server > 192.168.2.102,54385,1317264874629 not running, aborting > 2011-09-28 19:54:52,277 DEBUG [Master:0;192.168.2.102,54557,1317264885720] > catalog.CatalogTracker(316): new .META. server: > 192.168.2.102,54385,1317264874629 isn't valid. Cached .META. server: null > 2011-09-28 19:54:52,778 DEBUG [Master:0;192.168.2.102,54557,1317264885720] > zookeeper.ZKUtil(1003): master:54557-0x132b31adbb30005 Retrieved 33 byte(s) > of data from znode /hbase/root-region-server and set watcher; > 192.168.2.102,54383,131726487... > 2011-09-28 19:54:52,782 INFO [Master:0;192.168.2.102,54557,1317264885720] > catalog.CatalogTracker(476): Failed verification of .META.,,1 at > address=192.168.2.102,54385,1317264874629; > org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: > org.apache.hadoop.hbase.regionserver.RegionServerStoppedException: Server > 192.168.2.102,54385,1317264874629 not running, aborting > 2011-09-28 19:54:52,782 DEBUG [Master:0;192.168.2.102,54557,1317264885720] > catalog.CatalogTracker(316): new .META. server: > 192.168.2.102,54385,1317264874629 isn't valid. Cached .META. server: null > 2011-09-28 19:54:52,782 DEBUG [Master:0;192.168.2.102,54557,1317264885720] > zookeeper.ZKAssign(264): master:54557-0x132b31adbb30005 Creating (or > updating) unassigned node for 1028785192 with OFFLINE state > 2011-09-28 19:54:52,825 DEBUG [Thread-988-EventThread] > zookeeper.ZooKeeperWatcher(233): master:54557-0x132b31adbb30005 Received > ZooKeeper Event, type=NodeCreated, state=SyncConnected, > path=/hbase/unassigned/1028785192 > //It said that Master clean the cluster. > 2011-09-28 19:54:52,889 INFO [Master:0;192.168.2.102,54557,1317264885720] > master.AssignmentManager(383): Clean cluster startup. Assigning userregions > 2011-09-28 19:54:52,889 DEBUG [Master:0;192.168.2.102,54557,1317264885720] > zookeeper.ZKAssign(494): master:54557-0x132b31adbb30005 Deleting any existing > unassigned nodes -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira