[ https://issues.apache.org/jira/browse/HBASE-4455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113886#comment-13113886 ]
jirapos...@reviews.apache.org commented on HBASE-4455: ------------------------------------------------------ bq. On 2011-09-23 05:44:11, Michael Stack wrote: bq. > Really good stuff. I just need a bit of it explained to me below because I'm being a little slow. I think you've also nailed HBASE-3809 with this patch Ming (What you think?) Good stuff. That sounds right. HBASE-4245 could be the same as well. bq. On 2011-09-23 05:44:11, Michael Stack wrote: bq. > http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/MetaNodeTracker.java, line 62 bq. > <https://reviews.apache.org/r/2007/diff/2/?file=45308#file45308line62> bq. > bq. > Whats this? Fixed. bq. On 2011-09-23 05:44:11, Michael Stack wrote: bq. > http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java, line 299 bq. > <https://reviews.apache.org/r/2007/diff/2/?file=45299#file45299line299> bq. > bq. > Why this change? If we did not ask to refresh, why not return what we found? No callers call the function with refresh==false, took it out for now. bq. On 2011-09-23 05:44:11, Michael Stack wrote: bq. > http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java, line 222 bq. > <https://reviews.apache.org/r/2007/diff/2/?file=45299#file45299line222> bq. > bq. > Why make this public? Fixed. bq. On 2011-09-23 05:44:11, Michael Stack wrote: bq. > http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java, line 535 bq. > <https://reviews.apache.org/r/2007/diff/2/?file=45299#file45299line535> bq. > bq. > nit: No need of the parens Fixed. bq. On 2011-09-23 05:44:11, Michael Stack wrote: bq. > http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java, line 215 bq. > <https://reviews.apache.org/r/2007/diff/2/?file=45304#file45304line215> bq. > bq. > Oh, here I see the don't split logs flag. bq. > bq. > Funny. bq. > bq. > We used to do something like this in the old days. bq. > bq. > So, we come back in again, we don't split logs, but we are still a server that was carrying root or meta -- we reassign again? I don't get how it works here. If the server carries -ROOT- or .META, it will first do the log splitting and then submit another ServerShutDownHandler request without shouldSplitHLog==false. - Ming ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2007/#review2035 ----------------------------------------------------------- On 2011-09-24 01:50:02, Ming Ma wrote: bq. bq. ----------------------------------------------------------- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2007/ bq. ----------------------------------------------------------- bq. bq. (Updated 2011-09-24 01:50:02) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. ------- bq. bq. 1. Add more logging. bq. 2. Clean up CatalogTracker. waitForMeta waits for "timeout" value. When waitForMetaServerConnectionDefault is called by MetaNodeTracker, the timeout value is large. So it doesn't retry in case .ROOT. is updated; add the proper implementation for CatalogTracker.verifyMetaRegionLocation bq. 4. Check for the latest -ROOT- and .META. region location during the handling of server shutdown. bq. 5. Right after assigning the -ROOT- or .META. in ServerShutdownHandler, don't block and wait for .META. availability. Resubmit another ServerShutdownHandler for regular regions. bq. bq. bq. This addresses bug HBASE-4455. bq. https://issues.apache.org/jira/browse/HBASE-4455 bq. bq. bq. Diffs bq. ----- bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java 1172205 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperNodeTracker.java 1172205 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZooKeeperNodeTracker.java 1172205 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java 1172205 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/replication/ReplicationZookeeper.java 1172205 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterStatusTracker.java 1172205 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1172205 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/handler/CloseRegionHandler.java 1172205 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/MetaServerShutdownHandler.java 1172205 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/OpenedRegionHandler.java 1172205 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/handler/ServerShutdownHandler.java 1172205 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 1172205 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 1172205 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java 1172205 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java 1172205 bq. bq. Diff: https://reviews.apache.org/r/2007/diff bq. bq. bq. Testing bq. ------- bq. bq. Keep Master up all the time, do rolling restart of RSs like this - stop RS1, wait for 2 seconds, stop RS2, start RS1, wait for 2 seconds, stop RS3, start RS2, wait for 2 seconds, etc. The program can run for couple hours until it stops. -ROOT- and .META. are available during that time. bq. bq. bq. Thanks, bq. bq. Ming bq. bq. > Rolling restart RSs scenario, -ROOT-, .META. regions are lost in > AssignmentManager > ---------------------------------------------------------------------------------- > > Key: HBASE-4455 > URL: https://issues.apache.org/jira/browse/HBASE-4455 > Project: HBase > Issue Type: Bug > Reporter: Ming Ma > Assignee: Ming Ma > Fix For: 0.92.0 > > > Keep Master up all the time, do rolling restart of RSs like this - stop RS1, > wait for 2 seconds, stop RS2, start RS1, wait for 2 seconds, stop RS3, start > RS2, wait for 2 seconds, etc. After a while, you will find the -ROOT-, .META. > regions aren't in "regions in transtion" from AssignmentManager point of > view, but they aren't assigned to any regions. Here are the issues. > 1. .-ROOT- or .META. location is stale when MetaServerShutdownHandler is > invoked to check if it contains -ROOT- region. That is due to long delay from > ZK notification and async nature of the system. Here is an example, even > though new root region server sea-lab-1,60020,1316380133656 is set at T2, at > T3 the shutdown process for sea-lab-1,60020,1316380133656, the root location > still points to old server sea-lab-3,60020,1316380037898. > T1: 2011-09-18 14:08:52,470 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: > master:6 > 0000-0x1327e43175e0000 Retrieved 29 byte(s) of data from znode > /hbase/root-regio > n-server and set watcher; sea-lab-3,60020,1316380037898 > T2: 2011-09-18 14:08:57,173 INFO > org.apache.hadoop.hbase.catalog.RootLocationEditor: Setting ROOT region > location in ZooKeeper as sea-lab-1,60020,1316380133656 > T3: 2011-09-18 14:10:26,393 DEBUG > org.apache.hadoop.hbase.master.ServerManager: Adde > d=sea-lab-1,60020,1316380133656 to dead servers, submitted shutdown handler > to be executed, root=false, meta=true, current Root Location: > sea-lab-3,60020,1316380037898 > T4: 2011-09-18 14:12:37,314 DEBUG org.apache.hadoop.hbase.zookeeper.ZKUtil: > master:6 > 0000-0x1327e43175e0000 Retrieved 29 byte(s) of data from znode > /hbase/root-region-server and set watcher; sea-lab-1,60020,1316380133656 > 2. The MetaServerShutdownHandler worker thread that waits for -ROOT- or > .META. availability could be blocked. If meanwhile, the new server that > -ROOT- or .META. is being assigned restarted, another instance of > MetaServerShutdownHandler is queued. Eventually, all > MetaServerShutdownHandler worker threads are filled up. It looks like > HBASE-4245. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira