[ https://issues.apache.org/jira/browse/HBASE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13159851#comment-13159851 ]
chunhui shen commented on HBASE-4899: ------------------------------------- {code} Oct 20 12:51:36 dw75.kgb.sqa.cm4 2011-10-20 12:02:27,345 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting c70b69079782d421b34ac5e57ef06a35 on serverName=dw80.kgb.sqa.cm4,60020,1319083018615, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:37 dw75.kgb.sqa.cm4 2011-10-20 12:02:27,472 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting a07d62ba400f631c781d7232ee397ed3 on serverName=dw81.kgb.sqa.cm4,60020,1319083018636, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:39 dw75.kgb.sqa.cm4 2011-10-20 12:02:27,756 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 05fb66d83cf2e4294a3cbd3f6757ec32 on serverName=dw90.kgb.sqa.cm4,60020,1319083018625, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:40 dw75.kgb.sqa.cm4 2011-10-20 12:02:27,805 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting e53b5cb8fcaaa178270f2dc876ebdd9a on serverName=dw90.kgb.sqa.cm4,60020,1319083018625, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:43 dw75.kgb.sqa.cm4 2011-10-20 12:02:28,825 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting d9ae4de5ba9045f4b060191222f774fd on serverName=dw81.kgb.sqa.cm4,60020,1319083018636, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:44 dw75.kgb.sqa.cm4 2011-10-20 12:02:28,889 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 589e9269c31cdcd8aa073518c133f286 on serverName=dw90.kgb.sqa.cm4,60020,1319083018625, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:44 dw75.kgb.sqa.cm4 2011-10-20 12:02:28,892 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting f35352a47f076c33a0f5ea140f76c943 on serverName=dw90.kgb.sqa.cm4,60020,1319083018625, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:44 dw75.kgb.sqa.cm4 2011-10-20 12:02:28,964 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 5fecc8e3a17eed75ba7bb2de2bb5c762 on serverName=dw81.kgb.sqa.cm4,60020,1319083018636, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:44 dw75.kgb.sqa.cm4 2011-10-20 12:02:29,009 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 0cdf089748ecdc79123670bf76dc68d5 on serverName=dw79.kgb.sqa.cm4,60020,1319083018623, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:44 dw75.kgb.sqa.cm4 2011-10-20 12:02:29,071 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 6245b4e8b567755509ba6351ded55c92 on serverName=dw79.kgb.sqa.cm4,60020,1319083018623, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:44 dw75.kgb.sqa.cm4 2011-10-20 12:02:29,380 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting fd52c86737233950bf509fb9ecf524a9 on serverName=dw80.kgb.sqa.cm4,60020,1319083018615, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:44 dw75.kgb.sqa.cm4 2011-10-20 12:02:29,461 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting a30b7035359910a63d34b99402a9ef35 on serverName=dw81.kgb.sqa.cm4,60020,1319083018636, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:45 dw75.kgb.sqa.cm4 2011-10-20 12:02:29,918 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 363e647e5b9c96c2ad1acc6e44471b74 on serverName=dw80.kgb.sqa.cm4,60020,1319083018615, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:46 dw75.kgb.sqa.cm4 2011-10-20 12:02:30,130 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting e996ca8ca0eede508af4e02964ae3de6 on serverName=dw79.kgb.sqa.cm4,60020,1319083018623, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:46 dw75.kgb.sqa.cm4 2011-10-20 12:02:30,155 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting fb9fc31c5baa604fb76afd9df419ca8b on serverName=dw79.kgb.sqa.cm4,60020,1319083018623, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:46 dw75.kgb.sqa.cm4 2011-10-20 12:02:30,155 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting fb9fc31c5baa604fb76afd9df419ca8b on serverName=dw79.kgb.sqa.cm4,60020,1319083018623, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:46 dw75.kgb.sqa.cm4 2011-10-20 12:02:30,320 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting e9cbaa732a070c8947eb0603a84634c5 on serverName=dw80.kgb.sqa.cm4,60020,1319083018615, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:47 dw75.kgb.sqa.cm4 2011-10-20 12:02:30,417 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting a68d33b7bbbb4bfecee9679e8af6b991 on serverName=dw79.kgb.sqa.cm4,60020,1319083018623, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:47 dw75.kgb.sqa.cm4 2011-10-20 12:02:30,494 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting a708d26df7e2c62d09851eb0053cb567 on serverName=dw79.kgb.sqa.cm4,60020,1319083018623, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:47 dw75.kgb.sqa.cm4 2011-10-20 12:02:30,513 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting b8bc0ba5ef9fd610b38afd76891b5506 on serverName=dw79.kgb.sqa.cm4,60020,1319083018623, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:47 dw75.kgb.sqa.cm4 2011-10-20 12:02:30,835 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting adf8c774655437f14a72552eed6566c8 on serverName=dw81.kgb.sqa.cm4,60020,1319083018636, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 12:51:48 dw75.kgb.sqa.cm4 2011-10-20 12:02:30,984 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 555048e18d23e8d88ce34f3b2798d893 on serverName=dw79.kgb.sqa.cm4,60020,1319083018623, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 13:44:26 dw75.kgb.sqa.cm4 2011-10-20 13:12:00,731 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 4ff6a541f09be443db27482ac170077b on serverName=dw83.kgb.sqa.cm4,60020,1319087456454, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 13:44:26 dw75.kgb.sqa.cm4 2011-10-20 13:12:00,757 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 9203d1b6f17eaec9f62d3252cc0a4728 on serverName=dw90.kgb.sqa.cm4,60020,1319087454876, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 13:44:28 dw75.kgb.sqa.cm4 2011-10-20 13:12:00,963 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 05bd751ae0894c2618b431ee03e33536 on serverName=dw79.kgb.sqa.cm4,60020,1319087456067, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 13:44:28 dw75.kgb.sqa.cm4 2011-10-20 13:12:00,989 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 9d619820bc07ffbc6260fe8cac3ac8fb on serverName=dw79.kgb.sqa.cm4,60020,1319087456067, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 13:44:28 dw75.kgb.sqa.cm4 2011-10-20 13:12:01,192 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting e53b5cb8fcaaa178270f2dc876ebdd9a on serverName=dw79.kgb.sqa.cm4,60020,1319087456067, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 13:44:33 dw75.kgb.sqa.cm4 2011-10-20 13:12:02,401 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting aeb0ed71b9866626a79313d6f43e04d2 on serverName=dw90.kgb.sqa.cm4,60020,1319087454876, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 13:44:34 dw75.kgb.sqa.cm4 2011-10-20 13:12:02,487 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting c2e4e403709ab6c41d9a2d9bc74e758e on serverName=dw90.kgb.sqa.cm4,60020,1319087454876, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 13:44:34 dw75.kgb.sqa.cm4 2011-10-20 13:12:02,614 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 10fc5766fd94fd1e306c2f1b73e1476f on serverName=dw90.kgb.sqa.cm4,60020,1319087454876, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 13:44:34 dw75.kgb.sqa.cm4 2011-10-20 13:12:02,614 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 10fc5766fd94fd1e306c2f1b73e1476f on serverName=dw90.kgb.sqa.cm4,60020,1319087454876, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 13:44:37 dw75.kgb.sqa.cm4 2011-10-20 13:12:03,836 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 3e5da14300d6af039221c29564c5f3e3 on serverName=dw81.kgb.sqa.cm4,60020,1319087456329, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 13:44:37 dw75.kgb.sqa.cm4 2011-10-20 13:12:03,942 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting a68d33b7bbbb4bfecee9679e8af6b991 on serverName=dw81.kgb.sqa.cm4,60020,1319087456329, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 13:44:37 dw75.kgb.sqa.cm4 2011-10-20 13:12:04,240 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 2d26053bb10898a5f203ce96e93a35b1 on serverName=dw81.kgb.sqa.cm4,60020,1319087456329, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 13:44:38 dw75.kgb.sqa.cm4 2011-10-20 13:12:04,271 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting a708d26df7e2c62d09851eb0053cb567 on serverName=dw81.kgb.sqa.cm4,60020,1319087456329, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 13:44:38 dw75.kgb.sqa.cm4 2011-10-20 13:12:04,292 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting e996ca8ca0eede508af4e02964ae3de6 on serverName=dw81.kgb.sqa.cm4,60020,1319087456329, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 13:44:39 dw75.kgb.sqa.cm4 2011-10-20 13:12:04,463 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 901c30cb291925739451ef6997cd9750 on serverName=dw83.kgb.sqa.cm4,60020,1319087456454, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 13:44:39 dw75.kgb.sqa.cm4 2011-10-20 13:12:04,794 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting f39019fda9968274f0d6b44cc9281004 on serverName=dw79.kgb.sqa.cm4,60020,1319087456067, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 13:44:40 dw75.kgb.sqa.cm4 2011-10-20 13:12:05,019 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting b140350ff7de2b2d2fcd353ef1b565d2 on serverName=dw90.kgb.sqa.cm4,60020,1319087454876, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 13:44:40 dw75.kgb.sqa.cm4 2011-10-20 13:12:05,089 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 47b40d219b1dba4b210b00ac7de13a6c on serverName=dw83.kgb.sqa.cm4,60020,1319087456454, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 13:44:40 dw75.kgb.sqa.cm4 2011-10-20 13:12:05,306 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 47893f2ac30964f06f8f65219077b60f on serverName=dw79.kgb.sqa.cm4,60020,1319087456067, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 13:44:40 dw75.kgb.sqa.cm4 2011-10-20 13:12:05,351 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting a38ef110e6ef3a06f1fa1c25fba5be23 on serverName=dw79.kgb.sqa.cm4,60020,1319087456067, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 17:11:17 dw75.kgb.sqa.cm4 2011-10-20 17:11:17,048 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 70236052 on serverName=dw79.kgb.sqa.cm4,60020,1319101778268, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) Oct 20 21:35:24 dw75.kgb.sqa.cm4 2011-10-20 21:35:24,312 WARN org.apache.hadoop.hbase.master.AssignmentManager: Overwriting 88bd3816503151926f30916066750e08 on serverName=dw79.kgb.sqa.cm4,60020,1319114721135, load=(requests=0, regions=0, usedHeap=0, maxHeap=0) {code} > Region would be assigned twice easily with continually killing server and > moving region in testing environment > --------------------------------------------------------------------------------------------------------------- > > Key: HBASE-4899 > URL: https://issues.apache.org/jira/browse/HBASE-4899 > Project: HBase > Issue Type: Bug > Reporter: chunhui shen > > Before assigning region in ServerShutdownHandler#process, it will check > whether region is in RIT, > however, this checking doesn't work as the excepted in the following case: > 1.move region A from server B to server C > 2.kill server B > 3.start server B immediately > Let's see what happen in the code for the above case > {code} > for step1: > 1.1 server B close the region A, > 1.2 master setOffline for region > A,(AssignmentManager#setOffline:this.regions.remove(regionInfo)) > 1.3 server C start to open region A.(Not completed) > for step3: > master ServerShutdownHandler#process() for server B > { > .. > splitlog() > ... > List<RegionState> regionsInTransition = > this.services.getAssignmentManager() > .processServerShutdown(this.serverName); > ... > Skip regions that were in transition unless CLOSING or PENDING_CLOSE > ... > assign region > } > In fact, when running > ServerShutdownHandler#process()#this.services.getAssignmentManager().processServerShutdown(this.serverName), > region A is in RIT (step1.3 not completed), but the return List<RegionState> > regionsInTransition doesn't contain it, because region A has removed from > AssignmentManager.regions by AssignmentManager#setOffline in step 1.2 > Therefore, region A will be assigned twice. > {code} > Actually, one server killed and started twice will also easily cause region > assigned twice. > Exclude the above reason, another probability : > when execute ServerShutdownHandler#process()#MetaReader.getServerUserRegions > ,region is included which is in RIT now. > But after completing MetaReader.getServerUserRegions, the region has been > opened in other server and is not in RIT now. > In our testing environment where balancing,moving and killing are executed > periodly, assigning region twice often happens, and it is hateful because it > will affect other test cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira