[ https://issues.apache.org/jira/browse/HBASE-19501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290216#comment-16290216 ]
stack commented on HBASE-19501: ------------------------------- .002 fixes tests. Again it is on top of HBASE-18946. Let me do write up on what is in here. The subject is actually misleading now; this patch partially is about it. Will fix soon. > [AMv2] Retain assignment across restarts > ---------------------------------------- > > Key: HBASE-19501 > URL: https://issues.apache.org/jira/browse/HBASE-19501 > Project: HBase > Issue Type: Sub-task > Components: Region Assignment > Reporter: stack > Assignee: stack > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19501.master.001.patch, > HBASE-19501.master.002.patch, HBASE-19501.master.003.patch, HBASE-19501.patch > > > Working with replicas and the parent test in particular, I learned a few > interesting things: > # It is hard to test if we retain assignments because our little minicluster > gives RegionServers new ports on restart foiling our means of recognizing new > instance of a server by checking hostname+port (and ensuring the startcode is > larger). > # Some of our tests like the parent test depended on retaining assignment > across restarts. > # As said in parent issue, master used to be last to go down when we did a > controlled cluster shutdown. We lost that when we moved to AMv2. > # When we do a cluster shutdown, the RegionServers close down the Regions, > not the Master as is usual in AMv2 (Master wants to do all assign ops in > AMv2). This means that the Master is surprised when it gets notification of > CLOSE ops that it did not initiate. Usually on CLOSE, Master updates meta > with the CLOSE state. On cluster shutdown we are not doing this. > # So, on restart, we read meta and we see all regions still in OPEN state so > we think the cluster crashed down so we go and do ServerCrashProcedure. Which > hoses our ability to retain assign. > Some experiments: > # I can make the Master stay up so it is last to go down > # This makes it so we no longer spew the logs with failed transition > messages because Master is not up to receive the CLOSE transitions. > # I hacked in means of telling minicluster ports it should use on start; > helps fake case of new RS instances > # It is hard to tell the difference between a clean shutdown and a crash > down. It is dangerous if we get the call wrong. Currently, given that we just > let ServerCrashProcedure deal with it -- the safest option -- one experiment > is that when it goes to assign the regions that were on the crashed server, > rather than round robin, instead we should look and see if new instance of > old location and if so, just give it al lthe regions. That'd retain locality. > This seems to work. Problem is that SCP is doing assignment. Ideally balancer > would do it. > Let me put up a patch that retains assignment across restart (somehow). -- This message was sent by Atlassian JIRA (v6.4.14#64029)