[ 
https://issues.apache.org/jira/browse/HBASE-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16589370#comment-16589370
 ] 

stack commented on HBASE-21078:
-------------------------------

Not sure if this my patch but on cluster I see a strangeness where a MP starts 
up its UP and then it hangs here and is never scheduled again:
{code}
2018-08-22 11:05:29,929 INFO  
[RpcServer.default.FPBQ.Fifo.handler=3,queue=3,port=16000] master.HMaster: 
Client=stack//10.17.240.20 move hri=e5ed5607d30e9813aa7206048d5a94fd, 
source=ve0530.halxg.cloudera.com,16020,1534960226278, 
destination=ve0540.halxg.cloudera.com,16020,1534960226357, running balancer
2018-08-22 11:05:30,144 INFO  [PEWorker-1] procedure.MasterProcedureScheduler: 
pid=288, state=RUNNABLE:MOVE_REGION_UNASSIGN, hasLock=false; 
MoveRegionProcedure hri=e5ed5607d30e9813aa7206048d5a94fd, 
source=ve0530.halxg.cloudera.com,16020,1534960226278, 
destination=ve0540.halxg.cloudera.com,16020,1534960226357 checking lock on 
e5ed5607d30e9813aa7206048d5a94fd
2018-08-22 11:05:30,195 INFO  [PEWorker-1] procedure2.ProcedureExecutor: 
Initialized subprocedures=[{pid=289, ppid=288, 
state=RUNNABLE:REGION_TRANSITION_QUEUE, hasLock=false; UnassignProcedure 
table=IntegrationTestBigLinkedList, region=e5ed5607d30e9813aa7206048d5a94fd, 
server=ve0530.halxg.cloudera.com,16020,1534960226278}]
2018-08-22 11:05:30,252 INFO  [PEWorker-1] procedure.MasterProcedureScheduler: 
pid=289, ppid=288, state=RUNNABLE:REGION_TRANSITION_QUEUE, hasLock=false; 
UnassignProcedure table=IntegrationTestBigLinkedList, 
region=e5ed5607d30e9813aa7206048d5a94fd, 
server=ve0530.halxg.cloudera.com,16020,1534960226278 checking lock on 
e5ed5607d30e9813aa7206048d5a94fd
{code}

It has the lock on the region and won't let go (from UI):

{code}
REGION: e5ed5607d30e9813aa7206048d5a94fd
Lock type: EXCLUSIVE

Owner procedure: { ID => '288', PARENT_ID => '-1', STATE => 'WAITING', OWNER => 
'stack', TYPE => 'MoveRegionProcedure hri=e5ed5607d30e9813aa7206048d5a94fd, 
source=ve0530.halxg.cloudera.com,16020,1534960226278, 
destination=ve0540.halxg.cloudera.com,16020,1534960226357', START_TIME => 'Wed 
Aug 22 11:05:29 PDT 2018', LAST_UPDATE => 'Wed Aug 22 11:05:30 PDT 2018', 
PARAMETERS => [ { state => [ '1', '2' ] }, { regionId => '1534960773088', 
tableName => { namespace => 'ZGVmYXVsdA==', qualifier => 
'SW50ZWdyYXRpb25UZXN0QmlnTGlua2VkTGlzdA==' }, startKey => 'BhjRNw==', endKey => 
'DDDDDDDDDDA=', offline => 'false', split => 'false', replicaId => '0' }, { 
sourceServer => { hostName => 've0530.halxg.cloudera.com', port => '16020', 
startCode => '1534960226278' }, destinationServer => { hostName => 
've0540.halxg.cloudera.com', port => '16020', startCode => '1534960226357' } } 
] }
{code}

> [amv2] CODE-BUG NPE in RTP doing Unassign
> -----------------------------------------
>
>                 Key: HBASE-21078
>                 URL: https://issues.apache.org/jira/browse/HBASE-21078
>             Project: HBase
>          Issue Type: Bug
>          Components: amv2
>    Affects Versions: 2.0.1
>            Reporter: stack
>            Assignee: stack
>            Priority: Major
>             Fix For: 2.0.2
>
>         Attachments: HBASE-21078.branch-2.0.001.patch, 
> HBASE-21078.branch-2.0.002.patch, HBASE-21078.branch-2.0.003.patch
>
>
> Saw this is a run against tip of branch-2.0. The region had just finished 
> being split when the move goes to run.
> {code}
> 2018-08-18 16:55:14,908 INFO  [PEWorker-2] procedure2.ProcedureExecutor: 
> Finished pid=2028, state=SUCCESS, hasLock=false; SplitTableRegionProcedure 
> table=IntegrationTestBigLinkedList, parent=c3f199b5af62ae2ff8f8b6426b21d95d, 
> daughterA=31ccbf098ae615ce30f28ec84c956b8f, 
> daughterB=1890b4c96736f223f31efef11c817c90 in 9.0090sec
> 2018-08-18 16:55:14,908 INFO  [PEWorker-16] 
> procedure.MasterProcedureScheduler: pid=2038, ppid=2030, 
> state=RUNNABLE:MOVE_REGION_UNASSIGN, hasLock=false; MoveRegionProcedure 
> hri=c3f199b5af62ae2ff8f8b6426b21d95d, 
> source=ve0540.halxg.cloudera.com,16020,1534632630737, 
> destination=ve0540.halxg.cloudera.com,16020,1534632630737 checking lock on 
> c3f199b5af62ae2ff8f8b6426b21d95d
> 2018-08-18 16:55:14,958 INFO  [PEWorker-16] procedure2.ProcedureExecutor: 
> Initialized subprocedures=[{pid=2095, ppid=2038, 
> state=RUNNABLE:REGION_TRANSITION_DISPATCH, hasLock=false; UnassignProcedure 
> table=IntegrationTestBigLinkedList, region=c3f199b5af62ae2ff8f8b6426b21d95d, 
> server=ve0540.halxg.cloudera.com,16020,1534632630737}]
> 2018-08-18 16:55:15,008 INFO  [PEWorker-3] 
> procedure.MasterProcedureScheduler: pid=2095, ppid=2038, 
> state=RUNNABLE:REGION_TRANSITION_DISPATCH, hasLock=false; UnassignProcedure 
> table=IntegrationTestBigLinkedList, region=c3f199b5af62ae2ff8f8b6426b21d95d, 
> server=ve0540.halxg.cloudera.com,16020,1534632630737 checking lock on 
> c3f199b5af62ae2ff8f8b6426b21d95d
> 2018-08-18 16:55:15,085 ERROR [PEWorker-3] procedure2.ProcedureExecutor: 
> CODE-BUG: Uncaught runtime exception: pid=2095, ppid=2038, 
> state=RUNNABLE:REGION_TRANSITION_DISPATCH, hasLock=true; UnassignProcedure 
> table=IntegrationTestBigLinkedList, region=c3f199b5af62ae2ff8f8b6426b21d95d, 
> server=ve0540.halxg.cloudera.com,16020,1534632630737
> java.lang.NullPointerException
>   at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
>   at 
> org.apache.hadoop.hbase.master.assignment.RegionStates.getOrCreateServer(RegionStates.java:1097)
>   at 
> org.apache.hadoop.hbase.master.assignment.RegionStates.addRegionToServer(RegionStates.java:1125)
>   at 
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.markRegionAsClosing(AssignmentManager.java:1477)
>   at 
> org.apache.hadoop.hbase.master.assignment.UnassignProcedure.updateTransition(UnassignProcedure.java:204)
>   at 
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:345)
>   at 
> org.apache.hadoop.hbase.master.assignment.RegionTransitionProcedure.execute(RegionTransitionProcedure.java:97)
>   at 
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:873)
>   at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1556)
>   at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1344)
>   at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$900(ProcedureExecutor.java:76)
>   at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1854)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to