If you are using region replication, then you might be hitting https://issues.apache.org/jira/browse/HBASE-21644
Thanks, Ankit Singhal On Fri, Jul 5, 2019 at 9:53 AM Stack <[email protected]> wrote: > Looks like known issue. Can you update to later branch-2.0 release or patch > what you have? > Thanks, > S > > On Thu, Jul 4, 2019 at 6:59 PM Jacobo Coll <[email protected]> wrote: > > > Hi > > > > I'm trying to create a view against an existing hbase table, which adds > > some coprocessors to the table. I'm using Hortonworks 3.1.2, so it has > > HBase 2.0.2.3.1.2.1-1 > > < > > > https://repo.hortonworks.com/content/repositories/releases/org/apache/hbase/hbase-server/2.0.2.3.1.2.1-1/ > > > > > and Phoenix 5.0.0.3.1.2.1-1 > > < > > > https://repo.hortonworks.com/content/repositories/releases/org/apache/phoenix/phoenix/5.0.0.3.1.2.1-1/ > > >. > > The cluster is deployed in Azure. > > > > The issue seems to be related with HBASE-20817 > > <https://issues.apache.org/jira/browse/HBASE-20817>, but should be fixed > > in > > that version (I've checked that this patch was applied to that build) > > > > Just after creating a "view" in phoenix, the "ModifyTableProcedure" > > triggers a "ReopenTableRegionsProcedure" that enters into this infinite > > loop of "MoveRegionProcedure". This loop has a lapse of ~5s, and it fills > > up the list of procedures, and the procedure wal is not cleanup, as it > > never finishes the running procedure. > > > > This is a subset of the hbase-master log. The affected table has a > > pre-split of 100, so the log is quite large. I've shrunken some lines > with > > dots. > > > > > > > > 2019-07-03 16:12:27,924 INFO [PEWorker-8] > > procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=267, > > ppid=266, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; > > ReopenTableRegionsProcedure table=opencga_jcoll_grch38_variants}] > > 2019-07-03 16:12:28,059 INFO [PEWorker-2] > > procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=268, > > ppid=267, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure > > hri=fdad9893526ef840d117e6bea7c04bc5, > > source=wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960, > > destination= > wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960}, > > {pid=269, ppid=267, state=RUNNABLE:MOVE_REGION_UNASSIGN; > > MoveRegionProcedure hri=7b8b7dc99aee4f524af41a86e10ac945, > > source=wn0-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131867, > > destination= > wn0-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131867}, > > > > > ...................................................................................................., > > {pid=368, ppid=267, state=RUNNABLE:MOVE_REGION_UNASSIGN; > > MoveRegionProcedure hri=60ccd4513bc298b83d062cb0172ccba9, > > source=wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960, > > destination= > wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960}] > > 2019-07-03 16:12:28,096 INFO [PEWorker-5] > > procedure.MasterProcedureScheduler: Took xlock for pid=268, ppid=267, > > state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure > > hri=fdad9893526ef840d117e6bea7c04bc5, > > source=wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960, > > destination= > wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960 > > 2019-07-03 16:12:28,116 INFO [PEWorker-5] > > procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=370, > > ppid=268, state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure > > table=opencga_jcoll_grch38_variants, > > region=fdad9893526ef840d117e6bea7c04bc5, override=true, > > server=wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960}] > > 2019-07-03 16:12:28,247 INFO [PEWorker-4] > > procedure.MasterProcedureScheduler: Took xlock for pid=370, ppid=268, > > state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure > > table=opencga_jcoll_grch38_variants, > > region=fdad9893526ef840d117e6bea7c04bc5, override=true, > > server=wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960 > > 2019-07-03 16:12:28,280 INFO [PEWorker-4] > > assignment.RegionTransitionProcedure: Dispatch pid=370, ppid=268, > > state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; > > UnassignProcedure table=opencga_jcoll_grch38_variants, > > region=fdad9893526ef840d117e6bea7c04bc5, override=true, > > server=wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960 > > 2019-07-03 16:12:28,659 INFO [PEWorker-13] > > procedure2.ProcedureExecutor: Finished subprocedure pid=370, resume > > processing parent pid=268, ppid=267, > > state=RUNNABLE:MOVE_REGION_ASSIGN, locked=true; MoveRegionProcedure > > hri=fdad9893526ef840d117e6bea7c04bc5, > > source=wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960, > > destination= > wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960 > > 2019-07-03 16:12:28,659 INFO [PEWorker-13] > > procedure2.ProcedureExecutor: Finished pid=370, ppid=268, > > state=SUCCESS; UnassignProcedure table=opencga_jcoll_grch38_variants, > > region=fdad9893526ef840d117e6bea7c04bc5, override=true, > > server=wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960 > > in 458msec, unfinishedSiblingCount=0 > > 2019-07-03 16:12:28,662 INFO [PEWorker-8] > > procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=408, > > ppid=268, state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure > > table=opencga_jcoll_grch38_variants, > > region=fdad9893526ef840d117e6bea7c04bc5, > > target=wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960}] > > 2019-07-03 16:12:28,687 INFO [PEWorker-8] > > procedure.MasterProcedureScheduler: Took xlock for pid=408, ppid=268, > > state=RUNNABLE:REGION_TRANSITION_QUEUE; AssignProcedure > > table=opencga_jcoll_grch38_variants, > > region=fdad9893526ef840d117e6bea7c04bc5, > > target=wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960 > > 2019-07-03 16:12:28,713 INFO [PEWorker-8] assignment.AssignProcedure: > > Starting pid=408, ppid=268, state=RUNNABLE:REGION_TRANSITION_QUEUE, > > locked=true; AssignProcedure table=opencga_jcoll_grch38_variants, > > region=fdad9893526ef840d117e6bea7c04bc5, > > target=wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960; > > rit=OFFLINE, location= > > wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960; > > forceNewPlan=false, retain=false target > > svr=wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960 > > 2019-07-03 16:12:28,909 INFO [PEWorker-13] > > assignment.RegionTransitionProcedure: Dispatch pid=408, ppid=268, > > state=RUNNABLE:REGION_TRANSITION_DISPATCH, locked=true; > > AssignProcedure table=opencga_jcoll_grch38_variants, > > region=fdad9893526ef840d117e6bea7c04bc5, > > target=wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960 > > 2019-07-03 16:12:29,435 INFO [PEWorker-13] > > procedure2.ProcedureExecutor: Finished subprocedure pid=408, resume > > processing parent pid=268, ppid=267, state=RUNNABLE, locked=true; > > MoveRegionProcedure hri=fdad9893526ef840d117e6bea7c04bc5, > > source=wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960, > > destination= > wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960 > > 2019-07-03 16:12:29,436 INFO [PEWorker-13] > > procedure2.ProcedureExecutor: Finished pid=408, ppid=268, > > state=SUCCESS; AssignProcedure table=opencga_jcoll_grch38_variants, > > region=fdad9893526ef840d117e6bea7c04bc5, > > target=wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960 > > in 684msec, unfinishedSiblingCount=0 > > 2019-07-03 16:12:29,494 INFO [PEWorker-14] > > procedure2.ProcedureExecutor: Finished pid=268, ppid=267, > > state=SUCCESS; MoveRegionProcedure > > hri=fdad9893526ef840d117e6bea7c04bc5, > > source=wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960, > > destination= > wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960 > > in 1.3930sec, unfinishedSiblingCount=92 > > 2019-07-03 16:12:36,744 INFO [PEWorker-12] > > procedure2.ProcedureExecutor: Finished subprocedure pid=275, resume > > processing parent pid=267, ppid=266, > > state=RUNNABLE:REOPEN_TABLE_REGIONS_CONFIRM_REOPENED; > > ReopenTableRegionsProcedure table=opencga_jcoll_grch38_variants > > 2019-07-03 16:12:36,744 INFO [PEWorker-12] > > procedure2.ProcedureExecutor: Finished pid=275, ppid=267, > > state=SUCCESS; MoveRegionProcedure > > hri=f552eccd01cfd00bc30bec5e19f398df, > > source=wn2-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131806, > > destination= > wn2-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131806 > > in 8.4340sec, unfinishedSiblingCount=0 > > 2019-07-03 16:12:36,744 INFO [PEWorker-12] > > procedure2.ProcedureExecutor: Finished subprocedure pid=275, resume > > processing parent pid=267, ppid=266, > > state=RUNNABLE:REOPEN_TABLE_REGIONS_CONFIRM_REOPENED; > > ReopenTableRegionsProcedure table=o > > pencga_jcoll_grch38_variants > > 2019-07-03 16:12:36,744 INFO [PEWorker-12] > > procedure2.ProcedureExecutor: Finished pid=275, ppid=267, > > state=SUCCESS; MoveRegionProcedure > > hri=f552eccd01cfd00bc30bec5e19f398df, > > source=wn2-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131806, > > destination= > wn2-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131806 > > in 8.4340sec, unfinishedSiblingCount=0 > > 2019-07-03 16:12:36,791 INFO [PEWorker-5] > > procedure2.ProcedureExecutor: Initialized subprocedures=[{pid=571, > > ppid=267, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure > > hri=fdad9893526ef840d117e6bea7c04bc5, > > source=wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960, > > destination= > wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960}, > > {pid=572, ppid=267, state=RUNNABLE:MOVE_REGION_UNASSIGN; > > MoveRegionProced > > ure hri=7b8b7dc99aee4f524af41a86e10ac945, > > source=wn0-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131867, > > destination= > wn0-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131867}, > > {pid=573 > > , ppid=267, state=RUNNABLE:MOVE_REGION_UNASSIGN; MoveRegionProcedure > > hri=545caf13911c04263c8f84f2c14783b7, > > source=wn4-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131742, > > destination= > wn4-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131742}, > > {pid=574, ppid=267, state=RUNNABLE:MOVE_REGION_UNASSIGN; > > MoveRegionProcedure hri=6fd4397428a741d0fa67e1a2774f48d1, > > source=wn3-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131711, > > destination= > wn3-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131711}, > > {pid=575, ppid=267, state=RUNNABLE:MOVE_REGION_UNASSIGN; > > MoveRegionProcedure hri=053f3ff2a77982f98bb399d60aa0942b, > > source=wn2-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131806, > > destination= > wn2-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131806}, > > {pid=576, ppid=267, state=RUNNABLE:MOVE_REGION_UNASSIGN; > > MoveRegionProcedure hri=94415a23ead3e24367c12a0de1e90e28, > > source=wn4-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131742, > > destination= > wn4-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131742}, > > {pid=577, ppid=267, state=RUNNABLE:MOVE_REGION_UNASSIGN; > > MoveRegionProcedure hri=eacf81f79287d01f721a352407d5a1a5, > > source=wn2-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131806, > > destination= > wn2-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131806}, > > > > > .................................................................................................................................., > > {pid=670, ppid=267, state=RUNNABLE:MOVE_REGION_UNASSIGN; > > MoveRegionProcedure hri=941c5d8178257b7fc6bfa76b7d760468, > > source=wn4-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131742, > > destination= > wn4-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131742}, > > {pid=671, ppid=267, state=RUNNABLE:MOVE_REGION_UNASSIGN; > > MoveRegionProcedure hri=60ccd4513bc298b83d062cb0172ccba9, > > source=wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960, > > destination= > wn1-opencg.5w3ff4rocu0e1dpkokmkmgo5ib.zx.internal.cloudapp.net > > ,16020,1562169131960}] > > > > > > And then, the loop starts over > > > > The ReopenTableRegionsProcedure is stuck at > > REOPEN_TABLE_REGIONS_CONFIRM_REOPENED, where it starts over again, so, > > somehow, this should be relaed also with HBASE-20752 > > <https://issues.apache.org/jira/browse/HBASE-20752> > > > > > > Any idea what's going on? Is it related with the above tickets? > > > > > > Thanks > > >
