Have you also tried restarting active master? > regionserver.SplitRequest: Running rollback/cleanup of failed split
Can you provide few logs after this line? Also, master logs for this region when it was going through split will also be helpful. > My only > other thought is to disable the table, delete the parent region and > enable the table. This manual step should not be required. Do you see any logs with RS_ZK_REQUEST_REGION_SPLIT on master side? Without further logs after "Running rollback/cleanup of failed split", I am not sure if the issue is similar but you can try comparing logs from https://issues.apache.org/jira/browse/HBASE-23261 If active master failover doesn't solve RIT, please upload further logs as requested above on some public file hosting server and you can provide link here. That will be easier way to provide more logs than pasting all relevant logs over this thread. On 2020/12/11 14:36:53, Zach Johnson <[email protected]> wrote: > version: HBase 1.4.8 > > We have a stuck RIT that wont clear. The parent was offlined, and > daughter region were created and moved out of the parents .split dir. > But the parent region is never removed and the RIT doesn't clear. We > have tried to run `move '<parent_region_name>'`, `unassign > '<parent_region_name>'`, and |sudo stop hbase-regionserver && sudo start > hbase-regionserver |on the parent and daughter regionservers. My only > other thought is to disable the table, delete the parent region and > enable the table. Any advice on how to clear the RIT or insight on what > is causing it would be appreciated. I have included some logs bellow > that might be helpful. > > Thanks, > > Zach > > Region server logs: > > 2020-12-04 14:00:35,430 INFO > [RpcServer.priority.FPBQ.Fifo.handler=17,queue=7,port=16020] > regionserver.RSRpcServices: Close d0b06e343bc8ae49ef7e5b66089fcb2d, > moving to null > 2020-12-04 14:00:35,431 INFO > > [StoreCloserThread-synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.-1] > regionserver.HStore: Closed d > 2020-12-04 14:00:35,436 INFO [RS_CLOSE_REGION-ip-10-0-1-4:16020-0] > regionserver.HRegion: Closed > > synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. > 2020-12-04 14:00:35,438 INFO > [RpcServer.priority.FPBQ.Fifo.handler=17,queue=7,port=16020] > regionserver.RSRpcServices: Open > > synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. > 2020-12-04 14:00:35,703 INFO > [StoreOpener-d0b06e343bc8ae49ef7e5b66089fcb2d-1] hfile.CacheConfig: > Created cacheConfig for d: > blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@58ff4bf2, > cacheDataOnRead=true, cacheDataOnWrite=false, > cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, > cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false > 2020-12-04 14:00:35,703 INFO > [StoreOpener-d0b06e343bc8ae49ef7e5b66089fcb2d-1] > compactions.CompactionConfiguration: size [1409286144, 5000000000, > 5000000000); files [6, 10); ratio 1.000000; off-peak ratio 5.000000; > throttle point 28185722880; major period 0, major jitter 0.500000, > min locality to compact 0.000000; tiered compaction: max_age > 9223372036854775807, incoming window min 6, compaction policy for > tiered window > > org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, > single output for minor true, compaction window factory > > org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory > 2020-12-04 14:00:45,076 INFO [RS_OPEN_REGION-ip-10-0-1-4:16020-2] > regionserver.HRegion: Onlined d0b06e343bc8ae49ef7e5b66089fcb2d; next > sequenceid=2009960 > 2020-12-04 14:00:45,077 INFO > [PostOpenDeployTasks:d0b06e343bc8ae49ef7e5b66089fcb2d] > regionserver.HRegionServer: Post open deploy tasks for > > synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. > 2020-12-04 14:16:13,156 INFO > [RpcServer.priority.FPBQ.Fifo.handler=17,queue=7,port=16020] > regionserver.RSRpcServices: Splitting > > synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. > 2020-12-04 14:16:20,003 INFO > > [regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444] > regionserver.SplitTransaction: Starting split of region > > synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. > 2020-12-04 14:16:20,457 INFO > > [StoreCloserThread-synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.-1] > regionserver.HStore: Closed d > 2020-12-04 14:16:20,462 INFO > > [regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444] > regionserver.HRegion: Closed > > synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. > 2020-12-04 14:16:20,462 INFO > > [regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444] > regionserver.SplitTransaction: Preparing to split 14 storefiles for > region > > synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. > using 8 threads > 2020-12-04 14:16:22,228 INFO > > [regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444] > s3n2.S3NativeFileSystem2: rename > > s3://bucket/root/data/default/synthetic_PRM_attr_dtg_v8_00609/d0b06e343bc8ae49ef7e5b66089fcb2d/.splits/453d941ff97a10bcf28e6ffe22d1d0e2 > > s3://bucket/root/data/default/synthetic_PRM_attr_dtg_v8_00609/453d941ff97a10bcf28e6ffe22d1d0e2 > 2020-12-04 14:16:23,988 INFO > > [regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444] > s3n2.S3NativeFileSystem2: rename > > s3://bucket/root/data/default/synthetic_PRM_attr_dtg_v8_00609/d0b06e343bc8ae49ef7e5b66089fcb2d/.splits/9d210ddbe593c83c62ac2d02413b7d97 > > s3://bucket/root/data/default/synthetic_PRM_attr_dtg_v8_00609/9d210ddbe593c83c62ac2d02413b7d97 > 2020-12-04 14:16:26,391 INFO > > [regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444] > regionserver.SplitRequest: Running rollback/cleanup of failed split > of > > synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.; > Failed > > ip-10-0-1-4.ec2.internal,16020,1606786380554-daughterOpener=9d210ddbe593c83c62ac2d02413b7d97 > > > echo "scan 'hbase:meta'" grep file for region > > > synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. > column=info:regioninfo, timestamp=1607091385153, value={ENCODED => > d0b06e343bc8ae49ef7e5b66089fcb2d, NAME => > 'synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.', > STARTKEY => '\x018000017536e25cf0\x00e13', ENDKEY => '\x02', OFFLINE => > true, SPLIT => true} > > synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. > column=info:seqnumDuringOpen, timestamp=1607090445077, > value=\x00\x00\x00\x00\x00\x1E\xABh > > synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. > column=info:server, timestamp=1607090445077, > value=ip-10-0-1-4.ec2.internal:16020 > > synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. > column=info:serverstartcode, timestamp=1607090445077, value=1606786380554 > > synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. > column=info:sn, timestamp=1606786396087, > value=ip-10-0-1-4.ec2.internal,16020,1606786380554 > > synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. > column=info:splitA, timestamp=1607091385153, value={ENCODED => > 453d941ff97a10bcf28e6ffe22d1d0e2, NAME => > 'synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1607091380003.453d941ff97a10bcf28e6ffe22d1d0e2.', > STARTKEY => '\x018000017536e25cf0\x00e13', ENDKEY => > '\x01800001753f422970\x0063'} > > synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. > column=info:splitB, timestamp=1607091385153, value={ENCODED => > 9d210ddbe593c83c62ac2d02413b7d97, NAME => > 'synthetic_PRM_attr_dtg_v8_00609,\x01800001753f422970\x0063,1607091380003.9d210ddbe593c83c62ac2d02413b7d97.', > STARTKEY => '\x01800001753f422970\x0063', ENDKEY => '\x02'} > > synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d. > column=info:state, timestamp=1607091380003, value=SPLITTING > >
