Dear HBase Community,
We are currently encountering an issue in HBase version 2.5.4 where regions
are not splitting as expected, causing the region size to exceed the
configured limits.
*Issue Details:*
- *HBase Version: 2.5.4*
- *Hadoop Version: 3.3.5*
- *Problem*: Regions are not splitting according to the configured
thresholds, resulting in regions growing beyond the set size limits.
- *Impact*: This can potentially lead to increased I/O load and reduced
performance due to oversized regions.
We have attached the relevant trace for further context.
> Caused by: *org.apache.hadoop.hbase.io.hfile.CorruptHFileException:
> Problem reading data index and meta index from file*
> hdfs://clustername/hbasedata/data/namespace/Table1/00d39663375797659886f6cf865fe9bf/ATTACHMENT_STREAM/b2be642f2d894d3eb8837ec7e4f796bb
> at
> org.apache.hadoop.hbase.io.hfile.HFileInfo.initMetaAndIndex(HFileInfo.java:392)
> ~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at
> org.apache.hadoop.hbase.regionserver.HStoreFile.open(HStoreFile.java:394)
> ~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at
> org.apache.hadoop.hbase.regionserver.HStoreFile.initReader(HStoreFile.java:518)
> ~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.splitStoreFile(HRegionFileSystem.java:693)
> ~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at
> org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.splitStoreFile(SplitTableRegionProcedure.java:806)
> ~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at
> org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.access$000(SplitTableRegionProcedure.java:98)
> ~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at
> org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure$StoreFileSplitter.call(SplitTableRegionProcedure.java:839)
> ~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at
> org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure$StoreFileSplitter.call(SplitTableRegionProcedure.java:820)
> ~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
> *Caused by: java.lang.UnsatisfiedLinkError:
> org.apache.hadoop.io.compress.lz4.Lz4Decompressor.decompressBytesDirect()I*
> at
> org.apache.hadoop.io.compress.lz4.Lz4Decompressor.decompressBytesDirect(Native
> Method) ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.io.compress.lz4.Lz4Decompressor.decompress(Lz4Decompressor.java:231)
> ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:88)
> ~[hadoop-common-3.2.4.jar:?]
> at
> org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:105)
> ~[hadoop-common-3.2.4.jar:?]
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> ~[?:1.8.0_181]
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> ~[?:1.8.0_181]
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> ~[?:1.8.0_181]
> at
> org.apache.hadoop.hbase.io.util.BlockIOUtils.readFullyWithHeapBuffer(BlockIOUtils.java:151)
> ~[hbase-common-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at
> org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultDecodingContext.prepareDecoding(HFileBlockDefaultDecodingContext.java:104)
> ~[hbase-common-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.unpack(HFileBlock.java:644)
> ~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl$1.nextBlock(HFileBlock.java:1397)
> ~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl$1.nextBlockWithBlockType(HFileBlock.java:1407)
> ~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at
> org.apache.hadoop.hbase.io.hfile.HFileInfo.initMetaAndIndex(HFileInfo.java:365)
> ~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at
> org.apache.hadoop.hbase.regionserver.HStoreFile.open(HStoreFile.java:394)
> ~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at
> org.apache.hadoop.hbase.regionserver.HStoreFile.initReader(HStoreFile.java:518)
> ~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.splitStoreFile(HRegionFileSystem.java:693)
> ~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at
> org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.splitStoreFile(SplitTableRegionProcedure.java:806)
> ~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at
> org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.access$000(SplitTableRegionProcedure.java:98)
> ~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at
> org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure$StoreFileSplitter.call(SplitTableRegionProcedure.java:839)
> ~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at
> org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure$StoreFileSplitter.call(SplitTableRegionProcedure.java:820)
> ~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_181]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_181]
If anyone has encountered a similar issue or has suggestions, please share
any possible workarounds.
Thank you in advance.
*Regards,*
*Manimekalai K*
2024-10-29T13:38:26,562 INFO [PEWorker-4] procedure.MasterProcedureScheduler:
Took xlock for pid=245529, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE;
SplitTableRegionProcedure table=namespace:Table1,
parent=00d39663375797659886f6cf865fe9bf,
daughterA=9603dacb4a968ea40617ab309183dd22,
daughterB=22529bc41843e1fe7f89ce7935d895b0
2024-10-29T13:38:26,562 INFO [PEWorker-4] procedure.MasterProcedureScheduler:
Took xlock for pid=245529, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE;
SplitTableRegionProcedure table=namespace:Table1,
parent=00d39663375797659886f6cf865fe9bf,
daughterA=9603dacb4a968ea40617ab309183dd22,
daughterB=22529bc41843e1fe7f89ce7935d895b0
2024-10-29T13:38:26,562 INFO [PEWorker-4] procedure.MasterProcedureScheduler:
Took xlock for pid=245529, state=RUNNABLE:SPLIT_TABLE_REGION_PREPARE;
SplitTableRegionProcedure table=namespace:Table1,
parent=00d39663375797659886f6cf865fe9bf,
daughterA=9603dacb4a968ea40617ab309183dd22,
daughterB=22529bc41843e1fe7f89ce7935d895b0
2024-10-29T13:38:26,565 INFO [PEWorker-4] procedure2.ProcedureExecutor:
Initialized subprocedures=[{pid=245530, ppid=245529,
state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure
table=namespace:Table1, region=00d39663375797659886f6cf865fe9bf, UNASSIGN}]
2024-10-29T13:38:26,566 INFO [PEWorker-4] procedure.MasterProcedureScheduler:
Took xlock for pid=245530, ppid=245529,
state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure
table=namespace:Table1, region=00d39663375797659886f6cf865fe9bf, UNASSIGN
2024-10-29T13:38:26,566 INFO [PEWorker-4] assignment.RegionStateStore:
pid=245530 updating hbase:meta row=00d39663375797659886f6cf865fe9bf,
regionState=CLOSING, regionLocation=10.XXX.XX.XX,16020,1730041753236
2024-10-29T13:38:26,569 INFO [PEWorker-4] procedure2.ProcedureExecutor:
Initialized subprocedures=[{pid=245531, ppid=245530, state=RUNNABLE;
CloseRegionProcedure 00d39663375797659886f6cf865fe9bf,
server=10.XXX.XX.XX,16020,1730041753236}]
2024-10-29T13:38:26,868 INFO [PEWorker-7] assignment.RegionStateStore:
pid=245530 updating hbase:meta row=00d39663375797659886f6cf865fe9bf,
regionState=CLOSED
2024-10-29T13:38:26,874 INFO [PEWorker-7] procedure2.ProcedureExecutor:
Finished subprocedure pid=245531, resume processing ppid=245530
2024-10-29T13:38:26,874 INFO [PEWorker-7] procedure2.ProcedureExecutor:
Finished pid=245531, ppid=245530, state=SUCCESS; CloseRegionProcedure
00d39663375797659886f6cf865fe9bf, server=10.XXX.XX.XX,16020,1730041753236 in
303 msec
2024-10-29T13:38:26,876 INFO [PEWorker-14] procedure2.ProcedureExecutor:
Finished subprocedure pid=245530, resume processing ppid=245529
2024-10-29T13:38:26,876 INFO [PEWorker-14] procedure2.ProcedureExecutor:
Finished pid=245530, ppid=245529, state=SUCCESS; TransitRegionStateProcedure
table=namespace:Table1, region=00d39663375797659886f6cf865fe9bf, UNASSIGN in
309 msec
2024-10-29T13:38:26,894 INFO [PEWorker-1]
assignment.SplitTableRegionProcedure: pid=245529 splitting 3 storefiles,
region=00d39663375797659886f6cf865fe9bf, threads=3
2024-10-29T13:38:26,947 ERROR [PEWorker-1]
assignment.SplitTableRegionProcedure: Splitting
00d39663375797659886f6cf865fe9bf, pid=245529,
state=RUNNABLE:SPLIT_TABLE_REGION_CREATE_DAUGHTER_REGIONS, locked=true;
SplitTableRegionProcedure table=namespace:Table1,
parent=00d39663375797659886f6cf865fe9bf,
daughterA=9603dacb4a968ea40617ab309183dd22,
daughterB=22529bc41843e1fe7f89ce7935d895b0
java.io.IOException: java.util.concurrent.ExecutionException:
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading data
index and meta index from file
hdfs://clustername/hbasedata/data/namespace/Table1/00d39663375797659886f6cf865fe9bf/ATTACHMENT_STREAM/b2be642f2d894d3eb8837ec7e4f796bb
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.splitStoreFiles(SplitTableRegionProcedure.java:773)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.createDaughterRegions(SplitTableRegionProcedure.java:628)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:315)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.executeFromState(SplitTableRegionProcedure.java:97)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:188)
~[hbase-procedure-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:922)
~[hbase-procedure-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1650)
~[hbase-procedure-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1396)
~[hbase-procedure-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1000(ProcedureExecutor.java:75)
~[hbase-procedure-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.runProcedure(ProcedureExecutor.java:1962)
~[hbase-procedure-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
~[hbase-common-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1989)
~[hbase-procedure-2.5.4-hadoop3.jar:2.5.4-hadoop3]
Caused by: java.util.concurrent.ExecutionException:
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading data
index and meta index from file
hdfs://clustername/hbasedata/data/namespace/Table1/00d39663375797659886f6cf865fe9bf/ATTACHMENT_STREAM/b2be642f2d894d3eb8837ec7e4f796bb
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
~[?:1.8.0_181]
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
~[?:1.8.0_181]
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.splitStoreFiles(SplitTableRegionProcedure.java:763)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
... 11 more
Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
reading data index and meta index from file
hdfs://clustername/hbasedata/data/namespace/Table1/00d39663375797659886f6cf865fe9bf/ATTACHMENT_STREAM/b2be642f2d894d3eb8837ec7e4f796bb
at
org.apache.hadoop.hbase.io.hfile.HFileInfo.initMetaAndIndex(HFileInfo.java:392)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.regionserver.HStoreFile.open(HStoreFile.java:394)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.regionserver.HStoreFile.initReader(HStoreFile.java:518)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.splitStoreFile(HRegionFileSystem.java:693)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.splitStoreFile(SplitTableRegionProcedure.java:806)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.access$000(SplitTableRegionProcedure.java:98)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure$StoreFileSplitter.call(SplitTableRegionProcedure.java:839)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure$StoreFileSplitter.call(SplitTableRegionProcedure.java:820)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
Caused by: java.lang.UnsatisfiedLinkError:
org.apache.hadoop.io.compress.lz4.Lz4Decompressor.decompressBytesDirect()I
at
org.apache.hadoop.io.compress.lz4.Lz4Decompressor.decompressBytesDirect(Native
Method) ~[hadoop-common-3.2.4.jar:?]
at
org.apache.hadoop.io.compress.lz4.Lz4Decompressor.decompress(Lz4Decompressor.java:231)
~[hadoop-common-3.2.4.jar:?]
at
org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:88)
~[hadoop-common-3.2.4.jar:?]
at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:105)
~[hadoop-common-3.2.4.jar:?]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
~[?:1.8.0_181]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
~[?:1.8.0_181]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
~[?:1.8.0_181]
at
org.apache.hadoop.hbase.io.util.BlockIOUtils.readFullyWithHeapBuffer(BlockIOUtils.java:151)
~[hbase-common-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultDecodingContext.prepareDecoding(HFileBlockDefaultDecodingContext.java:104)
~[hbase-common-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.io.hfile.HFileBlock.unpack(HFileBlock.java:644)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl$1.nextBlock(HFileBlock.java:1397)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl$1.nextBlockWithBlockType(HFileBlock.java:1407)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.io.hfile.HFileInfo.initMetaAndIndex(HFileInfo.java:365)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.regionserver.HStoreFile.open(HStoreFile.java:394)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.regionserver.HStoreFile.initReader(HStoreFile.java:518)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.splitStoreFile(HRegionFileSystem.java:693)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.splitStoreFile(SplitTableRegionProcedure.java:806)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure.access$000(SplitTableRegionProcedure.java:98)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure$StoreFileSplitter.call(SplitTableRegionProcedure.java:839)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at
org.apache.hadoop.hbase.master.assignment.SplitTableRegionProcedure$StoreFileSplitter.call(SplitTableRegionProcedure.java:820)
~[hbase-server-2.5.4-hadoop3.jar:2.5.4-hadoop3]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_181]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
2024-10-29T13:38:26,958 WARN [PEWorker-1] regionserver.HRegionFileSystem:
Failed delete of
hdfs://clustername/hbasedata/data/namespace/Table1/9603dacb4a968ea40617ab309183dd22
2024-10-29T13:38:26,963 WARN [PEWorker-1] regionserver.HRegionFileSystem:
Failed delete of
hdfs://clustername/hbasedata/data/namespace/Table1/22529bc41843e1fe7f89ce7935d895b0
2024-10-29T13:38:26,965 INFO [PEWorker-15] procedure.MasterProcedureScheduler:
Waiting on xlock for pid=245532,
state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE;
TransitRegionStateProcedure table=namespace:Table1,
region=00d39663375797659886f6cf865fe9bf, ASSIGN held by pid=245529
2024-10-29T13:38:26,969 INFO [PEWorker-1] procedure2.ProcedureExecutor: Rolled
back pid=245529, state=ROLLEDBACK, exception=java.io.IOException via
master-split-regions:java.io.IOException:
java.util.concurrent.ExecutionException:
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading data
index and meta index from file
hdfs://clustername/hbasedata/data/namespace/Table1/00d39663375797659886f6cf865fe9bf/ATTACHMENT_STREAM/b2be642f2d894d3eb8837ec7e4f796bb;
SplitTableRegionProcedure table=namespace:Table1,
parent=00d39663375797659886f6cf865fe9bf,
daughterA=9603dacb4a968ea40617ab309183dd22,
daughterB=22529bc41843e1fe7f89ce7935d895b0 exec-time=409 msec
2024-10-29T13:38:26,969 INFO [PEWorker-6] procedure.MasterProcedureScheduler:
Took xlock for pid=245532,
state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE;
TransitRegionStateProcedure table=namespace:Table1,
region=00d39663375797659886f6cf865fe9bf, ASSIGN
2024-10-29T13:38:26,970 INFO [PEWorker-6]
assignment.TransitRegionStateProcedure: Starting pid=245532,
state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true;
TransitRegionStateProcedure table=namespace:Table1,
region=00d39663375797659886f6cf865fe9bf, ASSIGN; state=CLOSED,
location=10.XXX.XX.XX,16020,1730041753236; forceNewPlan=false, retain=true
2024-10-29T13:38:27,120 INFO [10.213.19.200:16020] balancer.BaseLoadBalancer:
Reassigned 1 regions. 1 retained the pre-restart assignment.
2024-10-29T13:38:27,120 INFO [PEWorker-9] assignment.RegionStateStore:
pid=245532 updating hbase:meta row=00d39663375797659886f6cf865fe9bf,
regionState=OPENING, regionLocation=10.XXX.XX.XX,16020,1730041753236
2024-10-29T13:38:27,124 INFO [PEWorker-9] procedure2.ProcedureExecutor:
Initialized subprocedures=[{pid=245533, ppid=245532, state=RUNNABLE;
OpenRegionProcedure 00d39663375797659886f6cf865fe9bf,
server=10.XXX.XX.XX,16020,1730041753236}]
2024-10-29T13:38:27,424 INFO [PEWorker-10] assignment.RegionStateStore:
pid=245532 updating hbase:meta row=00d39663375797659886f6cf865fe9bf,
regionState=OPEN, repBarrier=9530162, openSeqNum=9530162,
regionLocation=10.XXX.XX.XX,16020,1730041753236
2024-10-29T13:38:27,430 INFO [PEWorker-10] procedure2.ProcedureExecutor:
Finished subprocedure pid=245533, resume processing ppid=245532
2024-10-29T13:38:27,430 INFO [PEWorker-10] procedure2.ProcedureExecutor:
Finished pid=245533, ppid=245532, state=SUCCESS; OpenRegionProcedure
00d39663375797659886f6cf865fe9bf, server=10.XXX.XX.XX,16020,1730041753236 in
304 msec
2024-10-29T13:38:27,431 INFO [PEWorker-10] procedure2.ProcedureExecutor:
Finished pid=245532, state=SUCCESS; TransitRegionStateProcedure
table=namespace:Table1, region=00d39663375797659886f6cf865fe9bf, ASSIGN in 466
msec
