chaijunjie created HBASE-28541: ---------------------------------- Summary: RegionServer abort when max sequence id wrong Key: HBASE-28541 URL: https://issues.apache.org/jira/browse/HBASE-28541 Project: HBase Issue Type: Bug Affects Versions: 2.2.3 Reporter: chaijunjie
When we disable a table, some region close failed because max sequence id is less than the old max sequence id, then RS abort...Then the region close success on another RS.. we just use bulkload...is it releated? -------------------------------------------------------------------------------------------- 2024-04-20 23:44:20,611 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Bulk-load file hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/4f43f6cf569643da8ecc140668fa726e is on different filesystem than the destination store. Copying file over to destination filesystem. | org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:540) 2024-04-20 23:44:20,645 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Copied hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/4f43f6cf569643da8ecc140668fa726e to temporary path on destination filesystem: viewfs://ClusterX/hbase/data/default/boss20240420/dc30be37cbd0ddee2f49712c55fd36cd/.tmp/6f7e97f2e3994881a6e04b36759f495e | org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:544) 2024-04-20 23:44:20,648 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Loaded HFile viewfs://ClusterX/hbase/data/default/boss20240420/dc30be37cbd0ddee2f49712c55fd36cd/.tmp/6f7e97f2e3994881a6e04b36759f495e into dc30be37cbd0ddee2f49712c55fd36cd/info as viewfs://ClusterX/hbase/data/default/boss20240420/dc30be37cbd0ddee2f49712c55fd36cd/info/3946dc15ea1c49b582a2ed175fe12ec3_SeqId_26_ - updating store file list. | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:908) 2024-04-20 23:44:20,664 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Loaded HFile viewfs://ClusterX/hbase/data/default/boss20240420/dc30be37cbd0ddee2f49712c55fd36cd/info/3946dc15ea1c49b582a2ed175fe12ec3_SeqId_26_ into dc30be37cbd0ddee2f49712c55fd36cd/info | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:942) 2024-04-20 23:44:20,664 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Successfully loaded viewfs://ClusterX/hbase/data/default/boss20240420/dc30be37cbd0ddee2f49712c55fd36cd/.tmp/6f7e97f2e3994881a6e04b36759f495e into dc30be37cbd0ddee2f49712c55fd36cd/info (new location: viewfs://ClusterX/hbase/data/default/boss20240420/dc30be37cbd0ddee2f49712c55fd36cd/info/3946dc15ea1c49b582a2ed175fe12ec3_SeqId_26_) | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:914) 2024-04-20 23:44:23,592 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Validating hfile at viewfs://ClusterX/tenant/hw_cdr/dsttab/load_20240420232000/boss20240420/info/a97bb2bfcb5244279bcedbe20ec8322b for inclusion in b3d102e51ca3b78a1078580ed8a002de/info | org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:817) 2024-04-20 23:44:23,669 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Bulk-load file hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/a97bb2bfcb5244279bcedbe20ec8322b is on different filesystem than the destination store. Copying file over to destination filesystem. | org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:540) 2024-04-20 23:44:23,743 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Copied hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/a97bb2bfcb5244279bcedbe20ec8322b to temporary path on destination filesystem: viewfs://ClusterX/hbase/data/default/boss20240420/b3d102e51ca3b78a1078580ed8a002de/.tmp/f40ec16d10ab4474ace7af41d9126aa0 | org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:544) 2024-04-20 23:44:23,747 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Loaded HFile viewfs://ClusterX/hbase/data/default/boss20240420/b3d102e51ca3b78a1078580ed8a002de/.tmp/f40ec16d10ab4474ace7af41d9126aa0 into b3d102e51ca3b78a1078580ed8a002de/info as viewfs://ClusterX/hbase/data/default/boss20240420/b3d102e51ca3b78a1078580ed8a002de/info/7cfcf858a6b34bd1b9d47ccce5dde9e4_SeqId_41_ - updating store file list. | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:908) 2024-04-20 23:44:23,760 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Loaded HFile viewfs://ClusterX/hbase/data/default/boss20240420/b3d102e51ca3b78a1078580ed8a002de/info/7cfcf858a6b34bd1b9d47ccce5dde9e4_SeqId_41_ into b3d102e51ca3b78a1078580ed8a002de/info | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:942) 2024-04-20 23:44:23,760 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Successfully loaded viewfs://ClusterX/hbase/data/default/boss20240420/b3d102e51ca3b78a1078580ed8a002de/.tmp/f40ec16d10ab4474ace7af41d9126aa0 into b3d102e51ca3b78a1078580ed8a002de/info (new location: viewfs://ClusterX/hbase/data/default/boss20240420/b3d102e51ca3b78a1078580ed8a002de/info/7cfcf858a6b34bd1b9d47ccce5dde9e4_SeqId_41_) | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:914) 2024-04-20 23:44:25,443 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Validating hfile at viewfs://ClusterX/tenant/hw_cdr/dsttab/load_20240420232000/boss20240420/info/4dac6fe338ed45028dd4ac859d02c386 for inclusion in eec5518fe33d8c6b5f99f66e20bf782d/info | org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:817) 2024-04-20 23:44:25,720 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Bulk-load file hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/4dac6fe338ed45028dd4ac859d02c386 is on different filesystem than the destination store. Copying file over to destination filesystem. | org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:540) 2024-04-20 23:44:25,798 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Copied hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/4dac6fe338ed45028dd4ac859d02c386 to temporary path on destination filesystem: viewfs://ClusterX/hbase/data/default/boss20240420/eec5518fe33d8c6b5f99f66e20bf782d/.tmp/09cb80cf4b5d47b78ddf31eafce4f5f6 | org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:544) 2024-04-20 23:44:25,802 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Loaded HFile viewfs://ClusterX/hbase/data/default/boss20240420/eec5518fe33d8c6b5f99f66e20bf782d/.tmp/09cb80cf4b5d47b78ddf31eafce4f5f6 into eec5518fe33d8c6b5f99f66e20bf782d/info as viewfs://ClusterX/hbase/data/default/boss20240420/eec5518fe33d8c6b5f99f66e20bf782d/info/6d674b96ccec40339a4a1f486a65c4a7_SeqId_20_ - updating store file list. | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:908) 2024-04-20 23:44:25,823 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Loaded HFile viewfs://ClusterX/hbase/data/default/boss20240420/eec5518fe33d8c6b5f99f66e20bf782d/info/6d674b96ccec40339a4a1f486a65c4a7_SeqId_20_ into eec5518fe33d8c6b5f99f66e20bf782d/info | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:942) 2024-04-20 23:44:25,823 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Successfully loaded viewfs://ClusterX/hbase/data/default/boss20240420/eec5518fe33d8c6b5f99f66e20bf782d/.tmp/09cb80cf4b5d47b78ddf31eafce4f5f6 into eec5518fe33d8c6b5f99f66e20bf782d/info (new location: viewfs://ClusterX/hbase/data/default/boss20240420/eec5518fe33d8c6b5f99f66e20bf782d/info/6d674b96ccec40339a4a1f486a65c4a7_SeqId_20_) | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:914) 2024-04-20 23:44:28,837 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Validating hfile at viewfs://ClusterX/tenant/hw_cdr/dsttab/load_20240420232000/boss20240420/info/ef44caa4e05440569f76a05d351e4a62 for inclusion in 21d5667a8bcd59088ae865f5d9660283/info | org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:817) 2024-04-20 23:44:28,950 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Bulk-load file hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/ef44caa4e05440569f76a05d351e4a62 is on different filesystem than the destination store. Copying file over to destination filesystem. | org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:540) 2024-04-20 23:44:29,007 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Copied hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/ef44caa4e05440569f76a05d351e4a62 to temporary path on destination filesystem: viewfs://ClusterX/hbase/data/default/boss20240420/21d5667a8bcd59088ae865f5d9660283/.tmp/1be6287d04084bf48afd8eb0e0e63a6e | org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:544) 2024-04-20 23:44:29,010 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Loaded HFile viewfs://ClusterX/hbase/data/default/boss20240420/21d5667a8bcd59088ae865f5d9660283/.tmp/1be6287d04084bf48afd8eb0e0e63a6e into 21d5667a8bcd59088ae865f5d9660283/info as viewfs://ClusterX/hbase/data/default/boss20240420/21d5667a8bcd59088ae865f5d9660283/info/fb872c3f89e543c6b54fcb54898112d4_SeqId_16_ - updating store file list. | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:908) 2024-04-20 23:44:29,063 | INFO | RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Validating hfile at viewfs://ClusterX/tenant/hw_cdr/dsttab/load_20240420232000/boss20240420/info/6fdde06815624c5095d39e7b98cc81c7 for inclusion in 0bf7d39f1708c5e50444c810f60d3f9f/info | org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:817) 2024-04-20 23:44:29,066 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Loaded HFile viewfs://ClusterX/hbase/data/default/boss20240420/21d5667a8bcd59088ae865f5d9660283/info/fb872c3f89e543c6b54fcb54898112d4_SeqId_16_ into 21d5667a8bcd59088ae865f5d9660283/info | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:942) 2024-04-20 23:44:29,066 | INFO | RpcServer.default.FPBQ.Fifo.handler=483,queue=33,port=21302 | Successfully loaded viewfs://ClusterX/hbase/data/default/boss20240420/21d5667a8bcd59088ae865f5d9660283/.tmp/1be6287d04084bf48afd8eb0e0e63a6e into 21d5667a8bcd59088ae865f5d9660283/info (new location: viewfs://ClusterX/hbase/data/default/boss20240420/21d5667a8bcd59088ae865f5d9660283/info/fb872c3f89e543c6b54fcb54898112d4_SeqId_16_) | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:914) 2024-04-20 23:44:29,116 | INFO | RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Bulk-load file hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/6fdde06815624c5095d39e7b98cc81c7 is on different filesystem than the destination store. Copying file over to destination filesystem. | org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:540) 2024-04-20 23:44:29,182 | INFO | RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Copied hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/6fdde06815624c5095d39e7b98cc81c7 to temporary path on destination filesystem: viewfs://ClusterX/hbase/data/default/boss20240420/0bf7d39f1708c5e50444c810f60d3f9f/.tmp/0b6350c99fca4b06ba873f49a1ddb835 | org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:544) 2024-04-20 23:44:29,185 | INFO | RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Loaded HFile viewfs://ClusterX/hbase/data/default/boss20240420/0bf7d39f1708c5e50444c810f60d3f9f/.tmp/0b6350c99fca4b06ba873f49a1ddb835 into 0bf7d39f1708c5e50444c810f60d3f9f/info as viewfs://ClusterX/hbase/data/default/boss20240420/0bf7d39f1708c5e50444c810f60d3f9f/info/2ae95a60b63541538939adfef0ab2190_SeqId_34_ - updating store file list. | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:908) 2024-04-20 23:44:29,201 | INFO | RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Loaded HFile viewfs://ClusterX/hbase/data/default/boss20240420/0bf7d39f1708c5e50444c810f60d3f9f/info/2ae95a60b63541538939adfef0ab2190_SeqId_34_ into 0bf7d39f1708c5e50444c810f60d3f9f/info | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:942) 2024-04-20 23:44:29,201 | INFO | RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Successfully loaded viewfs://ClusterX/hbase/data/default/boss20240420/0bf7d39f1708c5e50444c810f60d3f9f/.tmp/0b6350c99fca4b06ba873f49a1ddb835 into 0bf7d39f1708c5e50444c810f60d3f9f/info (new location: viewfs://ClusterX/hbase/data/default/boss20240420/0bf7d39f1708c5e50444c810f60d3f9f/info/2ae95a60b63541538939adfef0ab2190_SeqId_34_) | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:914) 2024-04-20 23:44:31,993 | INFO | RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Validating hfile at viewfs://ClusterX/tenant/hw_cdr/dsttab/load_20240420232000/boss20240420/info/f886ffd9d3d44804bcf164ff14ff24a6 for inclusion in bf26eb5268636a04d990e9a99da516fa/info | org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:817) 2024-04-20 23:44:32,071 | INFO | RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Bulk-load file hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/f886ffd9d3d44804bcf164ff14ff24a6 is on different filesystem than the destination store. Copying file over to destination filesystem. | org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:540) 2024-04-20 23:44:32,179 | INFO | RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Copied hdfs://hacluster/hbase/staging/cdr__boss20240420__t20amapi48eb4ce0k0t80jatvnnhpm5rohaj26a5rdsp0s7kifokcdd32ko89u7n/info/f886ffd9d3d44804bcf164ff14ff24a6 to temporary path on destination filesystem: viewfs://ClusterX/hbase/data/default/boss20240420/bf26eb5268636a04d990e9a99da516fa/.tmp/bb3afca8b1c34f30a44fbc23416e43c7 | org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:544) 2024-04-20 23:44:32,182 | INFO | RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Loaded HFile viewfs://ClusterX/hbase/data/default/boss20240420/bf26eb5268636a04d990e9a99da516fa/.tmp/bb3afca8b1c34f30a44fbc23416e43c7 into bf26eb5268636a04d990e9a99da516fa/info as viewfs://ClusterX/hbase/data/default/boss20240420/bf26eb5268636a04d990e9a99da516fa/info/e7cafc760651400f95b9929bc4dfe09f_SeqId_24_ - updating store file list. | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:908) 2024-04-20 23:44:32,194 | INFO | RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Loaded HFile viewfs://ClusterX/hbase/data/default/boss20240420/bf26eb5268636a04d990e9a99da516fa/info/e7cafc760651400f95b9929bc4dfe09f_SeqId_24_ into bf26eb5268636a04d990e9a99da516fa/info | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:942) 2024-04-20 23:44:32,194 | INFO | RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Successfully loaded viewfs://ClusterX/hbase/data/default/boss20240420/bf26eb5268636a04d990e9a99da516fa/.tmp/bb3afca8b1c34f30a44fbc23416e43c7 into bf26eb5268636a04d990e9a99da516fa/info (new location: viewfs://ClusterX/hbase/data/default/boss20240420/bf26eb5268636a04d990e9a99da516fa/info/e7cafc760651400f95b9929bc4dfe09f_SeqId_24_) | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:914) 2024-04-20 23:44:32,643 | INFO | LruBlockCacheStatsExecutor | totalSize=810.84 MB, freeSize=2.37 GB, max=3.16 GB, blockCount=6432, accesses=2141008613, hits=42674300, hitRatio=1.99%, , cachingAccesses=42757862, cachingHits=42611408, cachingHitsRatio=99.66%, evictions=863177, evicted=0, evictedPerRun=0.0 | org.apache.hadoop.hbase.io.hfile.LruBlockCache.logStats(LruBlockCache.java:940) 2024-04-20 23:44:32,967 | INFO | BucketCacheStatsExecutor | failedBlockAdditions=0, totalSize=2.00 GB, freeSize=2.00 GB, usedSize=1.60 MB, cacheSize=1.55 MB, accesses=2103504114, hits=59423, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0.00%, cachingAccesses=290936, cachingHits=946, cachingHitsRatio=0.33%, evictions=49, evicted=289802, evictedPerRun=5914.326530612245 | org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.logStats(BucketCache.java:637) 2024-04-20 23:44:35,020 | INFO | org.apache.ranger.audit.queue.AuditBatchQueue0 | Audit Status Log: name=hbaseRegional.async.summary.batch, finalDestination=hbaseRegional.async.summary.batch.log4j, interval=01:02.099 minutes, events=18, succcessCount=8, totalEvents=10351899, totalSuccessCount=1046226 | org.apache.ranger.audit.provider.BaseAuditHandler.logStatus(BaseAuditHandler.java:332) 2024-04-20 23:44:38,469 | INFO | p1tdhdpdn-v-7:21302Replication Statistics #0 | Global stats: WAL Edits Buffer Used=0B, Limit=268435456B | org.apache.hadoop.hbase.replication.regionserver.Replication$ReplicationStatisticsTask.printStats(Replication.java:241) 2024-04-20 23:45:35,922 | INFO | org.apache.ranger.audit.queue.AuditBatchQueue0 | Audit Status Log: name=hbaseRegional.async.summary.batch, finalDestination=hbaseRegional.async.summary.batch.log4j, interval=01:00.902 minutes, events=2, succcessCount=2, totalEvents=10351901, totalSuccessCount=1046228 | org.apache.ranger.audit.provider.BaseAuditHandler.logStatus(BaseAuditHandler.java:332) 2024-04-20 23:45:52,951 | INFO | HBase-Metrics2-1 | Stopping HBase metrics system... | org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:210) 2024-04-20 23:45:52,958 | INFO | HBase-Metrics2-1 | HBase metrics system stopped. | org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:216) 2024-04-20 23:45:53,459 | WARN | HBase-Metrics2-1 | Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties | org.apache.hadoop.metrics2.impl.MetricsConfig.loadFirst(MetricsConfig.java:133) 2024-04-20 23:45:53,459 | INFO | HBase-Metrics2-1 | Scheduled Metric snapshot period at 10 second(s). | org.apache.hadoop.metrics2.impl.MetricsSystemImpl.startTimer(MetricsSystemImpl.java:378) 2024-04-20 23:45:53,460 | INFO | HBase-Metrics2-1 | HBase metrics system started | org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:191) 2024-04-20 23:47:38,739 | INFO | RpcServer.default.FPBQ.Fifo.handler=415,queue=15,port=21302 | Chore [ScheduledChore: Name: QuotaRefresherChore Period: 300000 Unit: MILLISECONDS] is enabled. | org.apache.hadoop.hbase.ChoreService.scheduleChore(ChoreService.java:157) 2024-04-20 23:47:41,926 | INFO | org.apache.ranger.audit.queue.AuditBatchQueue0 | Audit Status Log: name=hbaseRegional.async.summary.batch.log4j, interval=03:30.005 minutes, events=10, succcessCount=10, totalEvents=1046228, totalSuccessCount=1046228 | org.apache.ranger.audit.provider.BaseAuditHandler.logStatus(BaseAuditHandler.java:332) 2024-04-20 23:48:36,094 | INFO | org.apache.ranger.audit.queue.AuditBatchQueue0 | Audit Status Log: name=hbaseRegional.async.summary.batch, finalDestination=hbaseRegional.async.summary.batch.log4j, interval=01:00.169 minutes, events=3787, succcessCount=896, totalEvents=10355688, totalSuccessCount=1047124 | org.apache.ranger.audit.provider.BaseAuditHandler.logStatus(BaseAuditHandler.java:332) 2024-04-20 23:49:25,976 | INFO | RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Validating hfile at viewfs://ClusterX/tenant/hw_cdr/dsttab/load_20240420233000/boss20240420/info/6403649b9cff4e2e9a716edfb91eed02 for inclusion in 24b5f9451a7c43c78f510539ac3124b6/info | org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:817) 2024-04-20 23:49:26,330 | INFO | RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Bulk-load file hdfs://hacluster/hbase/staging/cdr__boss20240420__npj0dvc9jt9shu2lknr23t8d0l9qu9v84rjc1atk9m69ludreffg4plgpjeo5jlb/info/6403649b9cff4e2e9a716edfb91eed02 is on different filesystem than the destination store. Copying file over to destination filesystem. | org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:540) 2024-04-20 23:49:26,762 | INFO | RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Copied hdfs://hacluster/hbase/staging/cdr__boss20240420__npj0dvc9jt9shu2lknr23t8d0l9qu9v84rjc1atk9m69ludreffg4plgpjeo5jlb/info/6403649b9cff4e2e9a716edfb91eed02 to temporary path on destination filesystem: viewfs://ClusterX/hbase/data/default/boss20240420/24b5f9451a7c43c78f510539ac3124b6/.tmp/a3eb0160995742509ccd235ebfcfa281 | org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:544) 2024-04-20 23:49:26,768 | INFO | RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Loaded HFile viewfs://ClusterX/hbase/data/default/boss20240420/24b5f9451a7c43c78f510539ac3124b6/.tmp/a3eb0160995742509ccd235ebfcfa281 into 24b5f9451a7c43c78f510539ac3124b6/info as viewfs://ClusterX/hbase/data/default/boss20240420/24b5f9451a7c43c78f510539ac3124b6/info/775e33b5b20242f1821c8b52d9b93ddf_SeqId_16_ - updating store file list. | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:908) 2024-04-20 23:49:26,782 | INFO | RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Loaded HFile viewfs://ClusterX/hbase/data/default/boss20240420/24b5f9451a7c43c78f510539ac3124b6/info/775e33b5b20242f1821c8b52d9b93ddf_SeqId_16_ into 24b5f9451a7c43c78f510539ac3124b6/info | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:942) 2024-04-20 23:49:26,782 | INFO | RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Successfully loaded viewfs://ClusterX/hbase/data/default/boss20240420/24b5f9451a7c43c78f510539ac3124b6/.tmp/a3eb0160995742509ccd235ebfcfa281 into 24b5f9451a7c43c78f510539ac3124b6/info (new location: viewfs://ClusterX/hbase/data/default/boss20240420/24b5f9451a7c43c78f510539ac3124b6/info/775e33b5b20242f1821c8b52d9b93ddf_SeqId_16_) | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:914) 2024-04-20 23:49:27,148 | INFO | RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Validating hfile at viewfs://ClusterX/tenant/hw_cdr/dsttab/load_20240420233000/boss20240420/info/474cd88557de4a999c14173dd65c8817 for inclusion in 281284cc6ea724c51c83ce23e5249f7b/info | org.apache.hadoop.hbase.regionserver.HStore.assertBulkLoadHFileOk(HStore.java:817) 2024-04-20 23:49:27,219 | INFO | RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Bulk-load file hdfs://hacluster/hbase/staging/cdr__boss20240420__npj0dvc9jt9shu2lknr23t8d0l9qu9v84rjc1atk9m69ludreffg4plgpjeo5jlb/info/474cd88557de4a999c14173dd65c8817 is on different filesystem than the destination store. Copying file over to destination filesystem. | org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:540) 2024-04-20 23:49:27,257 | INFO | RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Copied hdfs://hacluster/hbase/staging/cdr__boss20240420__npj0dvc9jt9shu2lknr23t8d0l9qu9v84rjc1atk9m69ludreffg4plgpjeo5jlb/info/474cd88557de4a999c14173dd65c8817 to temporary path on destination filesystem: viewfs://ClusterX/hbase/data/default/boss20240420/281284cc6ea724c51c83ce23e5249f7b/.tmp/7d352cb6f77047e9a39bc35379f70a92 | org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:544) 2024-04-20 23:49:27,260 | INFO | RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Loaded HFile viewfs://ClusterX/hbase/data/default/boss20240420/281284cc6ea724c51c83ce23e5249f7b/.tmp/7d352cb6f77047e9a39bc35379f70a92 into 281284cc6ea724c51c83ce23e5249f7b/info as viewfs://ClusterX/hbase/data/default/boss20240420/281284cc6ea724c51c83ce23e5249f7b/info/8fb058fa1e4c48fb862e559f873a1390_SeqId_32_ - updating store file list. | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:908) 2024-04-20 23:49:27,276 | INFO | RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Loaded HFile viewfs://ClusterX/hbase/data/default/boss20240420/281284cc6ea724c51c83ce23e5249f7b/info/8fb058fa1e4c48fb862e559f873a1390_SeqId_32_ into 281284cc6ea724c51c83ce23e5249f7b/info | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:942) 2024-04-20 23:49:27,276 | INFO | RpcServer.default.FPBQ.Fifo.handler=301,queue=1,port=21302 | Successfully loaded viewfs://ClusterX/hbase/data/default/boss20240420/281284cc6ea724c51c83ce23e5249f7b/.tmp/7d352cb6f77047e9a39bc35379f70a92 into 281284cc6ea724c51c83ce23e5249f7b/info (new location: viewfs://ClusterX/hbase/data/default/boss20240420/281284cc6ea724c51c83ce23e5249f7b/info/8fb058fa1e4c48fb862e559f873a1390_SeqId_32_) | org.apache.hadoop.hbase.regionserver.HStore.bulkLoadHFile(HStore.java:914) 2024-04-20 23:49:27,417 | INFO | regionserver/p1tdhdpdn-v-7:21302.Chore.11 | MemstoreFlusherChore requesting flush of boss20240405,H3020209~,1712160114717.4e7bdeeb9b222b113307199da2810e18. because 4e7bdeeb9b222b113307199da2810e18/info has an old edit so flush to free WALs after random delay 239909 ms | org.apache.hadoop.hbase.regionserver.HRegionServer$PeriodicMemStoreFlusher.chore(HRegionServer.java:2059) 2024-04-20 23:49:30,094 | INFO | org.apache.ranger.audit.queue.AuditBatchQueue0 | Audit Status Log: name=hbaseRegional.async.summary.batch.log4j, interval=01:48.168 minutes, events=896, succcessCount=896, totalEvents=1047124, totalSuccessCount=1047124 | org.apache.ranger.audit.provider.BaseAuditHandler.logStatus(BaseAuditHandler.java:332) 2024-04-20 23:49:32,643 | INFO | LruBlockCacheStatsExecutor | totalSize=811.85 MB, freeSize=2.37 GB, max=3.16 GB, blockCount=6440, accesses=2141096719, hits=42676155, hitRatio=1.99%, , cachingAccesses=42759717, cachingHits=42613255, cachingHitsRatio=99.66%, evictions=863207, evicted=0, evictedPerRun=0.0 | org.apache.hadoop.hbase.io.hfile.LruBlockCache.logStats(LruBlockCache.java:940) 2024-04-20 23:49:32,967 | INFO | BucketCacheStatsExecutor | failedBlockAdditions=0, totalSize=2.00 GB, freeSize=2.00 GB, usedSize=1.60 MB, cacheSize=1.55 MB, accesses=2103590357, hits=59423, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0.00%, cachingAccesses=290936, cachingHits=946, cachingHitsRatio=0.33%, evictions=49, evicted=289802, evictedPerRun=5914.326530612245 | org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.logStats(BucketCache.java:637) 2024-04-20 23:49:38,469 | INFO | p1tdhdpdn-v-7:21302Replication Statistics #0 | Global stats: WAL Edits Buffer Used=0B, Limit=268435456B | org.apache.hadoop.hbase.replication.regionserver.Replication$ReplicationStatisticsTask.printStats(Replication.java:241) 2024-04-20 23:49:39,094 | INFO | org.apache.ranger.audit.queue.AuditBatchQueue0 | Audit Status Log: name=hbaseRegional.async.summary.batch, finalDestination=hbaseRegional.async.summary.batch.log4j, interval=01:03.000 minutes, events=4, succcessCount=2, totalEvents=10355692, totalSuccessCount=1047126 | org.apache.ranger.audit.provider.BaseAuditHandler.logStatus(BaseAuditHandler.java:332) 2024-04-20 23:50:52,951 | INFO | HBase-Metrics2-1 | Stopping HBase metrics system... | org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:210) 2024-04-20 23:50:52,959 | INFO | HBase-Metrics2-1 | HBase metrics system stopped. | org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:216) 2024-04-20 23:50:53,461 | WARN | HBase-Metrics2-1 | Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties | org.apache.hadoop.metrics2.impl.MetricsConfig.loadFirst(MetricsConfig.java:133) 2024-04-20 23:50:53,476 | INFO | HBase-Metrics2-1 | Scheduled Metric snapshot period at 10 second(s). | org.apache.hadoop.metrics2.impl.MetricsSystemImpl.startTimer(MetricsSystemImpl.java:378) 2024-04-20 23:50:53,476 | INFO | HBase-Metrics2-1 | HBase metrics system started | org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:191) 2024-04-20 23:51:14,537 | INFO | regionserver/p1tdhdpdn-v-7:21302-longCompactions-9 | 0bf7d39f1708c5e50444c810f60d3f9f#info#compaction#11852 average throughput is 3.49 MB/second, slept 1444 time(s) and total slept time is 3969809 ms. 8 active operations remaining, total limit is 18.00 MB/second | org.apache.hadoop.hbase.regionserver.throttle.PressureAwareThroughputController.finish(PressureAwareThroughputController.java:145) 2024-04-20 23:51:15,245 | INFO | regionserver/p1tdhdpdn-v-7:21302-longCompactions-9 | Completed compaction of 15 (all) file(s) in 0bf7d39f1708c5e50444c810f60d3f9f/info of 0bf7d39f1708c5e50444c810f60d3f9f into 8ae2d25f22a34a01a8b895b0bfa7394e(size=2.2 G), total size for store is 2.2 G. This selection was in queue for 0sec, and took 1hrs, 9mins, 7sec to execute. | org.apache.hadoop.hbase.regionserver.HStore.logCompactionEndMessage(HStore.java:1669) 2024-04-20 23:51:15,245 | INFO | regionserver/p1tdhdpdn-v-7:21302-longCompactions-9 | Completed compaction Request=regionName=boss20240420,H1030107628~,1713456095893.0bf7d39f1708c5e50444c810f60d3f9f., storeName=info, fileCount=15, fileSize=2.3 G (30.8 M, 1.2 G, 1003.2 M, 25.0 M, 7.1 M, 94.0 K, 83.3 K, 27.0 K, 23.4 K, 21.6 K, 14.7 K, 10.9 K, 17.4 K, 10.9 K, 12.8 K), priority=2, time=1713624128170; duration=1hrs, 9mins, 7sec | org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:632) 2024-04-20 23:53:27,327 | INFO | MemStoreFlusher.3 | Unable to get remote Address | org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214) 2024-04-20 23:53:27,376 | INFO | MemStoreFlusher.3 | Flushing 4e7bdeeb9b222b113307199da2810e18 1/1 column families, dataSize=1.87 MB heapSize=2.50 MB | org.apache.hadoop.hbase.regionserver.HRegion.logFatLineOnFlush(HRegion.java:2870) 2024-04-20 23:53:27,524 | INFO | MemStoreFlusher.3 | Flushed memstore data size=1.87 MB at sequenceid=146705 (bloomFilter=true), to=viewfs://ClusterX/hbase/data/default/boss20240405/4e7bdeeb9b222b113307199da2810e18/.tmp/info/98071aebc1e4481aafc41f8e1e8d2243 | org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:85) 2024-04-20 23:53:27,554 | INFO | MemStoreFlusher.3 | Added viewfs://ClusterX/hbase/data/default/boss20240405/4e7bdeeb9b222b113307199da2810e18/info/98071aebc1e4481aafc41f8e1e8d2243, entries=2940, sequenceid=146705, filesize=190.2 K | org.apache.hadoop.hbase.regionserver.HStore.commitFile(HStore.java:1145) 2024-04-20 23:53:27,555 | INFO | MemStoreFlusher.3 | Finished flush of dataSize ~1.87 MB/1960410, heapSize ~2.50 MB/2618256, currentSize=0 B/0 for 4e7bdeeb9b222b113307199da2810e18 in 179ms, sequenceid=146705, compaction requested=false | org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:3079) 2024-04-20 23:53:30,121 | INFO | org.apache.ranger.audit.queue.AuditBatchQueue0 | Audit Status Log: name=hbaseRegional.async.summary.batch.log4j, interval=04:00.027 minutes, events=2, succcessCount=2, totalEvents=1047126, totalSuccessCount=1047126 | org.apache.ranger.audit.provider.BaseAuditHandler.logStatus(BaseAuditHandler.java:332) 2024-04-20 23:53:39,121 | INFO | org.apache.ranger.audit.queue.AuditBatchQueue0 | Audit Status Log: name=hbaseRegional.async.summary.batch, finalDestination=hbaseRegional.async.summary.batch.log4j, interval=01:00.001 minutes, events=2, succcessCount=1, totalEvents=10355694, totalSuccessCount=1047127 | org.apache.ranger.audit.provider.BaseAuditHandler.logStatus(BaseAuditHandler.java:332) 2024-04-20 23:54:32,643 | INFO | LruBlockCacheStatsExecutor | totalSize=812.98 MB, freeSize=2.37 GB, max=3.16 GB, blockCount=6449, accesses=2141184975, hits=42678224, hitRatio=1.99%, , cachingAccesses=42761786, cachingHits=42615315, cachingHitsRatio=99.66%, evictions=863237, evicted=0, evictedPerRun=0.0 | org.apache.hadoop.hbase.io.hfile.LruBlockCache.logStats(LruBlockCache.java:940) 2024-04-20 23:54:32,967 | INFO | BucketCacheStatsExecutor | failedBlockAdditions=0, totalSize=2.00 GB, freeSize=2.00 GB, usedSize=1.60 MB, cacheSize=1.55 MB, accesses=2103676600, hits=59423, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0.00%, cachingAccesses=290936, cachingHits=946, cachingHitsRatio=0.33%, evictions=49, evicted=289802, evictedPerRun=5914.326530612245 | org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.logStats(BucketCache.java:637) 2024-04-20 23:54:32,972 | INFO | MobFileCache #0 | MobFileCache Statistics, access: 0, miss: 0, hit: 0, hit ratio: 0%, evicted files: 0 | org.apache.hadoop.hbase.mob.MobFileCache.printStatistics(MobFileCache.java:341) 2024-04-20 23:54:38,469 | INFO | p1tdhdpdn-v-7:21302Replication Statistics #0 | Global stats: WAL Edits Buffer Used=0B, Limit=268435456B | org.apache.hadoop.hbase.replication.regionserver.Replication$ReplicationStatisticsTask.printStats(Replication.java:241) 2024-04-20 23:55:10,095 | INFO | regionserver/p1tdhdpdn-v-7:21302.logRoller | Rolled WAL /hbase/WALs/p1tdhdpdn-v-7,21302,1704995671636/p1tdhdpdn-v-7%2C21302%2C1704995671636.1713624909375 with entries=584, filesize=1.11 MB; new WAL /hbase/WALs/p1tdhdpdn-v-7,21302,1704995671636/p1tdhdpdn-v-7%2C21302%2C1704995671636.1713628510024 | org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.logRollAndSetupWalProps(AbstractFSWAL.java:727) 2024-04-20 23:55:10,095 | INFO | Log-Archiver-0 | Archiving viewfs://ClusterX/hbase/WALs/p1tdhdpdn-v-7,21302,1704995671636/p1tdhdpdn-v-7%2C21302%2C1704995671636.1713621309238 to viewfs://ClusterX/hbase/oldWALs/p1tdhdpdn-v-7%2C21302%2C1704995671636.1713621309238 | org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.archiveLogFile(AbstractFSWAL.java:708) 2024-04-20 23:55:52,951 | INFO | HBase-Metrics2-1 | Stopping HBase metrics system... | org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:210) 2024-04-20 23:55:52,958 | INFO | HBase-Metrics2-1 | HBase metrics system stopped. | org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:216) 2024-04-20 23:55:53,460 | WARN | HBase-Metrics2-1 | Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties | org.apache.hadoop.metrics2.impl.MetricsConfig.loadFirst(MetricsConfig.java:133) 2024-04-20 23:55:53,461 | INFO | HBase-Metrics2-1 | Scheduled Metric snapshot period at 10 second(s). | org.apache.hadoop.metrics2.impl.MetricsSystemImpl.startTimer(MetricsSystemImpl.java:378) 2024-04-20 23:55:53,461 | INFO | HBase-Metrics2-1 | HBase metrics system started | org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:191) 2024-04-20 23:59:32,643 | INFO | LruBlockCacheStatsExecutor | totalSize=814.11 MB, freeSize=2.36 GB, max=3.16 GB, blockCount=6458, accesses=2141273489, hits=42680456, hitRatio=1.99%, , cachingAccesses=42764018, cachingHits=42617538, cachingHitsRatio=99.66%, evictions=863267, evicted=0, evictedPerRun=0.0 | org.apache.hadoop.hbase.io.hfile.LruBlockCache.logStats(LruBlockCache.java:940) 2024-04-20 23:59:32,967 | INFO | BucketCacheStatsExecutor | failedBlockAdditions=0, totalSize=2.00 GB, freeSize=2.00 GB, usedSize=1.60 MB, cacheSize=1.55 MB, accesses=2103762905, hits=59423, IOhitsPerSecond=0, IOTimePerHit=NaN, hitRatio=0.00%, cachingAccesses=290936, cachingHits=946, cachingHitsRatio=0.33%, evictions=49, evicted=289802, evictedPerRun=5914.326530612245 | org.apache.hadoop.hbase.io.hfile.bucket.BucketCache.logStats(BucketCache.java:637) 2024-04-20 23:59:38,469 | INFO | p1tdhdpdn-v-7:21302Replication Statistics #0 | Global stats: WAL Edits Buffer Used=0B, Limit=268435456B | org.apache.hadoop.hbase.replication.regionserver.Replication$ReplicationStatisticsTask.printStats(Replication.java:241) 2024-04-21 00:00:15,428 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-16 | Close 8975d95b7a69dc45b0055630545c2e25 | org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:107) 2024-04-21 00:00:15,428 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-16 | Unable to get remote Address | org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214) 2024-04-21 00:00:15,488 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-16 | Unable to get remote Address | org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214) 2024-04-21 00:00:15,488 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-16 | Waiting for flushes and compactions to finish for the region boss20240221,H5CC9923038~,1708358595748.8975d95b7a69dc45b0055630545c2e25. | org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1924) 2024-04-21 00:00:15,488 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-16 | Total wait time for flushes and compaction for the region boss20240221,H5CC9923038~,1708358595748.8975d95b7a69dc45b0055630545c2e25. is: 0ms | org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1960) 2024-04-21 00:00:15,488 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-16 | Closing region boss20240221,H5CC9923038~,1708358595748.8975d95b7a69dc45b0055630545c2e25. | org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1696) 2024-04-21 00:00:15,839 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-2 | Close 0d541b4f0bf68727cfd0f2dfffc4dde5 | org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:107) 2024-04-21 00:00:15,840 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-2 | Unable to get remote Address | org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214) 2024-04-21 00:00:15,840 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-2 | Unable to get remote Address | org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214) 2024-04-21 00:00:15,840 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-2 | Waiting for flushes and compactions to finish for the region boss20240221,H5CC9915320~,1708358595748.0d541b4f0bf68727cfd0f2dfffc4dde5. | org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1924) 2024-04-21 00:00:15,840 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-2 | Total wait time for flushes and compaction for the region boss20240221,H5CC9915320~,1708358595748.0d541b4f0bf68727cfd0f2dfffc4dde5. is: 0ms | org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1960) 2024-04-21 00:00:15,840 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-2 | Closing region boss20240221,H5CC9915320~,1708358595748.0d541b4f0bf68727cfd0f2dfffc4dde5. | org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1696) 2024-04-21 00:00:16,349 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-42 | Close b8083f48bb2d591ca5aff2931d6e1ff7 | org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:107) 2024-04-21 00:00:16,349 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-42 | Unable to get remote Address | org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214) 2024-04-21 00:00:16,349 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-42 | Unable to get remote Address | org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214) 2024-04-21 00:00:16,349 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-42 | Waiting for flushes and compactions to finish for the region boss20240221,H5CC9907525~,1708358595748.b8083f48bb2d591ca5aff2931d6e1ff7. | org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1924) 2024-04-21 00:00:16,349 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-42 | Total wait time for flushes and compaction for the region boss20240221,H5CC9907525~,1708358595748.b8083f48bb2d591ca5aff2931d6e1ff7. is: 0ms | org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1960) 2024-04-21 00:00:16,349 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-42 | Closing region boss20240221,H5CC9907525~,1708358595748.b8083f48bb2d591ca5aff2931d6e1ff7. | org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1696) 2024-04-21 00:00:16,631 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-16 | Closed boss20240221,H5CC9923038~,1708358595748.8975d95b7a69dc45b0055630545c2e25. | org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1913) 2024-04-21 00:00:16,820 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-47 | Close 00c03d83783ca21c7480b26b0f7b099e | org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:107) 2024-04-21 00:00:16,820 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-47 | Unable to get remote Address | org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214) 2024-04-21 00:00:16,820 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-47 | Unable to get remote Address | org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214) 2024-04-21 00:00:16,820 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-47 | Waiting for flushes and compactions to finish for the region boss20240221,H5309912782~,1708358595748.00c03d83783ca21c7480b26b0f7b099e. | org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1924) 2024-04-21 00:00:16,820 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-47 | Total wait time for flushes and compaction for the region boss20240221,H5309912782~,1708358595748.00c03d83783ca21c7480b26b0f7b099e. is: 0ms | org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1960) 2024-04-21 00:00:16,820 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-47 | Closing region boss20240221,H5309912782~,1708358595748.00c03d83783ca21c7480b26b0f7b099e. | org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1696) 2024-04-21 00:00:17,018 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-0 | Close 2b67a4ae42d57326ff9cf11a222b33ab | org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:107) 2024-04-21 00:00:17,018 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-0 | Unable to get remote Address | org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214) 2024-04-21 00:00:17,018 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-0 | Unable to get remote Address | org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.getRemoteAddress(RangerAuthorizationCoprocessor.java:214) 2024-04-21 00:00:17,026 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-0 | Waiting for flushes and compactions to finish for the region boss20240221,H52R9907~,1708515296800.2b67a4ae42d57326ff9cf11a222b33ab. | org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1924) 2024-04-21 00:00:17,027 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-0 | Total wait time for flushes and compaction for the region boss20240221,H52R9907~,1708515296800.2b67a4ae42d57326ff9cf11a222b33ab. is: 0ms | org.apache.hadoop.hbase.regionserver.HRegion.waitForFlushesAndCompactions(HRegion.java:1960) 2024-04-21 00:00:17,027 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-0 | Closing region boss20240221,H52R9907~,1708515296800.2b67a4ae42d57326ff9cf11a222b33ab. | org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1696) 2024-04-21 00:00:17,189 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-16 | Closed 8975d95b7a69dc45b0055630545c2e25 | org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:147) 2024-04-21 00:00:17,447 | INFO | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-2 | Closed boss20240221,H5CC9915320~,1708358595748.0d541b4f0bf68727cfd0f2dfffc4dde5. | org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1913) 2024-04-21 00:00:17,603 | WARN | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-0 | Fatal error occurred while closing region 2b67a4ae42d57326ff9cf11a222b33ab, aborting... | org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.handleException(UnassignRegionHandler.java:152) java.io.IOException: The new max sequence id 328 is less than the old max sequence id 330 at org.apache.hadoop.hbase.wal.WALSplitUtil.writeRegionSequenceIdFile(WALSplitUtil.java:461) at org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1265) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1893) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1625) at org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:116) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:106) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2024-04-21 00:00:17,610 | ERROR | RS_CLOSE_REGION-regionserver/p1tdhdpdn-v-7:21302-0 | ***** ABORTING region server p1tdhdpdn-v-7,21302,1704995671636: Failed to close region 2b67a4ae42d57326ff9cf11a222b33ab and can not recover ***** | org.slf4j.helpers.MarkerIgnoringBase.error(MarkerIgnoringBase.java:159) java.io.IOException: The new max sequence id 328 is less than the old max sequence id 330 at org.apache.hadoop.hbase.wal.WALSplitUtil.writeRegionSequenceIdFile(WALSplitUtil.java:461) at org.apache.hadoop.hbase.regionserver.HRegion.writeRegionCloseMarker(HRegion.java:1265) at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1893) at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1625) at org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:116) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:106) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.20.10#820010)