[
https://issues.apache.org/jira/browse/PHOENIX-6274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Viraj Jasani updated PHOENIX-6274:
----------------------------------
Description:
Test TableSnapshotReadsMapReduceIT.testMapReduceSnapshotsMultiRegion sometimes
fails to delete snapshot due to snapshot corruption.
{code:java}
ERROR [MASTER_SNAPSHOT_OPERATIONS-master/e09b7a102100:0-0]
org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier(187): Regions
moved during the snapshot '{ ss=FOO table=N000006 type=FLUSH }'. expected=2
snapshotted=3.ERROR [MASTER_SNAPSHOT_OPERATIONS-master/e09b7a102100:0-0]
org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier(187): Regions
moved during the snapshot '{ ss=FOO table=N000006 type=FLUSH }'. expected=2
snapshotted=3.2020-12-18 01:18:45,532 ERROR
[MASTER_SNAPSHOT_OPERATIONS-master/e09b7a102100:0-0]
org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler(222): Failed taking
snapshot { ss=FOO table=N000006 type=FLUSH } due to exception:Regions moved
during the snapshot '{ ss=FOO table=N000006 type=FLUSH }'. expected=2
snapshotted=3.org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException:
Regions moved during the snapshot '{ ss=FOO table=N000006 type=FLUSH }'.
expected=2 snapshotted=3. at
org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifyRegions(MasterSnapshotVerifier.java:205)
at
org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifySnapshot(MasterSnapshotVerifier.java:119)
at
org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.process(TakeSnapshotHandler.java:209)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
Due to this, deletion of snapshot fails:
{code:java}
org.apache.hadoop.hbase.snapshot.SnapshotDoesNotExistException: Snapshot 'FOO'
doesn't exist on the filesystem
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.deleteSnapshot(SnapshotManager.java:312)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteSnapshot(MasterRpcServices.java:694)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
{code}
Moreover, since the test is all about dealing with multi regions, which are
generated by splitting, as per test logs, apparently table is not getting split
correctly:
{code:java}
org.apache.hadoop.hbase.regionserver.StoreUtils(123): cannot split
hdfs://localhost:37149/user/jenkins/test-data/b2a954c8-abb9-5528-bca3-27fcaaab735c/data/default/N000006/903675aa1a12e58bb247916dd7d492dd/0/b4bdabddce254ea5a9fa3ca432e2bca3
because midkey is the same as first or last row
{code}
e.g
https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-mulitbranch/job/master/157/testReport/org.apache.phoenix.end2end/TableSnapshotReadsMapReduceIT/MatrixBuild___Matrix___HBASE_PROFILE____2_1____BuildAndTest___testMapReduceSnapshotsMultiRegion/
was:
Test TableSnapshotReadsMapReduceIT.testMapReduceSnapshotsMultiRegion sometimes
fails to delete snapshot due to snapshot corruption.
{code:java}
ERROR [MASTER_SNAPSHOT_OPERATIONS-master/e09b7a102100:0-0]
org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier(187): Regions
moved during the snapshot '{ ss=FOO table=N000006 type=FLUSH }'. expected=2
snapshotted=3.ERROR [MASTER_SNAPSHOT_OPERATIONS-master/e09b7a102100:0-0]
org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier(187): Regions
moved during the snapshot '{ ss=FOO table=N000006 type=FLUSH }'. expected=2
snapshotted=3.2020-12-18 01:18:45,532 ERROR
[MASTER_SNAPSHOT_OPERATIONS-master/e09b7a102100:0-0]
org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler(222): Failed taking
snapshot { ss=FOO table=N000006 type=FLUSH } due to exception:Regions moved
during the snapshot '{ ss=FOO table=N000006 type=FLUSH }'. expected=2
snapshotted=3.org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException:
Regions moved during the snapshot '{ ss=FOO table=N000006 type=FLUSH }'.
expected=2 snapshotted=3. at
org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifyRegions(MasterSnapshotVerifier.java:205)
at
org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifySnapshot(MasterSnapshotVerifier.java:119)
at
org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.process(TakeSnapshotHandler.java:209)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
Due to this, deletion of snapshot fails:
{code:java}
org.apache.hadoop.hbase.snapshot.SnapshotDoesNotExistException: Snapshot 'FOO'
doesn't exist on the filesystem
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.deleteSnapshot(SnapshotManager.java:312)
at
org.apache.hadoop.hbase.master.MasterRpcServices.deleteSnapshot(MasterRpcServices.java:694)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
{code}
Moreover, since the test is all about dealing with multi regions, which are
generated by splitting, as per test logs, apparently table is not getting split
correctly:
{code:java}
org.apache.hadoop.hbase.regionserver.StoreUtils(123): cannot split
hdfs://localhost:37149/user/jenkins/test-data/b2a954c8-abb9-5528-bca3-27fcaaab735c/data/default/N000006/903675aa1a12e58bb247916dd7d492dd/0/b4bdabddce254ea5a9fa3ca432e2bca3
because midkey is the same as first or last row
{code}
> Flaky test TableSnapshotReadsMapReduceIT.testMapReduceSnapshotsMultiRegion
> --------------------------------------------------------------------------
>
> Key: PHOENIX-6274
> URL: https://issues.apache.org/jira/browse/PHOENIX-6274
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 5.0.0, 4.15.0
> Reporter: Viraj Jasani
> Assignee: Viraj Jasani
> Priority: Major
> Fix For: 5.1.0, 4.16.0
>
> Attachments: PHOENIX-6274.4.x.000.patch, PHOENIX-6274.4.x.001.patch,
> PHOENIX-6274.master.000.patch, PHOENIX-6274.master.001.patch
>
>
> Test TableSnapshotReadsMapReduceIT.testMapReduceSnapshotsMultiRegion
> sometimes fails to delete snapshot due to snapshot corruption.
> {code:java}
> ERROR [MASTER_SNAPSHOT_OPERATIONS-master/e09b7a102100:0-0]
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier(187): Regions
> moved during the snapshot '{ ss=FOO table=N000006 type=FLUSH }'. expected=2
> snapshotted=3.ERROR [MASTER_SNAPSHOT_OPERATIONS-master/e09b7a102100:0-0]
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier(187): Regions
> moved during the snapshot '{ ss=FOO table=N000006 type=FLUSH }'. expected=2
> snapshotted=3.2020-12-18 01:18:45,532 ERROR
> [MASTER_SNAPSHOT_OPERATIONS-master/e09b7a102100:0-0]
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler(222): Failed
> taking snapshot { ss=FOO table=N000006 type=FLUSH } due to exception:Regions
> moved during the snapshot '{ ss=FOO table=N000006 type=FLUSH }'. expected=2
> snapshotted=3.org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException:
> Regions moved during the snapshot '{ ss=FOO table=N000006 type=FLUSH }'.
> expected=2 snapshotted=3. at
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifyRegions(MasterSnapshotVerifier.java:205)
> at
> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifySnapshot(MasterSnapshotVerifier.java:119)
> at
> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.process(TakeSnapshotHandler.java:209)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> Due to this, deletion of snapshot fails:
> {code:java}
> org.apache.hadoop.hbase.snapshot.SnapshotDoesNotExistException: Snapshot
> 'FOO' doesn't exist on the filesystem
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.deleteSnapshot(SnapshotManager.java:312)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.deleteSnapshot(MasterRpcServices.java:694)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> {code}
> Moreover, since the test is all about dealing with multi regions, which are
> generated by splitting, as per test logs, apparently table is not getting
> split correctly:
> {code:java}
> org.apache.hadoop.hbase.regionserver.StoreUtils(123): cannot split
> hdfs://localhost:37149/user/jenkins/test-data/b2a954c8-abb9-5528-bca3-27fcaaab735c/data/default/N000006/903675aa1a12e58bb247916dd7d492dd/0/b4bdabddce254ea5a9fa3ca432e2bca3
> because midkey is the same as first or last row
> {code}
> e.g
> https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-mulitbranch/job/master/157/testReport/org.apache.phoenix.end2end/TableSnapshotReadsMapReduceIT/MatrixBuild___Matrix___HBASE_PROFILE____2_1____BuildAndTest___testMapReduceSnapshotsMultiRegion/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
