[ https://issues.apache.org/jira/browse/PHOENIX-6274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17252786#comment-17252786 ]
Viraj Jasani commented on PHOENIX-6274: --------------------------------------- Not relevant to TableSnapshotReadsMapReduceIT: UpsertSelectIT failed 3 times on different test results on this Jira, but multibranch results don't show any failure for the past 10 runs, interesting. > Flaky test TableSnapshotReadsMapReduceIT.testMapReduceSnapshotsMultiRegion > -------------------------------------------------------------------------- > > Key: PHOENIX-6274 > URL: https://issues.apache.org/jira/browse/PHOENIX-6274 > Project: Phoenix > Issue Type: Bug > Affects Versions: 5.0.0, 4.15.0 > Reporter: Viraj Jasani > Assignee: Viraj Jasani > Priority: Major > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-6274.4.x.000.patch, PHOENIX-6274.4.x.001.patch, > PHOENIX-6274.master.000.patch, PHOENIX-6274.master.001.patch > > > Test TableSnapshotReadsMapReduceIT.testMapReduceSnapshotsMultiRegion > sometimes fails to delete snapshot due to snapshot corruption. > {code:java} > ERROR [MASTER_SNAPSHOT_OPERATIONS-master/e09b7a102100:0-0] > org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier(187): Regions > moved during the snapshot '{ ss=FOO table=N000006 type=FLUSH }'. expected=2 > snapshotted=3.ERROR [MASTER_SNAPSHOT_OPERATIONS-master/e09b7a102100:0-0] > org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier(187): Regions > moved during the snapshot '{ ss=FOO table=N000006 type=FLUSH }'. expected=2 > snapshotted=3.2020-12-18 01:18:45,532 ERROR > [MASTER_SNAPSHOT_OPERATIONS-master/e09b7a102100:0-0] > org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler(222): Failed > taking snapshot { ss=FOO table=N000006 type=FLUSH } due to exception:Regions > moved during the snapshot '{ ss=FOO table=N000006 type=FLUSH }'. expected=2 > snapshotted=3.org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: > Regions moved during the snapshot '{ ss=FOO table=N000006 type=FLUSH }'. > expected=2 snapshotted=3. at > org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifyRegions(MasterSnapshotVerifier.java:205) > at > org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.verifySnapshot(MasterSnapshotVerifier.java:119) > at > org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.process(TakeSnapshotHandler.java:209) > at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > Due to this, deletion of snapshot fails: > {code:java} > org.apache.hadoop.hbase.snapshot.SnapshotDoesNotExistException: Snapshot > 'FOO' doesn't exist on the filesystem > at > org.apache.hadoop.hbase.master.snapshot.SnapshotManager.deleteSnapshot(SnapshotManager.java:312) > at > org.apache.hadoop.hbase.master.MasterRpcServices.deleteSnapshot(MasterRpcServices.java:694) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) > at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) > {code} > Moreover, since the test is all about dealing with multi regions, which are > generated by splitting, as per test logs, apparently table is not getting > split correctly: > {code:java} > org.apache.hadoop.hbase.regionserver.StoreUtils(123): cannot split > hdfs://localhost:37149/user/jenkins/test-data/b2a954c8-abb9-5528-bca3-27fcaaab735c/data/default/N000006/903675aa1a12e58bb247916dd7d492dd/0/b4bdabddce254ea5a9fa3ca432e2bca3 > because midkey is the same as first or last row > {code} > e.g > https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-mulitbranch/job/master/157/testReport/org.apache.phoenix.end2end/TableSnapshotReadsMapReduceIT/MatrixBuild___Matrix___HBASE_PROFILE____2_1____BuildAndTest___testMapReduceSnapshotsMultiRegion/ -- This message was sent by Atlassian Jira (v8.3.4#803005)