[jira] [Assigned] (PHOENIX-6169) IT suite never finishes on 4.x with HBase 1.3 or 1.4
[
https://issues.apache.org/jira/browse/PHOENIX-6169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Istvan Toth reassigned PHOENIX-6169:
Assignee: Istvan Toth
> IT suite never finishes on 4.x with HBase 1.3 or 1.4
>
>
> Key: PHOENIX-6169
> URL: https://issues.apache.org/jira/browse/PHOENIX-6169
> Project: Phoenix
> Issue Type: Bug
> Components: core
>Affects Versions: 4.16.0
> Environment: ASF Jenkins (at least)
>Reporter: Istvan Toth
>Assignee: Istvan Toth
>Priority: Blocker
>
> running {{mvn verify}} on the current 4.x branch will hang indefinetly.
> Apart from making it impossible to run all tests, and get a successful test
> run, this also results in Yetus not being able to post precommit check
> results.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (PHOENIX-6169) IT suite never finishes on 4.x with HBase 1.3 or 1.4
Istvan Toth created PHOENIX-6169:
Summary: IT suite never finishes on 4.x with HBase 1.3 or 1.4
Key: PHOENIX-6169
URL: https://issues.apache.org/jira/browse/PHOENIX-6169
Project: Phoenix
Issue Type: Bug
Components: core
Affects Versions: 4.16.0
Environment: ASF Jenkins (at least)
Reporter: Istvan Toth
running {{mvn verify}} on the current 4.x branch will hang indefinetly.
Apart from making it impossible to run all tests, and get a successful test
run, this also results in Yetus not being able to post precommit check results.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (PHOENIX-6167) Adding maxMutationCellSizeBytes config and exception
[ https://issues.apache.org/jira/browse/PHOENIX-6167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinyi Yan updated PHOENIX-6167: --- Attachment: PHOENIX-6167.master.v2.patch > Adding maxMutationCellSizeBytes config and exception > > > Key: PHOENIX-6167 > URL: https://issues.apache.org/jira/browse/PHOENIX-6167 > Project: Phoenix > Issue Type: Improvement >Reporter: Xinyi Yan >Assignee: Xinyi Yan >Priority: Trivial > Fix For: 4.16.0 > > Attachments: PHOENIX-6167.4.x.patch, PHOENIX-6167.master.v2.patch, > PHOENIX-6167.patch > > > HBase can handle cells that are up to 10 MB, and we should have a better > error handling in Phoenix. Adding extra config so that we have better control > of column size, as well as helping us to understand which column exceeds the > max allowance. > This Jira will introduce config for maxMutationCellSizeBytes and throw an > exception if the column size is greater than the config allowance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-6159) Phoenix-pherf writes the result file even disableRuntimeResult flag is true
[ https://issues.apache.org/jira/browse/PHOENIX-6159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinyi Yan updated PHOENIX-6159: --- Attachment: PHOENIX-6159.patch > Phoenix-pherf writes the result file even disableRuntimeResult flag is true > --- > > Key: PHOENIX-6159 > URL: https://issues.apache.org/jira/browse/PHOENIX-6159 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.0 >Reporter: Xinyi Yan >Assignee: Xinyi Yan >Priority: Major > Fix For: 4.16.0 > > Attachments: PHOENIX-6159.patch > > > The ([MultiThreadedRunner|#L109])] writes the result to the file without > check the writeRuntimeResults value. > I'm not sure if anyone is using this [disableRuntimeResult param in > Pherf.java|https://github.com/apache/phoenix/blob/master/phoenix-pherf/src/main/java/org/apache/phoenix/pherf/Pherf.java#L156], > which is not presenting what it indicates. I prefer to fix the behavior and > match the meaning of disableRuntimeResult. > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (PHOENIX-6159) Phoenix-pherf writes the result file even disableRuntimeResult flag is true
[ https://issues.apache.org/jira/browse/PHOENIX-6159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinyi Yan reassigned PHOENIX-6159: -- Fix Version/s: 4.16.0 Assignee: Xinyi Yan > Phoenix-pherf writes the result file even disableRuntimeResult flag is true > --- > > Key: PHOENIX-6159 > URL: https://issues.apache.org/jira/browse/PHOENIX-6159 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.0 >Reporter: Xinyi Yan >Assignee: Xinyi Yan >Priority: Major > Fix For: 4.16.0 > > > The ([MultiThreadedRunner|#L109])] writes the result to the file without > check the writeRuntimeResults value. > I'm not sure if anyone is using this [disableRuntimeResult param in > Pherf.java|https://github.com/apache/phoenix/blob/master/phoenix-pherf/src/main/java/org/apache/phoenix/pherf/Pherf.java#L156], > which is not presenting what it indicates. I prefer to fix the behavior and > match the meaning of disableRuntimeResult. > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-6159) Phoenix-pherf writes the result file even disableRuntimeResult flag is true
[ https://issues.apache.org/jira/browse/PHOENIX-6159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinyi Yan updated PHOENIX-6159: --- Attachment: (was: PHOENIX-6159.patch) > Phoenix-pherf writes the result file even disableRuntimeResult flag is true > --- > > Key: PHOENIX-6159 > URL: https://issues.apache.org/jira/browse/PHOENIX-6159 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.0 >Reporter: Xinyi Yan >Assignee: Xinyi Yan >Priority: Major > Fix For: 4.16.0 > > > The ([MultiThreadedRunner|#L109])] writes the result to the file without > check the writeRuntimeResults value. > I'm not sure if anyone is using this [disableRuntimeResult param in > Pherf.java|https://github.com/apache/phoenix/blob/master/phoenix-pherf/src/main/java/org/apache/phoenix/pherf/Pherf.java#L156], > which is not presenting what it indicates. I prefer to fix the behavior and > match the meaning of disableRuntimeResult. > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-6167) Adding maxMutationCellSizeBytes config and exception
[ https://issues.apache.org/jira/browse/PHOENIX-6167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinyi Yan updated PHOENIX-6167: --- Attachment: PHOENIX-6167.4.x.patch > Adding maxMutationCellSizeBytes config and exception > > > Key: PHOENIX-6167 > URL: https://issues.apache.org/jira/browse/PHOENIX-6167 > Project: Phoenix > Issue Type: Improvement >Reporter: Xinyi Yan >Assignee: Xinyi Yan >Priority: Trivial > Fix For: 4.16.0 > > Attachments: PHOENIX-6167.4.x.patch, PHOENIX-6167.patch > > > HBase can handle cells that are up to 10 MB, and we should have a better > error handling in Phoenix. Adding extra config so that we have better control > of column size, as well as helping us to understand which column exceeds the > max allowance. > This Jira will introduce config for maxMutationCellSizeBytes and throw an > exception if the column size is greater than the config allowance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-6167) Adding maxMutationCellSizeBytes config and exception
[ https://issues.apache.org/jira/browse/PHOENIX-6167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinyi Yan updated PHOENIX-6167: --- Attachment: PHOENIX-6167.patch > Adding maxMutationCellSizeBytes config and exception > > > Key: PHOENIX-6167 > URL: https://issues.apache.org/jira/browse/PHOENIX-6167 > Project: Phoenix > Issue Type: Improvement >Reporter: Xinyi Yan >Assignee: Xinyi Yan >Priority: Trivial > Fix For: 4.16.0 > > Attachments: PHOENIX-6167.patch > > > HBase can handle cells that are up to 10 MB, and we should have a better > error handling in Phoenix. Adding extra config so that we have better control > of column size, as well as helping us to understand which column exceeds the > max allowance. > This Jira will introduce config for maxMutationCellSizeBytes and throw an > exception if the column size is greater than the config allowance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (PHOENIX-6167) Adding maxMutationCellSizeBytes config and exception
[ https://issues.apache.org/jira/browse/PHOENIX-6167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinyi Yan reassigned PHOENIX-6167: -- Fix Version/s: 4.16.0 Assignee: Xinyi Yan > Adding maxMutationCellSizeBytes config and exception > > > Key: PHOENIX-6167 > URL: https://issues.apache.org/jira/browse/PHOENIX-6167 > Project: Phoenix > Issue Type: Improvement >Reporter: Xinyi Yan >Assignee: Xinyi Yan >Priority: Trivial > Fix For: 4.16.0 > > Attachments: PHOENIX-6167.patch > > > HBase can handle cells that are up to 10 MB, and we should have a better > error handling in Phoenix. Adding extra config so that we have better control > of column size, as well as helping us to understand which column exceeds the > max allowance. > This Jira will introduce config for maxMutationCellSizeBytes and throw an > exception if the column size is greater than the config allowance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-6142) Make DDL operations resilient to orphan parent->child linking rows in SYSTEM.CHILD_LINK
[ https://issues.apache.org/jira/browse/PHOENIX-6142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni updated PHOENIX-6142: -- Attachment: PHOENIX-6142.4.x.v2.patch > Make DDL operations resilient to orphan parent->child linking rows in > SYSTEM.CHILD_LINK > --- > > Key: PHOENIX-6142 > URL: https://issues.apache.org/jira/browse/PHOENIX-6142 > Project: Phoenix > Issue Type: Bug >Affects Versions: 5.0.0, 4.15.0 >Reporter: Chinmay Kulkarni >Assignee: Chinmay Kulkarni >Priority: Blocker > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-6142-4.x-v1.patch, PHOENIX-6142.4.x.v2.patch > > > We are targeting PHOENIX-6141 for 4.17. Until we have it, we should aim at > making DDL operations resilient to orphan parent->child linking rows. DDL > operations identified which can fail due to orphan rows are: > # Any ALTER TABLE ADD/DROP/SET calls on the base table T will fail if there > are orphan links from T to some already dropped view. This happens because > the call to > [MetaDataEndpointImpl.findAllChildViews()|https://github.com/apache/phoenix/blob/fece8e69b9c03c80db7a0801d99e5de31fe15ffa/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java#L2142] > from > [MetaDataEndpointImpl.mutateColumn()|https://github.com/apache/phoenix/blob/fece8e69b9c03c80db7a0801d99e5de31fe15ffa/phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java#L259] > fails with a TableNotFoundException. > # Any DROP TABLE/VIEW call without CASCADE will fail even though there are > actually no child views since the orphan rows wrongly indicate that there are > child views. > # During the upgrade path for UpgradeUtil.syncUpdateCacheFreqAllIndexes(), > we will just ignore any orphan views (for ex, see > [this|https://github.com/apache/phoenix/blob/fece8e69b9c03c80db7a0801d99e5de31fe15ffa/phoenix-core/src/main/java/org/apache/phoenix/util/UpgradeUtil.java#L1368-L1374]), > but the call to UpgradeUtil.upgradeTable() will fail with a > TableNotFoundException for each orphan view. > # During a CREATE TABLE/VIEW, we try to drop any views from the previous life > of that table/view, however we might end up dropping a legitimate view (with > the same name) which is on another table/view because of this. > Before dropping any views that we see from a parent->child link, we need to > ensure that the view is in fact a child view of the same table/view we think > it is an orphan of. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-6158) create table/view should not update VIEW_INDEX_ID_DATA_TYPE column
[ https://issues.apache.org/jira/browse/PHOENIX-6158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinyi Yan updated PHOENIX-6158: --- Attachment: PHOENIX-6158.patch > create table/view should not update VIEW_INDEX_ID_DATA_TYPE column > -- > > Key: PHOENIX-6158 > URL: https://issues.apache.org/jira/browse/PHOENIX-6158 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.0 >Reporter: Xinyi Yan >Assignee: Xinyi Yan >Priority: Major > Fix For: 4.16.0 > > Attachments: PHOENIX-6158.patch, Screen Shot 2020-09-24 at 6.13.58 > PM.png > > > The VIEW_INDEX_ID_DATA_TYPE column should only give value when we create a > view index, which is presenting data type(BIGINT VS SMALLINT) with the > VIEW_INDEX_ID column. > !Screen Shot 2020-09-24 at 6.13.58 PM.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (PHOENIX-6158) create table/view should not update VIEW_INDEX_ID_DATA_TYPE column
[ https://issues.apache.org/jira/browse/PHOENIX-6158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xinyi Yan reassigned PHOENIX-6158: -- Assignee: Xinyi Yan > create table/view should not update VIEW_INDEX_ID_DATA_TYPE column > -- > > Key: PHOENIX-6158 > URL: https://issues.apache.org/jira/browse/PHOENIX-6158 > Project: Phoenix > Issue Type: Bug >Affects Versions: 4.15.0 >Reporter: Xinyi Yan >Assignee: Xinyi Yan >Priority: Major > Fix For: 4.16.0 > > Attachments: Screen Shot 2020-09-24 at 6.13.58 PM.png > > > The VIEW_INDEX_ID_DATA_TYPE column should only give value when we create a > view index, which is presenting data type(BIGINT VS SMALLINT) with the > VIEW_INDEX_ID column. > !Screen Shot 2020-09-24 at 6.13.58 PM.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[DISCUSS] Releasing Phoenix-thirdparty 1.0
Hi! We have been using phoenix-thirdparty snapshots in omid for a few weeks, and in phoenix (master) for about a week, and so far we haven't found any issues stemming from using Phoenix-thirdparty (effectively thirdparty Guava, the only current component). I think that we are ready to release Phoenix-thirdparty 1.0 now. My current plan is to cut an rc0 sometime next week, unless problems arise, or someone voices an objection. Looking forward to hearing your opinion and feedback. regards Istvan
[jira] [Updated] (PHOENIX-6153) Table Map Reduce job after a Snapshot based job fails with CorruptedSnapshotException
[ https://issues.apache.org/jira/browse/PHOENIX-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saksham Gangwar updated PHOENIX-6153: - Attachment: Screen Shot 2020-09-30 at 9.30.06 AM.png > Table Map Reduce job after a Snapshot based job fails with > CorruptedSnapshotException > - > > Key: PHOENIX-6153 > URL: https://issues.apache.org/jira/browse/PHOENIX-6153 > Project: Phoenix > Issue Type: Bug > Components: core >Affects Versions: 4.15.0, 4.14.3, master >Reporter: Saksham Gangwar >Assignee: Saksham Gangwar >Priority: Major > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-6153.4.x.v1.patch, PHOENIX-6153.master.v1.patch, > PHOENIX-6153.master.v2.patch, PHOENIX-6153.master.v3.patch, > PHOENIX-6153.master.v4.patch, PHOENIX-6153.master.v5.patch, Screen Shot > 2020-09-30 at 4.00.58 AM.png, Screen Shot 2020-09-30 at 4.01.10 AM.png, > Screen Shot 2020-09-30 at 4.01.10 AM.png, Screen Shot 2020-09-30 at 4.01.19 > AM.png, Screen Shot 2020-09-30 at 4.01.19 AM.png, Screen Shot 2020-09-30 at > 4.01.19 AM.png, Screen Shot 2020-09-30 at 4.01.34 AM.png, Screen Shot > 2020-09-30 at 4.01.52 AM.png, Screen Shot 2020-09-30 at 4.01.52 AM.png, > Screen Shot 2020-09-30 at 9.30.06 AM.png > > > Different MR job requests which reach [MapReduceParallelScanGrouper > getRegionBoundaries|https://github.com/apache/phoenix/blob/f9e304754bad886344a856dd2565e3f24e345ed2/phoenix-core/src/main/java/org/apache/phoenix/iterate/MapReduceParallelScanGrouper.java#L65] > we currently make use of shared configuration among jobs to figure out > snapshot names. > Example jobs' sequence: first two jobs work over snapshot and the third job > over a regular table. > Prininting hashcode of objects when entering: > [https://github.com/apache/phoenix/blob/f9e304754bad886344a856dd2565e3f24e345ed2/phoenix-core/src/main/java/org/apache/phoenix/iterate/MapReduceParallelScanGrouper.java#L65] > *Job 1:* (over snapshot of *ABC_TABLE_1* and is successful) > context.getConnection(): 521093916 > ConnectionQueryServices: 1772519705 > *Configuration conf: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY):*ABC_TABLE_1* > > *Job 2:* (over snapshot of *ABC_TABLE_2* and is successful) > context.getConnection(): 1928017473 > ConnectionQueryServices: 961279422 > *Configuration conf: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY): *ABC_TABLE_2* > > *Job 3:* (over the table *ABC_TABLE_3* but fails with > CorruptedSnapshotException while it got nothing to do with snapshot) > context.getConnection(): 28889670 > ConnectionQueryServices: 424389847 > *Configuration: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY): *ABC_TABLE_2* > > Exception which we get: > [2020:08:18 20:56:17.409] [MigrationRetryPoller-Executor-1] [ERROR] > [c.s.hgrate.mapreduce.MapReduceImpl] - Error submitting M/R job for Job 3 > java.lang.RuntimeException: > org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read > snapshot info > from:hdfs://.../hbase/.hbase-snapshot/ABC_TABLE_2_1597687413477/.snapshotinfo > at > org.apache.phoenix.iterate.MapReduceParallelScanGrouper.getRegionBoundaries(MapReduceParallelScanGrouper.java:81) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getRegionBoundaries(BaseResultIterators.java:541) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getParallelScans(BaseResultIterators.java:893) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getParallelScans(BaseResultIterators.java:641) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.(BaseResultIterators.java:511) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.ParallelIterators.(ParallelIterators.java:62) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.ScanPlan.newIterator(ScanPlan.java:278) > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:367) > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:218) >
[jira] [Updated] (PHOENIX-6153) Table Map Reduce job after a Snapshot based job fails with CorruptedSnapshotException
[ https://issues.apache.org/jira/browse/PHOENIX-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saksham Gangwar updated PHOENIX-6153: - Attachment: Screen Shot 2020-09-30 at 4.01.52 AM.png > Table Map Reduce job after a Snapshot based job fails with > CorruptedSnapshotException > - > > Key: PHOENIX-6153 > URL: https://issues.apache.org/jira/browse/PHOENIX-6153 > Project: Phoenix > Issue Type: Bug > Components: core >Affects Versions: 4.15.0, 4.14.3, master >Reporter: Saksham Gangwar >Assignee: Saksham Gangwar >Priority: Major > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-6153.4.x.v1.patch, PHOENIX-6153.master.v1.patch, > PHOENIX-6153.master.v2.patch, PHOENIX-6153.master.v3.patch, > PHOENIX-6153.master.v4.patch, PHOENIX-6153.master.v5.patch, Screen Shot > 2020-09-30 at 4.00.58 AM.png, Screen Shot 2020-09-30 at 4.01.10 AM.png, > Screen Shot 2020-09-30 at 4.01.10 AM.png, Screen Shot 2020-09-30 at 4.01.19 > AM.png, Screen Shot 2020-09-30 at 4.01.19 AM.png, Screen Shot 2020-09-30 at > 4.01.19 AM.png, Screen Shot 2020-09-30 at 4.01.34 AM.png, Screen Shot > 2020-09-30 at 4.01.52 AM.png, Screen Shot 2020-09-30 at 4.01.52 AM.png > > > Different MR job requests which reach [MapReduceParallelScanGrouper > getRegionBoundaries|https://github.com/apache/phoenix/blob/f9e304754bad886344a856dd2565e3f24e345ed2/phoenix-core/src/main/java/org/apache/phoenix/iterate/MapReduceParallelScanGrouper.java#L65] > we currently make use of shared configuration among jobs to figure out > snapshot names. > Example jobs' sequence: first two jobs work over snapshot and the third job > over a regular table. > Prininting hashcode of objects when entering: > [https://github.com/apache/phoenix/blob/f9e304754bad886344a856dd2565e3f24e345ed2/phoenix-core/src/main/java/org/apache/phoenix/iterate/MapReduceParallelScanGrouper.java#L65] > *Job 1:* (over snapshot of *ABC_TABLE_1* and is successful) > context.getConnection(): 521093916 > ConnectionQueryServices: 1772519705 > *Configuration conf: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY):*ABC_TABLE_1* > > *Job 2:* (over snapshot of *ABC_TABLE_2* and is successful) > context.getConnection(): 1928017473 > ConnectionQueryServices: 961279422 > *Configuration conf: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY): *ABC_TABLE_2* > > *Job 3:* (over the table *ABC_TABLE_3* but fails with > CorruptedSnapshotException while it got nothing to do with snapshot) > context.getConnection(): 28889670 > ConnectionQueryServices: 424389847 > *Configuration: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY): *ABC_TABLE_2* > > Exception which we get: > [2020:08:18 20:56:17.409] [MigrationRetryPoller-Executor-1] [ERROR] > [c.s.hgrate.mapreduce.MapReduceImpl] - Error submitting M/R job for Job 3 > java.lang.RuntimeException: > org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read > snapshot info > from:hdfs://.../hbase/.hbase-snapshot/ABC_TABLE_2_1597687413477/.snapshotinfo > at > org.apache.phoenix.iterate.MapReduceParallelScanGrouper.getRegionBoundaries(MapReduceParallelScanGrouper.java:81) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getRegionBoundaries(BaseResultIterators.java:541) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getParallelScans(BaseResultIterators.java:893) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getParallelScans(BaseResultIterators.java:641) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.(BaseResultIterators.java:511) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.ParallelIterators.(ParallelIterators.java:62) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.ScanPlan.newIterator(ScanPlan.java:278) > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:367) > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:218) >
[jira] [Updated] (PHOENIX-6153) Table Map Reduce job after a Snapshot based job fails with CorruptedSnapshotException
[ https://issues.apache.org/jira/browse/PHOENIX-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saksham Gangwar updated PHOENIX-6153: - Attachment: Screen Shot 2020-09-30 at 4.01.19 AM.png > Table Map Reduce job after a Snapshot based job fails with > CorruptedSnapshotException > - > > Key: PHOENIX-6153 > URL: https://issues.apache.org/jira/browse/PHOENIX-6153 > Project: Phoenix > Issue Type: Bug > Components: core >Affects Versions: 4.15.0, 4.14.3, master >Reporter: Saksham Gangwar >Assignee: Saksham Gangwar >Priority: Major > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-6153.4.x.v1.patch, PHOENIX-6153.master.v1.patch, > PHOENIX-6153.master.v2.patch, PHOENIX-6153.master.v3.patch, > PHOENIX-6153.master.v4.patch, PHOENIX-6153.master.v5.patch, Screen Shot > 2020-09-30 at 4.00.58 AM.png, Screen Shot 2020-09-30 at 4.01.10 AM.png, > Screen Shot 2020-09-30 at 4.01.10 AM.png, Screen Shot 2020-09-30 at 4.01.19 > AM.png, Screen Shot 2020-09-30 at 4.01.19 AM.png, Screen Shot 2020-09-30 at > 4.01.19 AM.png, Screen Shot 2020-09-30 at 4.01.34 AM.png, Screen Shot > 2020-09-30 at 4.01.52 AM.png, Screen Shot 2020-09-30 at 4.01.52 AM.png > > > Different MR job requests which reach [MapReduceParallelScanGrouper > getRegionBoundaries|https://github.com/apache/phoenix/blob/f9e304754bad886344a856dd2565e3f24e345ed2/phoenix-core/src/main/java/org/apache/phoenix/iterate/MapReduceParallelScanGrouper.java#L65] > we currently make use of shared configuration among jobs to figure out > snapshot names. > Example jobs' sequence: first two jobs work over snapshot and the third job > over a regular table. > Prininting hashcode of objects when entering: > [https://github.com/apache/phoenix/blob/f9e304754bad886344a856dd2565e3f24e345ed2/phoenix-core/src/main/java/org/apache/phoenix/iterate/MapReduceParallelScanGrouper.java#L65] > *Job 1:* (over snapshot of *ABC_TABLE_1* and is successful) > context.getConnection(): 521093916 > ConnectionQueryServices: 1772519705 > *Configuration conf: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY):*ABC_TABLE_1* > > *Job 2:* (over snapshot of *ABC_TABLE_2* and is successful) > context.getConnection(): 1928017473 > ConnectionQueryServices: 961279422 > *Configuration conf: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY): *ABC_TABLE_2* > > *Job 3:* (over the table *ABC_TABLE_3* but fails with > CorruptedSnapshotException while it got nothing to do with snapshot) > context.getConnection(): 28889670 > ConnectionQueryServices: 424389847 > *Configuration: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY): *ABC_TABLE_2* > > Exception which we get: > [2020:08:18 20:56:17.409] [MigrationRetryPoller-Executor-1] [ERROR] > [c.s.hgrate.mapreduce.MapReduceImpl] - Error submitting M/R job for Job 3 > java.lang.RuntimeException: > org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read > snapshot info > from:hdfs://.../hbase/.hbase-snapshot/ABC_TABLE_2_1597687413477/.snapshotinfo > at > org.apache.phoenix.iterate.MapReduceParallelScanGrouper.getRegionBoundaries(MapReduceParallelScanGrouper.java:81) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getRegionBoundaries(BaseResultIterators.java:541) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getParallelScans(BaseResultIterators.java:893) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getParallelScans(BaseResultIterators.java:641) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.(BaseResultIterators.java:511) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.ParallelIterators.(ParallelIterators.java:62) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.ScanPlan.newIterator(ScanPlan.java:278) > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:367) > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:218) >
[jira] [Updated] (PHOENIX-6153) Table Map Reduce job after a Snapshot based job fails with CorruptedSnapshotException
[ https://issues.apache.org/jira/browse/PHOENIX-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saksham Gangwar updated PHOENIX-6153: - Attachment: Screen Shot 2020-09-30 at 4.01.19 AM.png > Table Map Reduce job after a Snapshot based job fails with > CorruptedSnapshotException > - > > Key: PHOENIX-6153 > URL: https://issues.apache.org/jira/browse/PHOENIX-6153 > Project: Phoenix > Issue Type: Bug > Components: core >Affects Versions: 4.15.0, 4.14.3, master >Reporter: Saksham Gangwar >Assignee: Saksham Gangwar >Priority: Major > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-6153.4.x.v1.patch, PHOENIX-6153.master.v1.patch, > PHOENIX-6153.master.v2.patch, PHOENIX-6153.master.v3.patch, > PHOENIX-6153.master.v4.patch, PHOENIX-6153.master.v5.patch, Screen Shot > 2020-09-30 at 4.00.58 AM.png, Screen Shot 2020-09-30 at 4.01.10 AM.png, > Screen Shot 2020-09-30 at 4.01.10 AM.png, Screen Shot 2020-09-30 at 4.01.19 > AM.png, Screen Shot 2020-09-30 at 4.01.19 AM.png, Screen Shot 2020-09-30 at > 4.01.19 AM.png, Screen Shot 2020-09-30 at 4.01.34 AM.png, Screen Shot > 2020-09-30 at 4.01.52 AM.png, Screen Shot 2020-09-30 at 4.01.52 AM.png > > > Different MR job requests which reach [MapReduceParallelScanGrouper > getRegionBoundaries|https://github.com/apache/phoenix/blob/f9e304754bad886344a856dd2565e3f24e345ed2/phoenix-core/src/main/java/org/apache/phoenix/iterate/MapReduceParallelScanGrouper.java#L65] > we currently make use of shared configuration among jobs to figure out > snapshot names. > Example jobs' sequence: first two jobs work over snapshot and the third job > over a regular table. > Prininting hashcode of objects when entering: > [https://github.com/apache/phoenix/blob/f9e304754bad886344a856dd2565e3f24e345ed2/phoenix-core/src/main/java/org/apache/phoenix/iterate/MapReduceParallelScanGrouper.java#L65] > *Job 1:* (over snapshot of *ABC_TABLE_1* and is successful) > context.getConnection(): 521093916 > ConnectionQueryServices: 1772519705 > *Configuration conf: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY):*ABC_TABLE_1* > > *Job 2:* (over snapshot of *ABC_TABLE_2* and is successful) > context.getConnection(): 1928017473 > ConnectionQueryServices: 961279422 > *Configuration conf: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY): *ABC_TABLE_2* > > *Job 3:* (over the table *ABC_TABLE_3* but fails with > CorruptedSnapshotException while it got nothing to do with snapshot) > context.getConnection(): 28889670 > ConnectionQueryServices: 424389847 > *Configuration: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY): *ABC_TABLE_2* > > Exception which we get: > [2020:08:18 20:56:17.409] [MigrationRetryPoller-Executor-1] [ERROR] > [c.s.hgrate.mapreduce.MapReduceImpl] - Error submitting M/R job for Job 3 > java.lang.RuntimeException: > org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read > snapshot info > from:hdfs://.../hbase/.hbase-snapshot/ABC_TABLE_2_1597687413477/.snapshotinfo > at > org.apache.phoenix.iterate.MapReduceParallelScanGrouper.getRegionBoundaries(MapReduceParallelScanGrouper.java:81) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getRegionBoundaries(BaseResultIterators.java:541) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getParallelScans(BaseResultIterators.java:893) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getParallelScans(BaseResultIterators.java:641) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.(BaseResultIterators.java:511) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.ParallelIterators.(ParallelIterators.java:62) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.ScanPlan.newIterator(ScanPlan.java:278) > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:367) > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:218) >
[jira] [Updated] (PHOENIX-6153) Table Map Reduce job after a Snapshot based job fails with CorruptedSnapshotException
[ https://issues.apache.org/jira/browse/PHOENIX-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saksham Gangwar updated PHOENIX-6153: - Attachment: Screen Shot 2020-09-30 at 4.01.34 AM.png > Table Map Reduce job after a Snapshot based job fails with > CorruptedSnapshotException > - > > Key: PHOENIX-6153 > URL: https://issues.apache.org/jira/browse/PHOENIX-6153 > Project: Phoenix > Issue Type: Bug > Components: core >Affects Versions: 4.15.0, 4.14.3, master >Reporter: Saksham Gangwar >Assignee: Saksham Gangwar >Priority: Major > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-6153.4.x.v1.patch, PHOENIX-6153.master.v1.patch, > PHOENIX-6153.master.v2.patch, PHOENIX-6153.master.v3.patch, > PHOENIX-6153.master.v4.patch, PHOENIX-6153.master.v5.patch, Screen Shot > 2020-09-30 at 4.00.58 AM.png, Screen Shot 2020-09-30 at 4.01.10 AM.png, > Screen Shot 2020-09-30 at 4.01.10 AM.png, Screen Shot 2020-09-30 at 4.01.19 > AM.png, Screen Shot 2020-09-30 at 4.01.19 AM.png, Screen Shot 2020-09-30 at > 4.01.19 AM.png, Screen Shot 2020-09-30 at 4.01.34 AM.png, Screen Shot > 2020-09-30 at 4.01.52 AM.png, Screen Shot 2020-09-30 at 4.01.52 AM.png > > > Different MR job requests which reach [MapReduceParallelScanGrouper > getRegionBoundaries|https://github.com/apache/phoenix/blob/f9e304754bad886344a856dd2565e3f24e345ed2/phoenix-core/src/main/java/org/apache/phoenix/iterate/MapReduceParallelScanGrouper.java#L65] > we currently make use of shared configuration among jobs to figure out > snapshot names. > Example jobs' sequence: first two jobs work over snapshot and the third job > over a regular table. > Prininting hashcode of objects when entering: > [https://github.com/apache/phoenix/blob/f9e304754bad886344a856dd2565e3f24e345ed2/phoenix-core/src/main/java/org/apache/phoenix/iterate/MapReduceParallelScanGrouper.java#L65] > *Job 1:* (over snapshot of *ABC_TABLE_1* and is successful) > context.getConnection(): 521093916 > ConnectionQueryServices: 1772519705 > *Configuration conf: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY):*ABC_TABLE_1* > > *Job 2:* (over snapshot of *ABC_TABLE_2* and is successful) > context.getConnection(): 1928017473 > ConnectionQueryServices: 961279422 > *Configuration conf: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY): *ABC_TABLE_2* > > *Job 3:* (over the table *ABC_TABLE_3* but fails with > CorruptedSnapshotException while it got nothing to do with snapshot) > context.getConnection(): 28889670 > ConnectionQueryServices: 424389847 > *Configuration: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY): *ABC_TABLE_2* > > Exception which we get: > [2020:08:18 20:56:17.409] [MigrationRetryPoller-Executor-1] [ERROR] > [c.s.hgrate.mapreduce.MapReduceImpl] - Error submitting M/R job for Job 3 > java.lang.RuntimeException: > org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read > snapshot info > from:hdfs://.../hbase/.hbase-snapshot/ABC_TABLE_2_1597687413477/.snapshotinfo > at > org.apache.phoenix.iterate.MapReduceParallelScanGrouper.getRegionBoundaries(MapReduceParallelScanGrouper.java:81) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getRegionBoundaries(BaseResultIterators.java:541) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getParallelScans(BaseResultIterators.java:893) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getParallelScans(BaseResultIterators.java:641) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.(BaseResultIterators.java:511) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.ParallelIterators.(ParallelIterators.java:62) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.ScanPlan.newIterator(ScanPlan.java:278) > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:367) > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:218) >
[jira] [Updated] (PHOENIX-6153) Table Map Reduce job after a Snapshot based job fails with CorruptedSnapshotException
[ https://issues.apache.org/jira/browse/PHOENIX-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saksham Gangwar updated PHOENIX-6153: - Attachment: Screen Shot 2020-09-30 at 4.01.10 AM.png > Table Map Reduce job after a Snapshot based job fails with > CorruptedSnapshotException > - > > Key: PHOENIX-6153 > URL: https://issues.apache.org/jira/browse/PHOENIX-6153 > Project: Phoenix > Issue Type: Bug > Components: core >Affects Versions: 4.15.0, 4.14.3, master >Reporter: Saksham Gangwar >Assignee: Saksham Gangwar >Priority: Major > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-6153.4.x.v1.patch, PHOENIX-6153.master.v1.patch, > PHOENIX-6153.master.v2.patch, PHOENIX-6153.master.v3.patch, > PHOENIX-6153.master.v4.patch, PHOENIX-6153.master.v5.patch, Screen Shot > 2020-09-30 at 4.00.58 AM.png, Screen Shot 2020-09-30 at 4.01.10 AM.png, > Screen Shot 2020-09-30 at 4.01.10 AM.png, Screen Shot 2020-09-30 at 4.01.19 > AM.png, Screen Shot 2020-09-30 at 4.01.52 AM.png > > > Different MR job requests which reach [MapReduceParallelScanGrouper > getRegionBoundaries|https://github.com/apache/phoenix/blob/f9e304754bad886344a856dd2565e3f24e345ed2/phoenix-core/src/main/java/org/apache/phoenix/iterate/MapReduceParallelScanGrouper.java#L65] > we currently make use of shared configuration among jobs to figure out > snapshot names. > Example jobs' sequence: first two jobs work over snapshot and the third job > over a regular table. > Prininting hashcode of objects when entering: > [https://github.com/apache/phoenix/blob/f9e304754bad886344a856dd2565e3f24e345ed2/phoenix-core/src/main/java/org/apache/phoenix/iterate/MapReduceParallelScanGrouper.java#L65] > *Job 1:* (over snapshot of *ABC_TABLE_1* and is successful) > context.getConnection(): 521093916 > ConnectionQueryServices: 1772519705 > *Configuration conf: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY):*ABC_TABLE_1* > > *Job 2:* (over snapshot of *ABC_TABLE_2* and is successful) > context.getConnection(): 1928017473 > ConnectionQueryServices: 961279422 > *Configuration conf: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY): *ABC_TABLE_2* > > *Job 3:* (over the table *ABC_TABLE_3* but fails with > CorruptedSnapshotException while it got nothing to do with snapshot) > context.getConnection(): 28889670 > ConnectionQueryServices: 424389847 > *Configuration: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY): *ABC_TABLE_2* > > Exception which we get: > [2020:08:18 20:56:17.409] [MigrationRetryPoller-Executor-1] [ERROR] > [c.s.hgrate.mapreduce.MapReduceImpl] - Error submitting M/R job for Job 3 > java.lang.RuntimeException: > org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read > snapshot info > from:hdfs://.../hbase/.hbase-snapshot/ABC_TABLE_2_1597687413477/.snapshotinfo > at > org.apache.phoenix.iterate.MapReduceParallelScanGrouper.getRegionBoundaries(MapReduceParallelScanGrouper.java:81) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getRegionBoundaries(BaseResultIterators.java:541) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getParallelScans(BaseResultIterators.java:893) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getParallelScans(BaseResultIterators.java:641) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.(BaseResultIterators.java:511) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.ParallelIterators.(ParallelIterators.java:62) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.ScanPlan.newIterator(ScanPlan.java:278) > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:367) > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:218) > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:213) >
[jira] [Updated] (PHOENIX-6153) Table Map Reduce job after a Snapshot based job fails with CorruptedSnapshotException
[ https://issues.apache.org/jira/browse/PHOENIX-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saksham Gangwar updated PHOENIX-6153: - Attachment: Screen Shot 2020-09-30 at 4.00.58 AM.png Screen Shot 2020-09-30 at 4.01.10 AM.png Screen Shot 2020-09-30 at 4.01.19 AM.png Screen Shot 2020-09-30 at 4.01.52 AM.png > Table Map Reduce job after a Snapshot based job fails with > CorruptedSnapshotException > - > > Key: PHOENIX-6153 > URL: https://issues.apache.org/jira/browse/PHOENIX-6153 > Project: Phoenix > Issue Type: Bug > Components: core >Affects Versions: 4.15.0, 4.14.3, master >Reporter: Saksham Gangwar >Assignee: Saksham Gangwar >Priority: Major > Fix For: 5.1.0, 4.16.0 > > Attachments: PHOENIX-6153.4.x.v1.patch, PHOENIX-6153.master.v1.patch, > PHOENIX-6153.master.v2.patch, PHOENIX-6153.master.v3.patch, > PHOENIX-6153.master.v4.patch, PHOENIX-6153.master.v5.patch, Screen Shot > 2020-09-30 at 4.00.58 AM.png, Screen Shot 2020-09-30 at 4.01.10 AM.png, > Screen Shot 2020-09-30 at 4.01.19 AM.png, Screen Shot 2020-09-30 at 4.01.52 > AM.png > > > Different MR job requests which reach [MapReduceParallelScanGrouper > getRegionBoundaries|https://github.com/apache/phoenix/blob/f9e304754bad886344a856dd2565e3f24e345ed2/phoenix-core/src/main/java/org/apache/phoenix/iterate/MapReduceParallelScanGrouper.java#L65] > we currently make use of shared configuration among jobs to figure out > snapshot names. > Example jobs' sequence: first two jobs work over snapshot and the third job > over a regular table. > Prininting hashcode of objects when entering: > [https://github.com/apache/phoenix/blob/f9e304754bad886344a856dd2565e3f24e345ed2/phoenix-core/src/main/java/org/apache/phoenix/iterate/MapReduceParallelScanGrouper.java#L65] > *Job 1:* (over snapshot of *ABC_TABLE_1* and is successful) > context.getConnection(): 521093916 > ConnectionQueryServices: 1772519705 > *Configuration conf: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY):*ABC_TABLE_1* > > *Job 2:* (over snapshot of *ABC_TABLE_2* and is successful) > context.getConnection(): 1928017473 > ConnectionQueryServices: 961279422 > *Configuration conf: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY): *ABC_TABLE_2* > > *Job 3:* (over the table *ABC_TABLE_3* but fails with > CorruptedSnapshotException while it got nothing to do with snapshot) > context.getConnection(): 28889670 > ConnectionQueryServices: 424389847 > *Configuration: 813285994* > conf.get(PhoenixConfigurationUtil.SNAPSHOT_NAME_KEY): *ABC_TABLE_2* > > Exception which we get: > [2020:08:18 20:56:17.409] [MigrationRetryPoller-Executor-1] [ERROR] > [c.s.hgrate.mapreduce.MapReduceImpl] - Error submitting M/R job for Job 3 > java.lang.RuntimeException: > org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: Couldn't read > snapshot info > from:hdfs://.../hbase/.hbase-snapshot/ABC_TABLE_2_1597687413477/.snapshotinfo > at > org.apache.phoenix.iterate.MapReduceParallelScanGrouper.getRegionBoundaries(MapReduceParallelScanGrouper.java:81) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getRegionBoundaries(BaseResultIterators.java:541) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getParallelScans(BaseResultIterators.java:893) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.getParallelScans(BaseResultIterators.java:641) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.BaseResultIterators.(BaseResultIterators.java:511) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at > org.apache.phoenix.iterate.ParallelIterators.(ParallelIterators.java:62) > > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.ScanPlan.newIterator(ScanPlan.java:278) > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:367) > ~[phoenix-core-4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT.jar:4.14.3-hbase-1.6-sfdc-1.0.9-SNAPSHOT] > at org.apache.phoenix.execute.BaseQueryPlan.iterator(BaseQueryPlan.java:218) >
[jira] [Resolved] (PHOENIX-6168) PHOENIX-6143 breaks tests on linux
[ https://issues.apache.org/jira/browse/PHOENIX-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Toth resolved PHOENIX-6168. -- Fix Version/s: 4.16.0 5.1.0 Resolution: Fixed Committed. Thanks for the quick review [~rajeshbabu] > PHOENIX-6143 breaks tests on linux > -- > > Key: PHOENIX-6168 > URL: https://issues.apache.org/jira/browse/PHOENIX-6168 > Project: Phoenix > Issue Type: Bug > Components: core >Affects Versions: 5.1.0, 4.16.0 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > Fix For: 5.1.0, 4.16.0 > > > PHOENIX-6143 upgraded the jasmine plugin version to 2.2. > The default phantomjs version for that release in 2.0.0, which the plugin > cannot download on Linux. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (PHOENIX-6168) PHOENIX-6143 breaks tests on linux
Istvan Toth created PHOENIX-6168: Summary: PHOENIX-6143 breaks tests on linux Key: PHOENIX-6168 URL: https://issues.apache.org/jira/browse/PHOENIX-6168 Project: Phoenix Issue Type: Bug Components: core Affects Versions: 5.1.0, 4.16.0 Reporter: Istvan Toth Assignee: Istvan Toth PHOENIX-6143 upgraded the jasmine plugin version to 2.2. The default phantomjs version for that release in 2.0.0, which doesn't exist. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (PHOENIX-6168) PHOENIX-6143 breaks tests on linux
[ https://issues.apache.org/jira/browse/PHOENIX-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Istvan Toth updated PHOENIX-6168: - Description: PHOENIX-6143 upgraded the jasmine plugin version to 2.2. The default phantomjs version for that release in 2.0.0, which the plugin cannot download on Linux. was: PHOENIX-6143 upgraded the jasmine plugin version to 2.2. The default phantomjs version for that release in 2.0.0, which doesn't exist. > PHOENIX-6143 breaks tests on linux > -- > > Key: PHOENIX-6168 > URL: https://issues.apache.org/jira/browse/PHOENIX-6168 > Project: Phoenix > Issue Type: Bug > Components: core >Affects Versions: 5.1.0, 4.16.0 >Reporter: Istvan Toth >Assignee: Istvan Toth >Priority: Major > > PHOENIX-6143 upgraded the jasmine plugin version to 2.2. > The default phantomjs version for that release in 2.0.0, which the plugin > cannot download on Linux. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
