[jira] [Updated] (HBASE-6311) Data error after majorCompaction caused by keeping MVCC for opened scanners
[ https://issues.apache.org/jira/browse/HBASE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6311: - Fix Version/s: (was: 0.95.0) (was: 0.98.0) 0.96.0 > Data error after majorCompaction caused by keeping MVCC for opened scanners > --- > > Key: HBASE-6311 > URL: https://issues.apache.org/jira/browse/HBASE-6311 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.0 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Blocker > Fix For: 0.94.1, 0.96.0 > > Attachments: HBASE-6311-test.patch, HBASE-6311v1.patch, > HBASE-6311v2.patch > > > It is a big problem we found in 0.94, and you could reproduce the problem in > Trunk using the test case I uploaded. > When we do compaction, we will use region.getSmallestReadPoint() to keep MVCC > for opened scanners; > However,It will make data mistake after majorCompaction because we will skip > delete type KV but keep the put type kv in the compacted storefile. > The following is the reason from code: > In StoreFileScanner, enforceMVCC is false when compaction, so we could read > the delete type KV, > However, we will skip this delete type KV in ScanQueryMatcher because > following code > {code} > if (kv.isDelete()) > { > ... > if (includeDeleteMarker > && kv.getMemstoreTS() <= maxReadPointToTrackVersions) { > System.out.println("add deletes,maxReadPointToTrackVersions=" > + maxReadPointToTrackVersions); > this.deletes.add(bytes, offset, qualLength, timestamp, type); > } > ... > } > {code} > Here maxReadPointToTrackVersions = region.getSmallestReadPoint(); > and kv.getMemstoreTS() > maxReadPointToTrackVersions > So we won't add this to DeleteTracker. > Why test case passed if remove the line > MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); > Because in the StoreFileScanner#skipKVsNewerThanReadpoint > {code} > if (cur.getMemstoreTS() <= readPoint) { > cur.setMemstoreTS(0); > } > {code} > So if we remove the line > MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); > Here readPoint is LONG.MAX_VALUE, we will set memStore ts as 0, so we will > add it to DeleteTracker in ScanQueryMatcher > Solution: > We use smallestReadPoint of region when compaction to keep MVCC for OPENED > scanner, So we should retain delete type kv in output in the case(Already > deleted KV is retained in output to make old opened scanner could read this > KV) even if it is a majorcompaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6311) Data error after majorCompaction caused by keeping MVCC for opened scanners
[ https://issues.apache.org/jira/browse/HBASE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6311: - Fix Version/s: (was: 0.96.0) 0.95.0 > Data error after majorCompaction caused by keeping MVCC for opened scanners > --- > > Key: HBASE-6311 > URL: https://issues.apache.org/jira/browse/HBASE-6311 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.0 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Blocker > Fix For: 0.94.1, 0.95.0 > > Attachments: HBASE-6311-test.patch, HBASE-6311v1.patch, > HBASE-6311v2.patch > > > It is a big problem we found in 0.94, and you could reproduce the problem in > Trunk using the test case I uploaded. > When we do compaction, we will use region.getSmallestReadPoint() to keep MVCC > for opened scanners; > However,It will make data mistake after majorCompaction because we will skip > delete type KV but keep the put type kv in the compacted storefile. > The following is the reason from code: > In StoreFileScanner, enforceMVCC is false when compaction, so we could read > the delete type KV, > However, we will skip this delete type KV in ScanQueryMatcher because > following code > {code} > if (kv.isDelete()) > { > ... > if (includeDeleteMarker > && kv.getMemstoreTS() <= maxReadPointToTrackVersions) { > System.out.println("add deletes,maxReadPointToTrackVersions=" > + maxReadPointToTrackVersions); > this.deletes.add(bytes, offset, qualLength, timestamp, type); > } > ... > } > {code} > Here maxReadPointToTrackVersions = region.getSmallestReadPoint(); > and kv.getMemstoreTS() > maxReadPointToTrackVersions > So we won't add this to DeleteTracker. > Why test case passed if remove the line > MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); > Because in the StoreFileScanner#skipKVsNewerThanReadpoint > {code} > if (cur.getMemstoreTS() <= readPoint) { > cur.setMemstoreTS(0); > } > {code} > So if we remove the line > MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); > Here readPoint is LONG.MAX_VALUE, we will set memStore ts as 0, so we will > add it to DeleteTracker in ScanQueryMatcher > Solution: > We use smallestReadPoint of region when compaction to keep MVCC for OPENED > scanner, So we should retain delete type kv in output in the case(Already > deleted KV is retained in output to make old opened scanner could read this > KV) even if it is a majorcompaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6311) Data error after majorCompaction caused by keeping MVCC for opened scanners
[ https://issues.apache.org/jira/browse/HBASE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6311: - Fix Version/s: 0.98.0 > Data error after majorCompaction caused by keeping MVCC for opened scanners > --- > > Key: HBASE-6311 > URL: https://issues.apache.org/jira/browse/HBASE-6311 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.0 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Blocker > Fix For: 0.94.1, 0.98.0, 0.95.0 > > Attachments: HBASE-6311-test.patch, HBASE-6311v1.patch, > HBASE-6311v2.patch > > > It is a big problem we found in 0.94, and you could reproduce the problem in > Trunk using the test case I uploaded. > When we do compaction, we will use region.getSmallestReadPoint() to keep MVCC > for opened scanners; > However,It will make data mistake after majorCompaction because we will skip > delete type KV but keep the put type kv in the compacted storefile. > The following is the reason from code: > In StoreFileScanner, enforceMVCC is false when compaction, so we could read > the delete type KV, > However, we will skip this delete type KV in ScanQueryMatcher because > following code > {code} > if (kv.isDelete()) > { > ... > if (includeDeleteMarker > && kv.getMemstoreTS() <= maxReadPointToTrackVersions) { > System.out.println("add deletes,maxReadPointToTrackVersions=" > + maxReadPointToTrackVersions); > this.deletes.add(bytes, offset, qualLength, timestamp, type); > } > ... > } > {code} > Here maxReadPointToTrackVersions = region.getSmallestReadPoint(); > and kv.getMemstoreTS() > maxReadPointToTrackVersions > So we won't add this to DeleteTracker. > Why test case passed if remove the line > MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); > Because in the StoreFileScanner#skipKVsNewerThanReadpoint > {code} > if (cur.getMemstoreTS() <= readPoint) { > cur.setMemstoreTS(0); > } > {code} > So if we remove the line > MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); > Here readPoint is LONG.MAX_VALUE, we will set memStore ts as 0, so we will > add it to DeleteTracker in ScanQueryMatcher > Solution: > We use smallestReadPoint of region when compaction to keep MVCC for OPENED > scanner, So we should retain delete type kv in output in the case(Already > deleted KV is retained in output to make old opened scanner could read this > KV) even if it is a majorcompaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6311) Data error after majorCompaction caused by keeping MVCC for opened scanners
[ https://issues.apache.org/jira/browse/HBASE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6311: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to 0.94 and 0.96. > Data error after majorCompaction caused by keeping MVCC for opened scanners > --- > > Key: HBASE-6311 > URL: https://issues.apache.org/jira/browse/HBASE-6311 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.0 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Blocker > Fix For: 0.96.0, 0.94.1 > > Attachments: HBASE-6311-test.patch, HBASE-6311v1.patch, > HBASE-6311v2.patch > > > It is a big problem we found in 0.94, and you could reproduce the problem in > Trunk using the test case I uploaded. > When we do compaction, we will use region.getSmallestReadPoint() to keep MVCC > for opened scanners; > However,It will make data mistake after majorCompaction because we will skip > delete type KV but keep the put type kv in the compacted storefile. > The following is the reason from code: > In StoreFileScanner, enforceMVCC is false when compaction, so we could read > the delete type KV, > However, we will skip this delete type KV in ScanQueryMatcher because > following code > {code} > if (kv.isDelete()) > { > ... > if (includeDeleteMarker > && kv.getMemstoreTS() <= maxReadPointToTrackVersions) { > System.out.println("add deletes,maxReadPointToTrackVersions=" > + maxReadPointToTrackVersions); > this.deletes.add(bytes, offset, qualLength, timestamp, type); > } > ... > } > {code} > Here maxReadPointToTrackVersions = region.getSmallestReadPoint(); > and kv.getMemstoreTS() > maxReadPointToTrackVersions > So we won't add this to DeleteTracker. > Why test case passed if remove the line > MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); > Because in the StoreFileScanner#skipKVsNewerThanReadpoint > {code} > if (cur.getMemstoreTS() <= readPoint) { > cur.setMemstoreTS(0); > } > {code} > So if we remove the line > MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); > Here readPoint is LONG.MAX_VALUE, we will set memStore ts as 0, so we will > add it to DeleteTracker in ScanQueryMatcher > Solution: > We use smallestReadPoint of region when compaction to keep MVCC for OPENED > scanner, So we should retain delete type kv in output in the case(Already > deleted KV is retained in output to make old opened scanner could read this > KV) even if it is a majorcompaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6311) Data error after majorCompaction caused by keeping MVCC for opened scanners
[ https://issues.apache.org/jira/browse/HBASE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu updated HBASE-6311: -- Fix Version/s: 0.94.1 0.96.0 Hadoop Flags: Reviewed Status: Patch Available (was: Open) > Data error after majorCompaction caused by keeping MVCC for opened scanners > --- > > Key: HBASE-6311 > URL: https://issues.apache.org/jira/browse/HBASE-6311 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.0 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Blocker > Fix For: 0.96.0, 0.94.1 > > Attachments: HBASE-6311-test.patch, HBASE-6311v1.patch, > HBASE-6311v2.patch > > > It is a big problem we found in 0.94, and you could reproduce the problem in > Trunk using the test case I uploaded. > When we do compaction, we will use region.getSmallestReadPoint() to keep MVCC > for opened scanners; > However,It will make data mistake after majorCompaction because we will skip > delete type KV but keep the put type kv in the compacted storefile. > The following is the reason from code: > In StoreFileScanner, enforceMVCC is false when compaction, so we could read > the delete type KV, > However, we will skip this delete type KV in ScanQueryMatcher because > following code > {code} > if (kv.isDelete()) > { > ... > if (includeDeleteMarker > && kv.getMemstoreTS() <= maxReadPointToTrackVersions) { > System.out.println("add deletes,maxReadPointToTrackVersions=" > + maxReadPointToTrackVersions); > this.deletes.add(bytes, offset, qualLength, timestamp, type); > } > ... > } > {code} > Here maxReadPointToTrackVersions = region.getSmallestReadPoint(); > and kv.getMemstoreTS() > maxReadPointToTrackVersions > So we won't add this to DeleteTracker. > Why test case passed if remove the line > MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); > Because in the StoreFileScanner#skipKVsNewerThanReadpoint > {code} > if (cur.getMemstoreTS() <= readPoint) { > cur.setMemstoreTS(0); > } > {code} > So if we remove the line > MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); > Here readPoint is LONG.MAX_VALUE, we will set memStore ts as 0, so we will > add it to DeleteTracker in ScanQueryMatcher > Solution: > We use smallestReadPoint of region when compaction to keep MVCC for OPENED > scanner, So we should retain delete type kv in output in the case(Already > deleted KV is retained in output to make old opened scanner could read this > KV) even if it is a majorcompaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6311) Data error after majorCompaction caused by keeping MVCC for opened scanners
[ https://issues.apache.org/jira/browse/HBASE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-6311: Attachment: HBASE-6311v2.patch @ram What doubt do you have about my patch v2? I update the test case to verify MVCC for scanners after majorCompaction. > Data error after majorCompaction caused by keeping MVCC for opened scanners > --- > > Key: HBASE-6311 > URL: https://issues.apache.org/jira/browse/HBASE-6311 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.0 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Blocker > Attachments: HBASE-6311-test.patch, HBASE-6311v1.patch, > HBASE-6311v2.patch > > > It is a big problem we found in 0.94, and you could reproduce the problem in > Trunk using the test case I uploaded. > When we do compaction, we will use region.getSmallestReadPoint() to keep MVCC > for opened scanners; > However,It will make data mistake after majorCompaction because we will skip > delete type KV but keep the put type kv in the compacted storefile. > The following is the reason from code: > In StoreFileScanner, enforceMVCC is false when compaction, so we could read > the delete type KV, > However, we will skip this delete type KV in ScanQueryMatcher because > following code > {code} > if (kv.isDelete()) > { > ... > if (includeDeleteMarker > && kv.getMemstoreTS() <= maxReadPointToTrackVersions) { > System.out.println("add deletes,maxReadPointToTrackVersions=" > + maxReadPointToTrackVersions); > this.deletes.add(bytes, offset, qualLength, timestamp, type); > } > ... > } > {code} > Here maxReadPointToTrackVersions = region.getSmallestReadPoint(); > and kv.getMemstoreTS() > maxReadPointToTrackVersions > So we won't add this to DeleteTracker. > Why test case passed if remove the line > MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); > Because in the StoreFileScanner#skipKVsNewerThanReadpoint > {code} > if (cur.getMemstoreTS() <= readPoint) { > cur.setMemstoreTS(0); > } > {code} > So if we remove the line > MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); > Here readPoint is LONG.MAX_VALUE, we will set memStore ts as 0, so we will > add it to DeleteTracker in ScanQueryMatcher > Solution: > We use smallestReadPoint of region when compaction to keep MVCC for OPENED > scanner, So we should retain delete type kv in output in the case(Already > deleted KV is retained in output to make old opened scanner could read this > KV) even if it is a majorcompaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6311) Data error after majorCompaction caused by keeping MVCC for opened scanners
[ https://issues.apache.org/jira/browse/HBASE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-6311: Summary: Data error after majorCompaction caused by keeping MVCC for opened scanners (was: Data error after majorCompaction because keep MVCC for opened scanners) > Data error after majorCompaction caused by keeping MVCC for opened scanners > --- > > Key: HBASE-6311 > URL: https://issues.apache.org/jira/browse/HBASE-6311 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.0 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Blocker > Attachments: HBASE-6311-test.patch, HBASE-6311v1.patch > > > It is a big problem we found in 0.94, and you could reproduce the problem in > Trunk using the test case I uploaded. > When we do compaction, we will use region.getSmallestReadPoint() to keep MVCC > for opened scanners; > However,It will make data mistake after majorCompaction because we will skip > delete type KV but keep the put type kv in the compacted storefile. > The following is the reason from code: > In StoreFileScanner, enforceMVCC is false when compaction, so we could read > the delete type KV, > However, we will skip this delete type KV in ScanQueryMatcher because > following code > {code} > if (kv.isDelete()) > { > ... > if (includeDeleteMarker > && kv.getMemstoreTS() <= maxReadPointToTrackVersions) { > System.out.println("add deletes,maxReadPointToTrackVersions=" > + maxReadPointToTrackVersions); > this.deletes.add(bytes, offset, qualLength, timestamp, type); > } > ... > } > {code} > Here maxReadPointToTrackVersions = region.getSmallestReadPoint(); > and kv.getMemstoreTS() > maxReadPointToTrackVersions > So we won't add this to DeleteTracker. > Why test case passed if remove the line > MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); > Because in the StoreFileScanner#skipKVsNewerThanReadpoint > {code} > if (cur.getMemstoreTS() <= readPoint) { > cur.setMemstoreTS(0); > } > {code} > So if we remove the line > MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); > Here readPoint is LONG.MAX_VALUE, we will set memStore ts as 0, so we will > add it to DeleteTracker in ScanQueryMatcher > Solution: > We use smallestReadPoint of region when compaction to keep MVCC for OPENED > scanner, So we should retain delete type kv in output in the case(Already > deleted KV is retained in output to make old opened scanner could read this > KV) even if it is a majorcompaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira