[ https://issues.apache.org/jira/browse/HBASE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
chunhui shen updated HBASE-6311: -------------------------------- Description: It is a big problem found in 0.94, and you could reproduce the problem in Trunk using the test case I uploaded: When we do compaction, we will use region.getSmallestReadPoint(); It will make data mistake after compaction because we will skip delete type KV but keep the put type kv. The following is the reason: In StoreFileScanner, enforceMVCC is false when compaction, so we could read the delete type KV, However, we will skip this delete type KV in ScanQueryMatcher because following code {code} if (kv.isDelete()) { ... if (includeDeleteMarker && kv.getMemstoreTS() <= maxReadPointToTrackVersions) { System.out.println("add deletes,maxReadPointToTrackVersions=" + maxReadPointToTrackVersions); this.deletes.add(bytes, offset, qualLength, timestamp, type); } ... } {code} Here maxReadPointToTrackVersions = region.getSmallestReadPoint(); and kv.getMemstoreTS() > maxReadPointToTrackVersions So we won't add this to DeleteTracker. Why test case passed if remove the line MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); Because in the StoreFileScanner#skipKVsNewerThanReadpoint {code} if (cur.getMemstoreTS() <= readPoint) { cur.setMemstoreTS(0); } {code} So if we remove the line MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); Here readPoint is LONG.MAX_VALUE, we will set memStore ts as 0, so we will add it to DeleteTracker in ScanQueryMatcher was: It is a big problem: When we do compaction, we will use region.getSmallestReadPoint(); It will make data mistake after compaction because we will skip delete type KV but keep the put type kv. The following is the reason: In StoreFileScanner, enforceMVCC is false when compaction, so we could read the delete type KV, However, we will skip this delete type KV in ScanQueryMatcher because following code {code} if (kv.isDelete()) { ... if (includeDeleteMarker && kv.getMemstoreTS() <= maxReadPointToTrackVersions) { System.out.println("add deletes,maxReadPointToTrackVersions=" + maxReadPointToTrackVersions); this.deletes.add(bytes, offset, qualLength, timestamp, type); } ... } {code} Here maxReadPointToTrackVersions = region.getSmallestReadPoint(); and kv.getMemstoreTS() > maxReadPointToTrackVersions So we won't add this to DeleteTracker. Why test case passed if remove the line MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); Because in the StoreFileScanner#skipKVsNewerThanReadpoint {code} if (cur.getMemstoreTS() <= readPoint) { cur.setMemstoreTS(0); } {code} So if we remove the line MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); Here readPoint is LONG.MAX_VALUE, we will set memStore ts as 0, so we will add it to DeleteTracker in ScanQueryMatcher > Data error after compaction because of scanners > ----------------------------------------------- > > Key: HBASE-6311 > URL: https://issues.apache.org/jira/browse/HBASE-6311 > Project: HBase > Issue Type: Bug > Components: regionserver > Affects Versions: 0.94.0 > Reporter: chunhui shen > Assignee: chunhui shen > Priority: Blocker > Attachments: HBASE-6311-test.patch, HBASE-6311v1.patch > > > It is a big problem found in 0.94, and you could reproduce the problem in > Trunk using the test case I uploaded: > When we do compaction, we will use region.getSmallestReadPoint(); > It will make data mistake after compaction because we will skip delete type > KV but keep the put type kv. > The following is the reason: > In StoreFileScanner, enforceMVCC is false when compaction, so we could read > the delete type KV, > However, we will skip this delete type KV in ScanQueryMatcher because > following code > {code} > if (kv.isDelete()) > { > ... > if (includeDeleteMarker > && kv.getMemstoreTS() <= maxReadPointToTrackVersions) { > System.out.println("add deletes,maxReadPointToTrackVersions=" > + maxReadPointToTrackVersions); > this.deletes.add(bytes, offset, qualLength, timestamp, type); > } > ... > } > {code} > Here maxReadPointToTrackVersions = region.getSmallestReadPoint(); > and kv.getMemstoreTS() > maxReadPointToTrackVersions > So we won't add this to DeleteTracker. > Why test case passed if remove the line > MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); > Because in the StoreFileScanner#skipKVsNewerThanReadpoint > {code} > if (cur.getMemstoreTS() <= readPoint) { > cur.setMemstoreTS(0); > } > {code} > So if we remove the line > MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); > Here readPoint is LONG.MAX_VALUE, we will set memStore ts as 0, so we will > add it to DeleteTracker in ScanQueryMatcher -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira