[ https://issues.apache.org/jira/browse/HBASE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13408315#comment-13408315 ]
Hudson commented on HBASE-6311: ------------------------------- Integrated in HBase-0.94 #302 (See [https://builds.apache.org/job/HBase-0.94/302/]) HBASE-6311 Data error after majorCompaction caused by keeping MVCC for opened scanners (chunhui shen) (Revision 1358333) Result = FAILURE larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java > Data error after majorCompaction caused by keeping MVCC for opened scanners > --------------------------------------------------------------------------- > > Key: HBASE-6311 > URL: https://issues.apache.org/jira/browse/HBASE-6311 > Project: HBase > Issue Type: Bug > Components: regionserver > Affects Versions: 0.94.0 > Reporter: chunhui shen > Assignee: chunhui shen > Priority: Blocker > Fix For: 0.96.0, 0.94.1 > > Attachments: HBASE-6311-test.patch, HBASE-6311v1.patch, > HBASE-6311v2.patch > > > It is a big problem we found in 0.94, and you could reproduce the problem in > Trunk using the test case I uploaded. > When we do compaction, we will use region.getSmallestReadPoint() to keep MVCC > for opened scanners; > However,It will make data mistake after majorCompaction because we will skip > delete type KV but keep the put type kv in the compacted storefile. > The following is the reason from code: > In StoreFileScanner, enforceMVCC is false when compaction, so we could read > the delete type KV, > However, we will skip this delete type KV in ScanQueryMatcher because > following code > {code} > if (kv.isDelete()) > { > ... > if (includeDeleteMarker > && kv.getMemstoreTS() <= maxReadPointToTrackVersions) { > System.out.println("add deletes,maxReadPointToTrackVersions=" > + maxReadPointToTrackVersions); > this.deletes.add(bytes, offset, qualLength, timestamp, type); > } > ... > } > {code} > Here maxReadPointToTrackVersions = region.getSmallestReadPoint(); > and kv.getMemstoreTS() > maxReadPointToTrackVersions > So we won't add this to DeleteTracker. > Why test case passed if remove the line > MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); > Because in the StoreFileScanner#skipKVsNewerThanReadpoint > {code} > if (cur.getMemstoreTS() <= readPoint) { > cur.setMemstoreTS(0); > } > {code} > So if we remove the line > MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint); > Here readPoint is LONG.MAX_VALUE, we will set memStore ts as 0, so we will > add it to DeleteTracker in ScanQueryMatcher > Solution: > We use smallestReadPoint of region when compaction to keep MVCC for OPENED > scanner, So we should retain delete type kv in output in the case(Already > deleted KV is retained in output to make old opened scanner could read this > KV) even if it is a majorcompaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira