[ https://issues.apache.org/jira/browse/HBASE-5569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232349#comment-13232349 ]
Lars Hofhansl commented on HBASE-5569: -------------------------------------- Here's my theory... In ScanQueryMatcher we have this: {code} byte type = kv.getType(); if (kv.isDelete()) { if (!keepDeletedCells) { ... this.deletes.add(bytes, offset, qualLength, timestamp, type); } ... } else if (!this.deletes.isEmpty()) { DeleteResult deleteResult = deletes.isDeleted(bytes, offset, qualLength, timestamp); ... } {code} And in StoreScanner.resetScannerStack {code} // Reset the state of the Query Matcher and set to top row. // Only reset and call setRow if the row changes; avoids confusing the // query matcher if scanning intra-row. ... if ((matcher.row == null) || !kv.matchingRow(matcher.row)) { matcher.reset(); matcher.setRow(kv.getRow()); } {code} So, the SQM might already have a delete registered, or might miss a delete. With KEEP_DELETED_CELLS that race does not happen, because deletes are simply not registered. > Do not collect deleted KVs when they are still in use by a scanner. > ------------------------------------------------------------------- > > Key: HBASE-5569 > URL: https://issues.apache.org/jira/browse/HBASE-5569 > Project: HBase > Issue Type: Bug > Reporter: Lars Hofhansl > Assignee: Lars Hofhansl > Fix For: 0.94.0, 0.96.0 > > Attachments: 5569-v2.txt, 5569.txt, > TestAtomicOperation-output.trunk_120313.rar > > > I noticed this because TestAtomicOperation.testMultiRowMutationMultiThreads > fails rarely. > The solution is similar to HBASE-2856, where expired KVs are not collected > when in use by a scanner. > --- > What I pieced together so far is that it is the *scanning* side that has > problems sometimes. > Every time I see a assertion failure in the log I see this before: > {quote} > 2012-03-12 21:48:49,523 DEBUG [Thread-211] regionserver.StoreScanner(499): > Storescanner.peek() is changed where before = > rowB/colfamily11:qual1/75366/Put/vlen=6,and after = > rowB/colfamily11:qual1/75203/DeleteColumn/vlen=0 > {quote} > The order of if the Put and Delete is sometimes reversed. > The test threads should always see exactly one KV, if the "before" was the > Put the thread see 0 KVs, if the "before" was the Delete the threads see 2 > KVs. > This debug message comes from StoreScanner to checkReseek. It seems we still > some consistency issue with scanning sometimes :( -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira