[ https://issues.apache.org/jira/browse/HBASE-16232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394195#comment-15394195 ]
Mikhail Antonov commented on HBASE-16232: ----------------------------------------- With enough regions in the cluster overall (up to high tens thousands, though most were just empty) I got one failed run with lost keys on 1.2.1. > ITBLL fails on branch-1.3, now loosing actual keys > -------------------------------------------------- > > Key: HBASE-16232 > URL: https://issues.apache.org/jira/browse/HBASE-16232 > Project: HBase > Issue Type: Bug > Components: dataloss, integration tests > Affects Versions: 1.3.0 > Reporter: Mikhail Antonov > Assignee: Mikhail Antonov > Priority: Blocker > > So I'm running ITBLL off branch-1.3 on recent commit (after [~stack]'s fix > for fake keys showing up in the scans) with increased number of regions per > regionserver and seeing the following. > {quote} > $Verify$Counts > REFERENCED 0 4,999,999,994 4,999,999,994 > UNDEFINED 0 3 3 > UNREFERENCED 0 3 3 > {quote} > So we're loosing some keys. This time those aren't fake: > {quote} > undef > \x89\x10\xE0\xBBx\xF1\xC4\xBAY`\xC4\xD77\x87\x84\x0F 0 1 1 > \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE3 0 1 1 > \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A 0 1 1 > unref > \x15\x1F*f\x92i6\x86\x1D\x8E\xB7\xE1\xC1=\x96\xEF 0 1 1 > \xF4G\xC6E\xD6\xF1\xAB\xB7\xDB\xC0\x94\xF2\xE7mN\xEC 0 1 1 > U\x0F'\x88\x106\x19\x1C\x87Y"\xF3\xE6\xC1\xC8\x15 > {quote} > Re-running verify step with CM off still shows this issue. Search tool > reports: > {quote} > Total > \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE3 5 0 5 > \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A 4 0 4 > CELL_WITH_MISSING_ROW 15 0 15 > {quote} > Will post more as I dig into. -- This message was sent by Atlassian JIRA (v6.3.4#6332)