[ https://issues.apache.org/jira/browse/HBASE-10958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-10958: --------------------------------------- Attachment: HBASE-10958-less-intrusive-hack-0.96.patch Here's a less intrusive hack that shows what it looks like to request an asynchronous flush from {{bulkLoadHFiles}}. I also tested that it works like the previous patch. The benefits are that the bulk loader doesn't have to wait and it's a lot less code, but the blindspot is still there and it widens as the region server gets more load; for example, if a lot of flushing is happening, which is likely with this patch, then the flush requests are queued and a compacting can kick in and at that moment if the region server dies then data is lost. > [dataloss] Bulk loading with seqids can prevent some log entries from being > replayed > ------------------------------------------------------------------------------------ > > Key: HBASE-10958 > URL: https://issues.apache.org/jira/browse/HBASE-10958 > Project: HBase > Issue Type: Bug > Affects Versions: 0.96.2, 0.98.1, 0.94.18 > Reporter: Jean-Daniel Cryans > Priority: Blocker > Fix For: 0.99.0, 0.94.19, 0.98.2, 0.96.3 > > Attachments: HBASE-10958-less-intrusive-hack-0.96.patch, > HBASE-10958-quick-hack-0.96.patch > > > We found an issue with bulk loads causing data loss when assigning sequence > ids (HBASE-6630) that is triggered when replaying recovered edits. We're > nicknaming this issue *Blindspot*. > The problem is that the sequence id given to a bulk loaded file is higher > than those of the edits in the region's memstore. When replaying recovered > edits, the rule to skip some of them is that they have to be _lower than the > highest sequence id_. In other words, the edits that have a sequence id lower > than the highest one in the store files *should* have also been flushed. This > is not the case with bulk loaded files since we now have an HFile with a > sequence id higher than unflushed edits. > The log recovery code takes this into account by simply skipping the bulk > loaded files, but this "bulk loaded status" is *lost* on compaction. The > edits in the logs that have a sequence id lower than the bulk loaded file > that got compacted are put in a blind spot and are skipped during replay. > Here's the easiest way to recreate this issue: > - Create an empty table > - Put one row in it (let's say it gets seqid 1) > - Bulk load one file (it gets seqid 2). I used ImporTsv and set > hbase.mapreduce.bulkload.assign.sequenceNumbers. > - Bulk load a second file the same way (it gets seqid 3). > - Major compact the table (the new file has seqid 3 and isn't considered > bulk loaded). > - Kill the region server that holds the table's region. > - Scan the table once the region is made available again. The first row, at > seqid 1, will be missing since the HFile with seqid 3 makes us believe that > everything that came before it was flushed. -- This message was sent by Atlassian JIRA (v6.2#6252)