[
https://issues.apache.org/jira/browse/HBASE-10958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13965642#comment-13965642
]
Jean-Daniel Cryans commented on HBASE-10958:
--------------------------------------------
One workaround we found is to completely disable compactions, then when you
need to run them you have to force flush the regions that have bulk loaded file
first and ensure that bulk loads aren't coming in at the same time.
Workloads that are strictly doing incremental bulk loads aren't affected, you
need a mix of bulk loaded files and normal Puts.
A hacky solution could be to force flush when bulk loading with seqids and grab
the next sequence id that comes after the memstore flush to go to the bulk
loaded file. This means that bulk loading needs to initiate a flush, get the
sequence id under the region write lock, then do the bulk load. We don't need
to wait for the flush to happen... unless the possibility for the bulk loaded
file to be compacted before the flush is done is high enough.
> [dataloss] Bulk loading with seqids can prevent some log entries from being
> replayed
> ------------------------------------------------------------------------------------
>
> Key: HBASE-10958
> URL: https://issues.apache.org/jira/browse/HBASE-10958
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.96.2, 0.98.1, 0.94.18
> Reporter: Jean-Daniel Cryans
> Priority: Blocker
> Fix For: 0.99.0, 0.94.19, 0.98.2, 0.96.3
>
>
> We found an issue with bulk loads causing data loss when assigning sequence
> ids (HBASE-6630) that is triggered when replaying recovered edits. We're
> nicknaming this issue *Blindspot*.
> The problem is that the sequence id given to a bulk loaded file is higher
> than those of the edits in the region's memstore. When replaying recovered
> edits, the rule to skip some of them is that they have to be _lower than the
> highest sequence id_. In other words, the edits that have a sequence id lower
> than the highest one in the store files *should* have also been flushed. This
> is not the case with bulk loaded files since we now have an HFile with a
> sequence id higher than unflushed edits.
> The log recovery code takes this into account by simply skipping the bulk
> loaded files, but this "bulk loaded status" is *lost* on compaction. The
> edits in the logs that have a sequence id lower than the bulk loaded file
> that got compacted are put in a blind spot and are skipped during replay.
> Here's the easiest way to recreate this issue:
> - Create an empty table
> - Put one row in it (let's say it gets seqid 1)
> - Bulk load one file (it gets seqid 2). I used ImporTsv and set
> hbase.mapreduce.bulkload.assign.sequenceNumbers.
> - Bulk load a second file the same way (it gets seqid 3).
> - Major compact the table (the new file has seqid 3 and isn't considered
> bulk loaded).
> - Kill the region server that holds the table's region.
> - Scan the table once the region is made available again. The first row, at
> seqid 1, will be missing since the HFile with seqid 3 makes us believe that
> everything that came before it was flushed.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
