[
https://issues.apache.org/jira/browse/HBASE-13811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14569443#comment-14569443
]
stack commented on HBASE-13811:
-------------------------------
This has taken me a while. Issue is accounting of per-store sequenceids added
by the flush-by-column-family feature. This is enabled in branch-1, which is
what I am testing.
Issue happens if RS crashes when in the middle of a flush. What I am seeing is
that on split of the WALs subsequent to the crash, we are skipping in essence a
whole hfile of edits -- the hfile that would have been written out by the
crash-curtailed flush.
Happens, of course, only when a bit of scale involved... so chaos monkey has
chance to bring on the condition (happens about 1 in 3 runs).
After a couple of false trails, I now suspect this bit of code in FSHLog:
private void updateOldestUnflushedSequenceIds(byte[] encodedRegionName,
Set<byte[]> familyNameSet, Long lRegionSequenceId) {
ConcurrentMap<byte[], Long> m =
getOrCreateOldestUnflushedStoreSequenceIdsOfRegion(encodedRegionName);
for (byte[] familyName : familyNameSet) {
m.putIfAbsent(familyName, lRegionSequenceId);
}
}
When we go to flush, we REMOVE from the map of oldest outstanding sequenceids,
all reference to those we are about to flush -- we move them to a new map which
is used when the flush completes doing cleanup...
If the above code block runs -- it is called on each append to WAL -- and it
finds an empty map for oldest sequence ids, it will add the current
sequenceid... which will likely be far in excess of what the actual oldest
sequence id is.
If the heartbeat runs to report these new sequenceids to the master just before
we crash, then master will have these advanced seqids and they will be used
when split WAL runs.
Let me prove above is for sure the problem.
> Splitting WALs, we are filtering out too many edits -> DATALOSS
> ---------------------------------------------------------------
>
> Key: HBASE-13811
> URL: https://issues.apache.org/jira/browse/HBASE-13811
> Project: HBase
> Issue Type: Bug
> Components: wal
> Reporter: stack
> Priority: Critical
>
> I've been running ITBLLs against branch-1 around HBASE-13616 (move of
> ServerShutdownHandler to pv2). I have come across an instance of dataloss. My
> patch for HBASE-13616 was in place so can only think it the cause (but cannot
> see how). When we split the logs, we are skipping legit edits. Digging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
