[ 
https://issues.apache.org/jira/browse/HBASE-7006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13646864#comment-13646864
 ] 

stack commented on HBASE-7006:
------------------------------

Some comments on the design doc:

+ Nit: Add author, date, and add issue number so can go back to the hosting 
issue should I trip over the doc w/o any other context.
+ Is your assumption about out-of-order replay of edits new to this feature?  I 
suppose in the old/current way of log splitting, we do stuff in sequenceid 
order because we wrote the recovered.edits files named by sequenceid... so they 
were ordered when the regionserver read them in? We should highlight your 
assumption more.  I think if we move to multiple-WALs we'll want to also take 
on this assumption doing recovery.
+ Given the assumption, we should list the problematic scenarios (or point to 
where we list them already -- I think the 'Current Limitations' section here 
http://hbase.apache.org/book.html#version.delete should have the list we 
currently know).
+ "...check if all WALs of a failed region server have been successfully 
replayed."  How is this done?
+ How will a crashed regionserver "...... and appending itself into the list 
of...": i.e. append itself to list of crashed servers (am I reading this wrong)?

bq. "For each region per failed region server, we stores the last flushed 
sequence Id from the region server before it failed."

This is the mechanism that has the regionserver telling the master its current 
sequenceid everytime it flushes to an hfile?  So when server crashes, master 
writes a znode under the recovering-regions with the last reported seq id?    
if a new regionserver hosting a recovery of regions then crashes, it gets a new 
znode w/ its current sequenceid?  Now we have two crashed servers with 
(probably) two different sequenceids whose logs we are recovering.  The two 
sequenceids are never related right?  They are only applied to the logs of the 
server who passed the particular sequenceid to the master?


Question:  So it looks like we replay the WALs of a crashed regionserver by 
playing them into the new region host servers.  There does not seem to be a 
flush when the replay of the old crashed servers WALs is done.  Is your 
thinking that it is not needed since the old edits are now in the new servers 
WAL?  Would there be any advantage NOT writing the WAL on replay and only when 
done, then flush (I suppose not, thinking about it, and in fact, it would 
probably make replay more complicated since we'd have to have this new 
operation to do; a flush-when-all-WALS-recovered).

Good stuff.
                
> [MTTR] Study distributed log splitting to see how we can make it faster
> -----------------------------------------------------------------------
>
>                 Key: HBASE-7006
>                 URL: https://issues.apache.org/jira/browse/HBASE-7006
>             Project: HBase
>          Issue Type: Bug
>          Components: MTTR
>            Reporter: stack
>            Assignee: Jeffrey Zhong
>            Priority: Critical
>             Fix For: 0.95.1
>
>         Attachments: hbase-7006-combined.patch, hbase-7006-combined-v1.patch, 
> hbase-7006-combined-v2.patch, hbase-7006-combined-v3.patch, LogSplitting 
> Comparison.pdf, 
> ProposaltoimprovelogsplittingprocessregardingtoHBASE-7006-v2.pdf
>
>
> Just saw interesting issue where a cluster went down  hard and 30 nodes had 
> 1700 WALs to replay.  Replay took almost an hour.  It looks like it could run 
> faster that much of the time is spent zk'ing and nn'ing.
> Putting in 0.96 so it gets a look at least.  Can always punt.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Reply via email to