[
https://issues.apache.org/jira/browse/HDFS-13609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277680#comment-17277680
]
xuzq edited comment on HDFS-13609 at 2/3/21, 4:59 AM:
------------------------------------------------------
Give one example to illustrate what I think.
We have 5 journals, like jn1 ~ jn5.
And Active write edits like:
|Txid|SuccessWriteJournalId|FailedJournalId|
|TxId1|jn2, jn3, jn4, jn5|jn1(write into cache, write disk failed)|
|TxId2|jn2, jn3, jn4, jn5| |
|TxId3|jn2, jn3, jn4, jn5| |
|TxId4|jn2, jn3, jn4, jn5| |
|TxId5|jn2, jn3, jn4, jn5| |
When we attempt to failover standby to active, standby need to catchup all
edits from TxId1 ~ TxId5 from TxId1, and change to active.
But before to failover standby to active, jn4 and jn5 have some delay times,
caused responseCounts like (0(jn1), 5(jn2), 5(jn3)) when
_editLogTailer.catchupDuringFailover()._
Standby NameNode expect to get all edits from TxId1 ~ TxId5, but only get
txId1. TxId2 ~ TxId5 don't applied into fsImage.
And it will caused StandbyNameNode cashed when
_getFSImage().editLog.openForWrite()._
I think we should use responseCounts(2) ~ responseCounts(4) to ensure can
catchup all edits.
But the last edit in responseCounts(2) ~ responseCounts(4) maybe is writing by
active, maybe not on a quorum of JNs.
It will cause Obsever NameNode or Standby NameNode tail UnQuorum edits.
Or maybe we can write disk first, then write cache in Journal Node.
[~xkrogen] On this question, if you have some good ideas, please tell me,
thanks.
was (Author: xuzq_zander):
Give one example to illustrate what I think.
We have 5 journals, like jn1 ~ jn5.
And Active write edits like:
|Txid|SuccessWriteJournalId|FailedJournalId|
|TxId1|jn2, jn3, jn4, jn5|jn1(write into cache, write disk failed)|
|TxId2|jn2, jn3, jn4, jn5| |
|TxId3|jn2, jn3, jn4, jn5| |
|TxId4|jn2, jn3, jn4, jn5| |
|TxId5|jn2, jn3, jn4, jn5| |
When we attempt to failover standby to active, standby need to catchup all
edits from TxId1 ~ TxId5 from TxId1, and change to active.
But before to failover standby to active, jn4 and jn5 have some delay times,
caused responseCounts like (0(jn1), 5(jn2), 5(jn3)) when
_editLogTailer.catchupDuringFailover()._
Standby NameNode expect to get all edits from TxId1 ~ TxId5, but only get
txId1. TxId2 ~ TxId5 don't applied into fsImage.
And it will caused StandbyNameNode cashed when
_getFSImage().editLog.openForWrite()._
I think we should use responseCounts(2) ~ responseCounts(4) to ensure can
catchup all edits.
But the last edit in responseCounts(2) ~ responseCounts(4) maybe is writing by
active, maybe not on a quorum of JNs.
It will cause Obsever NameNode or Standby NameNode tail UnQuorum edits.
[~xkrogen] On this question, if you have some good ideas, please tell me,
thanks.
> [Edit Tail Fast Path Pt 3] NameNode-side changes to support tailing edits via
> RPC
> ---------------------------------------------------------------------------------
>
> Key: HDFS-13609
> URL: https://issues.apache.org/jira/browse/HDFS-13609
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, namenode
> Reporter: Erik Krogen
> Assignee: Erik Krogen
> Priority: Major
> Fix For: HDFS-12943, 3.3.0
>
> Attachments: HDFS-13609-HDFS-12943.000.patch,
> HDFS-13609-HDFS-12943.001.patch, HDFS-13609-HDFS-12943.002.patch,
> HDFS-13609-HDFS-12943.003.patch, HDFS-13609-HDFS-12943.004.patch
>
>
> See HDFS-13150 for the full design.
> This JIRA is targetted at the NameNode-side changes to enable tailing
> in-progress edits via the RPC mechanism added in HDFS-13608. Most changes are
> in the QuorumJournalManager.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
