[jira] [Commented] (HDFS-3313) create a protocol for journal service synchronziation
[ https://issues.apache.org/jira/browse/HDFS-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261929#comment-13261929 ] Aaron T. Myers commented on HDFS-3313: -- bq. One refactoring we can do is to remove getEditLogManifes() from namenode protocol and let namenode use JournalSyncProtocol. This is similar as the one Aaron suggested, with the difference that we won't have a protocol supportting only getEditLogManifes() just to reduce dup code. Sure, that'll work until you want to add more methods to the JournalSyncProtocol that don't make sense for the NN to implement. > create a protocol for journal service synchronziation > --- > > Key: HDFS-3313 > URL: https://issues.apache.org/jira/browse/HDFS-3313 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: Shared journals (HDFS-3092) > > Attachments: HDFS-3313.HDFS-3092.patch, HDFS-3313.HDFS-3092.patch > > > This protocol is used to synchronize lagging journal service with active > journal service with complete finalized edit segments. Currently it supports > one rpc call getEditLogManifest() which lists finalized edit segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3313) create a protocol for journal service synchronziation
[ https://issues.apache.org/jira/browse/HDFS-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261926#comment-13261926 ] Brandon Li commented on HDFS-3313: -- One refactoring we can do is to remove getEditLogManifes() from namenode protocol and let namenode use JournalSyncProtocol. This is similar as the one Aaron suggested, with the difference that we won't have a protocol supportting only getEditLogManifes() just to reduce dup code. > create a protocol for journal service synchronziation > --- > > Key: HDFS-3313 > URL: https://issues.apache.org/jira/browse/HDFS-3313 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Reporter: Brandon Li >Assignee: Brandon Li > Fix For: Shared journals (HDFS-3092) > > Attachments: HDFS-3313.HDFS-3092.patch, HDFS-3313.HDFS-3092.patch > > > This protocol is used to synchronize lagging journal service with active > journal service with complete finalized edit segments. Currently it supports > one rpc call getEditLogManifest() which lists finalized edit segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3313) create a protocol for journal service synchronziation
[ https://issues.apache.org/jira/browse/HDFS-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261894#comment-13261894 ] Tsz Wo (Nicholas), SZE commented on HDFS-3313: -- Hi Aaron, your suggestion makes sense but it is not easy to do the refactoring. I actually want to refactor all editlog related stuff. Let's get the journal service implementation done first and then do the refactoring later. > create a protocol for journal service synchronziation > --- > > Key: HDFS-3313 > URL: https://issues.apache.org/jira/browse/HDFS-3313 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-3313.HDFS-3092.patch, HDFS-3313.HDFS-3092.patch > > > This protocol is used to synchronize lagging journal service with active > journal service with complete finalized edit segments. Currently it supports > one rpc call getEditLogManifest() which lists finalized edit segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3313) create a protocol for journal service synchronziation
[ https://issues.apache.org/jira/browse/HDFS-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261190#comment-13261190 ] Aaron T. Myers commented on HDFS-3313: -- bq. As pointed out by Nicholas, the interface of getEditLogManifes() in JournalSyncProtocol should be different(at least adding JournalInfo into the request) with the one in namenode protocol. Why couldn't you send the JournalInfo in all cases, both when calling getEditLogManifest on the NN for checkpointing, and in the journal service? Even with this slight difference, there's still a lot of duplicated code between the two, and it seems a shame to implement two parallel, nearly identical methods in distinct protocols. > create a protocol for journal service synchronziation > --- > > Key: HDFS-3313 > URL: https://issues.apache.org/jira/browse/HDFS-3313 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-3313.HDFS-3092.patch, HDFS-3313.HDFS-3092.patch > > > This protocol is used to synchronize lagging journal service with active > journal service with complete finalized edit segments. Currently it supports > one rpc call getEditLogManifest() which lists finalized edit segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3313) create a protocol for journal service synchronziation
[
https://issues.apache.org/jira/browse/HDFS-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261162#comment-13261162
]
Brandon Li commented on HDFS-3313:
--
{quote}Seems like this patch duplicates a lot of code from
NamenodeProtocol#getEditLogManifest and related translators. Can this perhaps
be factored out into a generic RemoteEditLogProtocol that both the NN and
journal service implement?{quote}
Thanks for the comment.
As pointed out by Nicholas, the interface of getEditLogManifes() in
JournalSyncProtocol should be different(at least adding JournalInfo into the
request) with the one in namenode protocol.
> create a protocol for journal service synchronziation
> ---
>
> Key: HDFS-3313
> URL: https://issues.apache.org/jira/browse/HDFS-3313
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HDFS-3313.HDFS-3092.patch
>
>
> This protocol is used to synchronize lagging journal service with active
> journal service with complete finalized edit segments. Currently it supports
> one rpc call getEditLogManifest() which lists finalized edit segments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3313) create a protocol for journal service synchronziation
[ https://issues.apache.org/jira/browse/HDFS-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260992#comment-13260992 ] Aaron T. Myers commented on HDFS-3313: -- Seems like this patch duplicates a lot of code from NamenodeProtocol#getEditLogManifest and related translators. Can this perhaps be factored out into a generic RemoteEditLogProtocol that both the NN and journal service implement? > create a protocol for journal service synchronziation > --- > > Key: HDFS-3313 > URL: https://issues.apache.org/jira/browse/HDFS-3313 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-3313.HDFS-3092.patch > > > This protocol is used to synchronize lagging journal service with active > journal service with complete finalized edit segments. Currently it supports > one rpc call getEditLogManifest() which lists finalized edit segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3313) create a protocol for journal service synchronziation
[
https://issues.apache.org/jira/browse/HDFS-3313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260122#comment-13260122
]
Tsz Wo (Nicholas), SZE commented on HDFS-3313:
--
Looks good. Some comments:
- We should not change NameNodeProxies. NN is not involved in journal
synchronization.
- For the same reason, the principals in JournalSyncProtocolPB should not be
Namenode.
{code}
+serverPrincipal = DFSConfigKeys.DFS_NAMENODE_USER_NAME_KEY,
+clientPrincipal = DFSConfigKeys.DFS_NAMENODE_USER_NAME_KEY)
{code}
- Use FSEditLog.getEditLogManifest(long fromTxId) instead of FileJournalManager
and don't change FileJournalManager.
- Why DN is related to JournalSyncProtocol? Please check the comments.
{code}
+ * If you are adding/changing DN's interface then you need to change both
this
{code}
- I think we should pass JournalInfo in getEditLogManifest(..) so that it could
first verify the version, namespace id, etc.
- Why the test is commented out?
{code}
- @Test
+ //@Test
public void testHttpServer() throws Exception {
{code}
> create a protocol for journal service synchronziation
> ---
>
> Key: HDFS-3313
> URL: https://issues.apache.org/jira/browse/HDFS-3313
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
>Reporter: Brandon Li
>Assignee: Brandon Li
> Attachments: HDFS-3313.HDFS-3092.patch
>
>
> This protocol is used to synchronize lagging journal service with active
> journal service with complete finalized edit segments. Currently it supports
> one rpc call getEditLogManifest() which lists finalized edit segments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
