[jira] [Commented] (HDFS-3229) add JournalProtocol RPCs to list finalized edit segments, and read edit segment file from JournalNode.
[ https://issues.apache.org/jira/browse/HDFS-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249883#comment-13249883 ] Brandon Li commented on HDFS-3229: -- Todd, thanks for the comments! getEditLogManifest() call (or a likely one) could be used to list the finalized logs on a JournalNode. To copy edit segments, we may need to something different here. The proposal is to use the same Journal RPC server to download edit segments instead of resorting HTTP. In that, the JournalNode doesn't have to start an additional HTTP server which could cause some subtle issues and make debugging more difficult. add JournalProtocol RPCs to list finalized edit segments, and read edit segment file from JournalNode. --- Key: HDFS-3229 URL: https://issues.apache.org/jira/browse/HDFS-3229 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Brandon Li Assignee: Brandon Li -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3229) add JournalProtocol RPCs to list finalized edit segments, and read edit segment file from JournalNode.
[ https://issues.apache.org/jira/browse/HDFS-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249929#comment-13249929 ] Todd Lipcon commented on HDFS-3229: --- Can you give an example of the subtle issues you're referring to? The advantage of re-using HTTP is that we've already tested that code path, and it supports things like checksumming, etc. add JournalProtocol RPCs to list finalized edit segments, and read edit segment file from JournalNode. --- Key: HDFS-3229 URL: https://issues.apache.org/jira/browse/HDFS-3229 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Brandon Li Assignee: Brandon Li -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3229) add JournalProtocol RPCs to list finalized edit segments, and read edit segment file from JournalNode.
[ https://issues.apache.org/jira/browse/HDFS-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250249#comment-13250249 ] Brandon Li commented on HDFS-3229: -- Todd, Reusing existing code has certain benefits for sure. Have an HTTP server for JournalNode requires additional port. In a large cluster, it may not be trivial to manager one more port. However, if we believe we need web UI for JournalNode, we need the port anyways. Suppose we used HTTP server to synchronize the lagging JournalNode by downloading missed edit logs from another Journal Node. Firstly, the lagging JN needs to get (e.g., by asking for NN) a list of JNs with full set of edit logs. Then, it downloads the missed logs from a good JN through http, while it could accept streamed logs from NN through rpc at the same time. Given the two servers are working on different file sets(finalized logs vs in-progress log), synchronizing them seems not a concern. Debug-ability in this case has more to do with the developer's familiarity with the code/protocol, and seems not a good enough reason to me now. Please let me know what you think. add JournalProtocol RPCs to list finalized edit segments, and read edit segment file from JournalNode. --- Key: HDFS-3229 URL: https://issues.apache.org/jira/browse/HDFS-3229 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Brandon Li Assignee: Brandon Li -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3229) add JournalProtocol RPCs to list finalized edit segments, and read edit segment file from JournalNode.
[ https://issues.apache.org/jira/browse/HDFS-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250251#comment-13250251 ] Todd Lipcon commented on HDFS-3229: --- bq. However, if we believe we need web UI for JournalNode, we need the port anyways. I think it's a good idea, since we have other endpoints in our default HTTP server that are very useful for ops -- for example the /jmx servlet and the /conf servlet can both be very handy. I also think exposing a basic web UI is helpful to operators who might try to understand the current state of the system. bq. Suppose we used HTTP server to synchronize the lagging JournalNode by downloading missed edit logs from another Journal Node. Firstly, the lagging JN needs to get (e.g., by asking for NN) a list of JNs with full set of edit logs. Then, it downloads the missed logs from a good JN through http, while it could accept streamed logs from NN through rpc at the same time. Given the two servers are working on different file sets(finalized logs vs in-progress log), synchronizing them seems not a concern. Right - this is the same process that the 2NN uses to synchronize finalized log segments from the NN. See SecondaryNameNode.downloadCheckpointFiles for the code. add JournalProtocol RPCs to list finalized edit segments, and read edit segment file from JournalNode. --- Key: HDFS-3229 URL: https://issues.apache.org/jira/browse/HDFS-3229 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Brandon Li Assignee: Brandon Li -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3229) add JournalProtocol RPCs to list finalized edit segments, and read edit segment file from JournalNode.
[ https://issues.apache.org/jira/browse/HDFS-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249183#comment-13249183 ] Todd Lipcon commented on HDFS-3229: --- I'd recommend reusing the code/protobufs for the existing getEditLogManifest() calls that the 2NN uses to transfer logs, here. add JournalProtocol RPCs to list finalized edit segments, and read edit segment file from JournalNode. --- Key: HDFS-3229 URL: https://issues.apache.org/jira/browse/HDFS-3229 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Brandon Li Assignee: Brandon Li -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
