[jira] [Comment Edited] (HDFS-13767) Add msync server implementation.
[ https://issues.apache.org/jira/browse/HDFS-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16564576#comment-16564576 ] Chen Liang edited comment on HDFS-13767 at 8/1/18 12:59 AM: Post WIP.v004 patch. The main changes are: # changed to {{getCorrectLastAppliedOrWrittenTxId}} as Erik suggested. # Add a simple unit test and had some minor change to {{TestObserverNode}}. I was encountering some issues when running test, here are some suspects I had on TestObserverNode: Currently with msync, every single call to Observer needs to catch up state id. While when {{ObserverReadProxyProvider}} is created, it makes {{reportBadBlocks}} and {{checkAccess}} calls. These calls also need to catch up state id. This means every time when {{ObserverReadProxyProvider}} is created there will be a wait... And since every unit test creates a dfs cluster, for every single unit test in {{TestObserverNode}}, there will be such a wait. I could set the period to a small time, but that introduces other issues, because several tests depend on explicitly made edits tailing calls. So I set the periods to a smaller, but not too small number...and tests still quite some time to run. I think the right fix would be to have {{ObserverReadProxyProvider}} not make those calls on being created. But make a new special call that bypasses the catching up (if possible). This is one of those hacky code we are planing to optimize in the future. And also, the ongoing work HDFS-13523 may also simplify the tests. For now, I tend to believe having these slow tests may be fine. was (Author: vagarychen): Post WIP.v004 patch. The main changes are: # changed to {{getCorrectLastAppliedOrWrittenTxId}} as Erik suggested. # Add a simple unit test and had some minor change to {{TestObserverNode}}. I was encountering some issues when running test, here are some suspects I had on TestObserverNode: Currently with msync, every single call to Observer needs to catch up state id. While when {{ObserverReadProxyProvider}} is created, it makes {{reportBadBlocks}} and {{checkAccess}} calls. These calls are also read, so they also need to catch up state id. This means every time when {{ObserverReadProxyProvider}} is created there will be a wait... And since every unit test creates a dfs cluster, for every single unit test in {{TestObserverNode}}, there will be such a wait. I could set the period to a small time, but that introduces other issues, because several tests depend on explicitly made edits tailing calls. So I set the periods to a smaller, but not too small number...and tests still quite some time to run. I think the right fix would be to have {{ObserverReadProxyProvider}} not make those read calls on being created. This is one of those hacky code we are planing to optimize in the future. And also, the ongoing work HDFS-13523 may also simplify the tests. For now, I tend to believe having these slow tests may be fine. > Add msync server implementation. > > > Key: HDFS-13767 > URL: https://issues.apache.org/jira/browse/HDFS-13767 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-13767.WIP.001.patch, HDFS-13767.WIP.002.patch, > HDFS-13767.WIP.003.patch, HDFS-13767.WIP.004.patch > > > This is a followup on HDFS-13688, where msync API is introduced to > {{ClientProtocol}} but the server side implementation is missing. This is > Jira is to implement the server side logic. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13767) Add msync server implementation.
[ https://issues.apache.org/jira/browse/HDFS-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568674#comment-16568674 ] Plamen Jeliazkov edited comment on HDFS-13767 at 8/3/18 7:44 PM: - Hey Chen, Change looks pretty good so far; two things. (1) I like the idea of splitting AlignmentContext into client and server interfaces. I think originally Konstantin wanted just a single interface but I think its clear there is a difference. I would not mind picking up that JIRA if we want to go ahead with it. This would let us allow the server-side version of AlignmentContext to better handle defering / re-queue'ing calls. (2) A concern about the unit test -- I tried to add a 10 second sleep between the thread start and the assert: {code:java} assertFalse(readSucceed.get()); {code} However I found that the test would fail if I did. Which makes me question the unit test. I would expect the 'getFileStatus' to basically hang until the _rollEditLogAndTail _call but seems it is updating anyway. I tried also stopping the _EditLogTailer_ on the ObserverNode but it still updated anyway. Please let me know if I am missing something about this test; I will look further. was (Author: zero45): Hey Chen, Change looks pretty good so far; two things. (1) I like the idea of splitting AlignmentContext into client and server interfaces. I think originally Konstantin wanted just a single interface but I think its clear there is a difference. I would not mind picking up that JIRA if we want to go ahead with it. This would let us allow the server-side version of AlignmentContext to better handle defering / re-queue'ing calls. (2) A concern about the unit test -- I tried to add a 10 second sleep between the thread start and the assert: {code:java} assertFalse(readSucceed.get()); {code} However I found that the test would fail if I did. Which makes me question the unit test. I would expect the 'getFileStatus' to basically hang until the {{{rollEditLogAndTail}}} call but seems it is updating anyway. I tried also stopping the {{{EditLogTailer}}} on the ObserverNode but it still updated anyway. Please let me know if I am missing something about this test; I will look further. > Add msync server implementation. > > > Key: HDFS-13767 > URL: https://issues.apache.org/jira/browse/HDFS-13767 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-13767-HDFS-12943.001.patch, > HDFS-13767-HDFS-12943.002.patch, HDFS-13767.WIP.001.patch, > HDFS-13767.WIP.002.patch, HDFS-13767.WIP.003.patch, HDFS-13767.WIP.004.patch > > > This is a followup on HDFS-13688, where msync API is introduced to > {{ClientProtocol}} but the server side implementation is missing. This is > Jira is to implement the server side logic. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13767) Add msync server implementation.
[ https://issues.apache.org/jira/browse/HDFS-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568674#comment-16568674 ] Plamen Jeliazkov edited comment on HDFS-13767 at 8/3/18 7:44 PM: - Hey Chen, Change looks pretty good so far; two things. (1) I like the idea of splitting AlignmentContext into client and server interfaces. I think originally Konstantin wanted just a single interface but I think its clear there is a difference. I would not mind picking up that JIRA if we want to go ahead with it. This would let us allow the server-side version of AlignmentContext to better handle defering / re-queue'ing calls. (2) A concern about the unit test -- I tried to add a 10 second sleep between the thread start and the assert: {code:java} assertFalse(readSucceed.get()); {code} However I found that the test would fail if I did. Which makes me question the unit test. I would expect the 'getFileStatus' to basically hang until the _rollEditLogAndTail_ call but seems it is updating anyway. I tried also stopping the _EditLogTailer_ on the ObserverNode but it still updated anyway. Please let me know if I am missing something about this test; I will look further. was (Author: zero45): Hey Chen, Change looks pretty good so far; two things. (1) I like the idea of splitting AlignmentContext into client and server interfaces. I think originally Konstantin wanted just a single interface but I think its clear there is a difference. I would not mind picking up that JIRA if we want to go ahead with it. This would let us allow the server-side version of AlignmentContext to better handle defering / re-queue'ing calls. (2) A concern about the unit test -- I tried to add a 10 second sleep between the thread start and the assert: {code:java} assertFalse(readSucceed.get()); {code} However I found that the test would fail if I did. Which makes me question the unit test. I would expect the 'getFileStatus' to basically hang until the _rollEditLogAndTail _call but seems it is updating anyway. I tried also stopping the _EditLogTailer_ on the ObserverNode but it still updated anyway. Please let me know if I am missing something about this test; I will look further. > Add msync server implementation. > > > Key: HDFS-13767 > URL: https://issues.apache.org/jira/browse/HDFS-13767 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-13767-HDFS-12943.001.patch, > HDFS-13767-HDFS-12943.002.patch, HDFS-13767.WIP.001.patch, > HDFS-13767.WIP.002.patch, HDFS-13767.WIP.003.patch, HDFS-13767.WIP.004.patch > > > This is a followup on HDFS-13688, where msync API is introduced to > {{ClientProtocol}} but the server side implementation is missing. This is > Jira is to implement the server side logic. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13767) Add msync server implementation.
[ https://issues.apache.org/jira/browse/HDFS-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568674#comment-16568674 ] Plamen Jeliazkov edited comment on HDFS-13767 at 8/3/18 7:56 PM: - Hey Chen, Change looks pretty good so far; couple things. (1) I like the idea of splitting AlignmentContext into client and server interfaces. I think originally Konstantin wanted just a single interface but I think its clear there is a difference. I would not mind picking up that JIRA if we want to go ahead with it. This would let us allow the server-side version of AlignmentContext to better handle defering / re-queue'ing calls. (2) A concern about the unit test -- I tried to add a 10 second sleep between the thread start and the assert: {code:java} assertFalse(readSucceed.get()); {code} However I found that the test would fail if I did. Which makes me question the unit test. I would expect the 'getFileStatus' to basically hang until the _rollEditLogAndTail_ call but seems it is updating anyway. I tried also stopping the _EditLogTailer_ on the ObserverNode but it still updated anyway. Please let me know if I am missing something about this test; I will look further. (3) So far these patches don't actually implement _NameNodeRpcServer.msync()_. Should we rename it? (4) [~xkrogen], [~shv], and others -- I was curios about making msync pause at the server level. What if we had msync hang at the client side instead? This way we aren't impacting Observer queues with many deferred calls. We could simply have the client check up with the Observers' states and so rather than re-queue'ing we throw a StandbyException (or something) and force the client to maybe pause and retry the msync call on the same or a different Observer. If the msync call succeeds then we know we can use the particular Observer that it succeeded on. Thoughts? was (Author: zero45): Hey Chen, Change looks pretty good so far; two things. (1) I like the idea of splitting AlignmentContext into client and server interfaces. I think originally Konstantin wanted just a single interface but I think its clear there is a difference. I would not mind picking up that JIRA if we want to go ahead with it. This would let us allow the server-side version of AlignmentContext to better handle defering / re-queue'ing calls. (2) A concern about the unit test -- I tried to add a 10 second sleep between the thread start and the assert: {code:java} assertFalse(readSucceed.get()); {code} However I found that the test would fail if I did. Which makes me question the unit test. I would expect the 'getFileStatus' to basically hang until the _rollEditLogAndTail_ call but seems it is updating anyway. I tried also stopping the _EditLogTailer_ on the ObserverNode but it still updated anyway. Please let me know if I am missing something about this test; I will look further. > Add msync server implementation. > > > Key: HDFS-13767 > URL: https://issues.apache.org/jira/browse/HDFS-13767 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-13767-HDFS-12943.001.patch, > HDFS-13767-HDFS-12943.002.patch, HDFS-13767.WIP.001.patch, > HDFS-13767.WIP.002.patch, HDFS-13767.WIP.003.patch, HDFS-13767.WIP.004.patch > > > This is a followup on HDFS-13688, where msync API is introduced to > {{ClientProtocol}} but the server side implementation is missing. This is > Jira is to implement the server side logic. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13767) Add msync server implementation.
[ https://issues.apache.org/jira/browse/HDFS-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568674#comment-16568674 ] Plamen Jeliazkov edited comment on HDFS-13767 at 8/3/18 7:57 PM: - Hey Chen, Change looks pretty good so far; couple things. (1) I like the idea of splitting AlignmentContext into client and server interfaces. I think originally Konstantin wanted just a single interface but I think its clear there is a difference. I would not mind picking up that JIRA if we want to go ahead with it. This would let us allow the server-side version of AlignmentContext to better handle defering / re-queue'ing calls. (2) A concern about the unit test -- I tried to add a 10 second sleep between the thread start and the assert: {code:java} assertFalse(readSucceed.get()); {code} However I found that the test would fail if I did. Which makes me question the unit test. I would expect the 'getFileStatus' to basically hang until the _rollEditLogAndTail_ call but seems it is updating anyway. I tried also stopping the _EditLogTailer_ on the ObserverNode but it still updated anyway. Please let me know if I am missing something about this test; I will look further. (3) So far these patches don't actually implement _NameNodeRpcServer.msync()_. Should we rename it? (4) [~xkrogen], [~shv], and others -- I was curious about making msync pause at the server level. What if we had msync hang at the client side instead? This way we aren't impacting Observer queues with many deferred calls. We could simply have the client check up with the Observers' states and so rather than re-queue'ing we throw a StandbyException (or something) and force the client to maybe pause and retry the msync call on the same or a different Observer. If the msync call succeeds then we know we can use the particular Observer that it succeeded on. Thoughts? was (Author: zero45): Hey Chen, Change looks pretty good so far; couple things. (1) I like the idea of splitting AlignmentContext into client and server interfaces. I think originally Konstantin wanted just a single interface but I think its clear there is a difference. I would not mind picking up that JIRA if we want to go ahead with it. This would let us allow the server-side version of AlignmentContext to better handle defering / re-queue'ing calls. (2) A concern about the unit test -- I tried to add a 10 second sleep between the thread start and the assert: {code:java} assertFalse(readSucceed.get()); {code} However I found that the test would fail if I did. Which makes me question the unit test. I would expect the 'getFileStatus' to basically hang until the _rollEditLogAndTail_ call but seems it is updating anyway. I tried also stopping the _EditLogTailer_ on the ObserverNode but it still updated anyway. Please let me know if I am missing something about this test; I will look further. (3) So far these patches don't actually implement _NameNodeRpcServer.msync()_. Should we rename it? (4) [~xkrogen], [~shv], and others -- I was curios about making msync pause at the server level. What if we had msync hang at the client side instead? This way we aren't impacting Observer queues with many deferred calls. We could simply have the client check up with the Observers' states and so rather than re-queue'ing we throw a StandbyException (or something) and force the client to maybe pause and retry the msync call on the same or a different Observer. If the msync call succeeds then we know we can use the particular Observer that it succeeded on. Thoughts? > Add msync server implementation. > > > Key: HDFS-13767 > URL: https://issues.apache.org/jira/browse/HDFS-13767 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-13767-HDFS-12943.001.patch, > HDFS-13767-HDFS-12943.002.patch, HDFS-13767.WIP.001.patch, > HDFS-13767.WIP.002.patch, HDFS-13767.WIP.003.patch, HDFS-13767.WIP.004.patch > > > This is a followup on HDFS-13688, where msync API is introduced to > {{ClientProtocol}} but the server side implementation is missing. This is > Jira is to implement the server side logic. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13767) Add msync server implementation.
[ https://issues.apache.org/jira/browse/HDFS-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568674#comment-16568674 ] Plamen Jeliazkov edited comment on HDFS-13767 at 8/3/18 9:22 PM: - Hey Chen, Change looks pretty good so far; couple things. (1) I like the idea of splitting AlignmentContext into client and server interfaces. I think originally Konstantin wanted just a single interface but I think its clear there is a difference. I would not mind picking up that JIRA if we want to go ahead with it. This would let us allow the server-side version of AlignmentContext to better handle defering / re-queue'ing calls. (2) A concern about the unit test -- I tried to add a 10 second sleep between the thread start and the assert: {code:java} assertFalse(readSucceed.get()); {code} However I found that the test would fail if I did. Which makes me question the unit test. I would expect the 'getFileStatus' to basically hang until the _rollEditLogAndTail_ call but seems it is updating anyway. I tried also stopping the _EditLogTailer_ on the ObserverNode but it still updated anyway. Please let me know if I am missing something about this test; I will look further. (3) So far these patches don't actually implement _NameNodeRpcServer.msync()_. Should we re-title the JIRA? (4) [~xkrogen], [~shv], and others -- I was curious about making msync pause at the server level. What if we had msync hang at the client side instead? This way we aren't impacting Observer queues with many deferred calls. We could simply have the client check up with the Observers' states and so rather than re-queue'ing we throw a StandbyException (or something) and force the client to maybe pause and retry the msync call on the same or a different Observer. If the msync call succeeds then we know we can use the particular Observer that it succeeded on. Thoughts? was (Author: zero45): Hey Chen, Change looks pretty good so far; couple things. (1) I like the idea of splitting AlignmentContext into client and server interfaces. I think originally Konstantin wanted just a single interface but I think its clear there is a difference. I would not mind picking up that JIRA if we want to go ahead with it. This would let us allow the server-side version of AlignmentContext to better handle defering / re-queue'ing calls. (2) A concern about the unit test -- I tried to add a 10 second sleep between the thread start and the assert: {code:java} assertFalse(readSucceed.get()); {code} However I found that the test would fail if I did. Which makes me question the unit test. I would expect the 'getFileStatus' to basically hang until the _rollEditLogAndTail_ call but seems it is updating anyway. I tried also stopping the _EditLogTailer_ on the ObserverNode but it still updated anyway. Please let me know if I am missing something about this test; I will look further. (3) So far these patches don't actually implement _NameNodeRpcServer.msync()_. Should we rename it? (4) [~xkrogen], [~shv], and others -- I was curious about making msync pause at the server level. What if we had msync hang at the client side instead? This way we aren't impacting Observer queues with many deferred calls. We could simply have the client check up with the Observers' states and so rather than re-queue'ing we throw a StandbyException (or something) and force the client to maybe pause and retry the msync call on the same or a different Observer. If the msync call succeeds then we know we can use the particular Observer that it succeeded on. Thoughts? > Add msync server implementation. > > > Key: HDFS-13767 > URL: https://issues.apache.org/jira/browse/HDFS-13767 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-13767-HDFS-12943.001.patch, > HDFS-13767-HDFS-12943.002.patch, HDFS-13767.WIP.001.patch, > HDFS-13767.WIP.002.patch, HDFS-13767.WIP.003.patch, HDFS-13767.WIP.004.patch > > > This is a followup on HDFS-13688, where msync API is introduced to > {{ClientProtocol}} but the server side implementation is missing. This is > Jira is to implement the server side logic. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13767) Add msync server implementation.
[ https://issues.apache.org/jira/browse/HDFS-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568843#comment-16568843 ] Chen Liang edited comment on HDFS-13767 at 8/3/18 10:12 PM: Thanks for comments [~zero45]. bq. (2) A concern about the unit test... Thanks for trying this. I think it might be related to the recent update that enabled in-progress edit tail. Because if I disable in-progress edit tail for this test, then add the delay you mentioned, the test will succeed, but took longer (expected). I will need to look deeper to see how in-progress edit tail worked with this. [~xkrogen] any fast insights on this? bq. (3) So far these patches don't actually ... The only reason there is msync in {{NameNodeRpcServer}} is because msync is part of {{ClientProtocol}}, and NameNodeRpcServer implements it. So changing this in NameNodeRpcServer won't compile. I guess msync is a little bit different from other RPC calls, in that from client side, it's another RPC call like others, but from server side, all it's logic is handled in RPC layer, not in NN layer at all. was (Author: vagarychen): Thanks for comments [~zero45]. bq. (2) A concern about the unit test... Thanks for trying this. I think it might be related to the recent update that enabled in-progress edit tail. Because if I disable in-progress edit tail for this test, then add the delay you mentioned, the test will succeed, but took longer (expected). I will need to look deeper to see how in-progress edit tail worked with this. [~xkrogen] any fast insights on this? bq. (3) So far these patches don't actually ... The only reason there is msync in {{NameNodeRpcServer}} is because msync is part of {{ClientProtocol}}, and NameNodeRpcServer implements it. So changing this in NameNodeRpcServer won't compile. I guess msync is a little bit different from other RPC calls, in that from client side, it's another RPC call like others, but from server side, all it's logic is handled in RPC layer, not in NN layer at all. bq. (4) I was curious about making msync pause ... Just some thoughts of mine: it is true that there will be no deferred call in this case, but seems to me this will also add retry msync calls, so I don't see a particular advantage here in terms of call queue load. Actually it maybe even worse: if client times out and retries, the previous call could still be in the server call queue? In which case we could have multiple msync call in the server call queue, many clients doing this msync retry could fill up the queue unnecessarily. While if the client retry interval is long, it can easily be just wasting time. > Add msync server implementation. > > > Key: HDFS-13767 > URL: https://issues.apache.org/jira/browse/HDFS-13767 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-13767-HDFS-12943.001.patch, > HDFS-13767-HDFS-12943.002.patch, HDFS-13767.WIP.001.patch, > HDFS-13767.WIP.002.patch, HDFS-13767.WIP.003.patch, HDFS-13767.WIP.004.patch > > > This is a followup on HDFS-13688, where msync API is introduced to > {{ClientProtocol}} but the server side implementation is missing. This is > Jira is to implement the server side logic. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13767) Add msync server implementation.
[ https://issues.apache.org/jira/browse/HDFS-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16568843#comment-16568843 ] Chen Liang edited comment on HDFS-13767 at 8/3/18 10:21 PM: Thanks for comments [~zero45]. bq. (2) A concern about the unit test... Thanks for trying this. I think it might be related to the recent update that enabled in-progress edit tail. Because if I disable in-progress edit tail for this test, then add the delay you mentioned, the test will succeed, but took longer (expected). I will need to look deeper to see how in-progress edit tail worked with this. [~xkrogen] any fast insights on this? bq. (3) So far these patches don't actually ... The only reason there is msync in {{NameNodeRpcServer}} is because msync is part of {{ClientProtocol}}, and NameNodeRpcServer implements it. So changing this in NameNodeRpcServer won't compile. I guess msync is a little bit different from other RPC calls, in that from client side, it's another RPC call like others, but from server side, all it's logic is handled in RPC layer, not in NN layer at all. bq. (4) I was curious about making msync pause ... Just some thoughts of mine: it is true that there will be no deferred call in this case, but seems to me this will also add retry msync calls, so I don't see a particular advantage here in terms of call queue load. It seems to me it is actually the same as deferring the call, the only difference is whether let the client send another call, or just server viewing the existing call as a new one (by putting it back to queue). Actually, maybe we can have a mixed strategy, we did have some discussion offline, that if server sees itself too far behind when checking the call, don't re-queue, but just throw exception so that client don't bother to wait and can try a different Observer. I viewed this as an optimization. was (Author: vagarychen): Thanks for comments [~zero45]. bq. (2) A concern about the unit test... Thanks for trying this. I think it might be related to the recent update that enabled in-progress edit tail. Because if I disable in-progress edit tail for this test, then add the delay you mentioned, the test will succeed, but took longer (expected). I will need to look deeper to see how in-progress edit tail worked with this. [~xkrogen] any fast insights on this? bq. (3) So far these patches don't actually ... The only reason there is msync in {{NameNodeRpcServer}} is because msync is part of {{ClientProtocol}}, and NameNodeRpcServer implements it. So changing this in NameNodeRpcServer won't compile. I guess msync is a little bit different from other RPC calls, in that from client side, it's another RPC call like others, but from server side, all it's logic is handled in RPC layer, not in NN layer at all. > Add msync server implementation. > > > Key: HDFS-13767 > URL: https://issues.apache.org/jira/browse/HDFS-13767 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Attachments: HDFS-13767-HDFS-12943.001.patch, > HDFS-13767-HDFS-12943.002.patch, HDFS-13767.WIP.001.patch, > HDFS-13767.WIP.002.patch, HDFS-13767.WIP.003.patch, HDFS-13767.WIP.004.patch > > > This is a followup on HDFS-13688, where msync API is introduced to > {{ClientProtocol}} but the server side implementation is missing. This is > Jira is to implement the server side logic. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org