[jira] [Comment Edited] (HDDS-1868) Ozone pipelines should be marked as ready only after the leader election is complete
[ https://issues.apache.org/jira/browse/HDDS-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16939004#comment-16939004 ] Siddharth Wagle edited comment on HDDS-1868 at 9/26/19 10:31 PM: - HI [~ljain] can you take a look at 5, addressed all other comments except the _notifyLeader_ impl, IMO we don't require that. Thanks. was (Author: swagle): HI [~ljain] cna you take a look at 5, addressed all other comments except the _notifyLeader_ impl, IMO we don't require that. Thanks. > Ozone pipelines should be marked as ready only after the leader election is > complete > > > Key: HDDS-1868 > URL: https://issues.apache.org/jira/browse/HDDS-1868 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Siddharth Wagle >Priority: Major > Fix For: 0.5.0 > > Attachments: HDDS-1868.01.patch, HDDS-1868.02.patch, > HDDS-1868.03.patch, HDDS-1868.04.patch, HDDS-1868.05.patch > > > Ozone pipeline on restart start in allocated state, they are moved into open > state after all the pipeline have reported to it. However this potentially > can lead into an issue where the pipeline is still not ready to accept any > incoming IO operations. > The pipelines should be marked as ready only after the leader election is > complete and leader is ready to accept incoming IO. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1868) Ozone pipelines should be marked as ready only after the leader election is complete
[ https://issues.apache.org/jira/browse/HDDS-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937952#comment-16937952 ] Siddharth Wagle edited comment on HDDS-1868 at 9/25/19 6:00 PM: Hi [~ljain], regarding 3., the _setLeaderId_ is called for leader as well so doing anything for _notifyLeader_ is not needed, right? Agree to trigger HB right away. 4. Good catch, I missed the multi-raft point. " Can we keep it simple so that we call pipeline.reportDatanode(dn) once a pipeline report with leaderId set is received?" -> Not sure I follow, when we receive HB from 3rd DN with leaderID we need to open pipeline, so we do need to maintain PipelineID -> LeaderIDSet map to know if all 3 reported. That's what you meant right? was (Author: swagle): Hi [~ljain], regarding 3., the setLeaderId is called for leader as well so doing anything for _notifyLeader_ is not needed, right? Agree to trigger HB right away. 4. Good catch, I missed the multi-raft point. " Can we keep it simple so that we call pipeline.reportDatanode(dn) once a pipeline report with leaderId set is received?" -> Not sure I follow, when we receive HB from 3rd DN with leaderID we need to open pipeline, so we do need to maintain PipelineID -> LeaderIDSet map to know if all 3 reported. That's what you meant right? > Ozone pipelines should be marked as ready only after the leader election is > complete > > > Key: HDDS-1868 > URL: https://issues.apache.org/jira/browse/HDDS-1868 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Siddharth Wagle >Priority: Major > Fix For: 0.5.0 > > Attachments: HDDS-1868.01.patch, HDDS-1868.02.patch, > HDDS-1868.03.patch, HDDS-1868.04.patch > > > Ozone pipeline on restart start in allocated state, they are moved into open > state after all the pipeline have reported to it. However this potentially > can lead into an issue where the pipeline is still not ready to accept any > incoming IO operations. > The pipelines should be marked as ready only after the leader election is > complete and leader is ready to accept incoming IO. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1868) Ozone pipelines should be marked as ready only after the leader election is complete
[ https://issues.apache.org/jira/browse/HDDS-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937952#comment-16937952 ] Siddharth Wagle edited comment on HDDS-1868 at 9/25/19 6:00 PM: Hi [~ljain], regarding 3., the setLeaderId is called for leader as well so doing anything for _notifyLeader_ is not needed, right? Agree to trigger HB right away. 4. Good catch, I missed the multi-raft point. " Can we keep it simple so that we call pipeline.reportDatanode(dn) once a pipeline report with leaderId set is received?" -> Not sure I follow, when we receive HB from 3rd DN with leaderID we need to open pipeline, so we do need to maintain PipelineID -> LeaderIDSet map to know if all 3 reported. That's what you meant right? was (Author: swagle): Hi [~ljain], regarding 3., the setLeaderId is called for leader as well so _notifyLeader_ is not needed, right? Agree to trigger HB right away. 4. Good catch, I missed the multi-raft point. " Can we keep it simple so that we call pipeline.reportDatanode(dn) once a pipeline report with leaderId set is received?" -> Not sure I follow, when we receive HB from 3rd DN with leaderID we need to open pipeline, so we do need to maintain PipelineID -> LeaderIDSet map to know if all 3 reported. That's what you meant right? > Ozone pipelines should be marked as ready only after the leader election is > complete > > > Key: HDDS-1868 > URL: https://issues.apache.org/jira/browse/HDDS-1868 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Siddharth Wagle >Priority: Major > Fix For: 0.5.0 > > Attachments: HDDS-1868.01.patch, HDDS-1868.02.patch, > HDDS-1868.03.patch, HDDS-1868.04.patch > > > Ozone pipeline on restart start in allocated state, they are moved into open > state after all the pipeline have reported to it. However this potentially > can lead into an issue where the pipeline is still not ready to accept any > incoming IO operations. > The pipelines should be marked as ready only after the leader election is > complete and leader is ready to accept incoming IO. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1868) Ozone pipelines should be marked as ready only after the leader election is complete
[ https://issues.apache.org/jira/browse/HDDS-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936374#comment-16936374 ] Siddharth Wagle edited comment on HDDS-1868 at 9/24/19 4:47 AM: [~ljain] Can you please review new changes in 04 this patch and the RATIS one? I will write a unit test thereafter. was (Author: swagle): [~ljain] Can you please review new changes in this patch and the RATIS one? I will write a unit test thereafter. > Ozone pipelines should be marked as ready only after the leader election is > complete > > > Key: HDDS-1868 > URL: https://issues.apache.org/jira/browse/HDDS-1868 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Siddharth Wagle >Priority: Major > Fix For: 0.5.0 > > Attachments: HDDS-1868.01.patch, HDDS-1868.02.patch, > HDDS-1868.03.patch, HDDS-1868.04.patch > > > Ozone pipeline on restart start in allocated state, they are moved into open > state after all the pipeline have reported to it. However this potentially > can lead into an issue where the pipeline is still not ready to accept any > incoming IO operations. > The pipelines should be marked as ready only after the leader election is > complete and leader is ready to accept incoming IO. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1868) Ozone pipelines should be marked as ready only after the leader election is complete
[ https://issues.apache.org/jira/browse/HDDS-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932745#comment-16932745 ] Siddharth Wagle edited comment on HDDS-1868 at 9/18/19 6:46 PM: Hi [~ljain], but with my latest change, all peers send report to SCM. We only open pipeline on leader election. Isn't that taking care of the scenario you mentioned? was (Author: swagle): Hi [~ljain], but with my latest change, all peers send report to SCM. We only open pipeline on leader election. > Ozone pipelines should be marked as ready only after the leader election is > complete > > > Key: HDDS-1868 > URL: https://issues.apache.org/jira/browse/HDDS-1868 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Siddharth Wagle >Priority: Major > Fix For: 0.5.0 > > Attachments: HDDS-1868.01.patch, HDDS-1868.02.patch, > HDDS-1868.03.patch > > > Ozone pipeline on restart start in allocated state, they are moved into open > state after all the pipeline have reported to it. However this potentially > can lead into an issue where the pipeline is still not ready to accept any > incoming IO operations. > The pipelines should be marked as ready only after the leader election is > complete and leader is ready to accept incoming IO. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1868) Ozone pipelines should be marked as ready only after the leader election is complete
[ https://issues.apache.org/jira/browse/HDDS-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926174#comment-16926174 ] Siddharth Wagle edited comment on HDDS-1868 at 9/10/19 4:34 AM: [~ljain] you are very correct, my UT did catch it. Could you please review version 02? Thanks. The UT does not check negative scenario where no leader means no report, so I will change the name if you think code changes looks good. was (Author: swagle): [~ljain] you are very correct, my UT did catch it. Could you please review version 02? Thanks. The UT does not check negative scenario where no leader means no report, so I will change the name if you think change looks good. > Ozone pipelines should be marked as ready only after the leader election is > complete > > > Key: HDDS-1868 > URL: https://issues.apache.org/jira/browse/HDDS-1868 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Siddharth Wagle >Priority: Major > Fix For: 0.5.0 > > Attachments: HDDS-1868.01.patch, HDDS-1868.02.patch > > > Ozone pipeline on restart start in allocated state, they are moved into open > state after all the pipeline have reported to it. However this potentially > can lead into an issue where the pipeline is still not ready to accept any > incoming IO operations. > The pipelines should be marked as ready only after the leader election is > complete and leader is ready to accept incoming IO. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1868) Ozone pipelines should be marked as ready only after the leader election is complete
[ https://issues.apache.org/jira/browse/HDDS-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926174#comment-16926174 ] Siddharth Wagle edited comment on HDDS-1868 at 9/10/19 4:33 AM: [~ljain] you are very correct, my UT did catch it. Could you please review version 02? Thanks. The UT does not check negative scenario where no leader means no report, so I will change the name if you think change looks good. was (Author: swagle): [~ljain] you are very correct, my UT did catch it. Could you please review version 02? Thanks. > Ozone pipelines should be marked as ready only after the leader election is > complete > > > Key: HDDS-1868 > URL: https://issues.apache.org/jira/browse/HDDS-1868 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Siddharth Wagle >Priority: Major > Fix For: 0.5.0 > > Attachments: HDDS-1868.01.patch, HDDS-1868.02.patch > > > Ozone pipeline on restart start in allocated state, they are moved into open > state after all the pipeline have reported to it. However this potentially > can lead into an issue where the pipeline is still not ready to accept any > incoming IO operations. > The pipelines should be marked as ready only after the leader election is > complete and leader is ready to accept incoming IO. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1868) Ozone pipelines should be marked as ready only after the leader election is complete
[ https://issues.apache.org/jira/browse/HDDS-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923662#comment-16923662 ] Siddharth Wagle edited comment on HDDS-1868 at 9/5/19 6:31 PM: --- Yes, you are right, not sure why I thought report includes peers. I will try to verify if follower get RoleInfoProto, looking at the protos it seems followers should also be sending their roles. was (Author: swagle): Yes, you are right, nto sure why I thought report includes peers. > Ozone pipelines should be marked as ready only after the leader election is > complete > > > Key: HDDS-1868 > URL: https://issues.apache.org/jira/browse/HDDS-1868 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Datanode, SCM >Affects Versions: 0.4.0 >Reporter: Mukul Kumar Singh >Assignee: Siddharth Wagle >Priority: Major > Fix For: 0.5.0 > > Attachments: HDDS-1868.01.patch > > > Ozone pipeline on restart start in allocated state, they are moved into open > state after all the pipeline have reported to it. However this potentially > can lead into an issue where the pipeline is still not ready to accept any > incoming IO operations. > The pipelines should be marked as ready only after the leader election is > complete and leader is ready to accept incoming IO. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
