[jira] [Comment Edited] (FLINK-10429) Redesign Flink Scheduling, introducing dedicated Scheduler component
[ https://issues.apache.org/jira/browse/FLINK-10429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744746#comment-16744746 ] Tarush Grover edited comment on FLINK-10429 at 1/18/19 12:00 PM: - [~zhuzh] [~till.rohrmann] Whats the current status for this ticket, like which tasks are still open? I can pick some of them if they are still open :) was (Author: app-tarush): Whats the current status for this ticket, like which tasks are still open? I can pick some of them if they are still open :) > Redesign Flink Scheduling, introducing dedicated Scheduler component > > > Key: FLINK-10429 > URL: https://issues.apache.org/jira/browse/FLINK-10429 > Project: Flink > Issue Type: New Feature > Components: Distributed Coordination >Affects Versions: 1.7.0 >Reporter: Stefan Richter >Assignee: Stefan Richter >Priority: Major > > This epic tracks the redesign of scheduling in Flink. Scheduling is currently > a concern that is scattered across different components, mainly the > ExecutionGraph/Execution and the SlotPool. Scheduling also happens only on > the granularity of individual tasks, which make holistic scheduling > strategies hard to implement. In this epic we aim to introduce a dedicated > Scheduler component that can support use-case like auto-scaling, > local-recovery, and resource optimized batch. > The design for this feature is developed here: > https://docs.google.com/document/d/1q7NOqt05HIN-PlKEEPB36JiuU1Iu9fnxxVGJzylhsxU/edit?usp=sharing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (FLINK-10429) Redesign Flink Scheduling, introducing dedicated Scheduler component
[ https://issues.apache.org/jira/browse/FLINK-10429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744746#comment-16744746 ] Tarush Grover edited comment on FLINK-10429 at 1/18/19 12:00 PM: - [~zhuzh] [~till.rohrmann] [~srichter] Whats the current status for this ticket, like which tasks are still open? I can pick some of them if they are still open :) was (Author: app-tarush): [~zhuzh] [~till.rohrmann] Whats the current status for this ticket, like which tasks are still open? I can pick some of them if they are still open :) > Redesign Flink Scheduling, introducing dedicated Scheduler component > > > Key: FLINK-10429 > URL: https://issues.apache.org/jira/browse/FLINK-10429 > Project: Flink > Issue Type: New Feature > Components: Distributed Coordination >Affects Versions: 1.7.0 >Reporter: Stefan Richter >Assignee: Stefan Richter >Priority: Major > > This epic tracks the redesign of scheduling in Flink. Scheduling is currently > a concern that is scattered across different components, mainly the > ExecutionGraph/Execution and the SlotPool. Scheduling also happens only on > the granularity of individual tasks, which make holistic scheduling > strategies hard to implement. In this epic we aim to introduce a dedicated > Scheduler component that can support use-case like auto-scaling, > local-recovery, and resource optimized batch. > The design for this feature is developed here: > https://docs.google.com/document/d/1q7NOqt05HIN-PlKEEPB36JiuU1Iu9fnxxVGJzylhsxU/edit?usp=sharing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (FLINK-10429) Redesign Flink Scheduling, introducing dedicated Scheduler component
[ https://issues.apache.org/jira/browse/FLINK-10429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16630038#comment-16630038 ] Zhu Zhu edited comment on FLINK-10429 at 9/27/18 9:22 AM: -- +1 Really like this proposal. Having a centered scheduler makes the scheduling and related events easier to track, which can well support a pluggable scheduling and JM failover. A late expanded ExecutionGraph is also we've thought long to achieve better resource utilization based on runtime feedbacks. Glad we are moving in the same direction. Wish we can together make the scheduling better. was (Author: zhuzh): +1 Really like this proposal. Having a centered scheduler makes the scheduling and related events easier to track, which can support the pluggable scheduling the JM failover well. A late expanded ExecutionGraph is also we've thought long to achieve better resource utilization based on runtime feedbacks. Glad we are moving in the same direction. Wish we can together make the scheduling better. > Redesign Flink Scheduling, introducing dedicated Scheduler component > > > Key: FLINK-10429 > URL: https://issues.apache.org/jira/browse/FLINK-10429 > Project: Flink > Issue Type: New Feature > Components: Distributed Coordination >Affects Versions: 1.7.0 >Reporter: Stefan Richter >Assignee: Stefan Richter >Priority: Major > > This epic tracks the redesign of scheduling in Flink. Scheduling is currently > a concern that is scattered across different components, mainly the > ExecutionGraph/Execution and the SlotPool. Scheduling also happens only on > the granularity of individual tasks, which make holistic scheduling > strategies hard to implement. In this epic we aim to introduce a dedicated > Scheduler component that can support use-case like auto-scaling, > local-recovery, and resource optimized batch. > The design for this feature is developed here: > https://docs.google.com/document/d/1q7NOqt05HIN-PlKEEPB36JiuU1Iu9fnxxVGJzylhsxU/edit?usp=sharing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (FLINK-10429) Redesign Flink Scheduling, introducing dedicated Scheduler component
[ https://issues.apache.org/jira/browse/FLINK-10429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16629769#comment-16629769 ] tison edited comment on FLINK-10429 at 9/27/18 5:18 AM: I like the proposal treat schedule a separated component. It is helpful for further optimize on scheduling such as FLINK-10240. To [~zhuzh], [~tiemsn] I think the first step of the achieve of this redesign would be the extract part, maybe you can take this document into consideration? Also, the link to FLINK-10240 is broken, the correct one is https://docs.google.com/document/d/1zAseuBnqNXg3pst3vLBTc8yGOUo485J2LVWBAdFCW9I/ (without the "edit" part). And I afraid that it would prevent further discussion that this document is READ-ONLY, any thought could not be commented on. For more information, our users start looking for more flexible schedule strategy. Wish this redesign could help. was (Author: tison): I like the proposal treat schedule a separated component. It is helpful for further optimize on scheduling such as FLINK-10240. To [~zhuzh], [~tiemsn] I think the first step of the achieve of this redesign would be the extract part, maybe you can take this document into consideration? Also, the link to FLINK-10240 is broken, the correct one is https://docs.google.com/document/d/1zAseuBnqNXg3pst3vLBTc8yGOUo485J2LVWBAdFCW9I/ . And I afraid that it would prevent further discussion that this document is READ-ONLY, any thought could not be commented on. For more information, our users start looking for more flexible schedule strategy. Wish this redesign could help. > Redesign Flink Scheduling, introducing dedicated Scheduler component > > > Key: FLINK-10429 > URL: https://issues.apache.org/jira/browse/FLINK-10429 > Project: Flink > Issue Type: New Feature > Components: Distributed Coordination >Affects Versions: 1.7.0 >Reporter: Stefan Richter >Assignee: Stefan Richter >Priority: Major > > This epic tracks the redesign of scheduling in Flink. Scheduling is currently > a concern that is scattered across different components, mainly the > ExecutionGraph/Execution and the SlotPool. Scheduling also happens only on > the granularity of individual tasks, which make holistic scheduling > strategies hard to implement. In this epic we aim to introduce a dedicated > Scheduler component that can support use-case like auto-scaling, > local-recovery, and resource optimized batch. > The design for this feature is developed here: > https://docs.google.com/document/d/1q7NOqt05HIN-PlKEEPB36JiuU1Iu9fnxxVGJzylhsxU/edit?usp=sharing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (FLINK-10429) Redesign Flink Scheduling, introducing dedicated Scheduler component
[ https://issues.apache.org/jira/browse/FLINK-10429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16629769#comment-16629769 ] tison edited comment on FLINK-10429 at 9/27/18 5:18 AM: I like the proposal treat schedule a separated component. It is helpful for further optimize on scheduling such as FLINK-10240. To [~zhuzh], [~tiemsn] I think the first step of the achieve of this redesign would be the extract part, maybe you can take this document into consideration? Also, the link to FLINK-10240 is broken, the correct one is https://docs.google.com/document/d/1zAseuBnqNXg3pst3vLBTc8yGOUo485J2LVWBAdFCW9I/ (without the "edit" part). And I afraid that it would prevent further discussion that this document is READ-ONLY, any thought could not be commented on. For more information, our users start looking for more flexible schedule strategy[1]. Wish this redesign could help. [1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Scheduling-sources-td23344.html was (Author: tison): I like the proposal treat schedule a separated component. It is helpful for further optimize on scheduling such as FLINK-10240. To [~zhuzh], [~tiemsn] I think the first step of the achieve of this redesign would be the extract part, maybe you can take this document into consideration? Also, the link to FLINK-10240 is broken, the correct one is https://docs.google.com/document/d/1zAseuBnqNXg3pst3vLBTc8yGOUo485J2LVWBAdFCW9I/ (without the "edit" part). And I afraid that it would prevent further discussion that this document is READ-ONLY, any thought could not be commented on. For more information, our users start looking for more flexible schedule strategy. Wish this redesign could help. > Redesign Flink Scheduling, introducing dedicated Scheduler component > > > Key: FLINK-10429 > URL: https://issues.apache.org/jira/browse/FLINK-10429 > Project: Flink > Issue Type: New Feature > Components: Distributed Coordination >Affects Versions: 1.7.0 >Reporter: Stefan Richter >Assignee: Stefan Richter >Priority: Major > > This epic tracks the redesign of scheduling in Flink. Scheduling is currently > a concern that is scattered across different components, mainly the > ExecutionGraph/Execution and the SlotPool. Scheduling also happens only on > the granularity of individual tasks, which make holistic scheduling > strategies hard to implement. In this epic we aim to introduce a dedicated > Scheduler component that can support use-case like auto-scaling, > local-recovery, and resource optimized batch. > The design for this feature is developed here: > https://docs.google.com/document/d/1q7NOqt05HIN-PlKEEPB36JiuU1Iu9fnxxVGJzylhsxU/edit?usp=sharing -- This message was sent by Atlassian JIRA (v7.6.3#76005)
