[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[ https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-18295: --- Labels: pull-request-available (was: ) > Remove the hack logics of result consumers > -- > > Key: FLINK-18295 > URL: https://issues.apache.org/jira/browse/FLINK-18295 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Coordination >Reporter: Zhu Zhu >Priority: Major > Labels: pull-request-available > > Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}. > That's why the consumers of an {{IntermediateResultPartition}} is in the form > of {{List>}}. > However, in scheduler/{{ExecutionGraph}} there is assumption that one > {{IntermediateResultPartition}} can be consumed by one only > {{ExecutionJobVertex}}. This results in a lot of hack logics which assumes > partition consumers to contain a single list. > Given that there is no plan yet to support multiple consumer {{JobEdge}} of > one {{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to > have one only consumer {{JobEdge}}. Thus the scheduler can get rid of these > hack logics. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[ https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu updated FLINK-18295: Description: Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}. That's why the consumers of an {{IntermediateResultPartition}} is in the form of {{List>}}. However, in scheduler/{{ExecutionGraph}} there is assumption that one {{IntermediateResultPartition}} can be consumed by one only {{ExecutionJobVertex}}. This results in a lot of hack logics which assumes partition consumers to contain a single list. Given that there is no plan yet to support multiple consumer {{JobEdge}} of one {{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to have one only consumer {{JobEdge}}. Thus the scheduler can get rid of these hack logics. was: Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}. That's why the consumers of an {{IntermediateResultPartition}} is in the form of {{List>}}. However, in scheduler/{{ExecutionGraph}} there is assumption that one {{IntermediateResultPartition}} can be consumed by one only {{ExecutionJobVertex}}. This results in a lot of hack logics which assumes partition consumers to contain a single list. Given that there is no plan yet to support multiple consumer {{JobEdge}}s of one {{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to have one only consumer {{JobEdge}}. Thus the scheduler can get rid of these hack logics. > Remove the hack logics of result consumers > -- > > Key: FLINK-18295 > URL: https://issues.apache.org/jira/browse/FLINK-18295 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Coordination >Reporter: Zhu Zhu >Priority: Major > > Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}. > That's why the consumers of an {{IntermediateResultPartition}} is in the form > of {{List>}}. > However, in scheduler/{{ExecutionGraph}} there is assumption that one > {{IntermediateResultPartition}} can be consumed by one only > {{ExecutionJobVertex}}. This results in a lot of hack logics which assumes > partition consumers to contain a single list. > Given that there is no plan yet to support multiple consumer {{JobEdge}} of > one {{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to > have one only consumer {{JobEdge}}. Thus the scheduler can get rid of these > hack logics. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[ https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu updated FLINK-18295: Description: Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}. That's why the consumers of an {{IntermediateResultPartition}} is in the form of {{List>}}. However, in scheduler/{{ExecutionGraph}} there is assumption that one {{IntermediateResultPartition}} can be consumed by one only {{ExecutionJobVertex}}. This results in a lot of hack logics which assumes partition consumers to contain a single list. Given that there is no plan yet to support multiple consumer {{JobEdge}}s of one {{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to have one only consumer {{JobEdge}}. Thus the scheduler can get rid of these hack logics. was: Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}s. That's why the consumers of an {{IntermediateResultPartition}} is in the form of {{List>}}. However, in scheduler/{{ExecutionGraph}} there is assumption that one {{IntermediateResultPartition}} can be consumed by one only {{ExecutionJobVertex}}. This results in a lot of hack logics which assumes partition consumers to contain a single list. Given that there is no plan yet to support multiple consumer {{JobEdge}}s of one {{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to have one only consumer {{JobEdge}}. Thus the scheduler can get rid of these hack logics. > Remove the hack logics of result consumers > -- > > Key: FLINK-18295 > URL: https://issues.apache.org/jira/browse/FLINK-18295 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Coordination >Reporter: Zhu Zhu >Priority: Major > > Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}. > That's why the consumers of an {{IntermediateResultPartition}} is in the form > of {{List>}}. > However, in scheduler/{{ExecutionGraph}} there is assumption that one > {{IntermediateResultPartition}} can be consumed by one only > {{ExecutionJobVertex}}. This results in a lot of hack logics which assumes > partition consumers to contain a single list. > Given that there is no plan yet to support multiple consumer {{JobEdge}}s of > one {{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to > have one only consumer {{JobEdge}}. Thus the scheduler can get rid of these > hack logics. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[ https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu updated FLINK-18295: Description: Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}s. That's why the consumers of an {{IntermediateResultPartition}} is in the form of {{List>}}. However, in scheduler/{{ExecutionGraph}} there is assumption that one {{IntermediateResultPartition}} can be consumed by one only {{ExecutionJobVertex}}. This results in a lot of hack logics which assumes partition consumers to contain a single list. Given that there is no plan yet to support multiple consumer {{JobEdge}}s of one {{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to have one only consumer {{JobEdge}}. Thus the scheduler can get rid of these hack logics. was: Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its consumers. That's why the consumers of a `IntermediateResultPartition` is in the form of {{List>}}. However, in scheduler/{{ExecutionGraph}} there is assumption that one `IntermediateResultPartition` can be consumed by one only `ExecutionJobVertex`. This results in a lot of hack logics which assumes partition consumers to contain a single list. We should remove these hack logics. The idea is to change `IntermediateResultPartition#consumers` to be `List`. `ExecutionGraph` building logics should be adjusted accordingly with the assumption that an `IntermediateResult` can have one only consumer vertex. In `JobGraph`, there should also be check logics for this assumption. > Remove the hack logics of result consumers > -- > > Key: FLINK-18295 > URL: https://issues.apache.org/jira/browse/FLINK-18295 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Coordination >Reporter: Zhu Zhu >Priority: Major > > Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}s. > That's why the consumers of an {{IntermediateResultPartition}} is in the form > of {{List>}}. > However, in scheduler/{{ExecutionGraph}} there is assumption that one > {{IntermediateResultPartition}} can be consumed by one only > {{ExecutionJobVertex}}. This results in a lot of hack logics which assumes > partition consumers to contain a single list. > Given that there is no plan yet to support multiple consumer {{JobEdge}}s of > one {{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to > have one only consumer {{JobEdge}}. Thus the scheduler can get rid of these > hack logics. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[ https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu updated FLINK-18295: Priority: Major (was: Minor) > Remove the hack logics of result consumers > -- > > Key: FLINK-18295 > URL: https://issues.apache.org/jira/browse/FLINK-18295 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Coordination >Reporter: Zhu Zhu >Priority: Major > Labels: auto-deprioritized-major, auto-unassigned > > Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its > consumers. That's why the consumers of a `IntermediateResultPartition` is in > the form of {{List>}}. > However, in scheduler/{{ExecutionGraph}} there is assumption that one > `IntermediateResultPartition` can be consumed by one only > `ExecutionJobVertex`. This results in a lot of hack logics which assumes > partition consumers to contain a single list. > We should remove these hack logics. The idea is to change > `IntermediateResultPartition#consumers` to be `List`. > `ExecutionGraph` building logics should be adjusted accordingly with the > assumption that an `IntermediateResult` can have one only consumer vertex. In > `JobGraph`, there should also be check logics for this assumption. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[ https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu updated FLINK-18295: Labels: (was: auto-deprioritized-major auto-unassigned) > Remove the hack logics of result consumers > -- > > Key: FLINK-18295 > URL: https://issues.apache.org/jira/browse/FLINK-18295 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Coordination >Reporter: Zhu Zhu >Priority: Major > > Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its > consumers. That's why the consumers of a `IntermediateResultPartition` is in > the form of {{List>}}. > However, in scheduler/{{ExecutionGraph}} there is assumption that one > `IntermediateResultPartition` can be consumed by one only > `ExecutionJobVertex`. This results in a lot of hack logics which assumes > partition consumers to contain a single list. > We should remove these hack logics. The idea is to change > `IntermediateResultPartition#consumers` to be `List`. > `ExecutionGraph` building logics should be adjusted accordingly with the > assumption that an `IntermediateResult` can have one only consumer vertex. In > `JobGraph`, there should also be check logics for this assumption. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[ https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-18295: --- Labels: auto-deprioritized-major auto-unassigned (was: auto-unassigned stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Remove the hack logics of result consumers > -- > > Key: FLINK-18295 > URL: https://issues.apache.org/jira/browse/FLINK-18295 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Coordination >Reporter: Zhu Zhu >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned > > Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its > consumers. That's why the consumers of a `IntermediateResultPartition` is in > the form of {{List>}}. > However, in scheduler/{{ExecutionGraph}} there is assumption that one > `IntermediateResultPartition` can be consumed by one only > `ExecutionJobVertex`. This results in a lot of hack logics which assumes > partition consumers to contain a single list. > We should remove these hack logics. The idea is to change > `IntermediateResultPartition#consumers` to be `List`. > `ExecutionGraph` building logics should be adjusted accordingly with the > assumption that an `IntermediateResult` can have one only consumer vertex. In > `JobGraph`, there should also be check logics for this assumption. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[ https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-18295: --- Labels: auto-unassigned stale-major (was: auto-unassigned) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Remove the hack logics of result consumers > -- > > Key: FLINK-18295 > URL: https://issues.apache.org/jira/browse/FLINK-18295 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Coordination >Reporter: Zhu Zhu >Priority: Major > Labels: auto-unassigned, stale-major > > Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its > consumers. That's why the consumers of a `IntermediateResultPartition` is in > the form of {{List>}}. > However, in scheduler/{{ExecutionGraph}} there is assumption that one > `IntermediateResultPartition` can be consumed by one only > `ExecutionJobVertex`. This results in a lot of hack logics which assumes > partition consumers to contain a single list. > We should remove these hack logics. The idea is to change > `IntermediateResultPartition#consumers` to be `List`. > `ExecutionGraph` building logics should be adjusted accordingly with the > assumption that an `IntermediateResult` can have one only consumer vertex. In > `JobGraph`, there should also be check logics for this assumption. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[ https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-18295: -- Fix Version/s: (was: 1.14.0) > Remove the hack logics of result consumers > -- > > Key: FLINK-18295 > URL: https://issues.apache.org/jira/browse/FLINK-18295 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Coordination >Reporter: Zhu Zhu >Priority: Major > Labels: auto-unassigned > > Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its > consumers. That's why the consumers of a `IntermediateResultPartition` is in > the form of {{List>}}. > However, in scheduler/{{ExecutionGraph}} there is assumption that one > `IntermediateResultPartition` can be consumed by one only > `ExecutionJobVertex`. This results in a lot of hack logics which assumes > partition consumers to contain a single list. > We should remove these hack logics. The idea is to change > `IntermediateResultPartition#consumers` to be `List`. > `ExecutionGraph` building logics should be adjusted accordingly with the > assumption that an `IntermediateResult` can have one only consumer vertex. In > `JobGraph`, there should also be check logics for this assumption. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[ https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Knauf updated FLINK-18295: - Issue Type: Technical Debt (was: Improvement) > Remove the hack logics of result consumers > -- > > Key: FLINK-18295 > URL: https://issues.apache.org/jira/browse/FLINK-18295 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Coordination >Reporter: Zhu Zhu >Priority: Major > Labels: auto-unassigned > Fix For: 1.14.0 > > > Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its > consumers. That's why the consumers of a `IntermediateResultPartition` is in > the form of {{List>}}. > However, in scheduler/{{ExecutionGraph}} there is assumption that one > `IntermediateResultPartition` can be consumed by one only > `ExecutionJobVertex`. This results in a lot of hack logics which assumes > partition consumers to contain a single list. > We should remove these hack logics. The idea is to change > `IntermediateResultPartition#consumers` to be `List`. > `ExecutionGraph` building logics should be adjusted accordingly with the > assumption that an `IntermediateResult` can have one only consumer vertex. In > `JobGraph`, there should also be check logics for this assumption. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[ https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz updated FLINK-18295: - Fix Version/s: (was: 1.13.0) 1.14.0 > Remove the hack logics of result consumers > -- > > Key: FLINK-18295 > URL: https://issues.apache.org/jira/browse/FLINK-18295 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: Zhu Zhu >Priority: Major > Labels: auto-unassigned > Fix For: 1.14.0 > > > Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its > consumers. That's why the consumers of a `IntermediateResultPartition` is in > the form of {{List>}}. > However, in scheduler/{{ExecutionGraph}} there is assumption that one > `IntermediateResultPartition` can be consumed by one only > `ExecutionJobVertex`. This results in a lot of hack logics which assumes > partition consumers to contain a single list. > We should remove these hack logics. The idea is to change > `IntermediateResultPartition#consumers` to be `List`. > `ExecutionGraph` building logics should be adjusted accordingly with the > assumption that an `IntermediateResult` can have one only consumer vertex. In > `JobGraph`, there should also be check logics for this assumption. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[ https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-18295: --- Labels: auto-unassigned (was: stale-assigned) > Remove the hack logics of result consumers > -- > > Key: FLINK-18295 > URL: https://issues.apache.org/jira/browse/FLINK-18295 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: Zhu Zhu >Assignee: Zhu Zhu >Priority: Major > Labels: auto-unassigned > Fix For: 1.13.0 > > > Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its > consumers. That's why the consumers of a `IntermediateResultPartition` is in > the form of {{List>}}. > However, in scheduler/{{ExecutionGraph}} there is assumption that one > `IntermediateResultPartition` can be consumed by one only > `ExecutionJobVertex`. This results in a lot of hack logics which assumes > partition consumers to contain a single list. > We should remove these hack logics. The idea is to change > `IntermediateResultPartition#consumers` to be `List`. > `ExecutionGraph` building logics should be adjusted accordingly with the > assumption that an `IntermediateResult` can have one only consumer vertex. In > `JobGraph`, there should also be check logics for this assumption. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[ https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-18295: --- Labels: stale-assigned (was: ) > Remove the hack logics of result consumers > -- > > Key: FLINK-18295 > URL: https://issues.apache.org/jira/browse/FLINK-18295 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: Zhu Zhu >Assignee: Zhu Zhu >Priority: Major > Labels: stale-assigned > Fix For: 1.13.0 > > > Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its > consumers. That's why the consumers of a `IntermediateResultPartition` is in > the form of {{List>}}. > However, in scheduler/{{ExecutionGraph}} there is assumption that one > `IntermediateResultPartition` can be consumed by one only > `ExecutionJobVertex`. This results in a lot of hack logics which assumes > partition consumers to contain a single list. > We should remove these hack logics. The idea is to change > `IntermediateResultPartition#consumers` to be `List`. > `ExecutionGraph` building logics should be adjusted accordingly with the > assumption that an `IntermediateResult` can have one only consumer vertex. In > `JobGraph`, there should also be check logics for this assumption. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[ https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-18295: --- Fix Version/s: (was: 1.12.0) 1.13.0 > Remove the hack logics of result consumers > -- > > Key: FLINK-18295 > URL: https://issues.apache.org/jira/browse/FLINK-18295 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: Zhu Zhu >Assignee: Zhu Zhu >Priority: Major > Fix For: 1.13.0 > > > Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its > consumers. That's why the consumers of a `IntermediateResultPartition` is in > the form of {{List>}}. > However, in scheduler/{{ExecutionGraph}} there is assumption that one > `IntermediateResultPartition` can be consumed by one only > `ExecutionJobVertex`. This results in a lot of hack logics which assumes > partition consumers to contain a single list. > We should remove these hack logics. The idea is to change > `IntermediateResultPartition#consumers` to be `List`. > `ExecutionGraph` building logics should be adjusted accordingly with the > assumption that an `IntermediateResult` can have one only consumer vertex. In > `JobGraph`, there should also be check logics for this assumption. -- This message was sent by Atlassian Jira (v8.3.4#803005)