[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[
https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-18295:
---
Labels: pull-request-available (was: )
> Remove the hack logics of result consumers
> --
>
> Key: FLINK-18295
> URL: https://issues.apache.org/jira/browse/FLINK-18295
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Reporter: Zhu Zhu
>Priority: Major
> Labels: pull-request-available
>
> Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}.
> That's why the consumers of an {{IntermediateResultPartition}} is in the form
> of {{List>}}.
> However, in scheduler/{{ExecutionGraph}} there is assumption that one
> {{IntermediateResultPartition}} can be consumed by one only
> {{ExecutionJobVertex}}. This results in a lot of hack logics which assumes
> partition consumers to contain a single list.
> Given that there is no plan yet to support multiple consumer {{JobEdge}} of
> one {{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to
> have one only consumer {{JobEdge}}. Thus the scheduler can get rid of these
> hack logics.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[
https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu updated FLINK-18295:
Description:
Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}.
That's why the consumers of an {{IntermediateResultPartition}} is in the form
of {{List>}}.
However, in scheduler/{{ExecutionGraph}} there is assumption that one
{{IntermediateResultPartition}} can be consumed by one only
{{ExecutionJobVertex}}. This results in a lot of hack logics which assumes
partition consumers to contain a single list.
Given that there is no plan yet to support multiple consumer {{JobEdge}} of one
{{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to have
one only consumer {{JobEdge}}. Thus the scheduler can get rid of these hack
logics.
was:
Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}.
That's why the consumers of an {{IntermediateResultPartition}} is in the form
of {{List>}}.
However, in scheduler/{{ExecutionGraph}} there is assumption that one
{{IntermediateResultPartition}} can be consumed by one only
{{ExecutionJobVertex}}. This results in a lot of hack logics which assumes
partition consumers to contain a single list.
Given that there is no plan yet to support multiple consumer {{JobEdge}}s of
one {{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to
have one only consumer {{JobEdge}}. Thus the scheduler can get rid of these
hack logics.
> Remove the hack logics of result consumers
> --
>
> Key: FLINK-18295
> URL: https://issues.apache.org/jira/browse/FLINK-18295
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Reporter: Zhu Zhu
>Priority: Major
>
> Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}.
> That's why the consumers of an {{IntermediateResultPartition}} is in the form
> of {{List>}}.
> However, in scheduler/{{ExecutionGraph}} there is assumption that one
> {{IntermediateResultPartition}} can be consumed by one only
> {{ExecutionJobVertex}}. This results in a lot of hack logics which assumes
> partition consumers to contain a single list.
> Given that there is no plan yet to support multiple consumer {{JobEdge}} of
> one {{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to
> have one only consumer {{JobEdge}}. Thus the scheduler can get rid of these
> hack logics.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[
https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu updated FLINK-18295:
Description:
Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}.
That's why the consumers of an {{IntermediateResultPartition}} is in the form
of {{List>}}.
However, in scheduler/{{ExecutionGraph}} there is assumption that one
{{IntermediateResultPartition}} can be consumed by one only
{{ExecutionJobVertex}}. This results in a lot of hack logics which assumes
partition consumers to contain a single list.
Given that there is no plan yet to support multiple consumer {{JobEdge}}s of
one {{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to
have one only consumer {{JobEdge}}. Thus the scheduler can get rid of these
hack logics.
was:
Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}s.
That's why the consumers of an {{IntermediateResultPartition}} is in the form
of {{List>}}.
However, in scheduler/{{ExecutionGraph}} there is assumption that one
{{IntermediateResultPartition}} can be consumed by one only
{{ExecutionJobVertex}}. This results in a lot of hack logics which assumes
partition consumers to contain a single list.
Given that there is no plan yet to support multiple consumer {{JobEdge}}s of
one {{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to
have one only consumer {{JobEdge}}. Thus the scheduler can get rid of these
hack logics.
> Remove the hack logics of result consumers
> --
>
> Key: FLINK-18295
> URL: https://issues.apache.org/jira/browse/FLINK-18295
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Reporter: Zhu Zhu
>Priority: Major
>
> Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}.
> That's why the consumers of an {{IntermediateResultPartition}} is in the form
> of {{List>}}.
> However, in scheduler/{{ExecutionGraph}} there is assumption that one
> {{IntermediateResultPartition}} can be consumed by one only
> {{ExecutionJobVertex}}. This results in a lot of hack logics which assumes
> partition consumers to contain a single list.
> Given that there is no plan yet to support multiple consumer {{JobEdge}}s of
> one {{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to
> have one only consumer {{JobEdge}}. Thus the scheduler can get rid of these
> hack logics.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[
https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu updated FLINK-18295:
Description:
Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}s.
That's why the consumers of an {{IntermediateResultPartition}} is in the form
of {{List>}}.
However, in scheduler/{{ExecutionGraph}} there is assumption that one
{{IntermediateResultPartition}} can be consumed by one only
{{ExecutionJobVertex}}. This results in a lot of hack logics which assumes
partition consumers to contain a single list.
Given that there is no plan yet to support multiple consumer {{JobEdge}}s of
one {{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to
have one only consumer {{JobEdge}}. Thus the scheduler can get rid of these
hack logics.
was:
Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its
consumers. That's why the consumers of a `IntermediateResultPartition` is in
the form of {{List>}}.
However, in scheduler/{{ExecutionGraph}} there is assumption that one
`IntermediateResultPartition` can be consumed by one only `ExecutionJobVertex`.
This results in a lot of hack logics which assumes partition consumers to
contain a single list.
We should remove these hack logics. The idea is to change
`IntermediateResultPartition#consumers` to be `List`.
`ExecutionGraph` building logics should be adjusted accordingly with the
assumption that an `IntermediateResult` can have one only consumer vertex. In
`JobGraph`, there should also be check logics for this assumption.
> Remove the hack logics of result consumers
> --
>
> Key: FLINK-18295
> URL: https://issues.apache.org/jira/browse/FLINK-18295
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Reporter: Zhu Zhu
>Priority: Major
>
> Currently an {{IntermediateDataSet}} can have multiple consumer {{JobEdge}}s.
> That's why the consumers of an {{IntermediateResultPartition}} is in the form
> of {{List>}}.
> However, in scheduler/{{ExecutionGraph}} there is assumption that one
> {{IntermediateResultPartition}} can be consumed by one only
> {{ExecutionJobVertex}}. This results in a lot of hack logics which assumes
> partition consumers to contain a single list.
> Given that there is no plan yet to support multiple consumer {{JobEdge}}s of
> one {{IntermediateDataSet}}. I propose to refactor {{IntermediateDataSet}} to
> have one only consumer {{JobEdge}}. Thus the scheduler can get rid of these
> hack logics.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[
https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu updated FLINK-18295:
Priority: Major (was: Minor)
> Remove the hack logics of result consumers
> --
>
> Key: FLINK-18295
> URL: https://issues.apache.org/jira/browse/FLINK-18295
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Reporter: Zhu Zhu
>Priority: Major
> Labels: auto-deprioritized-major, auto-unassigned
>
> Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its
> consumers. That's why the consumers of a `IntermediateResultPartition` is in
> the form of {{List>}}.
> However, in scheduler/{{ExecutionGraph}} there is assumption that one
> `IntermediateResultPartition` can be consumed by one only
> `ExecutionJobVertex`. This results in a lot of hack logics which assumes
> partition consumers to contain a single list.
> We should remove these hack logics. The idea is to change
> `IntermediateResultPartition#consumers` to be `List`.
> `ExecutionGraph` building logics should be adjusted accordingly with the
> assumption that an `IntermediateResult` can have one only consumer vertex. In
> `JobGraph`, there should also be check logics for this assumption.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[
https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu updated FLINK-18295:
Labels: (was: auto-deprioritized-major auto-unassigned)
> Remove the hack logics of result consumers
> --
>
> Key: FLINK-18295
> URL: https://issues.apache.org/jira/browse/FLINK-18295
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Reporter: Zhu Zhu
>Priority: Major
>
> Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its
> consumers. That's why the consumers of a `IntermediateResultPartition` is in
> the form of {{List>}}.
> However, in scheduler/{{ExecutionGraph}} there is assumption that one
> `IntermediateResultPartition` can be consumed by one only
> `ExecutionJobVertex`. This results in a lot of hack logics which assumes
> partition consumers to contain a single list.
> We should remove these hack logics. The idea is to change
> `IntermediateResultPartition#consumers` to be `List`.
> `ExecutionGraph` building logics should be adjusted accordingly with the
> assumption that an `IntermediateResult` can have one only consumer vertex. In
> `JobGraph`, there should also be check logics for this assumption.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[
https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-18295:
---
Labels: auto-deprioritized-major auto-unassigned (was: auto-unassigned
stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Remove the hack logics of result consumers
> --
>
> Key: FLINK-18295
> URL: https://issues.apache.org/jira/browse/FLINK-18295
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Reporter: Zhu Zhu
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned
>
> Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its
> consumers. That's why the consumers of a `IntermediateResultPartition` is in
> the form of {{List>}}.
> However, in scheduler/{{ExecutionGraph}} there is assumption that one
> `IntermediateResultPartition` can be consumed by one only
> `ExecutionJobVertex`. This results in a lot of hack logics which assumes
> partition consumers to contain a single list.
> We should remove these hack logics. The idea is to change
> `IntermediateResultPartition#consumers` to be `List`.
> `ExecutionGraph` building logics should be adjusted accordingly with the
> assumption that an `IntermediateResult` can have one only consumer vertex. In
> `JobGraph`, there should also be check logics for this assumption.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[
https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-18295:
---
Labels: auto-unassigned stale-major (was: auto-unassigned)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Remove the hack logics of result consumers
> --
>
> Key: FLINK-18295
> URL: https://issues.apache.org/jira/browse/FLINK-18295
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Reporter: Zhu Zhu
>Priority: Major
> Labels: auto-unassigned, stale-major
>
> Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its
> consumers. That's why the consumers of a `IntermediateResultPartition` is in
> the form of {{List>}}.
> However, in scheduler/{{ExecutionGraph}} there is assumption that one
> `IntermediateResultPartition` can be consumed by one only
> `ExecutionJobVertex`. This results in a lot of hack logics which assumes
> partition consumers to contain a single list.
> We should remove these hack logics. The idea is to change
> `IntermediateResultPartition#consumers` to be `List`.
> `ExecutionGraph` building logics should be adjusted accordingly with the
> assumption that an `IntermediateResult` can have one only consumer vertex. In
> `JobGraph`, there should also be check logics for this assumption.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[
https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-18295:
--
Fix Version/s: (was: 1.14.0)
> Remove the hack logics of result consumers
> --
>
> Key: FLINK-18295
> URL: https://issues.apache.org/jira/browse/FLINK-18295
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Reporter: Zhu Zhu
>Priority: Major
> Labels: auto-unassigned
>
> Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its
> consumers. That's why the consumers of a `IntermediateResultPartition` is in
> the form of {{List>}}.
> However, in scheduler/{{ExecutionGraph}} there is assumption that one
> `IntermediateResultPartition` can be consumed by one only
> `ExecutionJobVertex`. This results in a lot of hack logics which assumes
> partition consumers to contain a single list.
> We should remove these hack logics. The idea is to change
> `IntermediateResultPartition#consumers` to be `List`.
> `ExecutionGraph` building logics should be adjusted accordingly with the
> assumption that an `IntermediateResult` can have one only consumer vertex. In
> `JobGraph`, there should also be check logics for this assumption.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[
https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Knauf updated FLINK-18295:
-
Issue Type: Technical Debt (was: Improvement)
> Remove the hack logics of result consumers
> --
>
> Key: FLINK-18295
> URL: https://issues.apache.org/jira/browse/FLINK-18295
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Reporter: Zhu Zhu
>Priority: Major
> Labels: auto-unassigned
> Fix For: 1.14.0
>
>
> Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its
> consumers. That's why the consumers of a `IntermediateResultPartition` is in
> the form of {{List>}}.
> However, in scheduler/{{ExecutionGraph}} there is assumption that one
> `IntermediateResultPartition` can be consumed by one only
> `ExecutionJobVertex`. This results in a lot of hack logics which assumes
> partition consumers to contain a single list.
> We should remove these hack logics. The idea is to change
> `IntermediateResultPartition#consumers` to be `List`.
> `ExecutionGraph` building logics should be adjusted accordingly with the
> assumption that an `IntermediateResult` can have one only consumer vertex. In
> `JobGraph`, there should also be check logics for this assumption.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[
https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Wysakowicz updated FLINK-18295:
-
Fix Version/s: (was: 1.13.0)
1.14.0
> Remove the hack logics of result consumers
> --
>
> Key: FLINK-18295
> URL: https://issues.apache.org/jira/browse/FLINK-18295
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Zhu Zhu
>Priority: Major
> Labels: auto-unassigned
> Fix For: 1.14.0
>
>
> Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its
> consumers. That's why the consumers of a `IntermediateResultPartition` is in
> the form of {{List>}}.
> However, in scheduler/{{ExecutionGraph}} there is assumption that one
> `IntermediateResultPartition` can be consumed by one only
> `ExecutionJobVertex`. This results in a lot of hack logics which assumes
> partition consumers to contain a single list.
> We should remove these hack logics. The idea is to change
> `IntermediateResultPartition#consumers` to be `List`.
> `ExecutionGraph` building logics should be adjusted accordingly with the
> assumption that an `IntermediateResult` can have one only consumer vertex. In
> `JobGraph`, there should also be check logics for this assumption.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[
https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-18295:
---
Labels: auto-unassigned (was: stale-assigned)
> Remove the hack logics of result consumers
> --
>
> Key: FLINK-18295
> URL: https://issues.apache.org/jira/browse/FLINK-18295
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Zhu Zhu
>Assignee: Zhu Zhu
>Priority: Major
> Labels: auto-unassigned
> Fix For: 1.13.0
>
>
> Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its
> consumers. That's why the consumers of a `IntermediateResultPartition` is in
> the form of {{List>}}.
> However, in scheduler/{{ExecutionGraph}} there is assumption that one
> `IntermediateResultPartition` can be consumed by one only
> `ExecutionJobVertex`. This results in a lot of hack logics which assumes
> partition consumers to contain a single list.
> We should remove these hack logics. The idea is to change
> `IntermediateResultPartition#consumers` to be `List`.
> `ExecutionGraph` building logics should be adjusted accordingly with the
> assumption that an `IntermediateResult` can have one only consumer vertex. In
> `JobGraph`, there should also be check logics for this assumption.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[
https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-18295:
---
Labels: stale-assigned (was: )
> Remove the hack logics of result consumers
> --
>
> Key: FLINK-18295
> URL: https://issues.apache.org/jira/browse/FLINK-18295
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Zhu Zhu
>Assignee: Zhu Zhu
>Priority: Major
> Labels: stale-assigned
> Fix For: 1.13.0
>
>
> Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its
> consumers. That's why the consumers of a `IntermediateResultPartition` is in
> the form of {{List>}}.
> However, in scheduler/{{ExecutionGraph}} there is assumption that one
> `IntermediateResultPartition` can be consumed by one only
> `ExecutionJobVertex`. This results in a lot of hack logics which assumes
> partition consumers to contain a single list.
> We should remove these hack logics. The idea is to change
> `IntermediateResultPartition#consumers` to be `List`.
> `ExecutionGraph` building logics should be adjusted accordingly with the
> assumption that an `IntermediateResult` can have one only consumer vertex. In
> `JobGraph`, there should also be check logics for this assumption.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18295) Remove the hack logics of result consumers
[
https://issues.apache.org/jira/browse/FLINK-18295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-18295:
---
Fix Version/s: (was: 1.12.0)
1.13.0
> Remove the hack logics of result consumers
> --
>
> Key: FLINK-18295
> URL: https://issues.apache.org/jira/browse/FLINK-18295
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Zhu Zhu
>Assignee: Zhu Zhu
>Priority: Major
> Fix For: 1.13.0
>
>
> Currently an {{IntermediateDataSet}} can have multiple {{JobVertex}} as its
> consumers. That's why the consumers of a `IntermediateResultPartition` is in
> the form of {{List>}}.
> However, in scheduler/{{ExecutionGraph}} there is assumption that one
> `IntermediateResultPartition` can be consumed by one only
> `ExecutionJobVertex`. This results in a lot of hack logics which assumes
> partition consumers to contain a single list.
> We should remove these hack logics. The idea is to change
> `IntermediateResultPartition#consumers` to be `List`.
> `ExecutionGraph` building logics should be adjusted accordingly with the
> assumption that an `IntermediateResult` can have one only consumer vertex. In
> `JobGraph`, there should also be check logics for this assumption.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
