[jira] [Commented] (FLINK-5197) Late JobStatusChanged messages can interfere with running jobs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708503#comment-15708503
 ] 

ASF GitHub Bot commented on FLINK-5197:
---

Github user tillrohrmann closed the pull request at:

https://github.com/apache/flink/pull/2895


> Late JobStatusChanged messages can interfere with running jobs
> --
>
> Key: FLINK-5197
> URL: https://issues.apache.org/jira/browse/FLINK-5197
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.2.0, 1.1.4
>
>
> When the {{JobManager}} receives a {{JobStatusChanged}} message, it will look 
> up the {{ExecutionGraph}} for the given {{JobID}}. If there is no 
> {{ExecutionGraph}}, then a {{RemoveJob}} message is sent to itself. In the 
> general case, this is not problematic, because the {{RemoveJob}} message 
> won't do anything if there is no {{ExecutionGraph}}. However, since this is 
> an asynchronous call, it can happen that the corresponding job of the 
> {{JobID}} is recovered before receiving the {{RemoveJob}} message. In this 
> case, the newly recovered job would be removed.
> I propose to change the behaviour such that a {{JobStatusChanged}} for a 
> non-existing {{ExecutionGraph}} will be simply ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5197) Late JobStatusChanged messages can interfere with running jobs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708500#comment-15708500
 ] 

ASF GitHub Bot commented on FLINK-5197:
---

Github user tillrohrmann closed the pull request at:

https://github.com/apache/flink/pull/2896


> Late JobStatusChanged messages can interfere with running jobs
> --
>
> Key: FLINK-5197
> URL: https://issues.apache.org/jira/browse/FLINK-5197
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.2.0, 1.1.4
>
>
> When the {{JobManager}} receives a {{JobStatusChanged}} message, it will look 
> up the {{ExecutionGraph}} for the given {{JobID}}. If there is no 
> {{ExecutionGraph}}, then a {{RemoveJob}} message is sent to itself. In the 
> general case, this is not problematic, because the {{RemoveJob}} message 
> won't do anything if there is no {{ExecutionGraph}}. However, since this is 
> an asynchronous call, it can happen that the corresponding job of the 
> {{JobID}} is recovered before receiving the {{RemoveJob}} message. In this 
> case, the newly recovered job would be removed.
> I propose to change the behaviour such that a {{JobStatusChanged}} for a 
> non-existing {{ExecutionGraph}} will be simply ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5197) Late JobStatusChanged messages can interfere with running jobs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708502#comment-15708502
 ] 

ASF GitHub Bot commented on FLINK-5197:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/2895
  
Forgot to include the closing tag in the commit.


> Late JobStatusChanged messages can interfere with running jobs
> --
>
> Key: FLINK-5197
> URL: https://issues.apache.org/jira/browse/FLINK-5197
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.2.0, 1.1.4
>
>
> When the {{JobManager}} receives a {{JobStatusChanged}} message, it will look 
> up the {{ExecutionGraph}} for the given {{JobID}}. If there is no 
> {{ExecutionGraph}}, then a {{RemoveJob}} message is sent to itself. In the 
> general case, this is not problematic, because the {{RemoveJob}} message 
> won't do anything if there is no {{ExecutionGraph}}. However, since this is 
> an asynchronous call, it can happen that the corresponding job of the 
> {{JobID}} is recovered before receiving the {{RemoveJob}} message. In this 
> case, the newly recovered job would be removed.
> I propose to change the behaviour such that a {{JobStatusChanged}} for a 
> non-existing {{ExecutionGraph}} will be simply ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5197) Late JobStatusChanged messages can interfere with running jobs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708494#comment-15708494
 ] 

ASF GitHub Bot commented on FLINK-5197:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/2895
  
Merging this PR. @uce reviewed the backport. 


> Late JobStatusChanged messages can interfere with running jobs
> --
>
> Key: FLINK-5197
> URL: https://issues.apache.org/jira/browse/FLINK-5197
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.2.0, 1.1.4
>
>
> When the {{JobManager}} receives a {{JobStatusChanged}} message, it will look 
> up the {{ExecutionGraph}} for the given {{JobID}}. If there is no 
> {{ExecutionGraph}}, then a {{RemoveJob}} message is sent to itself. In the 
> general case, this is not problematic, because the {{RemoveJob}} message 
> won't do anything if there is no {{ExecutionGraph}}. However, since this is 
> an asynchronous call, it can happen that the corresponding job of the 
> {{JobID}} is recovered before receiving the {{RemoveJob}} message. In this 
> case, the newly recovered job would be removed.
> I propose to change the behaviour such that a {{JobStatusChanged}} for a 
> non-existing {{ExecutionGraph}} will be simply ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5197) Late JobStatusChanged messages can interfere with running jobs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708485#comment-15708485
 ] 

ASF GitHub Bot commented on FLINK-5197:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/2896
  
Thanks for the review @uce. Failing test cases are unrelated. Merging this 
PR.


> Late JobStatusChanged messages can interfere with running jobs
> --
>
> Key: FLINK-5197
> URL: https://issues.apache.org/jira/browse/FLINK-5197
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.2.0, 1.1.4
>
>
> When the {{JobManager}} receives a {{JobStatusChanged}} message, it will look 
> up the {{ExecutionGraph}} for the given {{JobID}}. If there is no 
> {{ExecutionGraph}}, then a {{RemoveJob}} message is sent to itself. In the 
> general case, this is not problematic, because the {{RemoveJob}} message 
> won't do anything if there is no {{ExecutionGraph}}. However, since this is 
> an asynchronous call, it can happen that the corresponding job of the 
> {{JobID}} is recovered before receiving the {{RemoveJob}} message. In this 
> case, the newly recovered job would be removed.
> I propose to change the behaviour such that a {{JobStatusChanged}} for a 
> non-existing {{ExecutionGraph}} will be simply ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5197) Late JobStatusChanged messages can interfere with running jobs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705885#comment-15705885
 ] 

ASF GitHub Bot commented on FLINK-5197:
---

Github user uce commented on the issue:

https://github.com/apache/flink/pull/2896
  
Good catch! The `RemoveJob` cannot succeed since is also checking the 
`currentJobs` that are checked for `JobStatusChanged` already. So in the end, 
the only case where this actually triggers removal is when it interfers with a 
recovered job as you say.  

+1 to merge for 1.1 and #2895 for 1.2.


> Late JobStatusChanged messages can interfere with running jobs
> --
>
> Key: FLINK-5197
> URL: https://issues.apache.org/jira/browse/FLINK-5197
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.2.0, 1.1.4
>
>
> When the {{JobManager}} receives a {{JobStatusChanged}} message, it will look 
> up the {{ExecutionGraph}} for the given {{JobID}}. If there is no 
> {{ExecutionGraph}}, then a {{RemoveJob}} message is sent to itself. In the 
> general case, this is not problematic, because the {{RemoveJob}} message 
> won't do anything if there is no {{ExecutionGraph}}. However, since this is 
> an asynchronous call, it can happen that the corresponding job of the 
> {{JobID}} is recovered before receiving the {{RemoveJob}} message. In this 
> case, the newly recovered job would be removed.
> I propose to change the behaviour such that a {{JobStatusChanged}} for a 
> non-existing {{ExecutionGraph}} will be simply ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5197) Late JobStatusChanged messages can interfere with running jobs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705540#comment-15705540
 ] 

ASF GitHub Bot commented on FLINK-5197:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/2896

[FLINK-5197] [jm] Ignore outdated JobStatusChanged messages

Backport of #2895 for release 1.1 branch.

Outdated JobStatusChanged messages no longer trigger a RemoveJob message 
but are
logged and ignored. This has the advantage, that an outdated 
JobStatusChanged message
cannot interfere with a recovered job which can have the same job id.

Review @uce.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink 
backportFixJobStatusChangedMessage

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2896.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2896


commit 4a2f948224fe628c721adc4fae24199b0296c80f
Author: Till Rohrmann 
Date:   2016-11-29T15:02:29Z

[FLINK-5197] [jm] Ignore outdated JobStatusChanged messages

Outdated JobStatusChanged messages no longer trigger a RemoveJob message 
but are
logged and ignored. This has the advantage, that an outdated 
JobStatusChanged message
cannot interfere with a recovered job which can have the same job id.




> Late JobStatusChanged messages can interfere with running jobs
> --
>
> Key: FLINK-5197
> URL: https://issues.apache.org/jira/browse/FLINK-5197
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.2.0, 1.1.4
>
>
> When the {{JobManager}} receives a {{JobStatusChanged}} message, it will look 
> up the {{ExecutionGraph}} for the given {{JobID}}. If there is no 
> {{ExecutionGraph}}, then a {{RemoveJob}} message is sent to itself. In the 
> general case, this is not problematic, because the {{RemoveJob}} message 
> won't do anything if there is no {{ExecutionGraph}}. However, since this is 
> an asynchronous call, it can happen that the corresponding job of the 
> {{JobID}} is recovered before receiving the {{RemoveJob}} message. In this 
> case, the newly recovered job would be removed.
> I propose to change the behaviour such that a {{JobStatusChanged}} for a 
> non-existing {{ExecutionGraph}} will be simply ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5197) Late JobStatusChanged messages can interfere with running jobs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705538#comment-15705538
 ] 

ASF GitHub Bot commented on FLINK-5197:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/2895

[FLINK-5197] [jm] Ignore outdated JobStatusChanged messages

Outdated JobStatusChanged messages no longer trigger a RemoveJob message 
but are
logged and ignored. This has the advantage, that an outdated 
JobStatusChanged message
cannot interfere with a recovered job which can have the same job id.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink fixJobStatusChangedMessage

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2895.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2895


commit 490cef46380178a2296c2f743b9eb91154967463
Author: Till Rohrmann 
Date:   2016-11-29T15:02:29Z

[FLINK-5197] [jm] Ignore outdated JobStatusChanged messages

Outdated JobStatusChanged messages no longer trigger a RemoveJob message 
but are
logged and ignored. This has the advantage, that an outdated 
JobStatusChanged message
cannot interfere with a recovered job which can have the same job id.




> Late JobStatusChanged messages can interfere with running jobs
> --
>
> Key: FLINK-5197
> URL: https://issues.apache.org/jira/browse/FLINK-5197
> Project: Flink
>  Issue Type: Bug
>  Components: JobManager
>Affects Versions: 1.2.0, 1.1.3
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Minor
> Fix For: 1.2.0, 1.1.4
>
>
> When the {{JobManager}} receives a {{JobStatusChanged}} message, it will look 
> up the {{ExecutionGraph}} for the given {{JobID}}. If there is no 
> {{ExecutionGraph}}, then a {{RemoveJob}} message is sent to itself. In the 
> general case, this is not problematic, because the {{RemoveJob}} message 
> won't do anything if there is no {{ExecutionGraph}}. However, since this is 
> an asynchronous call, it can happen that the corresponding job of the 
> {{JobID}} is recovered before receiving the {{RemoveJob}} message. In this 
> case, the newly recovered job would be removed.
> I propose to change the behaviour such that a {{JobStatusChanged}} for a 
> non-existing {{ExecutionGraph}} will be simply ignored.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)