[
https://issues.apache.org/jira/browse/FLINK-5830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880168#comment-15880168
]
ASF GitHub Bot commented on FLINK-5830:
---------------------------------------
Github user zhijiangW commented on the issue:
https://github.com/apache/flink/pull/3360
@StephanEwen , if the exception is bubbled out, and cause TaskExecutor to
exit as a result, I think the JobMaster can be assumed in a sane state in final
based on detection of TaskExecutor failure.
The current solution just refers to OOM error, maybe it can extend to any
exceptions, because it is difficult to confirm the consistency of the internal
state and the conservative approach is to let it terminate as @tillrohrmann
said.
If I understand correctly from @tillrohrmann 's suggestions, the
**RpcEndpoint.runAsync** method would modify to return a **Future** that is
similar with **RpcEndpoint.callAsync**, but still a **Tell** action in akka.
And this **Future** should be set as a field in **RunAsync** in order to get it
when handle in **AkkaRpcActor**. The **Future** can help to determine whether
the message is executed successfully or lost to enhance the **Tell** mechanism.
If the **Future** with **Tell** action is better than current ** RpcEndpoint
.callAsync** which refers to **Ask** action, I will try to do for that. Or
another option is tolerating the lost message in current
**RpcEndpoint.runAsync**, and it should be used in such scenarios for
efficiency and not safe. For the important interaction, it should resort to
**RpcEndpoint.callAsync**. What do you think?
> OutOfMemoryError during notify final state in TaskExecutor may cause job stuck
> ------------------------------------------------------------------------------
>
> Key: FLINK-5830
> URL: https://issues.apache.org/jira/browse/FLINK-5830
> Project: Flink
> Issue Type: Bug
> Reporter: zhijiang
> Assignee: zhijiang
>
> The scenario is like this:
> {{JobMaster}} tries to cancel all the executions when process failed
> execution, and the task executor already acknowledge the cancel rpc message.
> When notify the final state in {{TaskExecutor}}, it causes OOM in
> {{AkkaRpcActor}} and this error is caught to log the info. The final state
> will not be sent any more.
> The {{JobMaster}} can not receive the final state and trigger the restart
> strategy.
> One solution is to catch the {{OutOfMemoryError}} and throw it, then it will
> cause to shut down the {{ActorSystem}} resulting in exiting the
> {{TaskExecutor}}. The {{JobMaster}} can be notified of {{TaskExecutor}}
> failure and fail all the tasks to trigger restart successfully.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
