As long as one task is in canceling state, then the job status might be still in canceling state.
@Joshua Do you confirm all of the tasks in topology were already in terminal state such as failed or canceled? Best, Zhijiang ------------------------------------------------------------------ From:Chesnay Schepler <ches...@apache.org> Send Time:2019年6月19日(星期三) 16:32 To:Joshua Fan <joshuafat...@gmail.com>; user <user@flink.apache.org>; Till Rohrmann <trohrm...@apache.org> Subject:Re: Maybe a flink bug. Job keeps in FAILING state @Till have you see something like this before? Despite all source tasks reaching a terminal state on a TM (FAILED) it does not send updates to the JM for all of them, but only a single one. On 18/06/2019 12:14, Joshua Fan wrote: > Hi All, > There is a topology of 3 operator, such as, source, parser, and > persist. Occasionally, 5 subtasks of the source encounters exception > and turns to failed, at the same time, one subtask of the parser runs > into exception and turns to failed too. The jobmaster gets a message > of the parser's failed. The jobmaster then try to cancel all the > subtask, most of the subtasks of the three operator turns to canceled > except the 5 subtasks of the source, because the state of the 5 ones > is already FAILED before jobmaster try to cancel it. Then the > jobmaster can not reach a final state but keeps in Failing state > meanwhile the subtask of the source kees in canceling state. > > The job run on a flink 1.7 cluster on yarn, and there is only one tm > with 10 slots. > > The attached files contains a jm log , tm log and the ui picture. > > The exception timestamp is about 2019-06-16 13:42:28. > > Yours > Joshua
