Hi all, I will have a look.
Kostas On Mon, Sep 28, 2020 at 3:56 PM Till Rohrmann <trohrm...@apache.org> wrote: > > Hi Cristian, > > thanks for reporting this issue. It looks indeed like a very critical problem. > > The problem seems to be that the ApplicationDispatcherBootstrap class > produces an exception (that the request job can no longer be found because of > a lost ZooKeeper connection) which will be interpreted as a job failure. Due > to this interpretation, the cluster will be shut down with a terminal state > of FAILED which will cause the HA data to be cleaned up. The exact problem > occurs in the JobStatusPollingUtils.getJobResult which is called by > ApplicationDispatcherBootstrap.getJobResult(). > > I think there are two problems here: First of all not every exception > bubbling up in the future returned by > ApplicationDispatcherBootstrap.fixJobIdAndRunApplicationAsync() indicates a > job failure. Some of them can also indicate a framework failure which should > not lead to the clean up of HA data. The other problem is that the polling > logic cannot properly handle a temporary connection loss to ZooKeeper which > is a normal situation. > > I am pulling in Aljoscha and Klou who worked on this feature and might be > able to propose a solution for these problems. I've also updated the JIRA > issue FLINK-19154. > > Cheers, > Till > > On Wed, Sep 9, 2020 at 9:00 AM Yang Wang <danrtsey...@gmail.com> wrote: >> >> > The job sub directory will be cleaned up when the job >> > finished/canceled/failed. >> Since we could submit multiple jobs into a Flink session, what i mean is >> when a job >> reached to the terminal state, the sub node(e.g. >> /flink/application_xxxx/running_job_registry/4d255397c7aeb5327adb567238c983c1) >> on the Zookeeper will be cleaned up. But the root >> directory(/flink/application_xxxx/) still exists. >> >> >> For your current case, it is a different case(perjob cluster). I think we >> need to figure out why the only >> running job reached the terminal state. For example, the restart attempts >> are exhausted. And you >> could find the following logs in your JobManager log. >> >> "org.apache.flink.runtime.JobException: Recovery is suppressed by >> NoRestartBackoffTimeStrategy" >> >> >> Best, >> Yang >> >> >> >> >> Cristian <k...@fastmail.fm> 于2020年9月9日周三 上午11:26写道： >>> >>> > The job sub directory will be cleaned up when the job >>> > finished/canceled/failed. >>> >>> What does this mean? >>> >>> Also, to clarify: I'm a very sloppy developer. My jobs crash ALL the >>> time... and yet, the jobs would ALWAYS resume from the last checkpoint. >>> >>> The only cases where I expect Flink to clean up the checkpoint data from ZK >>> is when I explicitly stop or cancel the job (in those cases the job manager >>> takes a savepoint before cleaning up zk and finishing the cluster). >>> >>> Which is not the case here. Flink was on autopilot here and decided to wipe >>> my poor, good checkpoint metadata as the logs show. >>> >>> On Tue, Sep 8, 2020, at 7:59 PM, Yang Wang wrote: >>> >>> AFAIK, the HA data, including Zookeeper meta data and real data on DFS, >>> will only be cleaned up >>> when the Flink cluster reached terminated state. >>> >>> So if you are using a session cluster, the root cluster node on Zk will be >>> cleaned up after you manually >>> stop the session cluster. The job sub directory will be cleaned up when the >>> job finished/canceled/failed. >>> >>> If you are using a job/application cluster, once the only running job >>> finished/failed, all the HA data will >>> be cleaned up. I think you need to check the job restart strategy you have >>> set. For example, the following >>> configuration will make the Flink cluster terminated after 10 attempts. >>> >>> restart-strategy: fixed-delay >>> restart-strategy.fixed-delay.attempts: 10 >>> >>> >>> Best, >>> Yang >>> >>> Cristian <k...@fastmail.fm> 于2020年9月9日周三 上午12:28写道： >>> >>> >>> I'm using the standalone script to start the cluster. >>> >>> As far as I can tell, it's not easy to reproduce. We found that zookeeper >>> lost a node around the time this happened, but all of our other 75 Flink >>> jobs which use the same setup, version and zookeeper, didn't have any >>> issues. They didn't even restart. >>> >>> So unfortunately I don't know how to reproduce this. All I know is I can't >>> sleep. I have nightmares were my precious state is deleted. I wake up >>> crying and quickly start manually savepointing all jobs just in case, >>> because I feel the day of reckon is near. Flinkpocalypse! >>> >>> On Tue, Sep 8, 2020, at 5:54 AM, Robert Metzger wrote: >>> >>> Thanks a lot for reporting this problem here Cristian! >>> >>> I am not super familiar with the involved components, but the behavior you >>> are describing doesn't sound right to me. >>> Which entrypoint are you using? This is logged at the beginning, like this: >>> "2020-09-08 14:45:32,807 INFO >>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Starting >>> StandaloneSessionClusterEntrypoint (Version: 1.11.1, Scala: 2.12, >>> Rev:7eb514a, Date:2020-07-15T07:02:09+02:00)" >>> >>> Do you know by chance if this problem is reproducible? With the >>> StandaloneSessionClusterEntrypoint I was not able to reproduce the problem. >>> >>> >>> >>> >>> On Tue, Sep 8, 2020 at 4:00 AM Husky Zeng <568793...@qq.com> wrote: >>> >>> Hi Cristian, >>> >>> >>> I don't know if it was designed to be like this deliberately. >>> >>> So I have already submitted an issue ,and wait for somebody to response. >>> >>> https://issues.apache.org/jira/browse/FLINK-19154 >>> >>> >>> >>> -- >>> Sent from: >>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ >>> >>>
