Hey all, My jobs that I am trying to write in Flink 1.5 are failing after a few minutes. I think its because the idle task managers are shutting down, which seems to kill the client and the running job. The running job itself was still going on one of the other task managers. I get:
org.apache.flink.client.program.ProgramInvocationException: org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException: Connection unexpectedly closed by remote task manager 'xxxx'. This might indicate that the remote task manager was lost. at org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.channelInactive(CreditBasedPartitionRequestClientHandler.java:143) Now I happen to have the last part of the flow paralleled to 1 right now for debugging, so the 4 task managers that are spun up, 3 of them hit the timeout period (currently set to 240000). I think as soon as the first one goes the client throws up and the whole job dies as a result. Is this expected behavior and if so, is there another way around it? Do I keep increasing the slotmanager.taskmanager-timeout to a really really large number? I have verified setting the timeout to 840000 lets the job complete without error. Thank you!
