We are using beam with samza runner - beam.version 2.19.0, samza.version 1.3.0
And we are seeing the following excption frequently. Should we be tweaking some configuration? Does this point to any network connectivity issue? 2020/03/21 21:42:09.896 INFO o.a.s.zk.ZkBarrierForVersionUpgrade - Subscribing data changes on the path: /app-clickstreamc360pl127-v1327c712f-fd45-4a67-ba80-7baae72e1f62/clickstreamc360pl127-v1327c712f-fd45-4a67-ba80-7baae72e1f62-coordinationData/jobModelGeneration/jobModelUpgradeBarrier/versionBarriers/barrier_151/barrier_state for barrier version: 151. 2020/03/21 21:42:09.896 ERROR o.a.s.zk.ScheduleAfterDebounceTime - Execution of action: JobModelVersionChange failed. java.lang.IllegalStateException: ZkClient already closed! at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:987) at org.I0Itec.zkclient.ZkClient.watchForData(ZkClient.java:1158) at org.I0Itec.zkclient.ZkClient.subscribeDataChanges(ZkClient.java:194) at org.apache.samza.zk.ZkUtils.subscribeDataChanges(ZkUtils.java:337) at org.apache.samza.zk.ZkBarrierForVersionUpgrade.join(ZkBarrierForVersionUpgrade.java:144) at org.apache.samza.zk.ZkJobCoordinator$ZkJobModelVersionChangeHandler.lambda$doHandleDataChange$0(ZkJobCoordinator.java:536) at org.apache.samza.zk.ScheduleAfterDebounceTime.lambda$getScheduleableAction$0(ScheduleAfterDebounceTime.java:169) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020/03/21 21:42:09.897 ERROR org.apache.samza.zk.ZkJobCoordinator - Received exception in debounce timer! Stopping the job coordinator java.lang.IllegalStateException: ZkClient already closed! at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:987) at org.I0Itec.zkclient.ZkClient.watchForData(ZkClient.java:1158) at org.I0Itec.zkclient.ZkClient.subscribeDataChanges(ZkClient.java:194) at org.apache.samza.zk.ZkUtils.subscribeDataChanges(ZkUtils.java:337) at org.apache.samza.zk.ZkBarrierForVersionUpgrade.join(ZkBarrierForVersionUpgrade.java:144) at org.apache.samza.zk.ZkJobCoordinator$ZkJobModelVersionChangeHandler.lambda$doHandleDataChange$0(ZkJobCoordinator.java:536) at org.apache.samza.zk.ScheduleAfterDebounceTime.lambda$getScheduleableAction$0(ScheduleAfterDebounceTime.java:169) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020/03/21 21:42:09.897 INFO org.apache.samza.zk.ZkJobCoordinator - Job Coordinator shutdown is in progress! 2020/03/21 21:42:09.898 ERROR o.a.samza.container.SamzaContainer - Caught exception/error in run loop. org.apache.samza.SamzaException: Run loop is interrupted at org.apache.samza.task.AsyncRunLoop.blockIfBusyOrNoNewWork(AsyncRunLoop.java:262) at org.apache.samza.task.AsyncRunLoop.run(AsyncRunLoop.java:160) at org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:778) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.InterruptedException: null at java.lang.Object.wait(Native Method) at org.apache.samza.task.AsyncRunLoop.blockIfBusyOrNoNewWork(AsyncRunLoop.java:259) ... 7 common frames omitted 2020/03/21 21:42:09.898 INFO o.a.samza.container.SamzaContainer - Shutting down SamzaContainer. 2020/03/21 21:42:09.899 ERROR o.a.b.r.samza.SamzaPipelineResult - Container shutdown timed out after 10000 ms. java.util.concurrent.TimeoutException: Container shutdown timed out after 10000 ms. at org.apache.samza.processor.StreamProcessor.stopSamzaContainer(StreamProcessor.java:371) at org.apache.samza.processor.StreamProcessor.access$300(StreamProcessor.java:104) at org.apache.samza.processor.StreamProcessor$1.onJobModelExpired(StreamProcessor.java:386) at org.apache.samza.zk.ZkJobCoordinator$ZkJobModelVersionChangeHandler.lambda$doHandleDataChange$0(ZkJobCoordinator.java:533) at org.apache.samza.zk.ScheduleAfterDebounceTime.lambda$getScheduleableAction$0(ScheduleAfterDebounceTime.java:169) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020/03/21 21:42:09.901 ERROR c.i.s.sdk.core.SppBaseProcessor - An illegal error occurred, forcibly terminating application org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.util.concurrent.TimeoutException: Container shutdown timed out after 10000 ms. at org.apache.beam.runners.samza.SamzaPipelineResult.getStateInfo(SamzaPipelineResult.java:113) at org.apache.beam.runners.samza.SamzaPipelineResult.waitUntilFinish(SamzaPipelineResult.java:77) at org.apache.beam.runners.samza.SamzaPipelineResult.waitUntilFinish(SamzaPipelineResult.java:92) at com.intuit.strmprocess.sdk.core.lifecycles.GracefulLifecycleManager.run(GracefulLifecycleManager.java:30) at com.intuit.strmprocess.sdk.core.SppBaseProcessor.run(SppBaseProcessor.java:74) at com.intuit.cgde.clickstream.c360.ProcessC360Data.main(ProcessC360Data.java:103) Caused by: java.util.concurrent.TimeoutException: Container shutdown timed out after 10000 ms. at org.apache.samza.processor.StreamProcessor.stopSamzaContainer(StreamProcessor.java:371) at org.apache.samza.processor.StreamProcessor.access$300(StreamProcessor.java:104) at org.apache.samza.processor.StreamProcessor$1.onJobModelExpired(StreamProcessor.java:386) at org.apache.samza.zk.ZkJobCoordinator$ZkJobModelVersionChangeHandler.lambda$doHandleDataChange$0(ZkJobCoordinator.java:533) at org.apache.samza.zk.ScheduleAfterDebounceTime.lambda$getScheduleableAction$0(ScheduleAfterDebounceTime.java:169) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020/03/21 21:42:09.902 INFO o.a.samza.container.SamzaContainer - Shutting down consumer multiplexer. 2020/03/21 21:42:09.902 INFO o.a.samza.container.SamzaContainer - Shutting down task instance stream tasks. 2020/03/21 21:42:09.902 INFO o.a.samza.container.SamzaContainer - Shutting down task thread pool 2020/03/21 21:42:09.902 INFO c.i.s.sdk.core.SppBaseProcessor - Application finished execution; terminating Cluster. 2020/03/21 21:42:09.903 ERROR o.a.b.r.samza.SamzaPipelineResult - Container shutdown timed out after 10000 ms. java.util.concurrent.TimeoutException: Container shutdown timed out after 10000 ms. at org.apache.samza.processor.StreamProcessor.stopSamzaContainer(StreamProcessor.java:371) at org.apache.samza.processor.StreamProcessor.access$300(StreamProcessor.java:104) at org.apache.samza.processor.StreamProcessor$1.onJobModelExpired(StreamProcessor.java:386) at org.apache.samza.zk.ZkJobCoordinator$ZkJobModelVersionChangeHandler.lambda$doHandleDataChange$0(ZkJobCoordinator.java:533) at org.apache.samza.zk.ScheduleAfterDebounceTime.lambda$getScheduleableAction$0(ScheduleAfterDebounceTime.java:169) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2020/03/21 21:42:09.903 INFO c.i.s.sdk.core.SppBaseProcessor - Application Killer sleeping for 30000 ms Exception in thread "Thread-8" org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.util.concurrent.TimeoutException: Container shutdown timed out after 10000 ms. at org.apache.beam.runners.samza.SamzaPipelineResult.getStateInfo(SamzaPipelineResult.java:113) at org.apache.beam.runners.samza.SamzaPipelineResult.waitUntilFinish(SamzaPipelineResult.java:77) at org.apache.beam.runners.samza.SamzaPipelineResult.waitUntilFinish(SamzaPipelineResult.java:92) at org.apache.beam.runners.samza.SamzaPipelineResult.cancel(SamzaPipelineResult.java:62) at com.intuit.strmprocess.sdk.core.lifecycles.GracefulLifecycleManager.lambda$run$0(GracefulLifecycleManager.java:23) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.TimeoutException: Container shutdown timed out after 10000 ms. at org.apache.samza.processor.StreamProcessor.stopSamzaContainer(StreamProcessor.java:371) at org.apache.samza.processor.StreamProcessor.access$300(StreamProcessor.java:104) at org.apache.samza.processor.StreamProcessor$1.onJobModelExpired(StreamProcessor.java:386) at org.apache.samza.zk.ZkJobCoordinator$ZkJobModelVersionChangeHandler.lambda$doHandleDataChange$0(ZkJobCoordinator.java:533) at org.apache.samza.zk.ScheduleAfterDebounceTime.lambda$getScheduleableAction$0(ScheduleAfterDebounceTime.java:169) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 2020/03/21 21:42:09.907 INFO o.a.samza.container.SamzaContainer - Shutting down timer executor 2020/03/21 21:42:09.908 INFO o.a.k.clients.producer.KafkaProducer - Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.