Hello flink community::
I am deploying flink application cluster standalone mode on kubernetes, but i
am facing some problems
the job starts normally and it continues to run but at some point in time it
crushes and gets restarted.
Does anyone facing the same problem or know how to resolve it.
I am using flink 1.12.5 version.
here's the log of the JM:
{"@timestamp":"2021-10-19T04:47:29.887+02:00","@version":"1","message":"WebSocket
successfully
opened","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:49:06.833+02:00","@version":"1","message":"Triggering
checkpoint 2930 (type=CHECKPOINT) @ 1634611746678 for job
00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"Checkpoint
Timer","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:49:07.000+02:00","@version":"1","message":"Completed
checkpoint 2930 for job 00000000000000000000000000000000 (474 bytes in 229
ms).","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"jobmanager-future-thread-1","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:51:40.861+02:00","@version":"1","message":"WebSocket
close received. code: 1000, reason:
","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
WebSocket https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Submitting
reconnect task to the
executor","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
WebSocket https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Scheduling
reconnect
task","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"scheduleReconnect|Executor
for Watch 1899640261","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:51:40.862+02:00","@version":"1","message":"Current
reconnect backoff is 1000 milliseconds
(T0)","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"scheduleReconnect|Executor
for Watch 1899640261","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:51:41.863+02:00","@version":"1","message":"Connecting
websocket ...
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@713a35c5","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"reconnectAttempt|Executor
for Watch 1899640261","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:51:41.864+02:00","@version":"1","message":"WebSocket
successfully
opened","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.916+02:00","@version":"1","message":"Renew
deadline reached after 120 seconds while renewing lock ConfigMapLock: flink-dev
- job1-dispatcher-leader
(8f7dddbb-236a-43fe-a706-1795b494838d)","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-19-thread-1","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.918+02:00","@version":"1","message":"Stopping
SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.919+02:00","@version":"1","message":"Stopping
dispatcher
akka.tcp://flink@172.24.10.160:6123/user/rpc/dispatcher_1.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.919+02:00","@version":"1","message":"Stopping
all currently running jobs of dispatcher
akka.tcp://flink@172.24.10.160:6123/user/rpc/dispatcher_1.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.920+02:00","@version":"1","message":"Stopping
the JobMaster for job
flinkJob(00000000000000000000000000000000).","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.920+02:00","@version":"1","message":"Stopping
DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Stopping
KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Force
closing the watch
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@367fad26","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"The
watcher is
closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.921+02:00","@version":"1","message":"Closing
websocket
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@28d6d80e","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"WebSocket
close received. code: 1000, reason:
","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"Ignoring
onClose for already closed/closing
websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.922+02:00","@version":"1","message":"Job
flinkJob (00000000000000000000000000000000) switched from state RUNNING to
SUSPENDED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000,"stack_trace":"org.apache.flink.util.FlinkException:
Stopping JobMaster for job flinkJob(00000000000000000000000000000000).
org.apache.flink.runtime.jobmaster.JobMaster.onStop(JobMaster.java:413)
org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:214)
org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:563)
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:186)
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:172)
akka.actor.Actor.aroundReceive(Actor.scala:517)
akka.actor.Actor.aroundReceive$(Actor.scala:515)
akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
akka.actor.ActorCell.invoke(ActorCell.scala:561)
akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
akka.dispatch.Mailbox.run(Mailbox.scala:225)
akka.dispatch.Mailbox.exec(Mailbox.scala:235)
akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
"}
{"@timestamp":"2021-10-19T04:53:16.925+02:00","@version":"1","message":"Source:
Rabbit MQ (1/1) (998a108c41f221566faca1d0e27530a6) switched from RUNNING to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.926+02:00","@version":"1","message":"Source:
Rabbit MQ (1/1) (998a108c41f221566faca1d0e27530a6) switched from CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.927+02:00","@version":"1","message":"Discarding
the results produced by task execution
998a108c41f221566faca1d0e27530a6.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.928+02:00","@version":"1","message":"Discarding
the results produced by task execution
998a108c41f221566faca1d0e27530a6.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Json
conversion (1/2) (e7779a5274b0d154b9ce1a6db8d03912) switched from RUNNING to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Json
conversion (1/2) (e7779a5274b0d154b9ce1a6db8d03912) switched from CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Discarding
the results produced by task execution
e7779a5274b0d154b9ce1a6db8d03912.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.930+02:00","@version":"1","message":"Discarding
the results produced by task execution
e7779a5274b0d154b9ce1a6db8d03912.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Json
conversion (2/2) (31a2506d16e7e5057d01c5b115e280b8) switched from RUNNING to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Json
conversion (2/2) (31a2506d16e7e5057d01c5b115e280b8) switched from CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Discarding
the results produced by task execution
31a2506d16e7e5057d01c5b115e280b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Discarding
the results produced by task execution
31a2506d16e7e5057d01c5b115e280b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.931+02:00","@version":"1","message":"Filter
with avro schema (1/2) (1190c97468ad43833ca0cfa7f46c4c50) switched from RUNNING
to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter
with avro schema (1/2) (1190c97468ad43833ca0cfa7f46c4c50) switched from
CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Discarding
the results produced by task execution
1190c97468ad43833ca0cfa7f46c4c50.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Discarding
the results produced by task execution
1190c97468ad43833ca0cfa7f46c4c50.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter
with avro schema (2/2) (923e11bd4045c9d5321a28507c640dc9) switched from RUNNING
to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.932+02:00","@version":"1","message":"Filter
with avro schema (2/2) (923e11bd4045c9d5321a28507c640dc9) switched from
CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding
the results produced by task execution
923e11bd4045c9d5321a28507c640dc9.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding
the results produced by task execution
923e11bd4045c9d5321a28507c640dc9.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Filter
Strategy Pattern (1/2) (62b200109f0b1395543c496a9c8c1790) switched from RUNNING
to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Filter
Strategy Pattern (1/2) (62b200109f0b1395543c496a9c8c1790) switched from
CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.933+02:00","@version":"1","message":"Discarding
the results produced by task execution
62b200109f0b1395543c496a9c8c1790.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Discarding
the results produced by task execution
62b200109f0b1395543c496a9c8c1790.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Filter
Strategy Pattern (2/2) (f47da1a29252a9d2bff505953699c012) switched from RUNNING
to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Filter
Strategy Pattern (2/2) (f47da1a29252a9d2bff505953699c012) switched from
CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.934+02:00","@version":"1","message":"Discarding
the results produced by task execution
f47da1a29252a9d2bff505953699c012.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Discarding
the results produced by task execution
f47da1a29252a9d2bff505953699c012.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Get
Subscriptions (1/2) (a7fdf13cf85ebc37be56b4990aacb980) switched from RUNNING to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Get
Subscriptions (1/2) (a7fdf13cf85ebc37be56b4990aacb980) switched from CANCELING
to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.935+02:00","@version":"1","message":"Discarding
the results produced by task execution
a7fdf13cf85ebc37be56b4990aacb980.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding
the results produced by task execution
a7fdf13cf85ebc37be56b4990aacb980.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Get
Subscriptions (2/2) (27b64dbb773ec86cc0d350ffd57da1d1) switched from RUNNING to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Get
Subscriptions (2/2) (27b64dbb773ec86cc0d350ffd57da1d1) switched from CANCELING
to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding
the results produced by task execution
27b64dbb773ec86cc0d350ffd57da1d1.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"Discarding
the results produced by task execution
27b64dbb773ec86cc0d350ffd57da1d1.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.936+02:00","@version":"1","message":"httpAd
call (1/2) (533763bc8108d3bcf2935aeb35970f0c) switched from RUNNING to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd
call (1/2) (533763bc8108d3bcf2935aeb35970f0c) switched from CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding
the results produced by task execution
533763bc8108d3bcf2935aeb35970f0c.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding
the results produced by task execution
533763bc8108d3bcf2935aeb35970f0c.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd
call (2/2) (601e7e96fc3a1af7a5d79421c75c9bc7) switched from RUNNING to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"httpAd
call (2/2) (601e7e96fc3a1af7a5d79421c75c9bc7) switched from CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.937+02:00","@version":"1","message":"Discarding
the results produced by task execution
601e7e96fc3a1af7a5d79421c75c9bc7.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding
the results produced by task execution
601e7e96fc3a1af7a5d79421c75c9bc7.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update
Main Record (1/2) (2004748b8a985c997bb1e96c21d7b7ea) switched from RUNNING to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update
Main Record (1/2) (2004748b8a985c997bb1e96c21d7b7ea) switched from CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding
the results produced by task execution
2004748b8a985c997bb1e96c21d7b7ea.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Discarding
the results produced by task execution
2004748b8a985c997bb1e96c21d7b7ea.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.938+02:00","@version":"1","message":"Update
Main Record (2/2) (13f2a23c52db212bcc531d1f9f7a41b8) switched from RUNNING to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.939+02:00","@version":"1","message":"Update
Main Record (2/2) (13f2a23c52db212bcc531d1f9f7a41b8) switched from CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.939+02:00","@version":"1","message":"Discarding
the results produced by task execution
13f2a23c52db212bcc531d1f9f7a41b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.982+02:00","@version":"1","message":"Discarding
the results produced by task execution
13f2a23c52db212bcc531d1f9f7a41b8.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.983+02:00","@version":"1","message":"Sink:
Send to Notifier (1/2) (4467a95a23f6a5e983f006fb12832b10) switched from RUNNING
to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.983+02:00","@version":"1","message":"Sink:
Send to Notifier (1/2) (4467a95a23f6a5e983f006fb12832b10) switched from
CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.984+02:00","@version":"1","message":"Discarding
the results produced by task execution
4467a95a23f6a5e983f006fb12832b10.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Sink:
Send to Notifier (2/2) (cf5ce6a12a584da4651a7d341cb1aeef) switched from RUNNING
to
CANCELING.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Sink:
Send to Notifier (2/2) (cf5ce6a12a584da4651a7d341cb1aeef) switched from
CANCELING to
CANCELED.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Discarding
the results produced by task execution
cf5ce6a12a584da4651a7d341cb1aeef.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.985+02:00","@version":"1","message":"Stopping
checkpoint coordinator for job
00000000000000000000000000000000.","logger_name":"org.apache.flink.runtime.checkpoint.CheckpointCoordinator","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Suspending","logger_name":"org.apache.flink.runtime.checkpoint.DefaultCompletedCheckpointStore","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Shutting
down.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.986+02:00","@version":"1","message":"Job
00000000000000000000000000000000 has been
suspended.","logger_name":"org.apache.flink.runtime.executiongraph.ExecutionGraph","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.990+02:00","@version":"1","message":"Suspending
SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.990+02:00","@version":"1","message":"Close
ResourceManager connection 81a0f3b34590e11df305b58878a355b1: Stopping JobMaster
for job
flinkJob(00000000000000000000000000000000)..","logger_name":"org.apache.flink.runtime.jobmaster.JobMaster","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.991+02:00","@version":"1","message":"Stopping
SlotPool.","logger_name":"org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.991+02:00","@version":"1","message":"Disconnect
job manager
bc901e3d25d406f70027b81673c74...@akka.tcp://flink@172.24.10.160:6123/user/rpc/jobmanager_2
for job 00000000000000000000000000000000 from the resource
manager.","logger_name":"org.apache.flink.runtime.resourcemanager.StandaloneResourceManager","thread_name":"flink-akka.actor.default-dispatcher-61","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Leader
election
started","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Stopping
DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.993+02:00","@version":"1","message":"Attempting
to acquire leader lease 'ConfigMapLock: flink-dev - job1-dispatcher-leader
(8f7dddbb-236a-43fe-a706-1795b494838d)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Closing
KubernetesLeaderElectionDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Force
closing the watch
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@4d181edd","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"The
watcher is
closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Closing
websocket
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@5efd28bd","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"AkkaRpcService-Supervisor-Termination-Future-Executor-thread-1","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.994+02:00","@version":"1","message":"Loop
thread
interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
null
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
"}
{"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"WebSocket
close received. code: 1000, reason:
","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"Job
00000000000000000000000000000000 reached terminal state
SUSPENDED.","logger_name":"org.apache.flink.runtime.dispatcher.StandaloneDispatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.995+02:00","@version":"1","message":"Ignoring
onClose for already closed/closing
websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:16.996+02:00","@version":"1","message":"Released
job graph 00000000000000000000000000000000 from
KubernetesStateHandleStore{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.runtime.jobmanager.DefaultJobGraphStore","thread_name":"cluster-io-thread-4","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:16.997+02:00","@version":"1","message":"Application
failed unexpectedly:
","logger_name":"org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap","thread_name":"cluster-io-thread-4","level":"WARN","level_value":30000,"stack_trace":"java.util.concurrent.CancellationException:
null
java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
"}
{"@timestamp":"2021-10-19T04:53:16.998+02:00","@version":"1","message":"Fatal
error occurred in the cluster
entrypoint.","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"cluster-io-thread-4","level":"ERROR","level_value":40000,"stack_trace":"org.apache.flink.util.FlinkException:
Application failed unexpectedly.
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAndShutdownClusterAsync$0(ApplicationDispatcherBootstrap.java:170)
java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2278)
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.stop(ApplicationDispatcherBootstrap.java:120)
org.apache.flink.runtime.dispatcher.Dispatcher.lambda$onStop$0(Dispatcher.java:274)
org.apache.flink.runtime.concurrent.FutureUtils.lambda$runAfterwardsAsync$18(FutureUtils.java:687)
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
org.apache.flink.runtime.concurrent.DirectExecutorService.execute(DirectExecutorService.java:217)
java.util.concurrent.CompletableFuture$UniCompletion.claim(CompletableFuture.java:543)
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:765)
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
org.apache.flink.runtime.concurrent.FutureUtils$CompletionConjunctFuture.completeFuture(FutureUtils.java:971)
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:575)
java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:704)
java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.CancellationException: null
java.util.concurrent.CompletableFuture.cancel(CompletableFuture.java:2276)
\t... 22 common frames omitted
"}
{"@timestamp":"2021-10-19T04:53:17.084+02:00","@version":"1","message":"Shutting
StandaloneApplicationClusterEntryPoint down with application status UNKNOWN.
Diagnostics Cluster entrypoint has been closed
externally..","logger_name":"org.apache.flink.runtime.entrypoint.ClusterEntrypoint","thread_name":"StandaloneApplicationClusterEntryPoint
shutdown hook","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.085+02:00","@version":"1","message":"Shutting
down rest
endpoint.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"StandaloneApplicationClusterEntryPoint
shutdown hook","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.086+02:00","@version":"1","message":"Stopped
BLOB server at
0.0.0.0:6124","logger_name":"org.apache.flink.runtime.blob.BlobServer","thread_name":"BlobServer
shutdown hook","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.090+02:00","@version":"1","message":"Successfully
Acquired leader lease 'ConfigMapLock: flink-dev - job1-dispatcher-leader
(8f7dddbb-236a-43fe-a706-1795b494838d)'","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"pool-25081-thread-1","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.090+02:00","@version":"1","message":"Attempting
to renew leader lease 'ConfigMapLock: flink-dev - job1-dispatcher-leader
(8f7dddbb-236a-43fe-a706-1795b494838d)'...","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.185+02:00","@version":"1","message":"Removing
cache directory
/tmp/flink-web-682b3e4f-ed72-4b00-96a0-da5c8248b62c/flink-web-ui","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Stopping
DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing
KubernetesLeaderElectionDriver{configMapName='job1-restserver-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Force
closing the watch
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@6f9e08d4","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"The
watcher is
closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing
websocket
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@42c1c89c","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Loop
thread
interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
null
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
"}
{"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Shut
down
complete.","logger_name":"org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.186+02:00","@version":"1","message":"Closing
components.","logger_name":"org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Stopping
DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Stopping
KubernetesLeaderRetrievalDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"WebSocket
close received. code: 1000, reason:
","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Force
closing the watch
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@11787b64","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Ignoring
onClose for already closed/closing
websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"The
watcher is
closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.187+02:00","@version":"1","message":"Closing
websocket
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@4c063eb4","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping
DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping
KubernetesLeaderRetrievalDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Force
closing the watch
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@713a35c5","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"The
watcher is
closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Closing
websocket
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@4839a3ec","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"WebSocket
close received. code: 1000, reason:
","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Ignoring
onClose for already closed/closing
websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Stopping
DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Closing
KubernetesLeaderElectionDriver{configMapName='job1-dispatcher-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Force
closing the watch
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@202d9236","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"The
watcher is
closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"WebSocket
close received. code: 1000, reason:
","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Closing
websocket
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@d3b348f","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"Thread-29","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Ignoring
onClose for already closed/closing
websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.188+02:00","@version":"1","message":"Loop
thread
interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
null
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
"}
{"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"Stopping
SessionDispatcherLeaderProcess.","logger_name":"org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess","thread_name":"Thread-29","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.189+02:00","@version":"1","message":"WebSocket
close received. code: 1000, reason:
","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Ignoring
onClose for already closed/closing
websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Closing
the
SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.190+02:00","@version":"1","message":"Suspending
the
SlotManager.","logger_name":"org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Stopping
DefaultLeaderElectionService.","logger_name":"org.apache.flink.runtime.leaderelection.DefaultLeaderElectionService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Closing
KubernetesLeaderElectionDriver{configMapName='job1-resourcemanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderElectionDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Force
closing the watch
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@7be008e9","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"The
watcher is
closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Loop
thread
interrupted","logger_name":"io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector","thread_name":"KubernetesLeaderElector-ExecutorService-thread-1","level":"DEBUG","level_value":10000,"stack_trace":"java.lang.InterruptedException:
null
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:998)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.loop(LeaderElector.java:200)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.renewWithTimeout(LeaderElector.java:100)
io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.run(LeaderElector.java:71)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
"}
{"@timestamp":"2021-10-19T04:53:17.191+02:00","@version":"1","message":"Closing
websocket
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@57ae1976","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Stopping
DefaultLeaderRetrievalService.","logger_name":"org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Stopping
KubernetesLeaderRetrievalDriver{configMapName='job1-00000000000000000000000000000000-jobmanager-leader'}.","logger_name":"org.apache.flink.kubernetes.highavailability.KubernetesLeaderRetrievalDriver","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"Force
closing the watch
io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager@3f6e134d","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"WebSocket
close received. code: 1000, reason:
","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.192+02:00","@version":"1","message":"The
watcher is
closing.","logger_name":"org.apache.flink.kubernetes.kubeclient.resources.KubernetesConfigMapWatcher","thread_name":"flink-akka.actor.default-dispatcher-68","level":"INFO","level_value":20000}
{"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Ignoring
onClose for already closed/closing
websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Closing
websocket
org.apache.flink.kubernetes.shaded.okhttp3.internal.ws.RealWebSocket@2a1892aa","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"flink-akka.actor.default-dispatcher-68","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"WebSocket
close received. code: 1000, reason:
","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000}
{"@timestamp":"2021-10-19T04:53:17.193+02:00","@version":"1","message":"Ignoring
onClose for already closed/closing
websocket","logger_name":"io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager","thread_name":"OkHttp
https://172.31.64.1/...","level":"DEBUG","level_value":10000}
