heiya07 created FLINK-35867:
-------------------------------
Summary: 99% state lost after task auto restart due to "rocksdb
block checksum mismatch" exception
Key: FLINK-35867
URL: https://issues.apache.org/jira/browse/FLINK-35867
Project: Flink
Issue Type: Bug
Components: API / DataStream, Runtime / State Backends
Affects Versions: 1.19.0
Environment: Flink version: 1.19.0
Linux: Alpine 3.20 runs on k8s
JDK17: OpenJDK 64-Bit Server VM - Alpine - 17/17.0.11+9-alpine-r0
Flink Configuration:
{code:java}
$internal.application.main com.xxxx.XXX
$internal.application.program-args []
$internal.pipeline.job-id 78a228640b847a30c4a398a95880ade8
blob.server.port 6124
dstl.dfs.base-path s3://xxxxxxxxxxxxxxxxxxxxxx/checkpoints
dstl.dfs.upload.max-attempts 6
dstl.dfs.upload.next-attempt-delay 5 s
dstl.dfs.upload.timeout 5 s
env.java.opts.all --add-exports=java.base/sun.net.util=ALL-UNNAMED
--add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED
--add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED
--add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED
--add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED
--add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED
--add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED
--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
--add-opens=java.base/java.nio=ALL-UNNAMED
--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
--add-opens=java.base/java.lang.reflect=ALL-UNNAMED
--add-opens=java.base/java.text=ALL-UNNAMED
--add-opens=java.base/java.time=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED
execution.checkpointing.aligned-checkpoint-timeout 15 s
execution.checkpointing.interval 30 s
execution.checkpointing.min-pause 30 s
execution.checkpointing.timeout 10 min
execution.checkpointing.tolerable-failed-checkpoints 5
execution.checkpointing.unaligned true
execution.savepoint-restore-mode CLAIM
execution.savepoint.ignore-unclaimed-state false
execution.target embedded
heartbeat.interval 60000
heartbeat.timeout 300000
internal.cluster.execution-mode NORMAL
jobmanager.archive.fs.dir s3://xxxxxxxxxxxxxxxxxx/history
jobmanager.memory.heap.size 469762048b
jobmanager.memory.jvm-metaspace.size 268435456b
jobmanager.memory.jvm-overhead.max 201326592b
jobmanager.memory.jvm-overhead.min 201326592b
jobmanager.memory.off-heap.size 134217728b
jobmanager.memory.process.size 1 gb
jobmanager.rpc.address xxxxxxx.xxxxxxx
jobmanager.rpc.port 6123
kubernetes.cluster-id xxxxxxxxxx
kubernetes.internal.jobmanager.entrypoint.class
org.apache.flink.kubernetes.entrypoint.KubernetesApplicationClusterEntrypoint
kubernetes.jobmanager.cpu.amount 0.5
kubernetes.jobmanager.labels xxxxxxxxxxxxxxxxxxxxxxxxxx
kubernetes.jobmanager.service-account xxxxxxxxxxx
kubernetes.namespace xxxxxxxx
kubernetes.pod-template-file.jobmanager /tmp/xxxxxxx.tmp
kubernetes.pod-template-file.taskmanager /tmp/xxxxxxxx.tmp
kubernetes.rest-service.exposed.type ClusterIP
kubernetes.taskmanager.cpu.amount 1
metrics.reporter.prom.class
org.apache.flink.metrics.prometheus.PrometheusReporter
metrics.reporter.prom.port 9249
parallelism.default 1
pipeline.classpaths []
pipeline.jars ['file:/opt/flink/usrlib/xxxxxxxx.jar']
pipeline.operator-chaining false
restart-strategy.failure-rate.delay 30 s
restart-strategy.failure-rate.failure-rate-interval 2 min
restart-strategy.failure-rate.max-failures-per-interval 5
restart-strategy.type failure-rate
s3.access-key xxxxxxxxx
s3.endpoint xxxxxxxxx
s3.path.style.access true
s3.secret-key ******
s3.ssl.enabled false
state.backend.changelog.enabled true
state.backend.changelog.periodic-materialize.interval 5 min
state.backend.changelog.storage filesystem
state.backend.local-recovery true
state.backend.rocksdb.log.dir /opt/flink/log/rocksdb
state.backend.rocksdb.log.level WARN_LEVEL
state.backend.type rocksdb
state.checkpoints.dir s3://xxxxxxxxxxx/checkpoints
taskmanager.memory.jvm-metaspace.size 150 mb
taskmanager.memory.jvm-overhead.max 200 mb
taskmanager.memory.managed.fraction 0.64
taskmanager.memory.network.max 20 mb
taskmanager.memory.network.min 16 mb
taskmanager.memory.process.size 3 gb
taskmanager.memory.task.off-heap.size 200 mb
taskmanager.numberOfTaskSlots 1
taskmanager.rpc.port 6122
web.tmpdir /tmp/flink-web-86fc04dd-19fb-4775-8084-52a263c827b7 {code}
Reporter: heiya07
I hava a Flink task encounters an "rocksdb block checksum mismatch" exception,
and it auto restart successfully as I have enabled the task failure restart
strategy, but after the task restarted, I found that almost all the state was
lost. here is the job log:
{code:java}
2024-07-12 18:28:24,818 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36218 for job 108a64cca6f69d6185679243cf2614c1 (5561263810 bytes,
checkpointDuration=69663 ms, finalizationTime=0 ms).
2024-07-12 18:28:54,819 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36219 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780134818 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:29:21,329 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36219 for job 108a64cca6f69d6185679243cf2614c1 (5557919089 bytes,
checkpointDuration=26510 ms, finalizationTime=1 ms).
2024-07-12 18:29:51,330 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36220 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780191330 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:29:51,669 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36220 for job 108a64cca6f69d6185679243cf2614c1 (5559503775 bytes,
checkpointDuration=339 ms, finalizationTime=0 ms).
2024-07-12 18:30:21,669 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36221 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780221669 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:30:21,881 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36221 for job 108a64cca6f69d6185679243cf2614c1 (5560859350 bytes,
checkpointDuration=211 ms, finalizationTime=1 ms).
2024-07-12 18:30:51,881 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36222 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780251881 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:31:01,769 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36222 for job 108a64cca6f69d6185679243cf2614c1 (5562228091 bytes,
checkpointDuration=9888 ms, finalizationTime=0 ms).
2024-07-12 18:31:31,769 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36223 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780291769 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:31:32,040 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36223 for job 108a64cca6f69d6185679243cf2614c1 (5561271317 bytes,
checkpointDuration=271 ms, finalizationTime=0 ms).
2024-07-12 18:32:02,041 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36224 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780322041 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:32:16,495 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36224 for job 108a64cca6f69d6185679243cf2614c1 (5562726599 bytes,
checkpointDuration=14454 ms, finalizationTime=0 ms).
2024-07-12 18:32:46,495 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36225 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780366495 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:32:46,762 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36225 for job 108a64cca6f69d6185679243cf2614c1 (5562130180 bytes,
checkpointDuration=267 ms, finalizationTime=0 ms).
2024-07-12 18:33:01,688 INFO org.apache.kafka.clients.NetworkClient
[] - [AdminClient
clientId=new_test_3eLbD7Wu-enumerator-admin-client] Node 2 disconnected.
2024-07-12 18:33:16,762 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36226 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780396762 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:34:23,110 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36226 for job 108a64cca6f69d6185679243cf2614c1 (5563534332 bytes,
checkpointDuration=66348 ms, finalizationTime=0 ms).
2024-07-12 18:34:53,110 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36227 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780493110 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:35:52,759 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36227 for job 108a64cca6f69d6185679243cf2614c1 (5564780617 bytes,
checkpointDuration=59649 ms, finalizationTime=0 ms).
2024-07-12 18:36:22,759 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36228 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780582759 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:36:41,048 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36228 for job 108a64cca6f69d6185679243cf2614c1 (5564705694 bytes,
checkpointDuration=18288 ms, finalizationTime=1 ms).
2024-07-12 18:37:11,048 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36229 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780631048 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:37:11,306 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36229 for job 108a64cca6f69d6185679243cf2614c1 (5561636978 bytes,
checkpointDuration=257 ms, finalizationTime=0 ms).
2024-07-12 18:37:41,305 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36230 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780661305 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:37:48,013 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36230 for job 108a64cca6f69d6185679243cf2614c1 (5562888186 bytes,
checkpointDuration=6708 ms, finalizationTime=0 ms).
2024-07-12 18:38:01,789 INFO org.apache.kafka.clients.NetworkClient
[] - [AdminClient
clientId=new_test_3eLbD7Wu-enumerator-admin-client] Node 3 disconnected.
2024-07-12 18:38:18,014 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36231 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780698013 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:38:18,281 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36231 for job 108a64cca6f69d6185679243cf2614c1 (5562168310 bytes,
checkpointDuration=268 ms, finalizationTime=0 ms).
2024-07-12 18:38:48,281 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36232 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780728281 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:38:48,490 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36232 for job 108a64cca6f69d6185679243cf2614c1 (5563567463 bytes,
checkpointDuration=209 ms, finalizationTime=0 ms).
2024-07-12 18:39:18,490 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36233 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780758490 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:39:18,711 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36233 for job 108a64cca6f69d6185679243cf2614c1 (5564807753 bytes,
checkpointDuration=221 ms, finalizationTime=0 ms).
2024-07-12 18:39:48,711 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36234 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780788711 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:39:54,232 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36234 for job 108a64cca6f69d6185679243cf2614c1 (5566129925 bytes,
checkpointDuration=5521 ms, finalizationTime=0 ms).
2024-07-12 18:40:24,232 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36235 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780824232 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:40:35,807 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36235 for job 108a64cca6f69d6185679243cf2614c1 (5561884010 bytes,
checkpointDuration=11574 ms, finalizationTime=1 ms).
2024-07-12 18:41:05,808 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36236 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780865807 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:42:24,718 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36236 for job 108a64cca6f69d6185679243cf2614c1 (5557617450 bytes,
checkpointDuration=78911 ms, finalizationTime=0 ms).
2024-07-12 18:42:54,718 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36237 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720780974718 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:43:01,891 INFO org.apache.kafka.clients.NetworkClient
[] - [AdminClient
clientId=new_test_3eLbD7Wu-enumerator-admin-client] Node 5 disconnected.
2024-07-12 18:43:24,560 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36237 for job 108a64cca6f69d6185679243cf2614c1 (5558955281 bytes,
checkpointDuration=29838 ms, finalizationTime=4 ms).
2024-07-12 18:43:54,560 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36238 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781034560 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:43:54,812 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36238 for job 108a64cca6f69d6185679243cf2614c1 (5558537398 bytes,
checkpointDuration=252 ms, finalizationTime=0 ms).
2024-07-12 18:44:24,812 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36239 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781064812 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:44:25,022 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36239 for job 108a64cca6f69d6185679243cf2614c1 (5559598066 bytes,
checkpointDuration=210 ms, finalizationTime=0 ms).
2024-07-12 18:44:55,023 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36240 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781095022 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:45:25,766 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36240 for job 108a64cca6f69d6185679243cf2614c1 (5560552320 bytes,
checkpointDuration=30744 ms, finalizationTime=0 ms).
2024-07-12 18:45:55,766 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36241 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781155766 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:46:07,411 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36241 for job 108a64cca6f69d6185679243cf2614c1 (5560190293 bytes,
checkpointDuration=11645 ms, finalizationTime=0 ms).
2024-07-12 18:46:37,411 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36242 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781197411 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:46:37,631 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36242 for job 108a64cca6f69d6185679243cf2614c1 (5559238902 bytes,
checkpointDuration=219 ms, finalizationTime=1 ms).
2024-07-12 18:47:07,632 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36243 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781227631 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:47:07,843 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36243 for job 108a64cca6f69d6185679243cf2614c1 (5560131966 bytes,
checkpointDuration=211 ms, finalizationTime=1 ms).
2024-07-12 18:47:37,843 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36244 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781257843 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:48:01,993 INFO org.apache.kafka.clients.NetworkClient
[] - [AdminClient
clientId=new_test_3eLbD7Wu-enumerator-admin-client] Node 4 disconnected.
2024-07-12 18:48:32,551 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 531 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_0)
switched from RUNNING to FAILED on
iop-new-test-3elbd7wu-4on9foiu0bd0-esoy-taskmanager-1-1 @ 10.18.218.80
(dataPort=36613).
org.apache.flink.util.SerializedThrowable: org.rocksdb.RocksDBException: block
checksum mismatch: stored = 2324967102, computed = 955729931 in
/tmp/tm_iop-new-test-3elbd7wu-4on9foiu0bd0-esoy-taskmanager-1-1/tmp/job_108a64cca6f69d6185679243cf2614c1_op_KeyedProcessOperator_996b015f3163ca08af0c30ed8f6247a7__1_1__uuid_e0d4925d-c24b-4d3a-a693-2df28f090798/db/012566.sst
offset 16764263 size 1149
at org.rocksdb.RocksDB.write0(Native Method) ~[flink-dist-1.19.0.jar:1.19.0]
at org.rocksdb.RocksDB.write(RocksDB.java:1784)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.contrib.streaming.state.RocksDBWriteBatchWrapper.flush(RocksDBWriteBatchWrapper.java:126)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.snapshot(RocksDBKeyedStateBackend.java:586)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.state.changelog.ChangelogKeyedStateBackend.initMaterialization(ChangelogKeyedStateBackend.java:796)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.state.common.PeriodicMaterializationManager.lambda$triggerMaterialization$1(PeriodicMaterializationManager.java:225)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
~[flink-dist-1.19.0.jar:1.19.0]
at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:383)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:345)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:909)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:858)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
~[flink-dist-1.19.0.jar:1.19.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:751)
~[flink-dist-1.19.0.jar:1.19.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
~[flink-dist-1.19.0.jar:1.19.0]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-07-12 18:48:32,569 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - 13 tasks will be restarted to recover the failed task
3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_0.
2024-07-12 18:48:32,569 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job Flink
Streaming Job (108a64cca6f69d6185679243cf2614c1) switched from state RUNNING to
RESTARTING.
2024-07-12 18:48:32,579 WARN
org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to
trigger or complete checkpoint 36244 for job 108a64cca6f69d6185679243cf2614c1.
(0 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint Coordinator
is suspending.
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.stopCheckpointScheduler(CheckpointCoordinator.java:2056)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinatorDeActivator.jobStatusChanges(CheckpointCoordinatorDeActivator.java:46)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.notifyJobStatusChange(DefaultExecutionGraph.java:1580)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.transitionState(DefaultExecutionGraph.java:1146)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.transitionState(DefaultExecutionGraph.java:1118)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.SchedulerBase.transitionExecutionGraphState(SchedulerBase.java:600)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.addVerticesToRestartPending(DefaultScheduler.java:382)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.restartTasksWithDelay(DefaultScheduler.java:358)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.maybeRestartTasks(DefaultScheduler.java:326)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:268)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.onTaskFailed(DefaultScheduler.java:261)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.SchedulerBase.onTaskExecutionStateUpdate(SchedulerBase.java:787)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:764)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:83)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:488)
~[flink-dist-1.19.0.jar:1.19.0]
at jdk.internal.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.lambda$handleRpcInvocation$1(PekkoRpcActor.java:309)
~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcInvocation(PekkoRpcActor.java:307)
~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcMessage(PekkoRpcActor.java:222)
~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.flink.runtime.rpc.pekko.FencedPekkoRpcActor.handleRpcMessage(FencedPekkoRpcActor.java:85)
~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleMessage(PekkoRpcActor.java:168)
~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:33)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:29)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at scala.PartialFunction.applyOrElse(PartialFunction.scala:127)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.pekko.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:29)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:545)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.pekko.actor.AbstractActor.aroundReceive(AbstractActor.scala:229)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:280)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:241)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:253)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) [?:?]
at
java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
[?:?]
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) [?:?]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) [?:?]
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
[?:?]
2024-07-12 18:48:32,584 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MySource (1/1)
(3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_0)
switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 528 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_0)
switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 529 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_0)
switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 532 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_0)
switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySink:
Writer (1/1)
(3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_0)
switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 500 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_0)
switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 497 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_0)
switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessA
(1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_0)
switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessB
(1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_0)
switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySource
(1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_0)
switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - GlobalWindows
(1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_0)
switched from RUNNING to CANCELING.
2024-07-12 18:48:32,585 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 530 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_0)
switched from RUNNING to CANCELING.
2024-07-12 18:48:36,934 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 529 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_0)
switched from CANCELING to CANCELED.
2024-07-12 18:48:36,934 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 532 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_0)
switched from CANCELING to CANCELED.
2024-07-12 18:48:36,934 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySink:
Writer (1/1)
(3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_0)
switched from CANCELING to CANCELED.
2024-07-12 18:48:36,934 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessA
(1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_0)
switched from CANCELING to CANCELED.
2024-07-12 18:48:36,935 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MySource (1/1)
(3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_0)
switched from CANCELING to CANCELED.
2024-07-12 18:48:36,935 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySource
(1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_0)
switched from CANCELING to CANCELED.
2024-07-12 18:48:36,935 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessB
(1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_0)
switched from CANCELING to CANCELED.
2024-07-12 18:48:36,935 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - GlobalWindows
(1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_0)
switched from CANCELING to CANCELED.
2024-07-12 18:48:38,440 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 497 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_0)
switched from CANCELING to CANCELED.
2024-07-12 18:48:39,931 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 530 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_0)
switched from CANCELING to CANCELED.
2024-07-12 18:48:40,447 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 500 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_0)
switched from CANCELING to CANCELED.
2024-07-12 18:48:40,646 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 528 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_0)
switched from CANCELING to CANCELED.
2024-07-12 18:48:40,648 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Clearing resource requirements of job 108a64cca6f69d6185679243cf2614c1
2024-07-12 18:48:40,648 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job Flink
Streaming Job (108a64cca6f69d6185679243cf2614c1) switched from state RESTARTING
to RUNNING.
2024-07-12 18:48:40,648 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedTaskManagerTracker
[] - Clear all pending allocations for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:48:40,650 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Restoring job
108a64cca6f69d6185679243cf2614c1 from Checkpoint 36243 @ 1720781227631 for
108a64cca6f69d6185679243cf2614c1 located at
s3://xxxxxxxxxxx/checkpoints/108a64cca6f69d6185679243cf2614c1/chk-36243.
2024-07-12 18:48:40,652 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - No master
state to restore
2024-07-12 18:48:40,654 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MySource (1/1)
(3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_1)
switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,654 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySource
(1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_1)
switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessB
(1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_1)
switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 497 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_1)
switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 530 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_1)
switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 531 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_1)
switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 532 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_1)
switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 500 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_1)
switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 528 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_1)
switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 529 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_1)
switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessA
(1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_1)
switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - GlobalWindows
(1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_1)
switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,655 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySink:
Writer (1/1)
(3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_1)
switched from CREATED to SCHEDULED.
2024-07-12 18:48:40,656 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Received resource requirements from job 108a64cca6f69d6185679243cf2614c1:
[ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
numberOfRequiredSlots=1}]
2024-07-12 18:48:40,657 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MySource (1/1)
(3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_1)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,657 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying
Source: MySource (1/1) (attempt #1) with attempt id
3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_1 and
vertex id bc764cd8ddf7a0cff126f51c16239658_0 to
iop-new-test-3elbd7wu-4on9foiu0bd0-esoy-taskmanager-1-1 @ 10.18.218.80
(dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,657 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySource
(1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_1)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,657 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying
MySource (1/1) (attempt #1) with attempt id
3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_1 and
vertex id d24cade8b1c6d9528ce19407131a7eda_0 to
iop-new-test-3elbd7wu-4on9foiu0bd0-esoy-taskmanager-1-1 @ 10.18.218.80
(dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,657 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessB
(1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_1)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,658 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying
ProcessB (1/1) (attempt #1) with attempt id
3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_1 and
vertex id b4596501f3bb302b1c7aedb1a85dd7d4_0 to
iop-new-test-3elbd7wu-4on9foiu0bd0-esoy-taskmanager-1-1 @ 10.18.218.80
(dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,658 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 497 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_1)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,658 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 497
(1/1) (attempt #1) with attempt id
3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_1 and
vertex id 0f257274a8483e4e1c50200b13c3ef28_0 to
iop-new-test-3elbd7wu-4on9foiu0bd0-esoy-taskmanager-1-1 @ 10.18.218.80
(dataPort=36613) with allocation id c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,658 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 530 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_1)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,658 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 530
(1/1) (attempt #1) with attempt id
3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_1 and
vertex id e43de375578634774108d582ac68fec1_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,659 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 531 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_1)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,659 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 531
(1/1) (attempt #1) with attempt id
3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_1 and
vertex id 996b015f3163ca08af0c30ed8f6247a7_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,659 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 532 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_1)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,659 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 532
(1/1) (attempt #1) with attempt id
3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_1 and
vertex id fa9967a8a4db458887f74772e7a3c05c_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,659 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 500 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_1)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,659 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 500
(1/1) (attempt #1) with attempt id
3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_1 and
vertex id 2f010967a103f39821dd1dc8dfbcabe7_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,660 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 528 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_1)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,660 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 528
(1/1) (attempt #1) with attempt id
3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_1 and
vertex id 005bcfbd5a31ac889c9be64a694eb949_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,660 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 529 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_1)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,660 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 529
(1/1) (attempt #1) with attempt id
3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_1 and
vertex id 361c54102873653ce8bbbfb96909315f_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,660 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessA
(1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_1)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,660 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying
ProcessA (1/1) (attempt #1) with attempt id
3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_1 and
vertex id 777214ea96c0589f14708a68600868b9_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,661 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - GlobalWindows
(1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_1)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,661 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying
GlobalWindows (1/1) (attempt #1) with attempt id
3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_1 and
vertex id 242e525045a332cf5da0422bf5582b5d_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,662 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySink:
Writer (1/1)
(3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_1)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:48:40,662 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying
MySink: Writer (1/1) (attempt #1) with attempt id
3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_1 and
vertex id a4eb636d5723c707af0c320f39db4e52_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:48:40,742 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MySource (1/1)
(3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_1)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:40,759 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySource
(1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_1)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:40,759 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessB
(1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_1)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:40,834 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 497 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_1)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:40,933 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MySource (1/1)
(3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_1)
switched from INITIALIZING to RUNNING.
2024-07-12 18:48:40,934 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 530 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_1)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:40,934 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 531 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_1)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:40,936 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 532 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_1)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:41,140 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 500 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_1)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:41,236 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 529 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_1)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:41,236 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 528 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_1)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:41,237 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySource
(1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_1)
switched from INITIALIZING to RUNNING.
2024-07-12 18:48:41,332 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessA
(1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_1)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:41,337 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - GlobalWindows
(1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_1)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:41,340 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySink:
Writer (1/1)
(3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_1)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:48:41,536 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessB
(1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_1)
switched from INITIALIZING to RUNNING.
2024-07-12 18:48:41,737 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySink:
Writer (1/1)
(3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_1)
switched from INITIALIZING to RUNNING.
2024-07-12 18:48:42,029 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessA
(1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_1)
switched from INITIALIZING to RUNNING.
2024-07-12 18:48:42,633 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - GlobalWindows
(1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_1)
switched from INITIALIZING to RUNNING.
2024-07-12 18:48:43,432 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 497 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_1)
switched from INITIALIZING to RUNNING.
2024-07-12 18:48:43,538 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 528 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_1)
switched from INITIALIZING to RUNNING.
2024-07-12 18:48:43,538 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 500 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_1)
switched from INITIALIZING to RUNNING.
2024-07-12 18:48:44,032 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 531 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_1)
switched from INITIALIZING to RUNNING.
2024-07-12 18:48:44,036 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 529 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_1)
switched from INITIALIZING to RUNNING.
2024-07-12 18:48:44,337 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 532 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_1)
switched from INITIALIZING to RUNNING.
2024-07-12 18:48:45,030 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 530 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_1)
switched from INITIALIZING to RUNNING.
2024-07-12 18:49:10,649 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36245 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781350649 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:49:11,196 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36245 for job 108a64cca6f69d6185679243cf2614c1 (5565178747 bytes,
checkpointDuration=546 ms, finalizationTime=0 ms).
2024-07-12 18:49:41,195 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36246 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781381195 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:49:41,384 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36246 for job 108a64cca6f69d6185679243cf2614c1 (5565960808 bytes,
checkpointDuration=189 ms, finalizationTime=0 ms).
2024-07-12 18:50:11,384 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36247 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781411384 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:50:11,580 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36247 for job 108a64cca6f69d6185679243cf2614c1 (5566901598 bytes,
checkpointDuration=196 ms, finalizationTime=0 ms).
2024-07-12 18:50:41,581 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36248 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781441580 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:51:16,833 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 531 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_1)
switched from RUNNING to FAILED on my-task-esoy-taskmanager-1-1 @ 10.18.218.80
(dataPort=36613).
org.apache.flink.util.SerializedThrowable: org.rocksdb.RocksDBException: block
checksum mismatch: stored = 2324967102, computed = 955729931 in
/tmp/tm_my-task-esoy-taskmanager-1-1/tmp/job_108a64cca6f69d6185679243cf2614c1_op_KeyedProcessOperator_996b015f3163ca08af0c30ed8f6247a7__1_1__uuid_97a224f6-fba4-4733-8709-55fba7b4f020/db/012566.sst
offset 16764263 size 1149
at org.rocksdb.RocksDB.write0(Native Method) ~[flink-dist-1.19.0.jar:1.19.0]
at org.rocksdb.RocksDB.write(RocksDB.java:1784)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.contrib.streaming.state.RocksDBWriteBatchWrapper.flush(RocksDBWriteBatchWrapper.java:126)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.snapshot(RocksDBKeyedStateBackend.java:586)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.state.changelog.ChangelogKeyedStateBackend.initMaterialization(ChangelogKeyedStateBackend.java:796)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.state.common.PeriodicMaterializationManager.lambda$triggerMaterialization$1(PeriodicMaterializationManager.java:225)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
~[flink-dist-1.19.0.jar:1.19.0]
at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:367)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:352)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:909)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:858)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
~[flink-dist-1.19.0.jar:1.19.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:751)
~[flink-dist-1.19.0.jar:1.19.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
~[flink-dist-1.19.0.jar:1.19.0]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-07-12 18:51:16,834 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - 13 tasks will be restarted to recover the failed task
3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_1.
2024-07-12 18:51:16,834 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job Flink
Streaming Job (108a64cca6f69d6185679243cf2614c1) switched from state RUNNING to
RESTARTING.
2024-07-12 18:51:16,834 WARN
org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to
trigger or complete checkpoint 36248 for job 108a64cca6f69d6185679243cf2614c1.
(0 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint Coordinator
is suspending.
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.stopCheckpointScheduler(CheckpointCoordinator.java:2056)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinatorDeActivator.jobStatusChanges(CheckpointCoordinatorDeActivator.java:46)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.notifyJobStatusChange(DefaultExecutionGraph.java:1580)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.transitionState(DefaultExecutionGraph.java:1146)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.transitionState(DefaultExecutionGraph.java:1118)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.SchedulerBase.transitionExecutionGraphState(SchedulerBase.java:600)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.addVerticesToRestartPending(DefaultScheduler.java:382)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.restartTasksWithDelay(DefaultScheduler.java:358)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.maybeRestartTasks(DefaultScheduler.java:326)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:268)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.onTaskFailed(DefaultScheduler.java:261)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.SchedulerBase.onTaskExecutionStateUpdate(SchedulerBase.java:787)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:764)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:83)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:488)
~[flink-dist-1.19.0.jar:1.19.0]
at jdk.internal.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.lambda$handleRpcInvocation$1(PekkoRpcActor.java:309)
~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcInvocation(PekkoRpcActor.java:307)
~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcMessage(PekkoRpcActor.java:222)
~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.flink.runtime.rpc.pekko.FencedPekkoRpcActor.handleRpcMessage(FencedPekkoRpcActor.java:85)
~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleMessage(PekkoRpcActor.java:168)
~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:33)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:29)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at scala.PartialFunction.applyOrElse(PartialFunction.scala:127)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.pekko.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:29)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:545)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.pekko.actor.AbstractActor.aroundReceive(AbstractActor.scala:229)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:280)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:241)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:253)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) [?:?]
at
java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
[?:?]
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) [?:?]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) [?:?]
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
[?:?]
2024-07-12 18:51:16,836 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MySource (1/1)
(3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_1)
switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 528 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_1)
switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 529 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_1)
switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 532 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_1)
switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySink:
Writer (1/1)
(3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_1)
switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 500 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_1)
switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 497 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_1)
switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessA
(1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_1)
switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessB
(1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_1)
switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySource
(1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_1)
switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - GlobalWindows
(1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_1)
switched from RUNNING to CANCELING.
2024-07-12 18:51:16,836 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 530 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_1)
switched from RUNNING to CANCELING.
2024-07-12 18:51:22,531 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MySource (1/1)
(3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_1)
switched from CANCELING to CANCELED.
2024-07-12 18:51:22,531 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 529 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_1)
switched from CANCELING to CANCELED.
2024-07-12 18:51:22,531 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySink:
Writer (1/1)
(3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_1)
switched from CANCELING to CANCELED.
2024-07-12 18:51:22,630 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessA
(1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_1)
switched from CANCELING to CANCELED.
2024-07-12 18:51:22,630 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySource
(1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_1)
switched from CANCELING to CANCELED.
2024-07-12 18:51:22,631 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessB
(1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_1)
switched from CANCELING to CANCELED.
2024-07-12 18:51:22,631 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - GlobalWindows
(1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_1)
switched from CANCELING to CANCELED.
2024-07-12 18:51:23,231 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 500 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_1)
switched from CANCELING to CANCELED.
2024-07-12 18:51:24,132 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 528 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_1)
switched from CANCELING to CANCELED.
2024-07-12 18:51:24,232 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 530 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_1)
switched from CANCELING to CANCELED.
2024-07-12 18:51:24,330 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 497 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_1)
switched from CANCELING to CANCELED.
2024-07-12 18:51:24,345 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 532 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_1)
switched from CANCELING to CANCELED.
2024-07-12 18:51:24,346 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Clearing resource requirements of job 108a64cca6f69d6185679243cf2614c1
2024-07-12 18:51:24,346 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job Flink
Streaming Job (108a64cca6f69d6185679243cf2614c1) switched from state RESTARTING
to RUNNING.
2024-07-12 18:51:24,346 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedTaskManagerTracker
[] - Clear all pending allocations for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:51:24,346 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Restoring job
108a64cca6f69d6185679243cf2614c1 from Checkpoint 36247 @ 1720781411384 for
108a64cca6f69d6185679243cf2614c1 located at
s3://xxxxxxxxxxx/checkpoints/108a64cca6f69d6185679243cf2614c1/chk-36247.
2024-07-12 18:51:24,348 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - No master
state to restore
2024-07-12 18:51:24,348 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MySource (1/1)
(3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_2)
switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySource
(1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_2)
switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessB
(1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_2)
switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 497 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_2)
switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 530 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_2)
switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 531 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_2)
switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 532 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_2)
switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 500 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_2)
switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 528 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_2)
switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 529 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_2)
switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessA
(1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_2)
switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - GlobalWindows
(1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_2)
switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,348 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySink:
Writer (1/1)
(3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_2)
switched from CREATED to SCHEDULED.
2024-07-12 18:51:24,349 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Received resource requirements from job 108a64cca6f69d6185679243cf2614c1:
[ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
numberOfRequiredSlots=1}]
2024-07-12 18:51:24,349 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MySource (1/1)
(3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_2)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,349 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying
Source: MySource (1/1) (attempt #2) with attempt id
3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_2 and
vertex id bc764cd8ddf7a0cff126f51c16239658_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,350 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySource
(1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_2)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,350 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying
MySource (1/1) (attempt #2) with attempt id
3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_2 and
vertex id d24cade8b1c6d9528ce19407131a7eda_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,350 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessB
(1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_2)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,350 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying
ProcessB (1/1) (attempt #2) with attempt id
3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_2 and
vertex id b4596501f3bb302b1c7aedb1a85dd7d4_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,350 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 497 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_2)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,350 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 497
(1/1) (attempt #2) with attempt id
3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_2 and
vertex id 0f257274a8483e4e1c50200b13c3ef28_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,351 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 530 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_2)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,351 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 530
(1/1) (attempt #2) with attempt id
3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_2 and
vertex id e43de375578634774108d582ac68fec1_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,351 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 531 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_2)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,351 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 531
(1/1) (attempt #2) with attempt id
3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_2 and
vertex id 996b015f3163ca08af0c30ed8f6247a7_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,351 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 532 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_2)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,351 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 532
(1/1) (attempt #2) with attempt id
3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_2 and
vertex id fa9967a8a4db458887f74772e7a3c05c_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,351 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 500 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_2)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,351 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 500
(1/1) (attempt #2) with attempt id
3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_2 and
vertex id 2f010967a103f39821dd1dc8dfbcabe7_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,351 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 528 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_2)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,351 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 528
(1/1) (attempt #2) with attempt id
3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_2 and
vertex id 005bcfbd5a31ac889c9be64a694eb949_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,352 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 529 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_2)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,352 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 529
(1/1) (attempt #2) with attempt id
3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_2 and
vertex id 361c54102873653ce8bbbfb96909315f_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,352 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessA
(1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_2)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,352 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying
ProcessA (1/1) (attempt #2) with attempt id
3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_2 and
vertex id 777214ea96c0589f14708a68600868b9_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,353 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - GlobalWindows
(1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_2)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,353 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying
GlobalWindows (1/1) (attempt #2) with attempt id
3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_2 and
vertex id 242e525045a332cf5da0422bf5582b5d_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,353 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySink:
Writer (1/1)
(3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_2)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:51:24,353 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying
MySink: Writer (1/1) (attempt #2) with attempt id
3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_2 and
vertex id a4eb636d5723c707af0c320f39db4e52_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:51:24,433 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MySource (1/1)
(3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_2)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,437 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySource
(1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_2)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,445 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 497 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_2)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,529 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessB
(1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_2)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,534 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 530 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_2)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,537 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 531 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_2)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,630 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 532 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_2)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,638 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 500 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_2)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,731 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MySource (1/1)
(3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_2)
switched from INITIALIZING to RUNNING.
2024-07-12 18:51:24,732 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 528 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_2)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,733 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySource
(1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_2)
switched from INITIALIZING to RUNNING.
2024-07-12 18:51:24,842 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 529 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_2)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,933 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessA
(1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_2)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,938 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - GlobalWindows
(1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_2)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:24,940 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySink:
Writer (1/1)
(3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_2)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:51:25,230 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessB
(1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_2)
switched from INITIALIZING to RUNNING.
2024-07-12 18:51:25,232 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySink:
Writer (1/1)
(3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_2)
switched from INITIALIZING to RUNNING.
2024-07-12 18:51:25,434 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessA
(1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_2)
switched from INITIALIZING to RUNNING.
2024-07-12 18:51:26,031 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - GlobalWindows
(1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_2)
switched from INITIALIZING to RUNNING.
2024-07-12 18:51:54,346 INFO
org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to
trigger checkpoint for job 108a64cca6f69d6185679243cf2614c1 since Checkpoint
triggering task 497 (1/1) of job 108a64cca6f69d6185679243cf2614c1 is not being
executed at the moment. Aborting checkpoint. Failure reason: Not all required
tasks are currently running..
2024-07-12 18:51:56,413 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 530 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_2)
switched from INITIALIZING to RUNNING.
2024-07-12 18:51:57,636 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 500 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_2)
switched from INITIALIZING to RUNNING.
2024-07-12 18:51:57,729 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 497 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_2)
switched from INITIALIZING to RUNNING.
2024-07-12 18:51:57,734 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 528 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_2)
switched from INITIALIZING to RUNNING.
2024-07-12 18:51:58,043 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 531 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_2)
switched from INITIALIZING to RUNNING.
2024-07-12 18:51:58,448 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 529 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_2)
switched from INITIALIZING to RUNNING.
2024-07-12 18:51:58,630 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 532 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_2)
switched from INITIALIZING to RUNNING.
2024-07-12 18:52:24,346 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36249 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781544346 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:52:24,763 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36249 for job 108a64cca6f69d6185679243cf2614c1 (5571063521 bytes,
checkpointDuration=417 ms, finalizationTime=0 ms).
2024-07-12 18:52:54,763 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36250 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781574763 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:52:55,134 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36250 for job 108a64cca6f69d6185679243cf2614c1 (5572066981 bytes,
checkpointDuration=370 ms, finalizationTime=1 ms).
2024-07-12 18:53:02,064 INFO org.apache.kafka.clients.NetworkClient
[] - [AdminClient
clientId=new_test_3eLbD7Wu-enumerator-admin-client] Node 1 disconnected.
2024-07-12 18:53:25,135 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36251 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781605134 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:53:25,336 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36251 for job 108a64cca6f69d6185679243cf2614c1 (5573138583 bytes,
checkpointDuration=201 ms, finalizationTime=1 ms).
2024-07-12 18:53:55,337 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36252 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781635337 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:54:30,532 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 531 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_2)
switched from RUNNING to FAILED on my-task-esoy-taskmanager-1-1 @ 10.18.218.80
(dataPort=36613).
org.apache.flink.util.SerializedThrowable: org.rocksdb.RocksDBException: block
checksum mismatch: stored = 2324967102, computed = 955729931 in
/tmp/tm_my-task-esoy-taskmanager-1-1/tmp/job_108a64cca6f69d6185679243cf2614c1_op_KeyedProcessOperator_996b015f3163ca08af0c30ed8f6247a7__1_1__uuid_7a62e596-fca3-4f5e-8905-55fa2f15254e/db/012566.sst
offset 16764263 size 1149
at org.rocksdb.RocksDB.write0(Native Method) ~[flink-dist-1.19.0.jar:1.19.0]
at org.rocksdb.RocksDB.write(RocksDB.java:1784)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.contrib.streaming.state.RocksDBWriteBatchWrapper.flush(RocksDBWriteBatchWrapper.java:126)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackend.snapshot(RocksDBKeyedStateBackend.java:586)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.state.changelog.ChangelogKeyedStateBackend.initMaterialization(ChangelogKeyedStateBackend.java:796)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.state.common.PeriodicMaterializationManager.lambda$triggerMaterialization$1(PeriodicMaterializationManager.java:225)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
~[flink-dist-1.19.0.jar:1.19.0]
at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:367)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:352)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:909)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:858)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
~[flink-dist-1.19.0.jar:1.19.0]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:751)
~[flink-dist-1.19.0.jar:1.19.0]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:566)
~[flink-dist-1.19.0.jar:1.19.0]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
2024-07-12 18:54:30,535 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - 13 tasks will be restarted to recover the failed task
3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_2.
2024-07-12 18:54:30,535 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job Flink
Streaming Job (108a64cca6f69d6185679243cf2614c1) switched from state RUNNING to
RESTARTING.
2024-07-12 18:54:30,535 WARN
org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to
trigger or complete checkpoint 36252 for job 108a64cca6f69d6185679243cf2614c1.
(0 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint Coordinator
is suspending.
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinator.stopCheckpointScheduler(CheckpointCoordinator.java:2056)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.checkpoint.CheckpointCoordinatorDeActivator.jobStatusChanges(CheckpointCoordinatorDeActivator.java:46)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.notifyJobStatusChange(DefaultExecutionGraph.java:1580)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.transitionState(DefaultExecutionGraph.java:1146)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.transitionState(DefaultExecutionGraph.java:1118)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.SchedulerBase.transitionExecutionGraphState(SchedulerBase.java:600)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.addVerticesToRestartPending(DefaultScheduler.java:382)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.restartTasksWithDelay(DefaultScheduler.java:358)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.maybeRestartTasks(DefaultScheduler.java:326)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:268)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.DefaultScheduler.onTaskFailed(DefaultScheduler.java:261)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.SchedulerBase.onTaskExecutionStateUpdate(SchedulerBase.java:787)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:764)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.scheduler.SchedulerNG.updateTaskExecutionState(SchedulerNG.java:83)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:488)
~[flink-dist-1.19.0.jar:1.19.0]
at jdk.internal.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.lambda$handleRpcInvocation$1(PekkoRpcActor.java:309)
~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:83)
~[flink-dist-1.19.0.jar:1.19.0]
at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcInvocation(PekkoRpcActor.java:307)
~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleRpcMessage(PekkoRpcActor.java:222)
~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.flink.runtime.rpc.pekko.FencedPekkoRpcActor.handleRpcMessage(FencedPekkoRpcActor.java:85)
~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.flink.runtime.rpc.pekko.PekkoRpcActor.handleMessage(PekkoRpcActor.java:168)
~[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:33)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.pekko.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:29)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at scala.PartialFunction.applyOrElse(PartialFunction.scala:127)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at scala.PartialFunction.applyOrElse$(PartialFunction.scala:126)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.pekko.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:29)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:176)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.actor.Actor.aroundReceive(Actor.scala:547)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.actor.Actor.aroundReceive$(Actor.scala:545)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at
org.apache.pekko.actor.AbstractActor.aroundReceive(AbstractActor.scala:229)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.actor.ActorCell.receiveMessage(ActorCell.scala:590)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.actor.ActorCell.invoke(ActorCell.scala:557)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.dispatch.Mailbox.processMailbox(Mailbox.scala:280)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.dispatch.Mailbox.run(Mailbox.scala:241)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at org.apache.pekko.dispatch.Mailbox.exec(Mailbox.scala:253)
[flink-rpc-akkaf7d3f76c-4aa1-40e3-83a3-06aa2eb107d9.jar:1.19.0]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) [?:?]
at
java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
[?:?]
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) [?:?]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) [?:?]
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
[?:?]
2024-07-12 18:54:30,537 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MySource (1/1)
(3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_2)
switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 528 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_2)
switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 529 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_2)
switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 532 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_2)
switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySink:
Writer (1/1)
(3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_2)
switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 500 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_2)
switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 497 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_2)
switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessA
(1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_2)
switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessB
(1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_2)
switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySource
(1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_2)
switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - GlobalWindows
(1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_2)
switched from RUNNING to CANCELING.
2024-07-12 18:54:30,538 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 530 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_2)
switched from RUNNING to CANCELING.
2024-07-12 18:54:37,032 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySink:
Writer (1/1)
(3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_2)
switched from CANCELING to CANCELED.
2024-07-12 18:54:37,032 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MySource (1/1)
(3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_2)
switched from CANCELING to CANCELED.
2024-07-12 18:54:37,033 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 529 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_2)
switched from CANCELING to CANCELED.
2024-07-12 18:54:37,033 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessA
(1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_2)
switched from CANCELING to CANCELED.
2024-07-12 18:54:37,033 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySource
(1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_2)
switched from CANCELING to CANCELED.
2024-07-12 18:54:37,033 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessB
(1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_2)
switched from CANCELING to CANCELED.
2024-07-12 18:54:37,034 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - GlobalWindows
(1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_2)
switched from CANCELING to CANCELED.
2024-07-12 18:54:37,439 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 530 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_2)
switched from CANCELING to CANCELED.
2024-07-12 18:54:39,845 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 528 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_2)
switched from CANCELING to CANCELED.
2024-07-12 18:54:40,047 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 532 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_2)
switched from CANCELING to CANCELED.
2024-07-12 18:54:40,630 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 497 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_2)
switched from CANCELING to CANCELED.
2024-07-12 18:54:41,218 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 500 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_2)
switched from CANCELING to CANCELED.
2024-07-12 18:54:41,218 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job Flink
Streaming Job (108a64cca6f69d6185679243cf2614c1) switched from state RESTARTING
to RUNNING.
2024-07-12 18:54:41,219 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Clearing resource requirements of job 108a64cca6f69d6185679243cf2614c1
2024-07-12 18:54:41,219 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedTaskManagerTracker
[] - Clear all pending allocations for job 108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:54:41,219 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Restoring job
108a64cca6f69d6185679243cf2614c1 from Checkpoint 36251 @ 1720781605134 for
108a64cca6f69d6185679243cf2614c1 located at
s3://xxxxxxxxxxx/checkpoints/108a64cca6f69d6185679243cf2614c1/chk-36251.
2024-07-12 18:54:41,220 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - No master
state to restore
2024-07-12 18:54:41,221 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MySource (1/1)
(3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_3)
switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySource
(1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_3)
switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessB
(1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_3)
switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 497 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_3)
switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 530 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_3)
switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 531 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_3)
switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 532 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_3)
switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 500 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_3)
switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 528 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_3)
switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 529 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_3)
switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessA
(1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_3)
switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - GlobalWindows
(1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_3)
switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySink:
Writer (1/1)
(3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_3)
switched from CREATED to SCHEDULED.
2024-07-12 18:54:41,221 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManager []
- Received resource requirements from job 108a64cca6f69d6185679243cf2614c1:
[ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
numberOfRequiredSlots=1}]
2024-07-12 18:54:41,222 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MySource (1/1)
(3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_3)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,222 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying
Source: MySource (1/1) (attempt #3) with attempt id
3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_3 and
vertex id bc764cd8ddf7a0cff126f51c16239658_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,222 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySource
(1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_3)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,222 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying
MySource (1/1) (attempt #3) with attempt id
3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_3 and
vertex id d24cade8b1c6d9528ce19407131a7eda_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,223 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessB
(1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_3)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,223 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying
ProcessB (1/1) (attempt #3) with attempt id
3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_3 and
vertex id b4596501f3bb302b1c7aedb1a85dd7d4_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,223 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 497 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_3)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,223 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 497
(1/1) (attempt #3) with attempt id
3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_3 and
vertex id 0f257274a8483e4e1c50200b13c3ef28_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,224 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 530 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_3)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,224 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 530
(1/1) (attempt #3) with attempt id
3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_3 and
vertex id e43de375578634774108d582ac68fec1_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,224 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 531 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_3)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,224 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 531
(1/1) (attempt #3) with attempt id
3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_3 and
vertex id 996b015f3163ca08af0c30ed8f6247a7_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,224 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 532 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_3)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,224 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 532
(1/1) (attempt #3) with attempt id
3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_3 and
vertex id fa9967a8a4db458887f74772e7a3c05c_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,224 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 500 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_3)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,225 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 500
(1/1) (attempt #3) with attempt id
3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_3 and
vertex id 2f010967a103f39821dd1dc8dfbcabe7_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,225 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 528 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_3)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,225 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 528
(1/1) (attempt #3) with attempt id
3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_3 and
vertex id 005bcfbd5a31ac889c9be64a694eb949_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,225 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 529 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_3)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,225 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying 529
(1/1) (attempt #3) with attempt id
3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_3 and
vertex id 361c54102873653ce8bbbfb96909315f_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,225 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessA
(1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_3)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,225 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying
ProcessA (1/1) (attempt #3) with attempt id
3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_3 and
vertex id 777214ea96c0589f14708a68600868b9_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,227 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - GlobalWindows
(1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_3)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,227 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying
GlobalWindows (1/1) (attempt #3) with attempt id
3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_3 and
vertex id 242e525045a332cf5da0422bf5582b5d_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,227 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySink:
Writer (1/1)
(3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_3)
switched from SCHEDULED to DEPLOYING.
2024-07-12 18:54:41,227 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Deploying
MySink: Writer (1/1) (attempt #3) with attempt id
3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_3 and
vertex id a4eb636d5723c707af0c320f39db4e52_0 to my-task-esoy-taskmanager-1-1 @
10.18.218.80 (dataPort=36613) with allocation id
c8a41fd9e0cd9e3f94af236c6d590c7b
2024-07-12 18:54:41,246 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MySource (1/1)
(3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_3)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,248 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySource
(1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_3)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,258 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessB
(1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_3)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,333 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MySource (1/1)
(3d2570153eb2b48554ffc1627a5e4881_bc764cd8ddf7a0cff126f51c16239658_0_3)
switched from INITIALIZING to RUNNING.
2024-07-12 18:54:41,334 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySource
(1/1) (3d2570153eb2b48554ffc1627a5e4881_d24cade8b1c6d9528ce19407131a7eda_0_3)
switched from INITIALIZING to RUNNING.
2024-07-12 18:54:41,335 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 497 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_3)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,339 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 530 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_3)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,431 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 531 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_3)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,547 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 532 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_3)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,548 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 500 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_3)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,631 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 528 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_3)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,633 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 529 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_3)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,637 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessA
(1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_3)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,732 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - GlobalWindows
(1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_3)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,733 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySink:
Writer (1/1)
(3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_3)
switched from DEPLOYING to INITIALIZING.
2024-07-12 18:54:41,739 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessB
(1/1) (3d2570153eb2b48554ffc1627a5e4881_b4596501f3bb302b1c7aedb1a85dd7d4_0_3)
switched from INITIALIZING to RUNNING.
2024-07-12 18:54:41,931 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - MySink:
Writer (1/1)
(3d2570153eb2b48554ffc1627a5e4881_a4eb636d5723c707af0c320f39db4e52_0_3)
switched from INITIALIZING to RUNNING.
2024-07-12 18:54:41,935 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 531 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_996b015f3163ca08af0c30ed8f6247a7_0_3)
switched from INITIALIZING to RUNNING.
2024-07-12 18:54:41,937 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 497 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_0f257274a8483e4e1c50200b13c3ef28_0_3)
switched from INITIALIZING to RUNNING.
2024-07-12 18:54:42,133 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - ProcessA
(1/1) (3d2570153eb2b48554ffc1627a5e4881_777214ea96c0589f14708a68600868b9_0_3)
switched from INITIALIZING to RUNNING.
2024-07-12 18:54:42,231 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 500 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_2f010967a103f39821dd1dc8dfbcabe7_0_3)
switched from INITIALIZING to RUNNING.
2024-07-12 18:54:42,332 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 528 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_005bcfbd5a31ac889c9be64a694eb949_0_3)
switched from INITIALIZING to RUNNING.
2024-07-12 18:54:42,652 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 529 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_361c54102873653ce8bbbfb96909315f_0_3)
switched from INITIALIZING to RUNNING.
2024-07-12 18:54:42,749 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 532 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_fa9967a8a4db458887f74772e7a3c05c_0_3)
switched from INITIALIZING to RUNNING.
2024-07-12 18:54:42,749 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - 530 (1/1)
(3d2570153eb2b48554ffc1627a5e4881_e43de375578634774108d582ac68fec1_0_3)
switched from INITIALIZING to RUNNING.
2024-07-12 18:54:43,072 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - GlobalWindows
(1/1) (3d2570153eb2b48554ffc1627a5e4881_242e525045a332cf5da0422bf5582b5d_0_3)
switched from INITIALIZING to RUNNING.
2024-07-12 18:55:11,219 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36253 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781711219 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:55:11,668 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36253 for job 108a64cca6f69d6185679243cf2614c1 (5579943671 bytes,
checkpointDuration=448 ms, finalizationTime=0 ms).
2024-07-12 18:55:41,667 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36254 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781741667 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:55:41,932 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36254 for job 108a64cca6f69d6185679243cf2614c1 (5581813906 bytes,
checkpointDuration=265 ms, finalizationTime=0 ms).
2024-07-12 18:56:11,932 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36255 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781771932 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:56:12,232 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36255 for job 108a64cca6f69d6185679243cf2614c1 (5583516024 bytes,
checkpointDuration=300 ms, finalizationTime=0 ms).
2024-07-12 18:56:42,233 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36256 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781802232 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:56:42,533 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36256 for job 108a64cca6f69d6185679243cf2614c1 (4016415422 bytes,
checkpointDuration=300 ms, finalizationTime=1 ms).
2024-07-12 18:57:12,534 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36257 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781832533 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:57:12,802 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36257 for job 108a64cca6f69d6185679243cf2614c1 (2652007752 bytes,
checkpointDuration=269 ms, finalizationTime=0 ms).
2024-07-12 18:57:42,802 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36258 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781862802 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:57:43,112 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36258 for job 108a64cca6f69d6185679243cf2614c1 (1773173152 bytes,
checkpointDuration=309 ms, finalizationTime=1 ms).
2024-07-12 18:58:02,165 INFO org.apache.kafka.clients.NetworkClient
[] - [AdminClient
clientId=new_test_3eLbD7Wu-enumerator-admin-client] Node 2 disconnected.
2024-07-12 18:58:13,111 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36259 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781893111 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:58:13,347 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36259 for job 108a64cca6f69d6185679243cf2614c1 (895142299 bytes,
checkpointDuration=236 ms, finalizationTime=0 ms).
2024-07-12 18:58:43,347 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36260 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781923347 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:58:43,638 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36260 for job 108a64cca6f69d6185679243cf2614c1 (894663617 bytes,
checkpointDuration=291 ms, finalizationTime=0 ms).
2024-07-12 18:59:13,638 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36261 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781953638 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:59:13,946 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36261 for job 108a64cca6f69d6185679243cf2614c1 (896368454 bytes,
checkpointDuration=308 ms, finalizationTime=0 ms).
2024-07-12 18:59:43,947 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36262 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720781983946 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 18:59:44,174 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36262 for job 108a64cca6f69d6185679243cf2614c1 (14651880 bytes,
checkpointDuration=228 ms, finalizationTime=0 ms).
2024-07-12 19:00:14,174 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36263 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782014174 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:00:14,407 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36263 for job 108a64cca6f69d6185679243cf2614c1 (16554103 bytes,
checkpointDuration=232 ms, finalizationTime=1 ms).
2024-07-12 19:00:44,407 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36264 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782044407 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:00:44,653 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36264 for job 108a64cca6f69d6185679243cf2614c1 (18614579 bytes,
checkpointDuration=246 ms, finalizationTime=0 ms).
2024-07-12 19:01:14,653 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36265 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782074653 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:01:14,890 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36265 for job 108a64cca6f69d6185679243cf2614c1 (20589849 bytes,
checkpointDuration=236 ms, finalizationTime=1 ms).
2024-07-12 19:01:44,891 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36266 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782104890 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:01:45,096 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36266 for job 108a64cca6f69d6185679243cf2614c1 (18483980 bytes,
checkpointDuration=206 ms, finalizationTime=0 ms).
2024-07-12 19:02:15,097 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36267 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782135097 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:02:15,330 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36267 for job 108a64cca6f69d6185679243cf2614c1 (15499141 bytes,
checkpointDuration=232 ms, finalizationTime=1 ms).
2024-07-12 19:02:45,330 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36268 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782165330 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:02:45,561 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36268 for job 108a64cca6f69d6185679243cf2614c1 (17239129 bytes,
checkpointDuration=230 ms, finalizationTime=1 ms).
2024-07-12 19:03:02,254 INFO org.apache.kafka.clients.NetworkClient
[] - [AdminClient
clientId=new_test_3eLbD7Wu-enumerator-admin-client] Node 3 disconnected.
2024-07-12 19:03:15,561 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36269 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782195561 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:03:15,806 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36269 for job 108a64cca6f69d6185679243cf2614c1 (17200016 bytes,
checkpointDuration=245 ms, finalizationTime=0 ms).
2024-07-12 19:03:45,806 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36270 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782225806 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:03:46,085 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36270 for job 108a64cca6f69d6185679243cf2614c1 (19206137 bytes,
checkpointDuration=278 ms, finalizationTime=1 ms).
2024-07-12 19:04:16,086 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36271 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782256085 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:04:16,298 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36271 for job 108a64cca6f69d6185679243cf2614c1 (20850283 bytes,
checkpointDuration=213 ms, finalizationTime=0 ms).
2024-07-12 19:04:46,299 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36272 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782286299 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:04:46,544 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36272 for job 108a64cca6f69d6185679243cf2614c1 (20564375 bytes,
checkpointDuration=245 ms, finalizationTime=0 ms).
2024-07-12 19:05:16,544 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36273 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782316544 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:05:16,805 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36273 for job 108a64cca6f69d6185679243cf2614c1 (22748704 bytes,
checkpointDuration=261 ms, finalizationTime=0 ms).
2024-07-12 19:05:46,805 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36274 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782346805 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:05:47,091 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36274 for job 108a64cca6f69d6185679243cf2614c1 (24839853 bytes,
checkpointDuration=286 ms, finalizationTime=0 ms).
2024-07-12 19:06:17,091 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 36275 (type=CheckpointType{name='Checkpoint',
sharingFilesStrategy=FORWARD_BACKWARD}) @ 1720782377091 for job
108a64cca6f69d6185679243cf2614c1.
2024-07-12 19:06:17,324 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Completed
checkpoint 36275 for job 108a64cca6f69d6185679243cf2614c1 (26767845 bytes,
checkpointDuration=233 ms, finalizationTime=0 ms). {code}
There is some exceptions when checkpointing, I have found a similar report on
[FLINK-19016|https://issues.apache.org/jira/browse/FLINK-19016], in my case, it
seems that rocksDB write local sst file failed.
But please see time from "2024-07-12 18:55:11,668" to "2024-07-12 19:00:14,174"
of checkpoint 36253-36262 , the task has restarted successfully but the state
size decreased from 5.5Gb to 14Mb, I didn't set any state ttlConfig for this
task and there is no state delete in my business code. I am sure the state was
realy lost as I have checked the sink data. My state is majority in the
KeyedProcessFunction, I am not sure wether it is due to the "block checksum
mismatch" exception and I am worried about this problem happens again, because
there isn't any other exceptions throws and Flink tell me the task is restarted
sucessfully :(, so when I realized this issue, it was already 3 days later,
there is no more logs left.
The task has enabled the changelog state feature and it runs well for half a
month before, I have set "state.backend.local-recovery=true", I don't know
whether the configuration of my task is appropriate, please point me how to
avoid this or even let the job crash down after it happens. I can't reproduce
this problem still now.
by the way, the task is running on k8s and there is no k8s pod restarted during
this problem happen.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
