[
https://issues.apache.org/jira/browse/FLINK-33186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18098065#comment-18098065
]
Efrat Levitan commented on FLINK-33186:
---------------------------------------
Hitting this again on another pipeline
[https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=77308]
{code:java}
Jul 21 18:52:35 18:52:35.110 [ERROR] Tests run: 3, Failures: 0, Errors: 1,
Skipped: 0, Time elapsed: 9.168 s <<< FAILURE! -- in
org.apache.flink.test.checkpointing.CheckpointAfterAllTasksFinishedITCase
Jul 21 18:52:35 18:52:35.110 [ERROR]
org.apache.flink.test.checkpointing.CheckpointAfterAllTasksFinishedITCase.testRestoreAfterSomeTasksFinished
-- Time elapsed: 1.846 s <<< ERROR!
Jul 21 18:52:35 java.util.concurrent.ExecutionException:
org.apache.flink.runtime.checkpoint.CheckpointException: Task local checkpoint
failure.{code}
As [~martijnvisser] mentioned, testRestoreAfterSomeTasksFinished triggers a
savepoint after [at least
one|https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/checkpointing/CheckpointAfterAllTasksFinishedITCase.java#L115]
subtask finished
In my case it looks like
{{69565783a411eb2ed0441dead38d4d7b_51397532e2d9c7a21097a30d590b3114_1_0}}
finished after the checkpoint plan was calculated but before
{{triggerCheckpoint}} was received:
{code:java}
18:52:34,345 [passA -> Sink: sinkA (2/4)#0] INFO
org.apache.flink.runtime.taskmanager.Task [] - passA ->
Sink: sinkA (2/4)#0
(69565783a411eb2ed0441dead38d4d7b_51397532e2d9c7a21097a30d590b3114_1_0)
switched from RUNNING to FINISHED.18:52:34,345 [passA -> Sink: sinkA (2/4)#0]
INFO org.apache.flink.runtime.taskmanager.Task [] - Freeing
task resources for passA -> Sink: sinkA (2/4)#0
(69565783a411eb2ed0441dead38d4d7b_51397532e2d9c7a21097a30d590b3114_1_0).18:52:34,345
[flink-pekko.actor.default-dispatcher-5] INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor [] -
Un-registering task and sending final execution state FINISHED to JobManager
for task passA -> Sink: sinkA (2/4)#0
69565783a411eb2ed0441dead38d4d7b_51397532e2d9c7a21097a30d590b3114_1_0.18:52:34,348
[flink-pekko.actor.default-dispatcher-7] INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - passA ->
Sink: sinkA (2/4)
(69565783a411eb2ed0441dead38d4d7b_51397532e2d9c7a21097a30d590b3114_1_0)
switched from RUNNING to FINISHED.18:52:34,348
[flink-pekko.actor.default-dispatcher-7] INFO
org.apache.flink.runtime.checkpoint.ExecutionTrackingCheckpointCoordinatorDeActivator
[] - Checkpoint scheduler is already started, ignoring status change (some
task has finished?)18:52:34,352 [ Checkpoint Timer] INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
checkpoint 2 (type=SavepointType{name='Savepoint', postCheckpointAction=NONE,
formatType=CANONICAL}) @ 1784659954344 for job
1f732934d5087ed49c858151e62c79cb.18:52:34,384 [jobmanager-io-thread-26] INFO
org.apache.flink.core.fs.FileSystem [] - Registering
filesystem factory org.apache.flink.core.fs.local.LocalFileSystemFactory for
scheme 'file' with priority 0 (highest registered priority:
-2147483648)18:52:34,385 [jobmanager-io-thread-26] INFO
org.apache.flink.core.fs.FileSystem [] - Registering
filesystem factory
org.apache.flink.test.checkpointing.SavepointITCase$PathFailingFileSystemFactory
for scheme 'failPath' with priority 0 (highest registered priority:
-2147483648)18:52:34,385 [jobmanager-io-thread-26] INFO
org.apache.flink.core.fs.FileSystem [] - Registering
filesystem factory
org.apache.flink.testutils.TestFileSystem$TestFileSystemFactory for scheme
'test' with priority 0 (highest registered priority: -2147483648)18:52:34,385
[jobmanager-io-thread-26] INFO org.apache.flink.core.fs.FileSystem
[] - Registering filesystem factory
org.apache.flink.testutils.EntropyInjectingTestFileSystem$EntropyInjectingTestFileSystemFactory
for scheme 'test-entropy' with priority 0 (highest registered priority:
-2147483648)18:52:34,400 [flink-pekko.actor.default-dispatcher-7] INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
Checkpoint 2 for job 1f732934d5087ed49c858151e62c79cb failed due to
org.apache.flink.runtime.checkpoint.CheckpointException: TaskManager received a
checkpoint request for unknown task
69565783a411eb2ed0441dead38d4d7b_51397532e2d9c7a21097a30d590b3114_1_0. Failure
reason: Task local checkpoint failure.18:52:34,405 [ Checkpoint Timer] WARN
org.apache.flink.runtime.checkpoint.CheckpointFailureManager [] - Failed to
trigger or complete checkpoint 2 for job 1f732934d5087ed49c858151e62c79cb. (0
consecutive failed attempts so
far)org.apache.flink.runtime.checkpoint.CheckpointException: TaskManager
received a checkpoint request for unknown task
69565783a411eb2ed0441dead38d4d7b_51397532e2d9c7a21097a30d590b3114_1_0. Failure
reason: Task local checkpoint failure. at
org.apache.flink.runtime.taskexecutor.TaskExecutor.triggerCheckpoint(TaskExecutor.java:1109)
{code}
Finished tasks are removed from taskSlotTable so TM can't actually know for
sure it was finished. so it fails the checkpoint.
> CheckpointAfterAllTasksFinishedITCase.testRestoreAfterSomeTasksFinished
> fails on AZP
> -------------------------------------------------------------------------------------
>
> Key: FLINK-33186
> URL: https://issues.apache.org/jira/browse/FLINK-33186
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
> Affects Versions: 1.19.0, 1.18.1, 2.4.0
> Reporter: Sergey Nuyanzin
> Assignee: Jiang Xin
> Priority: Critical
> Labels: test-stability
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53509&view=logs&j=baf26b34-3c6a-54e8-f93f-cf269b32f802&t=8c9d126d-57d2-5a9e-a8c8-ff53f7b35cd9&l=8762
> fails as
> {noformat}
> Sep 28 01:23:43 Caused by:
> org.apache.flink.runtime.checkpoint.CheckpointException: Task local
> checkpoint failure.
> Sep 28 01:23:43 at
> org.apache.flink.runtime.checkpoint.PendingCheckpoint.abort(PendingCheckpoint.java:550)
> Sep 28 01:23:43 at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2248)
> Sep 28 01:23:43 at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2235)
> Sep 28 01:23:43 at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.lambda$null$9(CheckpointCoordinator.java:817)
> Sep 28 01:23:43 at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> Sep 28 01:23:43 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Sep 28 01:23:43 at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> Sep 28 01:23:43 at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> Sep 28 01:23:43 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> Sep 28 01:23:43 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> Sep 28 01:23:43 at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
