Bom dia Schwalbe!
Thanks for the reply.
I'm using Flink 1.14.0. EMR is a managed cluster platform to run big data
applications on AWS. This way Flink services are running on YARN. I tried
to create another savepoint today and was able to retrieve the Job Manager
log:
2022-03-09 15:42:10,294 INFO org.apache.flink.runtime.jobmaster.JobMaster
[] - Triggering savepoint for job
6f9d71e57efba96dad7f5328ab9ac717.
2022-03-09 15:42:10,298 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] -
Triggering checkpoint 1378 (type=SAVEPOINT) @ 1646840530294 for job
6f9d71e57efba96dad7f5328ab9ac717.
2022-03-09 15:45:19,636 WARN akka.remote.transport.netty.NettyTransport
[] - Remote connection to [/172.30.0.169:57520] failed with
java.io.IOException: Connection reset by peer
2022-03-09 15:45:19,648 WARN akka.remote.ReliableDeliverySupervisor
[] - Association with remote system
[akka.tcp://flink@ip-172-30-0-169.ec2.internal:46639] has failed, address
is now gated for [50] ms. Reason: [Disassociated]
2022-03-09 15:45:19,652 WARN akka.remote.ReliableDeliverySupervisor
[] - Association with remote system
[akka.tcp://flink-metrics@ip-172-30-0-169.ec2.internal:41533] has failed,
address is now gated for [50] ms. Reason: [Disassociated]
2022-03-09 15:45:19,707 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
LEFT_JOIN_MESSAGE_BULK -> Map (1/3) (866e32468227f9f0adac82e9b83b970a)
switched from RUNNING to FAILED on container_1646341714746_0005_01_000004 @
ip-172-30-0-165.ec2.internal (dataPort=40231).
org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException:
Connection unexpectedly closed by remote task manager
'ip-172-30-0-169.ec2.internal/172.30.0.169:34413'. This might indicate that
the remote task manager was lost.
at
org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.channelInactive(CreditBasedPartitionRequestClientHandler.java:186)
~[flink-dist_2.12-1.14.0.jar:1.14.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
~[flink-dist_2.12-1.14.0.jar:1.14.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
~[flink-dist_2.12-1.14.0.jar:1.14.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
~[flink-dist_2.12-1.14.0.jar:1.14.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)
~[flink-dist_2.12-1.14.0.jar:1.14.0]
at
org.apache.flink.runtime.io.network.netty.NettyMessageClientDecoderDelegate.channelInactive(NettyMessageClientDecoderDelegate.java:94)
~[flink-dist_2.12-1.14.0.jar:1.14.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
~[flink-dist_2.12-1.14.0.jar:1.14.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
~[flink-dist_2.12-1.14.0.jar:1.14.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
~[flink-dist_2.12-1.14.0.jar:1.14.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
~[flink-dist_2.12-1.14.0.jar:1.14.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
~[flink-dist_2.12-1.14.0.jar:1.14.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
~[flink-dist_2.12-1.14.0.jar:1.14.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
~[flink-dist_2.12-1.14.0.jar:1.14.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831)
~[flink-dist_2.12-1.14.0.jar:1.14.0]
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
~[flink-dist_2.12-1.14.0.jar:1.14.0]
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
~[flink-dist_2.12-1.14.0.jar:1.14.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
~[flink-dist_2.12-1.14.0.jar:1.14.0]
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
~[flink-dist_2.12-1.14.0.jar:1.14.0]
at
org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
~[flink-dist_2.12-1.14.0.jar:1.14.0]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_322]
2022-03-09 15:45:19,720 INFO
org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy
[] - Calculating tasks to restart to recover the failed task
5d739cfcb34ba91e39db0d6db0a4f1a2_0.
2022-03-09 15:45:19,721 INFO
org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy
[] - 18 tasks should be restarted to recover the failed task
5d739cfcb34ba91e39db0d6db0a4f1a2_0.
2022-03-09 15:45:19,723 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job
TOOL_MESSAGE_STREAM (6f9d71e57efba96dad7f5328ab9ac717) switched from state
RUNNING to RESTARTING.
2022-03-09 15:45:19,728 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
LEFT_JOIN_MESSAGE_BULK -> Map (3/3) (df8df89abf1761a726dd4593387cbd76)
switched from RUNNING to CANCELING.
2022-03-09 15:45:19,732 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator [] -
Removing registered reader after failure for subtask 1 of source Source:
BULK_SENDER_EVENT -> Filter -> Map.
2022-03-09 15:45:19,732 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
BULK_SENDER_EVENT -> Filter -> Map (2/3) (2ba29adbeeceb86a69536082fbfb4931)
switched from RUNNING to CANCELING.
2022-03-09 15:45:19,732 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
LEFT_JOIN_MESSAGE_BULK -> Map (2/3) (3674e936354727457b06293592b65f20)
switched from RUNNING to CANCELING.
2022-03-09 15:45:19,733 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator [] -
Removing registered reader after failure for subtask 0 of source Source:
BULK_SENDER_EVENT -> Filter -> Map.
2022-03-09 15:45:19,733 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
BULK_SENDER_EVENT -> Filter -> Map (1/3) (117be79102d086ddb65fb0b31c245f49)
switched from RUNNING to CANCELING.
2022-03-09 15:45:19,733 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
LEFT_JOIN_MESSAGE_FLOW -> Map (3/3) (3add641adeebe4e14dd0111bd647aa75)
switched from RUNNING to CANCELING.
2022-03-09 15:45:19,733 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
LEFT_JOIN_MESSAGE_FLOW -> Map (2/3) (f5f4d0d49c1d2c2c65685ccb6c35eab4)
switched from RUNNING to CANCELING.
2022-03-09 15:45:19,733 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
DEDUP_TOOL_MESSAGE_EVENT -> Sink SINK_TOOL_MESSAGE_EVENT (3/3)
(07a1c10037596788aee6844603ab17a2) switched from RUNNING to CANCELING.
2022-03-09 15:45:19,733 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
LEFT_JOIN_MESSAGE_FLOW -> Map (1/3) (3caaf3a8072b29567d94691d1765a294)
switched from RUNNING to CANCELING.
2022-03-09 15:45:19,733 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
DEDUP_TOOL_MESSAGE_EVENT -> Sink SINK_TOOL_MESSAGE_EVENT (2/3)
(216e7b0a7d0f8c887a5c91a3e5267c73) switched from RUNNING to CANCELING.
2022-03-09 15:45:19,733 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
DEDUP_TOOL_MESSAGE_EVENT -> Sink SINK_TOOL_MESSAGE_EVENT (1/3)
(2e3973007d7c3d71f10456c3aca5a0b3) switched from RUNNING to CANCELING.
2022-03-09 15:45:19,733 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter,
Filter)) (3/3) (bcebf365a3e6da0281d80b3ab0e2cff8) switched from RUNNING to
CANCELING.
2022-03-09 15:45:19,733 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator [] -
Removing registered reader after failure for subtask 2 of source Source:
MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter,
Filter)).
2022-03-09 15:45:19,734 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator [] -
Removing registered reader after failure for subtask 1 of source Source:
MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter,
Filter)).
2022-03-09 15:45:19,734 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter,
Filter)) (2/3) (8d912317f46bf11f866bd70be09377ef) switched from RUNNING to
CANCELING.
2022-03-09 15:45:19,734 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator [] -
Removing registered reader after failure for subtask 0 of source Source:
MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter,
Filter)).
2022-03-09 15:45:19,734 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter,
Filter)) (1/3) (c4a8b6f2d0b7ea837dd4e69a107e652e) switched from RUNNING to
CANCELING.
2022-03-09 15:45:19,735 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator [] -
Removing registered reader after failure for subtask 2 of source Source:
FLOW_EVENT -> Filter -> Map.
2022-03-09 15:45:19,735 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
FLOW_EVENT -> Filter -> Map (3/3) (b811833b1ce35416fae61aba7cdbeb53)
switched from RUNNING to CANCELING.
2022-03-09 15:45:19,735 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator [] -
Removing registered reader after failure for subtask 1 of source Source:
FLOW_EVENT -> Filter -> Map.
2022-03-09 15:45:19,735 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
FLOW_EVENT -> Filter -> Map (2/3) (c5048ba45046caa3b954257602b4f0a4)
switched from RUNNING to CANCELING.
2022-03-09 15:45:19,735 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator [] -
Removing registered reader after failure for subtask 0 of source Source:
FLOW_EVENT -> Filter -> Map.
2022-03-09 15:45:19,735 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
FLOW_EVENT -> Filter -> Map (1/3) (4693404a84c87a07b1f35ed62f9edad7)
switched from RUNNING to CANCELING.
2022-03-09 15:45:19,735 INFO
org.apache.flink.runtime.source.coordinator.SourceCoordinator [] -
Removing registered reader after failure for subtask 2 of source Source:
BULK_SENDER_EVENT -> Filter -> Map.
2022-03-09 15:45:19,735 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
BULK_SENDER_EVENT -> Filter -> Map (3/3) (d4421c51bf8fb716c672e13fd249450f)
switched from RUNNING to CANCELING.
2022-03-09 15:45:19,744 WARN akka.remote.transport.netty.NettyTransport
[] - Remote connection to [null] failed with
java.net.ConnectException: Connection refused: ip-172-30-0-169.ec2.internal/
172.30.0.169:46639
2022-03-09 15:45:19,746 WARN akka.remote.ReliableDeliverySupervisor
[] - Association with remote system
[akka.tcp://flink@ip-172-30-0-169.ec2.internal:46639] has failed, address
is now gated for [50] ms. Reason: [Association failed with
[akka.tcp://flink@ip-172-30-0-169.ec2.internal:46639]] Caused by:
[java.net.ConnectException: Connection refused:
ip-172-30-0-169.ec2.internal/172.30.0.169:46639]
2022-03-09 15:45:19,751 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
LEFT_JOIN_MESSAGE_BULK -> Map (3/3) (df8df89abf1761a726dd4593387cbd76)
switched from CANCELING to CANCELED.
2022-03-09 15:45:19,751 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
Discarding the results produced by task execution
df8df89abf1761a726dd4593387cbd76.
2022-03-09 15:45:19,753 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
Discarding the results produced by task execution
df8df89abf1761a726dd4593387cbd76.
2022-03-09 15:45:19,754 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
LEFT_JOIN_MESSAGE_FLOW -> Map (3/3) (3add641adeebe4e14dd0111bd647aa75)
switched from CANCELING to CANCELED.
2022-03-09 15:45:19,755 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
Discarding the results produced by task execution
3add641adeebe4e14dd0111bd647aa75.
2022-03-09 15:45:19,755 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
Discarding the results produced by task execution
3add641adeebe4e14dd0111bd647aa75.
2022-03-09 15:45:19,755 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
DEDUP_TOOL_MESSAGE_EVENT -> Sink SINK_TOOL_MESSAGE_EVENT (3/3)
(07a1c10037596788aee6844603ab17a2) switched from CANCELING to CANCELED.
2022-03-09 15:45:19,755 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
Discarding the results produced by task execution
07a1c10037596788aee6844603ab17a2.
2022-03-09 15:45:19,755 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter,
Filter)) (3/3) (bcebf365a3e6da0281d80b3ab0e2cff8) switched from CANCELING
to CANCELED.
2022-03-09 15:45:19,755 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
Discarding the results produced by task execution
bcebf365a3e6da0281d80b3ab0e2cff8.
2022-03-09 15:45:19,755 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
Discarding the results produced by task execution
bcebf365a3e6da0281d80b3ab0e2cff8.
2022-03-09 15:45:19,755 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
FLOW_EVENT -> Filter -> Map (3/3) (b811833b1ce35416fae61aba7cdbeb53)
switched from CANCELING to CANCELED.
2022-03-09 15:45:19,755 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
Discarding the results produced by task execution
b811833b1ce35416fae61aba7cdbeb53.
2022-03-09 15:45:19,755 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
Discarding the results produced by task execution
b811833b1ce35416fae61aba7cdbeb53.
2022-03-09 15:45:19,756 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
BULK_SENDER_EVENT -> Filter -> Map (3/3) (d4421c51bf8fb716c672e13fd249450f)
switched from CANCELING to CANCELED.
2022-03-09 15:45:19,757 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
Discarding the results produced by task execution
d4421c51bf8fb716c672e13fd249450f.
2022-03-09 15:45:19,757 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
Discarding the results produced by task execution
d4421c51bf8fb716c672e13fd249450f.
2022-03-09 15:45:19,757 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager
[] - Received resource requirements from job
6f9d71e57efba96dad7f5328ab9ac717:
[ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN},
numberOfRequiredSlots=2}]
2022-03-09 15:45:19,790 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
BULK_SENDER_EVENT -> Filter -> Map (1/3) (117be79102d086ddb65fb0b31c245f49)
switched from CANCELING to CANCELED.
2022-03-09 15:45:19,797 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
FLOW_EVENT -> Filter -> Map (1/3) (4693404a84c87a07b1f35ed62f9edad7)
switched from CANCELING to CANCELED.
2022-03-09 15:45:19,801 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source:
MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter,
Filter)) (1/3) (c4a8b6f2d0b7ea837dd4e69a107e652e) switched from CANCELING
to CANCELED.
2022-03-09 15:45:20,026 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph [] -
DEDUP_TOOL_MESSAGE_EVENT -> Sink SINK_TOOL_MESSAGE_EVENT (1/3)
(2e3973007d7c3d71f10456c3aca5a0b3) switched from CANCELING to CANCELED.
2022-03-09 15:45:23,957 INFO
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
Worker container_1646341714746_0005_01_000003 is terminated. Diagnostics:
Container container_1646341714746_0005_01_000003 marked as failed.
Exit code:137.
Diagnostics:[2022-03-09 15:45:19.639]Container killed on request. Exit
code is 137
[2022-03-09 15:45:19.641]Container exited with a non-zero exit code 137.
[2022-03-09 15:45:19.642]Killed by external signal
2022-03-09 15:45:23,957 INFO
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
Closing TaskExecutor connection container_1646341714746_0005_01_000003
because: Container container_1646341714746_0005_01_000003 marked as failed.
Exit code:137.
Diagnostics:[2022-03-09 15:45:19.639]Container killed on request. Exit
code is 137
[2022-03-09 15:45:19.641]Container exited with a non-zero exit code 137.
[2022-03-09 15:45:19.642]Killed by external signal
Thanks,
On Tue, Mar 8, 2022 at 4:57 AM Schwalbe Matthias <
matthias.schwa...@viseca.ch> wrote:
> Bom Dia Vinicius,
>
>
>
> Can You still find (and post) the exception stack from your jobmanager
> log, the flink client log does not reveal enough information.
>
> Your situation reminds me of something similar I had.
>
> In the log you might find something like this or similar:
>
>
>
> 2022-03-07 02:15:41,347 INFO
> org.apache.flink.runtime.jobmaster.JobMaster [] -
> Triggering stop-with-savepoint for job e12f22653f79194863ab426312dd666a.
>
> 2022-03-07 02:15:41,380 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering
> checkpoint 4983974 (type=SAVEPOINT_SUSPEND) @ 1646615741347 for job
> e12f22653f79194863ab426312dd666a.
>
> 2022-03-07 02:15:43,042 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Decline
> checkpoint 4983974 by task 0e659ac720e3e0b3e4072dbc1cc85cd3 of job
> e12f22653f79194863ab426312dd666a at
> container_e1093_1646358077201_0002_01_000001 @ ulxxphaddtn02.adgr.net
> (dataPort=44767).
>
> org.apache.flink.util.SerializedThrowable: Asynchronous task checkpoint
> failed.
>
> at
> org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.handleExecutionException(AsyncCheckpointRunnable.java:279)
> ~[flink-dist_2.11-1.13.0.jar:1.13.0]
>
>
>
> BTW what Flink version are you running?
>
> What is EMR (what technology underneath).
>
>
>
>
>
>
>
> *From:* Vinicius Peracini <vinicius.perac...@zenvia.com>
> *Sent:* Montag, 7. März 2022 20:46
> *To:* Dawid Wysakowicz <dwysakow...@apache.org>
> *Cc:* user@flink.apache.org
> *Subject:* Re: Could not stop job with a savepoint
>
>
>
> Hi Dawid, thanks for the reply.
>
>
>
> The job was still in progress and producing events. Unfortunately I was
> not able to stop the job with a savepoint or to just create a savepoint. I
> had to stop the job without the savepoint and restore the state using the
> last checkpoint. Still reviewing my configuration and trying to figure out
> why this is happening. Any help would be appreciated.
>
>
>
> Thanks!
>
>
>
>
>
> On Mon, Mar 7, 2022 at 11:56 AM Dawid Wysakowicz <dwysakow...@apache.org>
> wrote:
>
> Hi,
>
> From the exception it seems the job has been already done when you're
> triggering the savepoint.
>
> Best,
>
> Dawid
>
> On 07/03/2022 14:56, Vinicius Peracini wrote:
>
> Hello everyone,
>
>
>
> I have a Flink job (version 1.14.0 running on EMR) and I'm having this
> issue while trying to stop a job with a savepoint on S3:
>
>
>
> org.apache.flink.util.FlinkException: Could not stop with a savepoint job
> "df3a3c590fabac737a17f1160c21094c".
> at
> org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:581)
> at
> org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:1002)
> at org.apache.flink.client.cli.CliFrontend.stop(CliFrontend.java:569)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1069)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
> Caused by: java.util.concurrent.ExecutionException:
> java.util.concurrent.CompletionException:
> org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint
> Coordinator is suspending.
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:579)
> ... 9 more
>
>
>
> I'm using incremental and unaligned checkpoints (aligned checkpoint
> timeout is 30 seconds). I also tried to create the savepoint without
> stopping the job (using flink savepoint command) and got the same error.
> Any idea what is happening here?
>
>
>
> Thanks in advance,
>
>
>
> Aviso Legal: Este documento pode conter informações confidenciais e/ou
> privilegiadas. Se você não for o destinatário ou a pessoa autorizada a
> receber este documento, não deve usar, copiar ou divulgar as informações
> nele contidas ou tomar qualquer ação baseada nessas informações.
>
>
>
> Disclaimer: The information contained in this document may be privileged
> and confidential and protected from disclosure. If the reader of this
> document is not the intended recipient, or an employee agent responsible
> for delivering this document to the intended recipient, you are hereby
> notified that any dissemination, distribution or copying of this
> communication is strictly prohibited.
>
>
>
> Aviso Legal: Este documento pode conter informações confidenciais e/ou
> privilegiadas. Se você não for o destinatário ou a pessoa autorizada a
> receber este documento, não deve usar, copiar ou divulgar as informações
> nele contidas ou tomar qualquer ação baseada nessas informações.
>
>
>
> Disclaimer: The information contained in this document may be privileged
> and confidential and protected from disclosure. If the reader of this
> document is not the intended recipient, or an employee agent responsible
> for delivering this document to the intended recipient, you are hereby
> notified that any dissemination, distribution or copying of this
> communication is strictly prohibited.
> Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und
> beinhaltet unter Umständen vertrauliche Mitteilungen. Da die
> Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann,
> übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und
> Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir
> Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie
> eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung
> dieser Informationen ist streng verboten.
>
> This message is intended only for the named recipient and may contain
> confidential or privileged information. As the confidentiality of email
> communication cannot be guaranteed, we do not accept any responsibility for
> the confidentiality and the intactness of this message. If you have
> received it in error, please advise the sender by return e-mail and delete
> this message and any attachments. Any unauthorised use or dissemination of
> this information is strictly prohibited.
>
--
Aviso Legal: Este documento pode conter informações confidenciais e/ou
privilegiadas. Se você não for o destinatário ou a pessoa autorizada a
receber este documento, não deve usar, copiar ou divulgar as informações
nele contidas ou tomar qualquer ação baseada nessas informações.
Disclaimer: The information contained in this document may be privileged
and confidential and protected from disclosure. If the reader of this
document is not the intended recipient, or an employee agent responsible
for delivering this document to the intended recipient, you are hereby
notified that any dissemination, distribution or copying of this
communication is strictly prohibited.
