Hi Schwalbe! Yes, I'm using RocksDBStateBackend. I guess your suspicion was right, I changed the memory allocator to jemalloc and the issue seems to be gone.
Here is what I did to change the memory allocator on EMR: 1. Installed the jemalloc package by using an EMR bootstrap action script: sudo amazon-linux-extras install -y epel sudo yum install -y jemalloc-devel 2. Configured Flink to use jemalloc: "containerized.master.env.LD_PRELOAD": "/usr/lib64/libjemalloc.so" "containerized.taskmanager.env.LD_PRELOAD": "/usr/lib64/libjemalloc.so" Thank you so much! Best, On Thu, Mar 10, 2022 at 4:43 AM Schwalbe Matthias < matthias.schwa...@viseca.ch> wrote: > Hi Vinicius, > > > > Your case, the taskmanager being actively killed by yarn was the other way > this happened. > > > > You are using RocksDBStateBackend, right? > > Not being sure, I’ve got the strong suspicion that this has got to do with > the glibc bug that is seemingly in the works. > > There is some documentation here [1] and a solution that has been > implemented for k8s containers [2] which replaces the glibc allocator with > libjemalloc.so . > > > > However we are not completely through with our encounter of the same > problem. > > Our intermediate solution is to reserve some unused extra memory, so the > problem is delayed but not completely prevented (we restart our jobs daily > by means of savepoint taking): > > > > flink-conf.yaml: > > … > > taskmanager.memory.managed.fraction: 0.2 > > #reserve 2GB extra unused space (out of 8GB per TM) in order to mitigate > the glibc memory leakage problem > > taskmanager.memory.task.off-heap.size: 2048mb > > … > > > > I’m not sure if and starting with which Flink version libjemalloc.so is > integrated by default into the flink runtime > > … Flink team to the rescue 😊! > > > > Hope this helps > > > > Thias > > > > [1] > https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/memory/mem_trouble/#container-memory-exceeded > > [2] https://issues.apache.org/jira/browse/FLINK-19125 > > > > *From:* Vinicius Peracini <vinicius.perac...@zenvia.com> > *Sent:* Mittwoch, 9. März 2022 17:56 > *To:* Schwalbe Matthias <matthias.schwa...@viseca.ch> > *Cc:* Dawid Wysakowicz <dwysakow...@apache.org>; user@flink.apache.org > *Subject:* Re: Could not stop job with a savepoint > > > > So apparently the YARN container for Task Manager is running out of memory > during the savepoint execution. Never had any problems with checkpoints > though. Task Manager configuration: > > > > "taskmanager.memory.process.size": "10240m", > "taskmanager.memory.managed.fraction": "0.6", > "taskmanager.memory.jvm-overhead.fraction": "0.07", > "taskmanager.memory.jvm-metaspace.size": "192mb", > "taskmanager.network.memory.buffer-debloat.enabled": "true", > > > > On Wed, Mar 9, 2022 at 1:33 PM Vinicius Peracini < > vinicius.perac...@zenvia.com> wrote: > > Bom dia Schwalbe! > > > > Thanks for the reply. > > > > I'm using Flink 1.14.0. EMR is a managed cluster platform to run big data > applications on AWS. This way Flink services are running on YARN. I tried > to create another savepoint today and was able to retrieve the Job Manager > log: > > > > 2022-03-09 15:42:10,294 INFO org.apache.flink.runtime.jobmaster.JobMaster > [] - Triggering savepoint for job > 6f9d71e57efba96dad7f5328ab9ac717. > 2022-03-09 15:42:10,298 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - > Triggering checkpoint 1378 (type=SAVEPOINT) @ 1646840530294 for job > 6f9d71e57efba96dad7f5328ab9ac717. > 2022-03-09 15:45:19,636 WARN akka.remote.transport.netty.NettyTransport > [] - Remote connection to [/172.30.0.169:57520] failed > with java.io.IOException: Connection reset by peer > 2022-03-09 15:45:19,648 WARN akka.remote.ReliableDeliverySupervisor > [] - Association with remote system > [akka.tcp://flink@ip-172-30-0-169.ec2.internal:46639] has failed, address > is now gated for [50] ms. Reason: [Disassociated] > 2022-03-09 15:45:19,652 WARN akka.remote.ReliableDeliverySupervisor > [] - Association with remote system > [akka.tcp://flink-metrics@ip-172-30-0-169.ec2.internal:41533] has failed, > address is now gated for [50] ms. Reason: [Disassociated] > 2022-03-09 15:45:19,707 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > LEFT_JOIN_MESSAGE_BULK -> Map (1/3) (866e32468227f9f0adac82e9b83b970a) > switched from RUNNING to FAILED on container_1646341714746_0005_01_000004 @ > ip-172-30-0-165.ec2.internal (dataPort=40231). > org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException: > Connection unexpectedly closed by remote task manager > 'ip-172-30-0-169.ec2.internal/172.30.0.169:34413'. This might indicate > that the remote task manager was lost. > at > org.apache.flink.runtime.io.network.netty.CreditBasedPartitionRequestClientHandler.channelInactive(CreditBasedPartitionRequestClientHandler.java:186) > ~[flink-dist_2.12-1.14.0.jar:1.14.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) > ~[flink-dist_2.12-1.14.0.jar:1.14.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) > ~[flink-dist_2.12-1.14.0.jar:1.14.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) > ~[flink-dist_2.12-1.14.0.jar:1.14.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81) > ~[flink-dist_2.12-1.14.0.jar:1.14.0] > at > org.apache.flink.runtime.io.network.netty.NettyMessageClientDecoderDelegate.channelInactive(NettyMessageClientDecoderDelegate.java:94) > ~[flink-dist_2.12-1.14.0.jar:1.14.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) > ~[flink-dist_2.12-1.14.0.jar:1.14.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) > ~[flink-dist_2.12-1.14.0.jar:1.14.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241) > ~[flink-dist_2.12-1.14.0.jar:1.14.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405) > ~[flink-dist_2.12-1.14.0.jar:1.14.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262) > ~[flink-dist_2.12-1.14.0.jar:1.14.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248) > ~[flink-dist_2.12-1.14.0.jar:1.14.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901) > ~[flink-dist_2.12-1.14.0.jar:1.14.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:831) > ~[flink-dist_2.12-1.14.0.jar:1.14.0] > at > org.apache.flink.shaded.netty4.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) > ~[flink-dist_2.12-1.14.0.jar:1.14.0] > at > org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) > ~[flink-dist_2.12-1.14.0.jar:1.14.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) > ~[flink-dist_2.12-1.14.0.jar:1.14.0] > at > org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) > ~[flink-dist_2.12-1.14.0.jar:1.14.0] > at > org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > ~[flink-dist_2.12-1.14.0.jar:1.14.0] > at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_322] > 2022-03-09 15:45:19,720 INFO > > org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy > [] - Calculating tasks to restart to recover the failed task > 5d739cfcb34ba91e39db0d6db0a4f1a2_0. > 2022-03-09 15:45:19,721 INFO > > org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionFailoverStrategy > [] - 18 tasks should be restarted to recover the failed task > 5d739cfcb34ba91e39db0d6db0a4f1a2_0. > 2022-03-09 15:45:19,723 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job > TOOL_MESSAGE_STREAM (6f9d71e57efba96dad7f5328ab9ac717) switched from state > RUNNING to RESTARTING. > 2022-03-09 15:45:19,728 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > LEFT_JOIN_MESSAGE_BULK -> Map (3/3) (df8df89abf1761a726dd4593387cbd76) > switched from RUNNING to CANCELING. > 2022-03-09 15:45:19,732 INFO > org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - > Removing registered reader after failure for subtask 1 of source Source: > BULK_SENDER_EVENT -> Filter -> Map. > 2022-03-09 15:45:19,732 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: > BULK_SENDER_EVENT -> Filter -> Map (2/3) (2ba29adbeeceb86a69536082fbfb4931) > switched from RUNNING to CANCELING. > 2022-03-09 15:45:19,732 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > LEFT_JOIN_MESSAGE_BULK -> Map (2/3) (3674e936354727457b06293592b65f20) > switched from RUNNING to CANCELING. > 2022-03-09 15:45:19,733 INFO > org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - > Removing registered reader after failure for subtask 0 of source Source: > BULK_SENDER_EVENT -> Filter -> Map. > 2022-03-09 15:45:19,733 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: > BULK_SENDER_EVENT -> Filter -> Map (1/3) (117be79102d086ddb65fb0b31c245f49) > switched from RUNNING to CANCELING. > 2022-03-09 15:45:19,733 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > LEFT_JOIN_MESSAGE_FLOW -> Map (3/3) (3add641adeebe4e14dd0111bd647aa75) > switched from RUNNING to CANCELING. > 2022-03-09 15:45:19,733 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > LEFT_JOIN_MESSAGE_FLOW -> Map (2/3) (f5f4d0d49c1d2c2c65685ccb6c35eab4) > switched from RUNNING to CANCELING. > 2022-03-09 15:45:19,733 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > DEDUP_TOOL_MESSAGE_EVENT -> Sink SINK_TOOL_MESSAGE_EVENT (3/3) > (07a1c10037596788aee6844603ab17a2) switched from RUNNING to CANCELING. > 2022-03-09 15:45:19,733 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > LEFT_JOIN_MESSAGE_FLOW -> Map (1/3) (3caaf3a8072b29567d94691d1765a294) > switched from RUNNING to CANCELING. > 2022-03-09 15:45:19,733 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > DEDUP_TOOL_MESSAGE_EVENT -> Sink SINK_TOOL_MESSAGE_EVENT (2/3) > (216e7b0a7d0f8c887a5c91a3e5267c73) switched from RUNNING to CANCELING. > 2022-03-09 15:45:19,733 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > DEDUP_TOOL_MESSAGE_EVENT -> Sink SINK_TOOL_MESSAGE_EVENT (1/3) > (2e3973007d7c3d71f10456c3aca5a0b3) switched from RUNNING to CANCELING. > 2022-03-09 15:45:19,733 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: > MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter, > Filter)) (3/3) (bcebf365a3e6da0281d80b3ab0e2cff8) switched from RUNNING to > CANCELING. > 2022-03-09 15:45:19,733 INFO > org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - > Removing registered reader after failure for subtask 2 of source Source: > MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter, > Filter)). > 2022-03-09 15:45:19,734 INFO > org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - > Removing registered reader after failure for subtask 1 of source Source: > MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter, > Filter)). > 2022-03-09 15:45:19,734 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: > MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter, > Filter)) (2/3) (8d912317f46bf11f866bd70be09377ef) switched from RUNNING to > CANCELING. > 2022-03-09 15:45:19,734 INFO > org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - > Removing registered reader after failure for subtask 0 of source Source: > MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter, > Filter)). > 2022-03-09 15:45:19,734 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: > MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter, > Filter)) (1/3) (c4a8b6f2d0b7ea837dd4e69a107e652e) switched from RUNNING to > CANCELING. > 2022-03-09 15:45:19,735 INFO > org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - > Removing registered reader after failure for subtask 2 of source Source: > FLOW_EVENT -> Filter -> Map. > 2022-03-09 15:45:19,735 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: > FLOW_EVENT -> Filter -> Map (3/3) (b811833b1ce35416fae61aba7cdbeb53) > switched from RUNNING to CANCELING. > 2022-03-09 15:45:19,735 INFO > org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - > Removing registered reader after failure for subtask 1 of source Source: > FLOW_EVENT -> Filter -> Map. > 2022-03-09 15:45:19,735 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: > FLOW_EVENT -> Filter -> Map (2/3) (c5048ba45046caa3b954257602b4f0a4) > switched from RUNNING to CANCELING. > 2022-03-09 15:45:19,735 INFO > org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - > Removing registered reader after failure for subtask 0 of source Source: > FLOW_EVENT -> Filter -> Map. > 2022-03-09 15:45:19,735 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: > FLOW_EVENT -> Filter -> Map (1/3) (4693404a84c87a07b1f35ed62f9edad7) > switched from RUNNING to CANCELING. > 2022-03-09 15:45:19,735 INFO > org.apache.flink.runtime.source.coordinator.SourceCoordinator [] - > Removing registered reader after failure for subtask 2 of source Source: > BULK_SENDER_EVENT -> Filter -> Map. > 2022-03-09 15:45:19,735 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: > BULK_SENDER_EVENT -> Filter -> Map (3/3) (d4421c51bf8fb716c672e13fd249450f) > switched from RUNNING to CANCELING. > 2022-03-09 15:45:19,744 WARN akka.remote.transport.netty.NettyTransport > [] - Remote connection to [null] failed with > java.net.ConnectException: Connection refused: ip-172-30-0-169.ec2.internal/ > 172.30.0.169:46639 > 2022-03-09 15:45:19,746 WARN akka.remote.ReliableDeliverySupervisor > [] - Association with remote system > [akka.tcp://flink@ip-172-30-0-169.ec2.internal:46639] has failed, address > is now gated for [50] ms. Reason: [Association failed with > [akka.tcp://flink@ip-172-30-0-169.ec2.internal:46639]] Caused by: > [java.net.ConnectException: Connection refused: > ip-172-30-0-169.ec2.internal/172.30.0.169:46639] > 2022-03-09 15:45:19,751 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > LEFT_JOIN_MESSAGE_BULK -> Map (3/3) (df8df89abf1761a726dd4593387cbd76) > switched from CANCELING to CANCELED. > 2022-03-09 15:45:19,751 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > Discarding the results produced by task execution > df8df89abf1761a726dd4593387cbd76. > 2022-03-09 15:45:19,753 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > Discarding the results produced by task execution > df8df89abf1761a726dd4593387cbd76. > 2022-03-09 15:45:19,754 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > LEFT_JOIN_MESSAGE_FLOW -> Map (3/3) (3add641adeebe4e14dd0111bd647aa75) > switched from CANCELING to CANCELED. > 2022-03-09 15:45:19,755 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > Discarding the results produced by task execution > 3add641adeebe4e14dd0111bd647aa75. > 2022-03-09 15:45:19,755 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > Discarding the results produced by task execution > 3add641adeebe4e14dd0111bd647aa75. > 2022-03-09 15:45:19,755 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > DEDUP_TOOL_MESSAGE_EVENT -> Sink SINK_TOOL_MESSAGE_EVENT (3/3) > (07a1c10037596788aee6844603ab17a2) switched from CANCELING to CANCELED. > 2022-03-09 15:45:19,755 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > Discarding the results produced by task execution > 07a1c10037596788aee6844603ab17a2. > 2022-03-09 15:45:19,755 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: > MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter, > Filter)) (3/3) (bcebf365a3e6da0281d80b3ab0e2cff8) switched from CANCELING > to CANCELED. > 2022-03-09 15:45:19,755 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > Discarding the results produced by task execution > bcebf365a3e6da0281d80b3ab0e2cff8. > 2022-03-09 15:45:19,755 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > Discarding the results produced by task execution > bcebf365a3e6da0281d80b3ab0e2cff8. > 2022-03-09 15:45:19,755 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: > FLOW_EVENT -> Filter -> Map (3/3) (b811833b1ce35416fae61aba7cdbeb53) > switched from CANCELING to CANCELED. > 2022-03-09 15:45:19,755 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > Discarding the results produced by task execution > b811833b1ce35416fae61aba7cdbeb53. > 2022-03-09 15:45:19,755 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > Discarding the results produced by task execution > b811833b1ce35416fae61aba7cdbeb53. > 2022-03-09 15:45:19,756 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: > BULK_SENDER_EVENT -> Filter -> Map (3/3) (d4421c51bf8fb716c672e13fd249450f) > switched from CANCELING to CANCELED. > 2022-03-09 15:45:19,757 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > Discarding the results produced by task execution > d4421c51bf8fb716c672e13fd249450f. > 2022-03-09 15:45:19,757 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > Discarding the results produced by task execution > d4421c51bf8fb716c672e13fd249450f. > 2022-03-09 15:45:19,757 INFO > org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager > [] - Received resource requirements from job > 6f9d71e57efba96dad7f5328ab9ac717: > [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, > numberOfRequiredSlots=2}] > 2022-03-09 15:45:19,790 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: > BULK_SENDER_EVENT -> Filter -> Map (1/3) (117be79102d086ddb65fb0b31c245f49) > switched from CANCELING to CANCELED. > 2022-03-09 15:45:19,797 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: > FLOW_EVENT -> Filter -> Map (1/3) (4693404a84c87a07b1f35ed62f9edad7) > switched from CANCELING to CANCELED. > 2022-03-09 15:45:19,801 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: > MESSAGE_EVENT -> Filter -> Map -> (Filter -> Map, Filter -> (Filter, > Filter)) (1/3) (c4a8b6f2d0b7ea837dd4e69a107e652e) switched from CANCELING > to CANCELED. > 2022-03-09 15:45:20,026 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph [] - > DEDUP_TOOL_MESSAGE_EVENT -> Sink SINK_TOOL_MESSAGE_EVENT (1/3) > (2e3973007d7c3d71f10456c3aca5a0b3) switched from CANCELING to CANCELED. > 2022-03-09 15:45:23,957 INFO > org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - > Worker container_1646341714746_0005_01_000003 is terminated. Diagnostics: > Container container_1646341714746_0005_01_000003 marked as failed. > Exit code:137. > Diagnostics:[2022-03-09 15:45:19.639]Container killed on request. Exit > code is 137 > [2022-03-09 15:45:19.641]Container exited with a non-zero exit code 137. > [2022-03-09 15:45:19.642]Killed by external signal > > 2022-03-09 15:45:23,957 INFO > org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - > Closing TaskExecutor connection container_1646341714746_0005_01_000003 > because: Container container_1646341714746_0005_01_000003 marked as failed. > Exit code:137. > Diagnostics:[2022-03-09 15:45:19.639]Container killed on request. Exit > code is 137 > [2022-03-09 15:45:19.641]Container exited with a non-zero exit code 137. > [2022-03-09 15:45:19.642]Killed by external signal > > > > Thanks, > > > > On Tue, Mar 8, 2022 at 4:57 AM Schwalbe Matthias < > matthias.schwa...@viseca.ch> wrote: > > Bom Dia Vinicius, > > > > Can You still find (and post) the exception stack from your jobmanager > log, the flink client log does not reveal enough information. > > Your situation reminds me of something similar I had. > > In the log you might find something like this or similar: > > > > 2022-03-07 02:15:41,347 INFO > org.apache.flink.runtime.jobmaster.JobMaster [] - > Triggering stop-with-savepoint for job e12f22653f79194863ab426312dd666a. > > 2022-03-07 02:15:41,380 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering > checkpoint 4983974 (type=SAVEPOINT_SUSPEND) @ 1646615741347 for job > e12f22653f79194863ab426312dd666a. > > 2022-03-07 02:15:43,042 INFO > org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Decline > checkpoint 4983974 by task 0e659ac720e3e0b3e4072dbc1cc85cd3 of job > e12f22653f79194863ab426312dd666a at > container_e1093_1646358077201_0002_01_000001 @ ulxxphaddtn02.adgr.net > (dataPort=44767). > > org.apache.flink.util.SerializedThrowable: Asynchronous task checkpoint > failed. > > at > org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable.handleExecutionException(AsyncCheckpointRunnable.java:279) > ~[flink-dist_2.11-1.13.0.jar:1.13.0] > > > > BTW what Flink version are you running? > > What is EMR (what technology underneath). > > > > > > > > *From:* Vinicius Peracini <vinicius.perac...@zenvia.com> > *Sent:* Montag, 7. März 2022 20:46 > *To:* Dawid Wysakowicz <dwysakow...@apache.org> > *Cc:* user@flink.apache.org > *Subject:* Re: Could not stop job with a savepoint > > > > Hi Dawid, thanks for the reply. > > > > The job was still in progress and producing events. Unfortunately I was > not able to stop the job with a savepoint or to just create a savepoint. I > had to stop the job without the savepoint and restore the state using the > last checkpoint. Still reviewing my configuration and trying to figure out > why this is happening. Any help would be appreciated. > > > > Thanks! > > > > > > On Mon, Mar 7, 2022 at 11:56 AM Dawid Wysakowicz <dwysakow...@apache.org> > wrote: > > Hi, > > From the exception it seems the job has been already done when you're > triggering the savepoint. > > Best, > > Dawid > > On 07/03/2022 14:56, Vinicius Peracini wrote: > > Hello everyone, > > > > I have a Flink job (version 1.14.0 running on EMR) and I'm having this > issue while trying to stop a job with a savepoint on S3: > > > > org.apache.flink.util.FlinkException: Could not stop with a savepoint job > "df3a3c590fabac737a17f1160c21094c". > at > org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:581) > at > org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:1002) > at org.apache.flink.client.cli.CliFrontend.stop(CliFrontend.java:569) > at > org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1069) > at > org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) > at > org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) > at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132) > Caused by: java.util.concurrent.ExecutionException: > java.util.concurrent.CompletionException: > org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint > Coordinator is suspending. > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) > at > org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:579) > ... 9 more > > > > I'm using incremental and unaligned checkpoints (aligned checkpoint > timeout is 30 seconds). I also tried to create the savepoint without > stopping the job (using flink savepoint command) and got the same error. > Any idea what is happening here? > > > > Thanks in advance, > > > > Aviso Legal: Este documento pode conter informações confidenciais e/ou > privilegiadas. Se você não for o destinatário ou a pessoa autorizada a > receber este documento, não deve usar, copiar ou divulgar as informações > nele contidas ou tomar qualquer ação baseada nessas informações. > > > > Disclaimer: The information contained in this document may be privileged > and confidential and protected from disclosure. If the reader of this > document is not the intended recipient, or an employee agent responsible > for delivering this document to the intended recipient, you are hereby > notified that any dissemination, distribution or copying of this > communication is strictly prohibited. > > > > Aviso Legal: Este documento pode conter informações confidenciais e/ou > privilegiadas. Se você não for o destinatário ou a pessoa autorizada a > receber este documento, não deve usar, copiar ou divulgar as informações > nele contidas ou tomar qualquer ação baseada nessas informações. > > > > Disclaimer: The information contained in this document may be privileged > and confidential and protected from disclosure. If the reader of this > document is not the intended recipient, or an employee agent responsible > for delivering this document to the intended recipient, you are hereby > notified that any dissemination, distribution or copying of this > communication is strictly prohibited. > > Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und > beinhaltet unter Umständen vertrauliche Mitteilungen. Da die > Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann, > übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und > Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir > Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie > eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung > dieser Informationen ist streng verboten. > > This message is intended only for the named recipient and may contain > confidential or privileged information. As the confidentiality of email > communication cannot be guaranteed, we do not accept any responsibility for > the confidentiality and the intactness of this message. If you have > received it in error, please advise the sender by return e-mail and delete > this message and any attachments. Any unauthorised use or dissemination of > this information is strictly prohibited. > > > > Aviso Legal: Este documento pode conter informações confidenciais e/ou > privilegiadas. Se você não for o destinatário ou a pessoa autorizada a > receber este documento, não deve usar, copiar ou divulgar as informações > nele contidas ou tomar qualquer ação baseada nessas informações. > > > > Disclaimer: The information contained in this document may be privileged > and confidential and protected from disclosure. If the reader of this > document is not the intended recipient, or an employee agent responsible > for delivering this document to the intended recipient, you are hereby > notified that any dissemination, distribution or copying of this > communication is strictly prohibited. > Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und > beinhaltet unter Umständen vertrauliche Mitteilungen. Da die > Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann, > übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und > Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir > Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie > eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung > dieser Informationen ist streng verboten. > > This message is intended only for the named recipient and may contain > confidential or privileged information. As the confidentiality of email > communication cannot be guaranteed, we do not accept any responsibility for > the confidentiality and the intactness of this message. If you have > received it in error, please advise the sender by return e-mail and delete > this message and any attachments. Any unauthorised use or dissemination of > this information is strictly prohibited. > -- Aviso Legal: Este documento pode conter informações confidenciais e/ou privilegiadas. Se você não for o destinatário ou a pessoa autorizada a receber este documento, não deve usar, copiar ou divulgar as informações nele contidas ou tomar qualquer ação baseada nessas informações. Disclaimer: The information contained in this document may be privileged and confidential and protected from disclosure. If the reader of this document is not the intended recipient, or an employee agent responsible for delivering this document to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited.
