Hi Flink users,
After upgrading Flink ( from 1.13.1 -> 1.18.0), I noticed the an issue when HA
is enabled.( see exception below). I am using k8s deployment and I clean the
previous configmaps, like leader files etc. I know the pekko is a recently
thing. Can someone share doc on how to use or set it? When I disable HA, the
deployment was successful. I also noticed a new configmap called
“xxxx-cluster-config-map”, can someone provide reference on what it is for? I
don’t see it in the 1.13.1 version.
Thanks a lot
Ivan
org.apache.flink.runtime.rpc.exceptions.RecipientUnreachableException: Could
not send message
[LocalRpcInvocation(RestfulGateway.requestMultipleJobDetails(Time))] from
sender [unknown] to recipient [pe
kko.tcp://flink@flink-secondary-jobmanager:6123/user/rpc/dispatcher_1], because
the recipient is unreachable. This can either mean that the recipient has been
terminated or that the remote RpcService i
s currently not reachable.
at com.sun.proxy.$Proxy55.requestMultipleJobDetails(Unknown Source)
~[?:?]
at
org.apache.flink.runtime.rest.handler.job.JobsOverviewHandler.handleRequest(JobsOverviewHandler.java:65)
~[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:83)
~[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:196)
~[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83)
~[flink-dist-1.18.0.jar:1.18.0]
at java.util.Optional.ifPresent(Unknown Source) [?:?]
at
org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:208)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:69)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
[flink-dist-1.18.0.jar:1.1
8.0]
at
org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
[flink-dist-1.18.0.jar:1.18.0]
at
org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[flink-dist-1.18.0.jar:1.18.0]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: org.apache.pekko.pattern.AskTimeoutException: Recipient
[Actor[pekko://flink/user/rpc/dispatcher_1#1708522126]] had already been
terminated. Message of type [org.apache.flink.runtime.rpc.mes
sages.LocalFencedMessage].
