Never mind. The issue was fix due to the service account permission missing “patch” verb. Which lead to RPC service not started.
> On Dec 5, 2023, at 1:40 PM, Ethan T Yang <ivanygy...@gmail.com> wrote: > > Hi Flink users, > After upgrading Flink ( from 1.13.1 -> 1.18.0), I noticed the an issue when > HA is enabled.( see exception below). I am using k8s deployment and I clean > the previous configmaps, like leader files etc. I know the pekko is a > recently thing. Can someone share doc on how to use or set it? When I disable > HA, the deployment was successful. I also noticed a new configmap called > “xxxx-cluster-config-map”, can someone provide reference on what it is for? I > don’t see it in the 1.13.1 version. > > Thanks a lot > Ivan > > > org.apache.flink.runtime.rpc.exceptions.RecipientUnreachableException: Could > not send message > [LocalRpcInvocation(RestfulGateway.requestMultipleJobDetails(Time))] from > sender [unknown] to recipient [pe > kko.tcp://flink@flink-secondary-jobmanager:6123/user/rpc/dispatcher_1], > because the recipient is unreachable. This can either mean that the recipient > has been terminated or that the remote RpcService i > s currently not reachable. > at com.sun.proxy.$Proxy55.requestMultipleJobDetails(Unknown Source) > ~[?:?] > at > org.apache.flink.runtime.rest.handler.job.JobsOverviewHandler.handleRequest(JobsOverviewHandler.java:65) > ~[flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:83) > ~[flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:196) > ~[flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83) > ~[flink-dist-1.18.0.jar:1.18.0] > at java.util.Optional.ifPresent(Unknown Source) [?:?] > at > org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:208) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:69) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) > [flink-dist-1.18.0.jar:1.1 > 8.0] > at > org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > [flink-dist-1.18.0.jar:1.18.0] > at java.lang.Thread.run(Unknown Source) [?:?] > Caused by: org.apache.pekko.pattern.AskTimeoutException: Recipient > [Actor[pekko://flink/user/rpc/dispatcher_1#1708522126]] had already been > terminated. Message of type [org.apache.flink.runtime.rpc.mes > sages.LocalFencedMessage]. >
