The fabric8 K8s client is using PATCH to replace get-and-update in v6.6.2. That's why you also need to give PATCH permission for the K8s service account. This would help to decrease the pressure of K8s APIServer. You could find more information here[1].
[1]. https://issues.apache.org/jira/browse/FLINK-32678 Best, Yang On Wed, Dec 6, 2023 at 10:54 PM Ethan T Yang <ivanygy...@gmail.com> wrote: > Never mind. The issue was fix due to the service account permission > missing “patch” verb. Which lead to RPC service not started. > > On Dec 5, 2023, at 1:40 PM, Ethan T Yang <ivanygy...@gmail.com> wrote: > > Hi Flink users, > After upgrading Flink ( from 1.13.1 -> 1.18.0), I noticed the an issue > when HA is enabled.( see exception below). I am using k8s deployment and I > clean the previous configmaps, like leader files etc. I know the pekko is a > recently thing. Can someone share doc on how to use or set it? When I > disable HA, the deployment was successful. I also noticed a new configmap > called “xxxx-cluster-config-map”, can someone provide reference on what > it is for? I don’t see it in the 1.13.1 version. > > Thanks a lot > Ivan > > > org.apache.flink.runtime.rpc.exceptions.RecipientUnreachableException: > Could not send message > [LocalRpcInvocation(RestfulGateway.requestMultipleJobDetails(Time))] from > sender [unknown] to recipient [pe > kko.tcp://flink@flink-secondary-jobmanager:6123/user/rpc/dispatcher_1], > because the recipient is unreachable. This can either mean that the > recipient has been terminated or that the remote RpcService i > s currently not reachable. > at com.sun.proxy.$Proxy55.requestMultipleJobDetails(Unknown Source) ~[?:?] > at > org.apache.flink.runtime.rest.handler.job.JobsOverviewHandler.handleRequest(JobsOverviewHandler.java:65) > ~[flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:83) > ~[flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:196) > ~[flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:83) > ~[flink-dist-1.18.0.jar:1.18.0] > at java.util.Optional.ifPresent(Unknown Source) [?:?] > at > org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:45) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:80) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:115) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:94) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:55) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:208) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.runtime.rest.FileUploadHandler.channelRead0(FileUploadHandler.java:69) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) > [flink-dist-1.18.0.jar:1.1 > 8.0] > at > org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) > [flink-dist-1.18.0.jar:1.18.0] > at > org.apache.flink.shaded.netty4.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > [flink-dist-1.18.0.jar:1.18.0] > at java.lang.Thread.run(Unknown Source) [?:?] > Caused by: org.apache.pekko.pattern.AskTimeoutException: Recipient > [Actor[pekko://flink/user/rpc/dispatcher_1#1708522126]] had already been > terminated. Message of type [org.apache.flink.runtime.rpc.mes > sages.LocalFencedMessage]. > > >
