moelhoussein opened a new issue, #6751: URL: https://github.com/apache/kyuubi/issues/6751
### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues. ### Describe the bug Kyuubi UI fails to render Servers  ### Affects Version(s) 1.9.1 ### Kyuubi Server Log Output ```logtalk 2024-10-17 16:06:45.703 INFO KyuubiRestFrontendService-12286 org.apache.kyuubi.server.api.v1.AdminResource: Received list all live kyuubi servers request from aMYUSER/10.251.228.11 2024-10-17 16:06:45.708 WARN grpc-default-executor-34 io.netty.bootstrap.Bootstrap: Unknown channel option 'io.netty.channel.epoll.EpollChannelOption#TCP_USER_TIMEOUT' for channel '[id: 0xa812e53b]' 2024-10-17 16:06:45.715 INFO KyuubiRestFrontendService-12286 org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient: Get service instance:100.64.139.185:10015 and version:1.9.1 under /kyuubi 2024-10-17 16:06:45.716 INFO KyuubiRestFrontendService-12286 org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient: Get service instance:100.64.40.106:10015 and version:1.9.1 under /kyuubi 2024-10-17 16:06:45.718 ERROR KyuubiRestFrontendService-12286 org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient: Failed to get service node info org.apache.kyuubi.KyuubiException: Key[/kyuubi/_1.9.1_USER_SPARK_SQL/A162003/default] not exists in ETCD, please check it. at org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient.getData(EtcdDiscoveryClient.scala:123) at org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient.$anonfun$getServiceNodesInfo$2(EtcdDiscoveryClient.scala:225) at scala.collection.immutable.List.map(List.scala:297) at org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient.getServiceNodesInfo(EtcdDiscoveryClient.scala:223) at org.apache.kyuubi.server.api.v1.AdminResource.$anonfun$listServers$2(AdminResource.scala:381) at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36) at org.apache.kyuubi.server.api.v1.AdminResource.listServers(AdminResource.scala:380) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:244) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) at org.apache.kyuubi.server.http.authentication.AuthenticationFilter.doFilter(AuthenticationFilter.scala:157) at org.apache.kyuubi.server.http.authentication.AuthenticationFilter.doFilter(AuthenticationFilter.scala:125) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.apache.kyuubi.server.http.authentication.KyuubiHttpAuthenticationFactory$HttpHandlerWrapperFactory$$anon$1.handle(KyuubiHttpAuthenticationFactory.scala:68) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.lang.Thread.run(Thread.java:750) 2024-10-17 16:06:45.720 INFO KyuubiRestFrontendService-12286 org.apache.kyuubi.server.http.authentication.AuthenticationAuditLogger: user=aMYUSER(auth:BASIC) ip=100.64.228.9 proxyIp=10.251.228.11 method=GET uri=/api/v1/admin/server params=null protocol=HTTP/1.1 status=200 ``` ### Kyuubi Engine Log Output ```logtalk N/A ``` ### Kyuubi Server Configurations ```yaml # kyuubi HA servers kyuubi.ha.addresses=http://etcd:2379 kyuubi.ha.client.class=org.apache.kyuubi.ha.client.etcd.EtcdDiscoveryClient ``` ### Kyuubi Engine Configurations ```yaml N/A ``` ### Additional context kyuubi is deployed on AKS through helm chart, with cnpg postgres as a metadata store, and ETCD for HA deployed with something like this https://etcd.io/docs/v3.6/op-guide/kubernetes/ -- I would like to mention that the set-up was initially with Zookeeper but moved to ETCD. ### Are you willing to submit PR? - [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix. - [X] No. I cannot submit a PR at this time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
