jing-c-tyagi opened a new issue, #501: URL: https://github.com/apache/pulsar-helm-chart/issues/501
we are running pulsar chart version pulsar-3.3.1 /app version 3.0.3 in aws eks. $ helm list -n pulsar NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION pulsar pulsar 8 2024-05-24 13:10:17.2221715 -0400 EDT deployed pulsar-3.3.1 3.0.3 due to some application side error, we are trying to restart pulsar pods. we ran kubectl scale statefulsets for pulsar-broker but it failed to start with CrashLoopBackOff status and describe the pod shows Liveness probe failed: HTTP probe failed with statuscode: 404 and Readiness probe failed: HTTP probe failed with statuscode: 404 **To Reproduce** kubectl scale statefulsets pulsar-broker --replicas=0 -n pulsar then kubectl scale statefulsets pulsar-broker --replicas=3 -n pulsar But the pulsar-broker pods failed to start, $ kgpo -n pulsar + kubectl get pods -n pulsar NAME READY STATUS RESTARTS AGE pulsar-bookie-0 1/1 Running 0 65m pulsar-bookie-1 1/1 Running 0 56m pulsar-bookie-2 1/1 Running 0 56m pulsar-bookie-3 1/1 Running 0 30m pulsar-broker-0 0/1 CrashLoopBackOff 14 (75s ago) 50m pulsar-broker-1 0/1 CrashLoopBackOff 14 (83s ago) 50m pulsar-broker-2 0/1 CrashLoopBackOff 9 (100s ago) 30m pulsar-proxy-0 0/1 Init:1/2 0 49m pulsar-proxy-1 0/1 Init:1/2 0 49m pulsar-proxy-2 0/1 Init:1/2 0 49m pulsar-pulsar-manager-0 1/1 Running 0 55m pulsar-recovery-0 1/1 Running 0 55m pulsar-toolset-0 1/1 Running 0 65m pulsar-zookeeper-0 1/1 Running 0 65m pulsar-zookeeper-1 1/1 Running 0 56m pulsar-zookeeper-2 1/1 Running 0 56m Describe the pulsar-broker-2 pod has the following events: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 29m default-scheduler Successfully assigned pulsar/pulsar-broker-2 to ip-10-249-18-54.ec2.internal Normal Pulled 29m kubelet Container image "apachepulsar/pulsar-all:3.0.3" already present on machine Normal Created 29m kubelet Created container wait-zookeeper-ready Normal Started 29m kubelet Started container wait-zookeeper-ready Normal Pulled 29m kubelet Container image "apachepulsar/pulsar-all:3.0.3" already present on machine Normal Created 29m kubelet Created container wait-bookkeeper-ready Normal Started 29m kubelet Started container wait-bookkeeper-ready Normal Pulled 29m (x2 over 29m) kubelet Container image "apachepulsar/pulsar-all:3.0.3" already present on machine Normal Created 29m (x2 over 29m) kubelet Created container pulsar-broker Normal Started 29m (x2 over 29m) kubelet Started container pulsar-broker Warning Unhealthy 28m (x2 over 28m) kubelet Liveness probe failed: Get "http://100.64.15.103:8080/status.html": dial tcp 100.64.15.103:8080: connect: connection refused Warning Unhealthy 28m (x2 over 28m) kubelet Readiness probe failed: Get "http://100.64.15.103:8080/status.html": dial tcp 100.64.15.103:8080: connect: connection refused Warning Unhealthy 28m (x4 over 28m) kubelet Liveness probe failed: HTTP probe failed with statuscode: 404 Warning Unhealthy 24m (x18 over 28m) kubelet Readiness probe failed: HTTP probe failed with statuscode: 404 Warning BackOff 4m39s (x78 over 27m) kubelet Back-off restarting failed container pulsar-broker in pod pulsar-broker-2_pulsar(a41a0f02-17bd-48ca-9e8f-a93afc673fc4) $ k logs -n pulsar pulsar-broker-2 --tail=100 -p + kubectl logs -n pulsar pulsar-broker-2 --tail=100 -p Defaulted container "pulsar-broker" out of: pulsar-broker, wait-zookeeper-ready (init), wait-bookkeeper-ready (init) at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222) ~[org.asynchttpclient-async-http-client-2.12.1.j] at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?] at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241) ~[org.asynchttpclient-async-http-nt-2.12.1.jar:?] at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:114) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?] at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:143) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?] at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78) ~[org.asynchttpclient-async-http-cl-2.12.1.jar:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[io.netty-netty-transport-4.1.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.10nal.jar:4.1.100.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[io.netty-netty-codec-4.1.100.Final.jar:4.0.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[io.netty-netty-transport-4.1.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.10nal.jar:4.1.100.Final] at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[itty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[io.netty-netty-codec-4.1.100.Final.jar:4.1.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[io.netty-netty-codec-4.1.100.Final.jar:4.1.100.l] at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[io.netty-netty-transport-4.1.100.Finar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[io.netty-netty-transport-4.1.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.10nal.jar:4.1.100.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.100.Finar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[io.netty-netty-transport-4.1.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.Final.jar:4.1.100.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty-netty-transport-4.1.100.Final.jar:4.0.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty-netty-transport-4.1.100.Final:4.1.100.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.1inal] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty-netty-common-4.1.100.Final.jar:400.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.l] at java.lang.Thread.run(Thread.java:840) ~[?:?] Caused by: javax.ws.rs.InternalServerErrorException: HTTP 500 {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: Topic cron encountered an exception by initialize topic policies service. topic_name=persistent://public/functions/assignments error_message={\"errorMsg\":\"Error e reading ledger - ledger=74597 - operation=Failed to read entry - entry=0\",\"reqId\":4520564592423817567, \"remote\":\"pulsar-broker-2.pulsar-broker.pulsvc.cluster.local/100.64.15.105:6650\", \"local\":\"/100.64.15.105:38832\"}\n\nStacktrace:\n\norg.apache.pulsar.broker.service.BrokerServiceException$ServiitNotReadyException: Topic creation encountered an exception by initialize topic policies service. topic_name=persistent://public/functions/assignments erressage={\"errorMsg\":\"Error while reading ledger - ledger=74597 - operation=Failed to read entry - entry=0\",\"reqId\":4520564592423817567, \"remote\":\"ar-broker-2.pulsar-broker.pulsar.svc.cluster.local/100.64.15.105: 6650\", \"local\":\"/100.64.15.105:38832\"}\n\tat org.apache.pulsar.broker.service.BrokerSce.lambda$getTopic$28(BrokerService.java:1065)\n\tat java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990)\n\tat jaase/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:974)\n\tat java.base/java.util.concurrent.CompletableFuture.postlete(CompletableFuture.java:510)\n\tat java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)\n\tat org.apachesar.client.impl.PulsarClientImpl.lambda$createSingleTopicReaderAsync$14(PulsarClientImpl.java:687)\n\tat java.base/java.util.concurrent.CompletableFuture.uceptionally(CompletableFuture.java:990)\n\tat java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:974)\n\tat jbase/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)\n\tat java.base/java.util.concurrent.Completabl eFuture.completeExceptily(CompletableFuture.java:2162)\n\tat org.apache.pulsar.client.impl.ConsumerImpl.lambda$connectionOpened$17(ConsumerImpl.java:893)\n\tat java.base/java.utincurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990)\n\tat java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(ComableFuture.java:974)\n\tat java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)\n\tat java.base/java.util.concurrent.CetableFuture.completeExceptionally(CompletableFuture.java:2162)\n\tat org.apache.pulsar.client.impl.ClientCnx.handleError(ClientCnx.java:793)\n\tat org.apapulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:192)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChaHandlerContext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.chanAbstractChannelHandlerContext.fireChannelRea d(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(BytessageDecoder.java:346)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)\n\tat io.netty.channel.AbstractChannellerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannellerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.flFlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChlHandlerContext.java:442)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.cha.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.channel.D efaultChannelPipeline$HeadContext.channelReafaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tat io.nethannel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelReafaultChannelPipeline.java:919)\n\tat io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)\t io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509)\n\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)\n\ta.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecuap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:84"} at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:960) ~[org.glassfish.jersey.core-jersey-client-2.3r:?] at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:82) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?] ... 54 more Caused by: javax.ws.rs.InternalServerErrorException: HTTP 500 {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: Topic creation entered an exception by initialize topic policies service. topic_name=persistent://public/functions/assignments error_message={\"errorMsg\":\"Error while reg ledger - ledger=74597 - operation=Failed to read entry - entry=0\",\"reqId\":4520564592423817567, \"remote\":\"pulsar-broker-2.pulsar-broker.pulsar.svc.ter.local/100.64.15.105:6650\", \"local\":\"/100.64.15.105:38832\"}\n\nStacktrace:\n\norg.apache.pulsar.broker.service.BrokerServiceException$ServiceUnitNodyException: Topic creation encountered an exception by initialize topic policies service. topic_name=persistent://public/functions/assignments error_messa\"errorMsg\":\"Error while reading ledger - ledger=74597 - operation=Failed to read entry - entry=0\",\"reqId\":4520564592423817567, \"remote\":\"pulsar-br-2.pulsar-broker.pulsar.svc.cluster.local/100.64.15.105:6650\", \"local\":\"/100.64.15.105:38832\"}\n\tat org.apache.pulsar.broker.service.BrokerService.la$getTopic$28(BrokerService.java:1065)\n\tat java.base/java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:990)\n\tat java.base/.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:974)\n\tat java.base/java.util.concurrent.CompletableFuture.postCompletepletableFuture.java:510)\n\tat java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)\n\tat org.apache.pulsar.nt.impl.PulsarClientImpl.lambda$createSingleTopicReaderAsync$14(PulsarClientImpl.java:687)\n\tat java.base/java.util.concurrent.CompletableFuture.uniExceptlly(CompletableFuture.java:990)\n\tat java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:974)\n\tat java.basea.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)\n\tat java.base/java.util.concurrent.CompletableFuture. completeExceptionally(CetableFuture.java:2162)\n\tat org.apache.pulsar.client.impl.ConsumerImpl.lambda$connectionOpened$17(ConsumerImpl.java:893)\n\tat java.base/java.util.concur.CompletableFuture.uniExceptionally(CompletableFuture.java:990)\n\tat java.base/java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(Completablere.java:974)\n\tat java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)\n\tat java.base/java.util.concurrent.Completabture.completeExceptionally(CompletableFuture.java:2162)\n\tat org.apache.pulsar.client.impl.ClientCnx.handleError(ClientCnx.java:793)\n\tat org.apache.pulsommon.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:192)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandontext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstChannelHandlerContext.fireChannelRead(Abstra ctChannelHandlerContext.java:412)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessacoder.java:346)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)\n\tat io.netty.channel.AbstractChannelHandlerCxt.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerCxt.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.handler.flush.FlussolidationHandler.channelRead(FlushConsolidationHandler.java:152)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHanContext.java:442)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.channel.DefaultCh annelPipeline$HeadContext.channelRead(DefaulnnelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tat io.netty.channbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaulnnelPipeline.java:919)\n\tat io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)\n\tat ioty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509)\n\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)\n\tat io.nettil.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.j74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\n"} at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:960) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?] at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:82) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?] at org.glassfish.jersey.client.JerseyInvocation$1.completed(JerseyInvocation.java:863) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?] at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:229) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?] at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:62) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?] at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:173) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?] at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:173) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?] at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:254) ~[org.apache.pulsar-pulsar-client-adoriginal-3.0.3.jar:3.0.3] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?] at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:296) ~[org.apache.pulsar-pulsar-nt-admin-original-3.0.3.jar:3.0.3] at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?] at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?] at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?] at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?] at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241) ~[org.asynchttpclient-async-http-client-21.jar:?] at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:114) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?] at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:143) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?] at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78) ~[org.asynchttpclient-async-http-client-2.1jar:?] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[io.netty-netty-transport-4.1.100.Finar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.100.Finar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.100.Final.4.1.100.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[io.netty-netty-codec-4.1.100.Final.jar:4.1.100.Fi at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[io.netty-netty-transport-4.1.100.Finar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.100.Finar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.100.Final.4.1.100.Final] at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[io.netty-y-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[io.netty-netty-codec-4.1.100.Final.jar:4.1.100.Fina 100.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[io.netty-netty-transport-4.1.100.Final.jar:4.1.100.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.100.Final.jar:4.1.100.Final] at java.lang.Thread.run(Thread.java:840) ~[?:?] 2024-05-24T16:58:28,764+0000 [main] WARN org.apache.pulsar.common.util.ShutdownUtil - Triggering immediate shutdown of current process with status 1 java.lang.Exception: Stacktrace for immediate shutdown at org.apache.pulsar.common.util.ShutdownUtil.triggerImmediateForcefulShutdown(ShutdownUtil.java:55) ~[org.apache.pulsar-pulsar-common-3.0.3.jar:3.0.3] at org.apache.pulsar.common.util.ShutdownUtil.triggerImmediateForcefulShutdown(ShutdownUtil.java:50) ~[org.apache.pulsar-pulsar-common-3.0.3.jar:3.0.3] at org.apache.pulsar.common.util.ShutdownUtil.triggerImmediateForcefulShutdown(ShutdownUtil.java:82) ~[org.apache.pulsar-pulsar-common-3.0.3.jar:3.0.3] at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:359) ~[org.apache.pulsar-pulsar-broker-3.0.3.jar:3.0.3] -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org