lhotari commented on issue #551:
URL:
https://github.com/apache/pulsar-helm-chart/issues/551#issuecomment-2493802246
> @merlimat @mattisonchao Do you have a chance to help in addressing this
issue with the Oxia support in Apache Pulsar Helm chart? The chart currently
works with 1 oxia server replica, but when that's 3, it fails.
I found the issue and made a PR in #553. Now Oxia cluster starts up without
problems, but the Oxia Java client fails to connect.
There are errors like this:
`2024-11-22T13:41:28,857+0000 [grpc-default-worker-ELG-2-4] WARN
io.streamnative.oxia.client.notify.ShardNotificationReceiver - Error while
receiving notifications for shard=8: UNKNOWN: node is not leader for shard 8 -
Retrying in 0.11 seconds`
@mattisonchao @merlimat What's the reason for this problem?
```
at
io.grpc.internal.AbstractStream$TransportState.closeDeframer(AbstractStream.java:201)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.transportReportStatus(AbstractClientStream.java:444)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.transportReportStatus(AbstractClientStream.java:400)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.inboundTrailersReceived(AbstractClientStream.java:383)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.Http2ClientStreamTransportState.transportTrailersReceived(Http2ClientStreamTransportState.java:183)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientStream$TransportState.transportHeadersReceived(NettyClientStream.java:334)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.onHeadersRead(NettyClientHandler.java:379)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.access$1200(NettyClientHandler.java:93)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler$FrameListener.onHeadersRead(NettyClientHandler.java:936)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:409)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:337)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onHeadersRead(Http2InboundFrameLogger.java:56)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader$2.processFragment(DefaultHttp2FrameReader.java:476)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:484)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:253)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at java.base/java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: io.grpc.StatusRuntimeException: UNKNOWN: node is not leader for
shard 8
at io.grpc.Status.asRuntimeException(Status.java:539)
~[io.grpc-grpc-api-1.56.1.jar:1.56.1]
at
io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:491)
~[io.grpc-grpc-stub-1.56.1.jar:1.56.1]
at
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.SerializeReentrantCallsDirectExecutor.execute(SerializeReentrantCallsDirectExecutor.java:49)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.closedInternal(ClientCallImpl.java:743)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.closed(ClientCallImpl.java:683)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at io.grpc.internal.RetriableStream$4.run(RetriableStream.java:830)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at io.grpc.SynchronizationContext.drain(SynchronizationContext.java:94)
~[io.grpc-grpc-api-1.56.1.jar:1.56.1]
at
io.grpc.SynchronizationContext.execute(SynchronizationContext.java:126)
~[io.grpc-grpc-api-1.56.1.jar:1.56.1]
at
io.grpc.internal.RetriableStream.safeCloseMasterListener(RetriableStream.java:825)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at io.grpc.internal.RetriableStream.access$2200(RetriableStream.java:55)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.RetriableStream$Sublistener.closed(RetriableStream.java:1018)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.ForwardingClientStreamListener.closed(ForwardingClientStreamListener.java:34)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.InternalSubchannel$CallTracingTransport$1$1.closed(InternalSubchannel.java:691)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.closeListener(AbstractClientStream.java:458)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.access$400(AbstractClientStream.java:221)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState$1.run(AbstractClientStream.java:441)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.deframerClosed(AbstractClientStream.java:278)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.Http2ClientStreamTransportState.deframerClosed(Http2ClientStreamTransportState.java:31)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at io.grpc.internal.MessageDeframer.close(MessageDeframer.java:234)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.MessageDeframer.closeWhenComplete(MessageDeframer.java:192)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractStream$TransportState.closeDeframer(AbstractStream.java:201)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.transportReportStatus(AbstractClientStream.java:444)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.transportReportStatus(AbstractClientStream.java:400)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.inboundTrailersReceived(AbstractClientStream.java:383)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.Http2ClientStreamTransportState.transportTrailersReceived(Http2ClientStreamTransportState.java:183)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientStream$TransportState.transportHeadersReceived(NettyClientStream.java:334)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.onHeadersRead(NettyClientHandler.java:379)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.access$1200(NettyClientHandler.java:93)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler$FrameListener.onHeadersRead(NettyClientHandler.java:936)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:409)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:337)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onHeadersRead(Http2InboundFrameLogger.java:56)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader$2.processFragment(DefaultHttp2FrameReader.java:476)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:484)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:253)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at java.base/java.lang.Thread.run(Unknown Source) ~[?:?]
2024-11-22T13:41:24,088+0000 [main] INFO
org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager
initialized
Nov 22, 2024 1:41:24 PM io.opentelemetry.api.GlobalOpenTelemetry
maybeAutoConfigureAndSetGlobal
INFO: AutoConfiguredOpenTelemetrySdk found on classpath but automatic
configuration is disabled. To enable, run your JVM with
-Dotel.java.global-autoconfigure.enabled=true
2024-11-22T13:41:24,406+0000 [grpc-default-worker-ELG-2-4] WARN
io.streamnative.oxia.client.notify.ShardNotificationReceiver - Error while
receiving notifications for shard=8: UNKNOWN: node is not leader for shard 8 -
Retrying in 0.112 seconds
2024-11-22T13:41:24,420+0000 [main] INFO
org.apache.pulsar.common.util.GracefulExecutorServicesTerminationHandler -
Starting termination handler for 1 executors.
2024-11-22T13:41:24,421+0000 [main] INFO
org.apache.pulsar.common.util.GracefulExecutorServicesTerminationHandler -
Shutdown completed.
2024-11-22T13:41:24,423+0000 [main] INFO
org.apache.pulsar.common.util.GracefulExecutorServicesTerminationHandler -
Starting termination handler for 1 executors.
2024-11-22T13:41:24,423+0000 [main] INFO
org.apache.pulsar.common.util.GracefulExecutorServicesTerminationHandler -
Shutdown completed.
at
io.grpc.internal.AbstractStream$TransportState.closeDeframer(AbstractStream.java:201)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.transportReportStatus(AbstractClientStream.java:444)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.transportReportStatus(AbstractClientStream.java:400)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.inboundTrailersReceived(AbstractClientStream.java:383)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.Http2ClientStreamTransportState.transportTrailersReceived(Http2ClientStreamTransportState.java:183)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientStream$TransportState.transportHeadersReceived(NettyClientStream.java:334)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.onHeadersRead(NettyClientHandler.java:379)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.access$1200(NettyClientHandler.java:93)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler$FrameListener.onHeadersRead(NettyClientHandler.java:936)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:409)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:337)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onHeadersRead(Http2InboundFrameLogger.java:56)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader$2.processFragment(DefaultHttp2FrameReader.java:476)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:484)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:253)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at java.base/java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: io.grpc.StatusRuntimeException: UNKNOWN: node is not leader for
shard 8
at io.grpc.Status.asRuntimeException(Status.java:539)
~[io.grpc-grpc-api-1.56.1.jar:1.56.1]
at
io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:491)
~[io.grpc-grpc-stub-1.56.1.jar:1.56.1]
at
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.SerializeReentrantCallsDirectExecutor.execute(SerializeReentrantCallsDirectExecutor.java:49)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.closedInternal(ClientCallImpl.java:743)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.closed(ClientCallImpl.java:683)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at io.grpc.internal.RetriableStream$4.run(RetriableStream.java:830)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at io.grpc.SynchronizationContext.drain(SynchronizationContext.java:94)
~[io.grpc-grpc-api-1.56.1.jar:1.56.1]
at
io.grpc.SynchronizationContext.execute(SynchronizationContext.java:126)
~[io.grpc-grpc-api-1.56.1.jar:1.56.1]
at
io.grpc.internal.RetriableStream.safeCloseMasterListener(RetriableStream.java:825)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at io.grpc.internal.RetriableStream.access$2200(RetriableStream.java:55)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.RetriableStream$Sublistener.closed(RetriableStream.java:1018)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.ForwardingClientStreamListener.closed(ForwardingClientStreamListener.java:34)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.InternalSubchannel$CallTracingTransport$1$1.closed(InternalSubchannel.java:691)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.closeListener(AbstractClientStream.java:458)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.access$400(AbstractClientStream.java:221)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState$1.run(AbstractClientStream.java:441)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.deframerClosed(AbstractClientStream.java:278)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.Http2ClientStreamTransportState.deframerClosed(Http2ClientStreamTransportState.java:31)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at io.grpc.internal.MessageDeframer.close(MessageDeframer.java:234)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.MessageDeframer.closeWhenComplete(MessageDeframer.java:192)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractStream$TransportState.closeDeframer(AbstractStream.java:201)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.transportReportStatus(AbstractClientStream.java:444)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.transportReportStatus(AbstractClientStream.java:400)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.inboundTrailersReceived(AbstractClientStream.java:383)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.Http2ClientStreamTransportState.transportTrailersReceived(Http2ClientStreamTransportState.java:183)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientStream$TransportState.transportHeadersReceived(NettyClientStream.java:334)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.onHeadersRead(NettyClientHandler.java:379)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.access$1200(NettyClientHandler.java:93)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler$FrameListener.onHeadersRead(NettyClientHandler.java:936)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:409)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:337)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onHeadersRead(Http2InboundFrameLogger.java:56)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader$2.processFragment(DefaultHttp2FrameReader.java:476)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:484)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:253)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at java.base/java.lang.Thread.run(Unknown Source) ~[?:?]
2024-11-22T13:41:28,561+0000 [main] INFO
org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager
initialized
Nov 22, 2024 1:41:28 PM io.opentelemetry.api.GlobalOpenTelemetry
maybeAutoConfigureAndSetGlobal
INFO: AutoConfiguredOpenTelemetrySdk found on classpath but automatic
configuration is disabled. To enable, run your JVM with
-Dotel.java.global-autoconfigure.enabled=true
2024-11-22T13:41:28,857+0000 [grpc-default-worker-ELG-2-4] WARN
io.streamnative.oxia.client.notify.ShardNotificationReceiver - Error while
receiving notifications for shard=8: UNKNOWN: node is not leader for shard 8 -
Retrying in 0.11 seconds
2024-11-22T13:41:28,869+0000 [main] INFO
org.apache.pulsar.common.util.GracefulExecutorServicesTerminationHandler -
Starting termination handler for 1 executors.
2024-11-22T13:41:28,870+0000 [main] INFO
org.apache.pulsar.common.util.GracefulExecutorServicesTerminationHandler -
Shutdown completed.
2024-11-22T13:41:28,871+0000 [main] INFO
org.apache.pulsar.common.util.GracefulExecutorServicesTerminationHandler -
Starting termination handler for 1 executors.
2024-11-22T13:41:28,871+0000 [main] INFO
org.apache.pulsar.common.util.GracefulExecutorServicesTerminationHandler -
Shutdown completed.
at
io.grpc.internal.AbstractStream$TransportState.closeDeframer(AbstractStream.java:201)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.transportReportStatus(AbstractClientStream.java:444)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.transportReportStatus(AbstractClientStream.java:400)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.inboundTrailersReceived(AbstractClientStream.java:383)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.Http2ClientStreamTransportState.transportTrailersReceived(Http2ClientStreamTransportState.java:183)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientStream$TransportState.transportHeadersReceived(NettyClientStream.java:334)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.onHeadersRead(NettyClientHandler.java:379)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.access$1200(NettyClientHandler.java:93)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler$FrameListener.onHeadersRead(NettyClientHandler.java:936)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:409)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:337)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onHeadersRead(Http2InboundFrameLogger.java:56)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader$2.processFragment(DefaultHttp2FrameReader.java:476)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:484)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:253)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at java.base/java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: io.grpc.StatusRuntimeException: UNKNOWN: node is not leader for
shard 8
at io.grpc.Status.asRuntimeException(Status.java:539)
~[io.grpc-grpc-api-1.56.1.jar:1.56.1]
at
io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:491)
~[io.grpc-grpc-stub-1.56.1.jar:1.56.1]
at
io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.SerializeReentrantCallsDirectExecutor.execute(SerializeReentrantCallsDirectExecutor.java:49)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.closedInternal(ClientCallImpl.java:743)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.closed(ClientCallImpl.java:683)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at io.grpc.internal.RetriableStream$4.run(RetriableStream.java:830)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at io.grpc.SynchronizationContext.drain(SynchronizationContext.java:94)
~[io.grpc-grpc-api-1.56.1.jar:1.56.1]
at
io.grpc.SynchronizationContext.execute(SynchronizationContext.java:126)
~[io.grpc-grpc-api-1.56.1.jar:1.56.1]
at
io.grpc.internal.RetriableStream.safeCloseMasterListener(RetriableStream.java:825)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at io.grpc.internal.RetriableStream.access$2200(RetriableStream.java:55)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.RetriableStream$Sublistener.closed(RetriableStream.java:1018)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.ForwardingClientStreamListener.closed(ForwardingClientStreamListener.java:34)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.InternalSubchannel$CallTracingTransport$1$1.closed(InternalSubchannel.java:691)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.closeListener(AbstractClientStream.java:458)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.access$400(AbstractClientStream.java:221)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState$1.run(AbstractClientStream.java:441)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.deframerClosed(AbstractClientStream.java:278)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.Http2ClientStreamTransportState.deframerClosed(Http2ClientStreamTransportState.java:31)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at io.grpc.internal.MessageDeframer.close(MessageDeframer.java:234)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.MessageDeframer.closeWhenComplete(MessageDeframer.java:192)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractStream$TransportState.closeDeframer(AbstractStream.java:201)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.transportReportStatus(AbstractClientStream.java:444)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.transportReportStatus(AbstractClientStream.java:400)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.AbstractClientStream$TransportState.inboundTrailersReceived(AbstractClientStream.java:383)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.internal.Http2ClientStreamTransportState.transportTrailersReceived(Http2ClientStreamTransportState.java:183)
~[io.grpc-grpc-core-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientStream$TransportState.transportHeadersReceived(NettyClientStream.java:334)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.onHeadersRead(NettyClientHandler.java:379)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.access$1200(NettyClientHandler.java:93)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler$FrameListener.onHeadersRead(NettyClientHandler.java:936)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:409)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:337)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onHeadersRead(Http2InboundFrameLogger.java:56)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader$2.processFragment(DefaultHttp2FrameReader.java:476)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:484)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:253)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at
io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]
at java.base/java.lang.Thread.run(Unknown Source) ~[?:?]
2024-11-22T13:41:33,038+0000 [main] INFO
org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager
initialized
Nov 22, 2024 1:41:33 PM io.opentelemetry.api.GlobalOpenTelemetry
maybeAutoConfigureAndSetGlobal
INFO: AutoConfiguredOpenTelemetrySdk found on classpath but automatic
configuration is disabled. To enable, run your JVM with
-Dotel.java.global-autoconfigure.enabled=true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
