leiless opened a new issue #11491:
URL: https://github.com/apache/pulsar/issues/11491
# **Describe the bug**
When I utilize Pulsar Java Admin API to get the topic's last message-id, I
got 500 HTTP status code from Java API.
As a comparison, I use `pulsar-admin topics get-message-by-id ...`, it seems
OK.
# **To Reproduce**
I cannot reproduce it, it'll happen occasionally after several hours running.
# Behavior stated
## Java API side
My `getTopicLastMessage` API snippet:
```java
static Optional<Message<byte[]>> getTopicLastMessage(
PulsarAdmin admin,
String fqTopic) throws PulsarAdminException
{
MessageIdImpl msgId = (MessageIdImpl)
admin.topics().getLastMessageId(fqTopic);
System.out.println("Topic last msg id: " + msgId);
Message<byte[]> msg = admin.topics().getMessageById(fqTopic,
msgId.getLedgerId(), msgId.getEntryId());
return Optional.of(msg);
}
```
When I call this API with topic `persistent://foobar/mysql-cdc/binlog`, I
got the following logs:
<details>
<summary>My App logs...</summary>
<p>
```
2021-07-29 10:28:02.612 INFO foobar.mysql.cdc.binlog.CdcClient: Getting
topic last GTID from Pulsar ...
Topic last msg id: -1:-1:-1
2021-07-29 10:28:03.415 ERROR
foobar.mysql.cdc.binlog.CdcClient$EnsureThread$1: Cannot ensure binlog liveness
foobar.mysql.cdc.exception.FatalException: Cannot get last message GTID from
persistent://foobar/mysql-cdc/binlog
at
foobar.mysql.cdc.pulsar.PulsarAdminClient.getTopicLastGtidOrPanic(PulsarAdminClient.java:72)
~[classes/:?]
at
foobar.mysql.cdc.binlog.CdcClient.newBinlogClient(CdcClient.java:188)
~[classes/:?]
at foobar.mysql.cdc.binlog.CdcClient.access$900(CdcClient.java:51)
~[classes/:?]
at
foobar.mysql.cdc.binlog.CdcClient$EnsureThread.ensure(CdcClient.java:364)
~[classes/:?]
at
foobar.mysql.cdc.binlog.CdcClient$EnsureThread.access$1000(CdcClient.java:342)
~[classes/:?]
at
foobar.mysql.cdc.binlog.CdcClient$EnsureThread$1.run(CdcClient.java:376)
[classes/:?]
at java.util.TimerThread.mainLoop(Timer.java:555) [?:1.8.0_292]
at java.util.TimerThread.run(Timer.java:505) [?:1.8.0_292]
Caused by:
org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException:
HTTP 500 Internal Server Errorjava.io.ByteArrayInputStream@46f2a7f5
at
org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:208)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:251)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.client.admin.internal.TopicsImpl.getMessagesFromHttpResponse(TopicsImpl.java:1419)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.client.admin.internal.TopicsImpl.access$100(TopicsImpl.java:90)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.client.admin.internal.TopicsImpl$17.completed(TopicsImpl.java:1143)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.client.admin.internal.TopicsImpl$17.completed(TopicsImpl.java:1139)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyInvocation$1.completed(JerseyInvocation.java:814)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:229)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:62)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:173)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:292)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:274)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:244)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:173)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:212)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
~[?:1.8.0_292]
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
~[?:1.8.0_292]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
~[?:1.8.0_292]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
~[?:1.8.0_292]
at
org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$3(AsyncHttpConnector.java:253)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
~[?:1.8.0_292]
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
~[?:1.8.0_292]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
~[?:1.8.0_292]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
~[?:1.8.0_292]
at
org.apache.pulsar.shade.org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:114)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:143)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
~[pulsar-client-2.7.2.jar:2.7.2]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292]
Caused by: org.apache.pulsar.shade.javax.ws.rs.ServerErrorException: HTTP
500 Internal Server Error
at
org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:244)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.client.admin.internal.TopicsImpl.getMessagesFromHttpResponse(TopicsImpl.java:1419)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.client.admin.internal.TopicsImpl.access$100(TopicsImpl.java:90)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.client.admin.internal.TopicsImpl$17.completed(TopicsImpl.java:1143)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.client.admin.internal.TopicsImpl$17.completed(TopicsImpl.java:1139)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.client.JerseyInvocation$1.completed(JerseyInvocation.java:814)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:229)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:62)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:173)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:292)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:274)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.internal.Errors.process(Errors.java:244)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:173)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:212)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
~[?:1.8.0_292]
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
~[?:1.8.0_292]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
~[?:1.8.0_292]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
~[?:1.8.0_292]
at
org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$3(AsyncHttpConnector.java:253)
~[pulsar-client-admin-2.7.2.jar:2.7.2]
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
~[?:1.8.0_292]
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
~[?:1.8.0_292]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
~[?:1.8.0_292]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
~[?:1.8.0_292]
at
org.apache.pulsar.shade.org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:114)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:143)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
~[pulsar-client-2.7.2.jar:2.7.2]
at
org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
~[pulsar-client-2.7.2.jar:2.7.2]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_292]
```
</p>
</details>
<details>
<summary>Pulsar logs...</summary>
<p>
```
02:28:01.603 [pulsar-web-63-24] INFO org.eclipse.jetty.server.RequestLog -
127.0.0.1 - - [29/Jul/2021:02:28:01 +0000] "GET
/admin/v2/persistent/public/functions/coordinate/stats?getPreciseBacklog=false&subscriptionBacklogSize=false
HTTP/1.1" 200 1950 "-" "Pulsar-Java-v2.8.0" 1
02:28:01.604 [pulsar-web-63-14] INFO org.eclipse.jetty.server.RequestLog -
127.0.0.1 - - [29/Jul/2021:02:28:01 +0000] "GET
/admin/v2/persistent/public/functions/coordinate/stats?getPreciseBacklog=false&subscriptionBacklogSize=false
HTTP/1.1" 200 1950 "-" "Pulsar-Java-v2.8.0" 1
02:28:03.347 [pulsar-web-63-24] INFO org.eclipse.jetty.server.RequestLog -
10.0.2.100 - - [29/Jul/2021:02:28:03 +0000] "GET
/admin/v2/persistent/foobar/mysql-cdc/binlog/lastMessageId HTTP/1.1" 200 48 "-"
"Pulsar-Java-v2.7.2" 1
02:28:03.409 [pulsar-web-63-14] ERROR
org.apache.bookkeeper.client.impl.OpenBuilderBase - invalid ledgerId -1 < 0
02:28:03.409 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[foobar/mysql-cdc/persistent/binlog] Error opening ledger for reading at
position -1:-1 -
org.apache.bookkeeper.mledger.ManagedLedgerException$NonRecoverableLedgerException:
No such ledger exists on Metadata Server
02:28:03.410 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.eclipse.jetty.server.RequestLog - 10.0.2.100 - - [29/Jul/2021:02:28:03
+0000] "GET /admin/v2/persistent/foobar/mysql-cdc/binlog/ledger/-1/entry/-1
HTTP/1.1" 500 243 "-" "Pulsar-Java-v2.7.2" 5
```
</p>
</details>
Note the `HTTP/1.1" 500` error log.
## Pulsar Admin command-line tool side
```
$ pulsar-admin topics last-message-id persistent://foobar/mysql-cdc/binlog
Warning: Nashorn engine is planned to be removed from a future JDK release
{
"ledgerId" : -1,
"entryId" : -1,
"partitionIndex" : -1
}
02:52:16.047 [pulsar-web-63-14] INFO org.eclipse.jetty.server.RequestLog -
10.0.2.100 - - [29/Jul/2021:02:52:16 +0000] "GET
/admin/v2/persistent/foobar/mysql-cdc/binlog/lastMessageId HTTP/1.1" 200 48 "-"
"Pulsar-Java-v2.7.2" 2
$ pulsar-admin topics get-message-by-id persistent://foobar/myqsl-cdc/binlog
-e -1 -l -1
Warning: Nashorn engine is planned to be removed from a future JDK release
10:53:08.534 [AsyncHttpClient-7-1] WARN
org.apache.pulsar.client.admin.internal.TopicsImpl - Exception 'Policies not
found for foobar/myqsl-cdc namespace' occurred while trying to get message.
Cannot find any messages based on ledgerId:-1 entryId:-1
02:53:08.486 [ForkJoinPool.commonPool-worker-5] WARN
org.apache.pulsar.broker.web.PulsarWebResource - Policies not found for
foobar/myqsl-cdc namespace
02:53:08.486 [pulsar-web-63-14] ERROR
org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [null] Failed to get
message with ledgerId -1 entryId -1 from persistent://foobar/myqsl-cdc/binlog
org.apache.pulsar.broker.web.RestException: Policies not found for
foobar/myqsl-cdc namespace
at
org.apache.pulsar.broker.web.PulsarWebResource.lambda$checkLocalOrGetPeerReplicationCluster$11(PulsarWebResource.java:760)
~[org.apache.pulsar-pulsar-broker-2.8.0.jar:2.8.0]
at
java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714)
~[?:?]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
~[?:?]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
~[?:?]
at
org.apache.pulsar.zookeeper.ZooKeeperDataCache.lambda$getAsync$0(ZooKeeperDataCache.java:68)
~[org.apache.pulsar-pulsar-zookeeper-utils-2.8.0.jar:2.8.0]
at
java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714)
~[?:?]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
~[?:?]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
~[?:?]
at
org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$getDataAsync$17(ZooKeeperCache.java:388)
~[org.apache.pulsar-pulsar-zookeeper-utils-2.8.0.jar:2.8.0]
at
java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:714)
~[?:?]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
~[?:?]
at
java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
~[?:?]
at
org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$getDataAsync$13(ZooKeeperCache.java:373)
~[org.apache.pulsar-pulsar-zookeeper-utils-2.8.0.jar:2.8.0]
at
java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426)
~[?:?]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
~[?:?]
at
java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
~[?:?]
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) ~[?:?]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
~[?:?]
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
~[?:?]
02:53:08.487 [pulsar-web-63-14] INFO org.eclipse.jetty.server.RequestLog -
10.0.2.100 - - [29/Jul/2021:02:53:08 +0000] "GET
/admin/v2/persistent/foobar/myqsl-cdc/binlog/ledger/-1/entry/-1 HTTP/1.1" 404
64 "-" "Pulsar-Java-v2.7.2" 2
```
What is weird to me is that `last-message-id` returned a `ledgerId` which is
negative.
# **Desktop (please complete the following information):**
- OS: Ubuntu 20.04 LTS amd64
- Maven artifacts: `pulsar-client`, `pulsar-client-admin` 2.7.2
# **Additional context**
The Pulsar is running in Podman 3.2.3 with no special config
```
podman run \
--name pulsar-cdc \
--hostname pulsar-cdc \
-d --restart=always \
-p 6650:6650 -p 8080:8080 \
apachepulsar/pulsar:latest \
bin/pulsar standalone
$ podman image ls | grep pulsar
docker.io/apachepulsar/pulsar latest
0622425a7cd8 6 weeks ago 1.48 GB
$ podman logs pulsar-cdc 2> /dev/null | grep -i "version: "
02:42:58.484 [main] INFO org.apache.pulsar.broker.PulsarService - Starting
Pulsar Broker service; version: '2.8.0'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
