lhotari commented on issue #22601: URL: https://github.com/apache/pulsar/issues/22601#issuecomment-2122642708
I was able to reproduce a few issues with the test setup. logs are at https://gist.github.com/lhotari/8302131cde5a0f0999e39f8fbd391f09 . ``` [pulsar-testenv-deployment-broker-2] 2024-05-21T13:17:12,146+0000 [pulsar-io-3-8] ERROR org.apache.pulsar.broker.service.Consumer - [PersistentSubscription{topic=persistent://my-tenant/my-namespace/my-topic-1, name=angus_test}] [4] Received ack for corrupted message at 94:98341 - Reason: ChecksumMismatch [pulsar-testenv-deployment-broker-2] 2024-05-21T13:17:12,146+0000 [pulsar-io-3-8] ERROR org.apache.pulsar.broker.service.Consumer - [PersistentSubscription{topic=persistent://my-tenant/my-namespace/my-topic-1, name=angus_test}] [8] Received ack for corrupted message at 94:98373 - Reason: ChecksumMismatch ``` also ``` [pulsar-testenv-deployment-broker-2] 2024-05-21T13:17:12,862+0000 [pulsar-io-3-1] INFO org.apache.pulsar.broker.service.persistent.PersistentSubscription - backlog for persistent://my-tenant/ my-namespace/my-topic-1 - 1076 [pulsar-testenv-deployment-broker-2] 2024-05-21T13:17:12,862+0000 [pulsar-io-3-1] INFO org.apache.pulsar.broker.service.ServerCnx - [/192.168.122.1:42264] Created subscription on topic persis tent://my-tenant/my-namespace/my-topic-1 / lari_test3 [pulsar-testenv-deployment-broker-2] 2024-05-21T13:17:12,862+0000 [pulsar-io-3-1] INFO org.apache.pulsar.broker.service.ServerCnx - [[id: 0xaf5ded60, L:/10.1.179.90:6651 - R:/192.168.122.1:42 264]] Subscribing on topic persistent://my-tenant/my-namespace/my-topic-1 / lari_test3. consumerId: 5 [pulsar-testenv-deployment-broker-2] 2024-05-21T13:17:12,862+0000 [pulsar-io-3-1] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://my-tenant/my-namespace/my-to pic-1] Disabled replicated subscriptions controller [pulsar-testenv-deployment-broker-2] 2024-05-21T13:17:12,862+0000 [pulsar-io-3-1] INFO org.apache.pulsar.broker.service.persistent. PersistentSubscription - backlog for persistent://my-tenant/my-namespace/my-topic-1 - 1076 [pulsar-testenv-deployment-broker-2] 2024-05-21T13:17:12,862+0000 [pulsar-io-3-1] INFO org.apache.pulsar.broker.service.ServerCnx - [/192.168.122.1:42264] Created subscription on topic persistent://my-tenant/my-namespace/my-topic-1 / lari_test3 [pulsar-testenv-deployment-broker-2] 2024-05-21T13:17:13,188+0000 [pulsar-io-3-1] WARN org.apache.pulsar.broker.service.ServerCnx - [/192.168.122.1:42264] Got exception java.lang.IllegalArgumentException: newPosition > limit: (2097 > 91) [pulsar-testenv-deployment-broker-2] at java.base/java.nio.Buffer.createPositionException(Buffer.java:341) [pulsar-testenv-deployment-broker-2] at java.base/java.nio.Buffer.position(Buffer.java:316) [pulsar-testenv-deployment-broker-2] at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516) [pulsar-testenv-deployment-broker-2] at java.base/java.nio.HeapByteBuffer.get(HeapByteBuffer.java:185) [pulsar-testenv-deployment-broker-2] at io.netty.buffer.UnpooledHeapByteBuf.setBytes(UnpooledHeapByteBuf.java:268) [pulsar-testenv-deployment-broker-2] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1113) [pulsar-testenv-deployment-broker-2] at io.netty.buffer.ReadOnlyByteBufferBuf.copy(ReadOnlyByteBufferBuf.java:431) [pulsar-testenv-deployment-broker-2] at io.netty.buffer.DuplicatedByteBuf.copy(DuplicatedByteBuf.java:210) [pulsar-testenv-deployment-broker-2] at io.netty.buffer.AbstractByteBuf.copy(AbstractByteBuf.java:1194) [pulsar-testenv-deployment-broker-2] at org.apache.pulsar.common.protocol.ByteBufPair$CopyingEncoder.write(ByteBufPair.java:149) [pulsar-testenv-deployment-broker-2] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:893) [pulsar-testenv-deployment-broker-2] at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:875) [pulsar-testenv-deployment-broker-2] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:984) [pulsar-testenv-deployment-broker-2] at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:868) [pulsar-testenv-deployment-broker-2] at org.apache.pulsar.broker.service.PulsarCommandSenderImpl.lambda$sendMessagesToConsumer$1(PulsarCommandSenderImpl.java:277) [pulsar-testenv-deployment-broker-2] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) [pulsar-testenv-deployment-broker-2] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) [pulsar-testenv-deployment-broker-2] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) [pulsar-testenv-deployment-broker-2] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) [pulsar-testenv-deployment-broker-2] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [pulsar-testenv-deployment-broker-2] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [pulsar-testenv-deployment-broker-2] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [pulsar-testenv-deployment-broker-2] at java.base/java.lang.Thread.run(Thread.java:840) ``` The way I reproduced this was that I used the scripts in https://github.com/lhotari/pulsar-playground/tree/master/issues/issue22601 . I modified `consume.sh` to take a subscription name as a parameter. I created multiple subscriptions. `./consume.sh lari_test`, `./consume.sh lari_test2` and `./consume.sh lari_test3` and after creating a consumer, I killed it immediately to let some backlog collect since the producer was producing. After a few simultaneous restarts of the consumers, the problems reproduced immediately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
