Dave,
IIUC (from offline discussion) those tests started to fail only recently,
as we run those tests against the master branch of apache/pulsar repo.
Does the beginning of failures match the upgrade to BK 4.16 ?
The problem may be related to the handling of ByteBufs, IIRC there are a
few changes in the latest
BK release.
Enrico
Il giorno gio 13 apr 2023 alle ore 05:02 Hang Chen <chenh...@apache.org> ha
scritto:
> Hi Dave,
> Thanks for verifying this release candidate. I have setup the Pulsar
> cluster with 3.0 on my laptop and test the produce and consume, it works
> fine. We also deploy the it on k8s, the produce, consume and health check
> also works fine. Would you please share your broker configurations?
>
> Thanks,
> Hang
>
> Dave Fisher <wave4d...@comcast.net> 于2023年4月13日周四 06:49写道：
>
>> Hi -
>>
>> We do End to end testing of Pulsar versions on GCP and with Pulsar 3.0
>> today we have the following problem.
>>
>> This image shows the broker deployments not coming up. This is a non-TLS
>> setup.
>>
>> [image: Screen Shot 2023-04-12 at 1.09.59 PM.png]
>>
>> The Broker Pods have errors like this one:
>>
>> 2023-04-12T20:10:35,995+0000 [broker-topic-workers-OrderedExecutor-3-0]
>> ERROR org.apache.pulsar.common.protocol.Commands -
>> [PersistentSubscription{topic=persistent://pulsar/pulsar/10.236.0.4:8080/healthcheck,
>> name=healthCheck-04a8e22e-3e0b-4aa3-9141-d8970c1b0712}] [-1] Failed to
>> parse message metadata
>> java.lang.IndexOutOfBoundsException: readerIndex(96) + length(2) exceeds
>> writerIndex(96): UnpooledDuplicatedByteBuf(ridx: 96, widx: 96, cap: 96,
>> unwrapped: CompositeByteBuf(ridx: 96, widx: 96, cap: 96, components=2))
>> at
>> io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1442)
>> ~[io.netty-netty-buffer-4.1.89.Final.jar:4.1.89.Final]
>> at io.netty.buffer.AbstractByteBuf.readShort(AbstractByteBuf.java:749)
>> ~[io.netty-netty-buffer-4.1.89.Final.jar:4.1.89.Final]
>> at
>> org.apache.pulsar.common.protocol.Commands.skipBrokerEntryMetadataIfExist(Commands.java:1692)
>> ~[org.apache.pulsar-pulsar-common-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
>> at
>> org.apache.pulsar.common.protocol.Commands.parseMessageMetadata(Commands.java:452)
>> ~[org.apache.pulsar-pulsar-common-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
>> at
>> org.apache.pulsar.common.protocol.Commands.parseMessageMetadata(Commands.java:445)
>> ~[org.apache.pulsar-pulsar-common-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
>> at
>> org.apache.pulsar.common.protocol.Commands.peekMessageMetadata(Commands.java:1899)
>> ~[org.apache.pulsar-pulsar-common-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
>> at
>> org.apache.pulsar.common.protocol.Commands.peekAndCopyMessageMetadata(Commands.java:1918)
>> ~[org.apache.pulsar-pulsar-common-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
>> at
>> org.apache.pulsar.broker.service.AbstractBaseDispatcher.filterEntriesForConsumer(AbstractBaseDispatcher.java:142)
>> ~[org.apache.pulsar-pulsar-broker-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
>> at
>> org.apache.pulsar.broker.service.AbstractBaseDispatcher.filterEntriesForConsumer(AbstractBaseDispatcher.java:100)
>> ~[org.apache.pulsar-pulsar-broker-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
>> at
>> org.apache.pulsar.broker.service.persistent.PersistentDispatcherSingleActiveConsumer.internalReadEntriesComplete(PersistentDispatcherSingleActiveConsumer.java:210)
>> ~[org.apache.pulsar-pulsar-broker-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
>> at
>> org.apache.pulsar.broker.service.persistent.PersistentDispatcherSingleActiveConsumer.lambda$readEntriesComplete$1(PersistentDispatcherSingleActiveConsumer.java:151)
>> ~[org.apache.pulsar-pulsar-broker-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
>> at
>> org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137)
>> ~[org.apache.bookkeeper-bookkeeper-common-4.16.0.jar:4.16.0]
>> at
>> org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:113)
>> ~[org.apache.bookkeeper-bookkeeper-common-4.16.0.jar:4.16.0]
>> at
>> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
>> ~[io.netty-netty-common-4.1.89.Final.jar:4.1.89.Final]
>> at java.lang.Thread.run(Thread.java:833) ~[?:?]
>>
>> We looked into this but aren’t sure what would cause this to suddenly be
>> an issue. The candidate would be differences in the new version of BK …
>>
>> Best,
>> Dave
>>
>
