2020-10-01 10:46:30 UTC - Linton: Looks like there’s also one a Flink Forward
<https://www.flink-forward.org/global-2020/conference-program#flink---pulsar--the-path-to-unified-batch-and-streaming->
----
2020-10-01 10:57:12 UTC - Ward Loos: @Ward Loos has joined the channel
----
2020-10-01 13:45:25 UTC - dipali bhat: pulsar using persistent volume claim
----
2020-10-01 13:45:34 UTC - dipali bhat: any body using this
----
2020-10-01 13:46:27 UTC - dipali bhat: the values,yaml file does not mention
about pv amd pvc . please advise
----
2020-10-01 14:29:50 UTC - Evan Furman: Yep, that is the one
`apachepulsar/pulsar-manager:v0.2.0`
----
2020-10-01 14:37:41 UTC - Sijie Guo: Yes @Addison Higham will given a
representation in Flink Forward Global.
----
2020-10-01 14:38:47 UTC - Evan Furman: What exactly does the bookkeeper visual
manager do and how is it different than this part of the
`application.properties` config?
```# bookie
bookie.host=<http://bookie-1>-{{ cluster_name }}.{{ route53_zone }}:8000
bookie.enable=false```
----
2020-10-01 14:45:07 UTC - Christopher Faini: Thanks for getting back to me! In
my case, I'm looking at converting an existing C++ message processing project
to use Apache Pulsar. While it is true that the main reason I asked this was
because there is existing code I'd like to put into the pulsar functions, this
is also a very large and complex project where we need the most optimal per
message performance possible. I'm not sure that any of the current language
options for pulsar functions would work for us.
----
2020-10-01 15:40:36 UTC - Christopher Faini: Just an example of how important
low latency is for this project: We started looking at Pulsar because there are
concerns that Kafka isn't fast enough for our requirements.
----
2020-10-01 17:18:46 UTC - Alan Hoffmeister: has pulsar dropped support for
protobuf schemas? I couldn't find anything related to it in the latest docs
----
2020-10-01 17:59:57 UTC - Addison Higham: No, still supported:
<http://pulsar.apache.org/docs/en/concepts-schema-registry/#supported-schema-formats>
----
2020-10-01 18:01:00 UTC - Pushkar Sawant: We are running Pulsar 2.6.0 deployed
using helm charts. For couple of topics i get following exception. Has anyone
seen this? i get a 504 timeouts when i try to delete these two topics. Rest of
the topics seem to be working fine
----
2020-10-01 18:01:00 UTC - Pushkar Sawant:
```java.util.concurrent.CompletionException:
java.util.concurrent.TimeoutException: Failed to load topic within timeout
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
~[?:1.8.0_252]
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
~[?:1.8.0_252]
at
java.util.concurrent.CompletableFuture.uniCompose(CompletableFuture.java:957)
~[?:1.8.0_252]
at
java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:940)
~[?:1.8.0_252]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
~[?:1.8.0_252]
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
~[?:1.8.0_252]
at
org.apache.pulsar.broker.service.BrokerService.lambda$futureWithDeadline$18(BrokerService.java:836)
~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
[io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
[io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
[io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
[io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
[io.netty-netty-transport-native-epoll-4.1.48.Final-linux-x86_64.jar:4.1.48.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
[io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.48.Final.jar:4.1.48.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
Caused by: java.util.concurrent.TimeoutException: Failed to load topic within
timeout
at
org.apache.pulsar.broker.service.BrokerService.loadOrCreatePersistentTopic(BrokerService.java:955)
~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0]
at
org.apache.pulsar.broker.service.BrokerService.lambda$getTopic$13(BrokerService.java:731)
~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0]
at
org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.put(ConcurrentOpenHashMap.java:277)
~[org.apache.pulsar-pulsar-common-2.6.0.jar:2.6.0]
at
org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.computeIfAbsent(ConcurrentOpenHashMap.java:130)
~[org.apache.pulsar-pulsar-common-2.6.0.jar:2.6.0]
at
org.apache.pulsar.broker.service.BrokerService.getTopic(BrokerService.java:730)
~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0]
at
org.apache.pulsar.broker.service.ServerCnx.lambda$null$14(ServerCnx.java:841)
~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0]
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
~[?:1.8.0_252]
at
java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:628)
~[?:1.8.0_252]
at
java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1996)
~[?:1.8.0_252]
at
org.apache.pulsar.broker.service.ServerCnx.lambda$handleSubscribe$16(ServerCnx.java:792)
~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0]
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
~[?:1.8.0_252]
at
java.util.concurrent.CompletableFuture.uniApplyStage(CompletableFuture.java:628)
~[?:1.8.0_252]
at
java.util.concurrent.CompletableFuture.thenApply(CompletableFuture.java:1996)
~[?:1.8.0_252]
at
org.apache.pulsar.broker.service.ServerCnx.handleSubscribe(ServerCnx.java:777)
~[org.apache.pulsar-pulsar-broker-2.6.0.jar:2.6.0]
at
org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:215)
~[org.apache.pulsar-pulsar-common-2.6.0.jar:2.6.0]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.handler.flow.FlowControlHandler.dequeue(FlowControlHandler.java:191)
~[io.netty-netty-handler-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.handler.flow.FlowControlHandler.channelRead(FlowControlHandler.java:153)
~[io.netty-netty-handler-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
~[io.netty-netty-codec-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
~[io.netty-netty-codec-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
~[io.netty-netty-transport-4.1.48.Final.jar:4.1.48.Final]
at
io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
~[io.netty-netty-transport-native-epoll-4.1.48.Final-linux-x86_64.jar:4.1.48.Final]
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
~[io.netty-netty-transport-native-epoll-4.1.48.Final-linux-x86_64.jar:4.1.48.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
~[io.netty-netty-transport-native-epoll-4.1.48.Final-linux-x86_64.jar:4.1.48.Final]
... 4 more```
----
2020-10-01 18:02:51 UTC - Addison Higham: There is support, see this example:
<https://github.com/apache/pulsar-helm-chart/blob/master/examples/values-bookkeeper-aws.yaml#L73>
It allows you to configure the storageclasses and then pulsar uses statefulsets
with volumeclaimtemplates
----
2020-10-01 19:45:30 UTC - Jim M.: I keep getting ErrorNativeIoException
failed:connection rest by peer
----
2020-10-02 00:44:52 UTC - Addison Higham: Hi @Pushkar Sawant there are some
known issues that can happen sometimes if something goes wrong with zookeeper
and bookkeeper and the topic fails to load once it can sometimes get stuck in a
bad state. There are fixes for this I believe in 2.6.1 (or will be released in
2.6.2). In the interim, you can unload the topic and that ~may~ should fix it
----
2020-10-02 05:59:02 UTC - Priyath Gregory: @Priyath Gregory has joined the
channel
----
2020-10-02 06:07:22 UTC - Priyath Gregory: Hey guys, apologies in advance if
this isn't the right place for this.
Was going through the pulsar-io kinesis consumer source code and noticed that
checkpointing is periodically done in an *asynchronous* manner after records
are pushed to an in-memory blocking queue. How is the possibility of record
loss handled with this approach? (eg: checkpoint completed but instance failure
while records are waiting in blocking queue)
<https://github.com/apache/pulsar/blob/master/pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/KinesisRecordProcessor.java>
----
2020-10-02 08:44:33 UTC - Shivji Kumar Jha: Hi, I made some CPP / Python client
changes in <https://github.com/apache/pulsar/pull/8173|my PR> and I see the
same suite of
<https://pipelines.actions.githubusercontent.com/EXIZfX8ixSbQEfqDA00dMJrEtloTMSwjlUs9NqJqrTcJfd7NaR/_apis/pipelines/1/runs/109338/signedlogcontent/3?urlExpires=2020-10-02T08%3A29%3A52.0688216Z&urlSigningMethod=HMACV1&urlSignature=EMEmcnANRlznJ0g5jEuQv6FU1m7tdr3tTH14AqOJPX0%3D|tests
failing>. Don't quite see why though, the errors seem completely unrelated. I
see the tests run passing in a different PR though. Need help understanding
this.
----
2020-10-02 08:46:44 UTC - Shivji Kumar Jha: Hi @Penghui Li Is this a known
issue?
----
2020-10-02 08:49:04 UTC - Shivji Kumar Jha:
<https://pipelines.actions.githubusercontent.com/EXIZfX8ixSbQEfqDA00dMJrEtloTMSwjlUs9NqJqrTcJfd7NaR/_apis/pipelines/1/runs/109354/signedlogcontent/3?urlExpires=2020-10-02T08%3A33%3A36.9834223Z&urlSigningMethod=HMACV1&urlSignature=TQvSxixR9wgHyeRxjSE09yHpckzki2ztZP8oI1SBZgw%3D|Another
failing run>, again apparently unrelated. But these tests have failed in
multiple runs so I wanted to check them.
----
2020-10-02 08:49:07 UTC - Penghui Li: Looks you can try to rerun the test
----
2020-10-02 08:50:13 UTC - Penghui Li: This one looks related to the github
action environment
----
2020-10-02 08:51:16 UTC - Shivji Kumar Jha: ```/pulsarbot run-failure-checks```
?
----
2020-10-02 08:51:25 UTC - Penghui Li: yes
+1 : Shivji Kumar Jha
----
2020-10-02 08:52:45 UTC - Shivji Kumar Jha: @Penghui Li Can you assign me the
<https://github.com/apache/pulsar/issues/8176|issue> /
<https://github.com/apache/pulsar/pull/8173|PR> and Sijie, @Yunze Xu as
reviewers please? Happy to have other reviewers if required.
----
2020-10-02 08:54:58 UTC - Shivji Kumar Jha: @Penghui Li didnt trigger test runs
with that
----
2020-10-02 08:56:30 UTC - Penghui Li: done
----
2020-10-02 08:57:26 UTC - Penghui Li: Since yunze is no a committer yet, you
can ping him if need a review.
----
2020-10-02 08:57:57 UTC - Shivji Kumar Jha: sure, I will do that, Because he
has been involved :-)
----
2020-10-02 08:58:20 UTC - Shivji Kumar Jha: started now, may be it was queued.
----
