2020-04-07 09:22:02 UTC - Penghui Li: Looks related to
<https://github.com/apache/pulsar/pull/6489> .
----
2020-04-07 13:13:04 UTC - Greg: Ok thanks, meaning that this will work in 2.5.1
?
----
2020-04-07 13:38:48 UTC - Ian: If anyone here has used Pulsar for messages on
the larger end (250KB, 500KB, 1MB, 5MB), what sort of throughput have you seen?
----
2020-04-07 14:17:32 UTC - Ebere Abanonu: `<http://java.io|java.io>.IOException:
Failed to initialize ledger manager factory` is driving me nuts. How can I fix
it. I have trried both Pulsar 2.5.0 and StreamNative weekly build same thing.
Just started getting this error like two days ago
----
2020-04-07 14:18:41 UTC - Ebere Abanonu: Publishing and Consuming works fine
----
2020-04-07 15:48:27 UTC - Matteo Merli: Once you go over ~128KB, there's no
practical difference in byte/s throughput. It basically depends on having
enough disk IO and NIC bandwidth.
----
2020-04-07 16:23:36 UTC - Sijie Guo: hmm interesting. where did you receive
this error? Did you have the full stacktrace?
----
2020-04-07 16:25:23 UTC - Sijie Guo: Hmm. I have checked the code path. This
doesn’t seems to be directly impacting brokers. Did you have any other logs in
the broker?
----
2020-04-07 16:33:28 UTC - Dzmitry Kazimirchyk: there is another similar NPE
that follows, but with a slightly different stack trace
```15:47:34.690 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO
org.apache.bookkeeper.client.PendingReadOp - Error: Bookie handle is not
available while reading L356 E318 from bookie:
robot-pulsar-bookkeeper-1.robot-pulsar-bookkeeper.robot.svc.cluster.local:3181
15:47:34.693 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN
org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (356, 317):
Bookie handle is not available
15:47:34.697 [BookKeeperClientWorker-OrderedExecutor-0-0] ERROR
org.apache.bookkeeper.common.util.SafeRunnable - Unexpected throwable caught
java.lang.NullPointerException: null
at
<http://org.apache.bookkeeper.net|org.apache.bookkeeper.net>.NetUtils.resolveNetworkLocation(NetUtils.java:77)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy.resolveNetworkLocation(TopologyAwareEnsemblePlacementPolicy.java:779)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy.createBookieNode(TopologyAwareEnsemblePlacementPolicy.java:775)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:450)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:117)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:289)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.client.ReadOnlyLedgerHandle.handleBookieFailure(ReadOnlyLedgerHandle.java:226)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:377)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.proto.BookieClientImpl$1.safeRun(BookieClientImpl.java:292)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36)
[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_232]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_232]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]```
----
2020-04-07 16:38:09 UTC - Dzmitry Kazimirchyk: then on every consumer
connection attempt broker is logging this warning:
```16:36:20.204 [pulsar-io-22-1] WARN
org.apache.pulsar.broker.service.ServerCnx -
[/10.8.8.18:53928][<persistent://robot/_system/_signals_ingest][_system--_signals_ingest]>
Consumer with id 0 is already present on the connection
16:37:01.379 [pulsar-io-22-2] WARN org.apache.pulsar.broker.service.ServerCnx
- [/10.8.31.193:48732][1] Producer with id
<persistent://robot/_system/_signals_ingest> is already present on the
connection```
the clients are timing out with this error:
```org.apache.pulsar.client.api.PulsarClientException$TimeoutException: 2
lookup request timedout after ms 30000
at
org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:667)
at
org.apache.pulsar.client.impl.ConsumerBuilderImpl.subscribe(ConsumerBuilderImpl.java:97)```
----
2020-04-07 16:38:28 UTC - Ian: Thanks!
----
2020-04-07 16:52:10 UTC - Sijie Guo: oh I see. so there was an ensemble change,
which it will try to resolve the network address and got NPE.
----
2020-04-07 16:52:22 UTC - Sijie Guo: make sense now. so it is a problem of the
NPE.
----
2020-04-07 16:52:50 UTC - Sijie Guo: I will make sure that change can be merged
today.
+1 : Dzmitry Kazimirchyk
----
2020-04-07 17:37:34 UTC - Rattanjot Singh: ```I am trying to setup functions
worker (run with brokers)
curl <broker-ip>:8080/admin/v2/worker/cluster
{"reason":"Function worker service is not done initializing. Please try again
in a little while."}⏎
Any ideas how to resolve this```
----
2020-04-07 17:40:04 UTC - Sijie Guo: how did you setup the function worker?
----
2020-04-07 17:42:58 UTC - Rattanjot Singh: 1. functionsWorkerEnabled=true in
<http://broker.co|broker.conf>
2. enable Configure Kubernetes runtime in functions_worker.yml
----
2020-04-07 17:50:25 UTC - Ebere Abanonu: ```
2020-04-07T17:45:20.813486968Z 2020-04-07T17:45:20.813Z DEBUG
query-execution-8 com.facebook.presto.execution.QueryStateMachine Query
20200407_174519_00003_2a5ir is FAILED
2020-04-07T17:45:20.817463053Z 2020-04-07T17:45:20.815Z DEBUG
Query-20200407_174519_00003_2a5ir-236
com.facebook.presto.execution.QueryStateMachine Query
20200407_174519_00003_2a5ir failed
2020-04-07T17:45:20.817482955Z java.lang.RuntimeException:
<http://java.io|java.io>.IOException: Failed to initialize ledger manager
factory
2020-04-07T17:45:20.817489155Z at
org.apache.pulsar.sql.presto.PulsarSplitManager.getSplits(PulsarSplitManager.java:143)
2020-04-07T17:45:20.817513058Z at
com.facebook.presto.split.SplitManager.getSplits(SplitManager.java:64)
2020-04-07T17:45:20.817520058Z at
com.facebook.presto.sql.planner.DistributedExecutionPlanner$Visitor.visitTableScan(DistributedExecutionPlanner.java:146)
2020-04-07T17:45:20.817525059Z at
com.facebook.presto.sql.planner.DistributedExecutionPlanner$Visitor.visitTableScan(DistributedExecutionPlanner.java:122)
2020-04-07T17:45:20.817530259Z at
com.facebook.presto.sql.planner.plan.TableScanNode.accept(TableScanNode.java:136)
2020-04-07T17:45:20.81753506Z at
com.facebook.presto.sql.planner.DistributedExecutionPlanner.doPlan(DistributedExecutionPlanner.java:108)
2020-04-07T17:45:20.81753996Z at
com.facebook.presto.sql.planner.DistributedExecutionPlanner.doPlan(DistributedExecutionPlanner.java:113)
2020-04-07T17:45:20.817544361Z at
com.facebook.presto.sql.planner.DistributedExecutionPlanner.plan(DistributedExecutionPlanner.java:85)
2020-04-07T17:45:20.817548661Z at
com.facebook.presto.execution.SqlQueryExecution.planDistribution(SqlQueryExecution.java:385)
2020-04-07T17:45:20.817553162Z at
com.facebook.presto.execution.SqlQueryExecution.start(SqlQueryExecution.java:287)
2020-04-07T17:45:20.817557462Z at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2020-04-07T17:45:20.817561562Z at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2020-04-07T17:45:20.817565963Z at java.lang.Thread.run(Thread.java:748)
2020-04-07T17:45:20.817570363Z Caused by: <http://java.io|java.io>.IOException:
Failed to initialize ledger manager factory
2020-04-07T17:45:20.817574564Z at
org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:520)
2020-04-07T17:45:20.817579264Z at
org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:368)
2020-04-07T17:45:20.817583965Z at
org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$DefaultBkFactory.<init>(ManagedLedgerFactoryImpl.java:187)
2020-04-07T17:45:20.817588765Z at
org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.<init>(ManagedLedgerFactoryImpl.java:125)
2020-04-07T17:45:20.817593465Z at
org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.<init>(ManagedLedgerFactoryImpl.java:117)
2020-04-07T17:45:20.817599666Z at
org.apache.pulsar.sql.presto.PulsarConnectorCache.initManagedLedgerFactory(PulsarConnectorCache.java:117)
2020-04-07T17:45:20.817604967Z at
org.apache.pulsar.sql.presto.PulsarConnectorCache.<init>(PulsarConnectorCache.java:73)
2020-04-07T17:45:20.817610167Z at
org.apache.pulsar.sql.presto.PulsarConnectorCache.getConnectorCache(PulsarConnectorCache.java:92)
2020-04-07T17:45:20.817614968Z at
org.apache.pulsar.sql.presto.PulsarSplitManager.getSplitsPartitionedTopic(PulsarSplitManager.java:164)
2020-04-07T17:45:20.817619868Z at
org.apache.pulsar.sql.presto.PulsarSplitManager.getSplits(PulsarSplitManager.java:137)
2020-04-07T17:45:20.817624168Z ... 12 more
2020-04-07T17:45:20.817633269Z Caused by:
org.apache.bookkeeper.meta.exceptions.MetadataException: Failed to initialized
ledger manager factory
2020-04-07T17:45:20.81763757Z at
org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.getLedgerManagerFactory(ZKMetadataDriverBase.java:243)
2020-04-07T17:45:20.81764257Z at
org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:518)
2020-04-07T17:45:20.817646971Z ... 21 more
2020-04-07T17:45:20.817651371Z Caused by: <http://java.io|java.io>.IOException:
Empty Ledger Root Path.
2020-04-07T17:45:20.817655671Z at
org.apache.bookkeeper.meta.AbstractZkLedgerManagerFactory.newLedgerManagerFactory(AbstractZkLedgerManagerFactory.java:158)
2020-04-07T17:45:20.817660672Z at
org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase.getLedgerManagerFactory(ZKMetadataDriverBase.java:239)
2020-04-07T17:45:20.817665172Z ... 22 more```
----
2020-04-07 17:52:59 UTC - Kirill Kosenko: Hi guys
Is it possible to read all the messages of a topic that are stored in
bookkeeper and s3 using Reader?
----
2020-04-07 18:23:41 UTC - Matteo Merli: Yes, the purpose of tiered storage is
to completely transparent to clients
----
2020-04-07 18:35:49 UTC - Kirill Kosenko: Thanks
----
2020-04-07 18:44:37 UTC - Tim Corbett: When using a consumer, most docs have
the read messages loop continue infinitely, but obviously in the real world
services performing reads may need to exit gracefully. Is there a preferred
chain of events aside from just closing the consumer/connection and allowing
all locally buffered but unacked messages be redelivered to a new consumer?
----
2020-04-07 18:45:38 UTC - Matteo Merli: No, nothing more. Just close consumer
and (especially) the client instance when you're done.
+1 : Tim Corbett
----
2020-04-07 18:57:35 UTC - Sijie Guo: oh I see. it is from presto worker.
----
2020-04-07 18:57:43 UTC - Sijie Guo: How did you start presto worker?
----
2020-04-07 19:13:12 UTC - Gokul Sarangapani: @Gokul Sarangapani has joined the
channel
----
2020-04-07 20:43:13 UTC - Sijie Guo: @Rattanjot Singh are you configuring it on
a standalone instance or on a cluster?
----
2020-04-07 21:29:39 UTC - Greg Methvin: Are there any suggestions for writing
integration tests with topic pattern auto-discovery? At the moment I basically
have to wait at least a minute since that’s the minimum auto-discovery
interval, which can make the tests quite slow. I see
<https://github.com/apache/pulsar/issues/5927> was fixed for 2.6.0, but I’m not
on that version yet. Is there a way to manually tell the consumer to re-check,
so my tests can run more quickly?
----
2020-04-07 22:38:27 UTC - Sijie Guo: Currently the logic was triggered by a
timer only. It doesn’t trigger by other conditions.
One solution is to use mokito to spy a pulsar client instance and replace the
timer in the client. Then you can use that timer to control pattern auto
discovery check. But I am not sure if it is worth for you or not.
----
2020-04-08 00:13:44 UTC - Rattanjot Singh: On a cluster
----
2020-04-08 00:38:42 UTC - Sijie Guo: Are you running on Kubernetes? Did you
check the broker log to see if it is started correctly?
I guess the function worker is not started correctly. Did you set
`pulsarFunctionsCluster` in your function_worker.yml?
----
2020-04-08 01:00:09 UTC - Rattanjot Singh: yes set the
```pulsarFunctionsCluster```
----
2020-04-08 01:00:37 UTC - Rattanjot Singh: ```workerId: standalone
workerHostname: localhost```
i thin these two also need to be changed?
----
2020-04-08 01:11:27 UTC - Nicolas Ha: What is the purpose of `consumerName` ?
<http://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ConsumerBuilder.html#subscriptionName-java.lang.String->
the doc only says “Set the consumer name”
----
2020-04-08 01:11:48 UTC - Nicolas Ha: ie. is it only for debugging or does this
influence the subscribing in any way?
----
2020-04-08 01:17:52 UTC - Matteo Merli: It mainly serves for:
• Debugging (it shows up in the topic stats)
• It can be used in the failover subscriptions dispatcher to have
deterministic partition assignments (eg: brokers will sort by consumer name and
assign partitions based on that)
+1 : Nicolas Ha
----
2020-04-08 01:18:37 UTC - Penghui Li: Yes, 2.5.1-candidate-1 is out.
----
2020-04-08 01:19:31 UTC - Ken Huang: Hi, does pulsar support s3 compatible
storage for Tiered Storage?
----
2020-04-08 01:19:51 UTC - Ali Ahmed: yes
----
2020-04-08 01:20:08 UTC - Ali Ahmed: s3 is the default tiered storge layer
+1 : Ken Huang
----
2020-04-08 02:29:42 UTC - Damien: @Damien has joined the channel
----
2020-04-08 05:32:38 UTC - Poul Henriksen: Will message retention settings have
any effect on non-persistent topics, or will they just be ignored?
----
2020-04-08 05:40:39 UTC - Hiroyuki Yamada: Hi guys, just a question.
When deploying a Pulsar cluster in Mulit-AZ in one region, should I use
geo-replication or should I just create a normal cluster without
geo-replication ?
I’m wondering which is recommended.
----
2020-04-08 06:11:47 UTC - Ebere Abanonu: I am using the streamnative chart with
my pull request. Is there a way to start worker node?
----
2020-04-08 06:16:40 UTC - Greg Methvin: yeah, ok, it may not be worth it. I
might just precreate the topics in the test for now.
----
2020-04-08 07:23:36 UTC - Ken Huang: Hi, I try to offload the topic to s3
storage. Here is my command.
I got "Offload was a success", but my s3 storage still empty
```root@fun-octopus-pulsar-broker-755b845bc8-2hx26:/pulsar# ./bin/pulsar-admin
topics offload --size-threshold 1K public/default/eat
Offload triggered for <persistent://public/default/eat> for messages before
6:0:-1
root@fun-octopus-pulsar-broker-755b845bc8-2hx26:/pulsar# ./bin/pulsar-admin
topics offload-status public/default/eat
Offload was a success
root@fun-octopus-pulsar-broker-755b845bc8-2hx26:/pulsar# ./bin/pulsar-admin
topics stats-internal public/default/eat
{
"entriesAddedCounter" : 3500,
"numberOfEntries" : 3500,
"totalSize" : 363580,
"currentLedgerEntries" : 3500,
"currentLedgerSize" : 363580,
"lastLedgerCreatedTimestamp" : "2020-04-08T06:56:46.466Z",
"waitingCursorsCount" : 0,
"pendingAddEntriesCount" : 0,
"lastConfirmedEntry" : "6:3499",
"state" : "LedgerOpened",
"ledgers" : [ {
"ledgerId" : 6,
"entries" : 0,
"size" : 0,
"offloaded" : false
} ],
"cursors" : {
"ken" : {
"markDeletePosition" : "6:2999",
"readPosition" : "6:3000",
"waitingReadOp" : false,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 3000,
"cursorLedger" : 7,
"cursorLedgerLastEntry" : 11,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2020-04-08T06:56:46.469Z",
"state" : "Open",
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
"properties" : { }
}
}```
----
2020-04-08 07:47:54 UTC - Ebere Abanonu:
```<http://java.io|java.io>.IOException: Empty Ledger Root Path```
----
2020-04-08 07:48:28 UTC - Ebere Abanonu: That is likely the root cause
----
