2020-03-13 09:18:13 UTC - xue: ---- 2020-03-13 09:48:31 UTC - Ebere Abanonu: @Ebere Abanonu has joined the channel ---- 2020-03-13 09:49:48 UTC - Ebere Abanonu: Hi all ---- 2020-03-13 09:50:10 UTC - Ebere Abanonu: Does partitioned topic support schema registration? ---- 2020-03-13 09:51:07 UTC - Ebere Abanonu: I am getting null schema in GetSchemaResponse ---- 2020-03-13 09:59:08 UTC - Pavel Tishkevich: Hello!
Question about `ZooKeeperCache` / `ZooKeeperChildrenCache`. I’ve already asked about performance of Pulsar/Zookeeper: <https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1583249646353000?thread_ts=1582547885.083500&cid=C5Z4T36F7> (mentioned performance issues are reproduced for 3 brokers as well) and after some analysis found out that there is a lot of requests to zookeeper initiated by cache reloading for z-nodes like: /managed-ledgers/<tenant_name>/<cluster_name>/<namespace1>/persistent - children of which are topic names. This makes sense, because as I see zookeeper cache works as follows: 1. Child node added/removed (topic created/deleted in our case) 2. Watcher triggered 3. Cache invalidated 4. Child nodes loaded in cache by performing request to zookeeper with watcher set and this repeats for each topic creation/deletion which in our case around 2000 operations in a minute! Another thing that I’ve noticed is that these topics names are not used that actively by Pulsar broker. So when I restart all zookeeper instances one-by-one (thus zookeeper sessions are reopened and watchers are closed): - there are almost no requests to zookeeper - much less zk cache reloading - performance of Zookeeper and Pulsar improved dramatically Which makes me think that probably we don’t need to cache topic names and refresh this cache so aggressively. It looks like initially topic names are loaded during `NamespaceService#findBrokerServiceUrl`: ```// Schedule the task to pre-load topics pulsar.loadNamespaceTopics(bundle);``` How these cached topic names are used after that? Could you verify if this analysis is reasonable? Maybe it’s possible to improve performance by not caching topic names or making cache reloading lazy (skipping step 4) in the algorithm above? ---- 2020-03-13 14:42:28 UTC - Kyle Arean-Raines: @Kyle Arean-Raines has joined the channel ---- 2020-03-13 14:47:29 UTC - Kyle Arean-Raines: Hello, I was wondering whether there is a planned (even tentative) date for 2.6 release? Thanks ---- 2020-03-13 16:34:31 UTC - Ebere Abanonu: I have a topic with four partitions but pulsar is only able to create two producers ---- 2020-03-13 16:37:42 UTC - Ebere Abanonu: is that normal? ---- 2020-03-13 16:57:29 UTC - David Kjerrumgaard: @Ebere Abanonu You should be able to create as many producers as you like. What is the error you get when creating the 3rd+ producer? ---- 2020-03-13 17:14:48 UTC - Ebere Abanonu: BadVersion ---- 2020-03-13 17:15:46 UTC - Ebere Abanonu: org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion ---- 2020-03-13 17:17:01 UTC - Ebere Abanonu: with 4 partition the topic is as follows: <persistent://public/default/partitioned-topic-0>, <persistent://public/default/partitioned-topic-1>, <persistent://public/default/partitioned-topic-2>, <persistent://public/default/partitioned-topic-3> ---- 2020-03-13 17:23:33 UTC - Ebere Abanonu: am running pulsar in standalone mode ---- 2020-03-13 17:24:08 UTC - David Kjerrumgaard: Can you post the producer generation code? ---- 2020-03-13 17:29:12 UTC - Ebere Abanonu: for (var i = 0; i < configuration.Partitions; i++) { var partitionName = TopicName.Get(topic).GetPartition(i).ToString(); var produceid = Interlocked.Increment(ref IdGenerators.ProducerId); var c = Context.ActorOf(Producer.Prop(clientConfiguration, partitionName, configuration, produceid, network, true, Self)); routees.Add(c.Path.ToString()); } ---- 2020-03-13 17:29:13 UTC - Ebere Abanonu: Porting java partitioned producer ---- 2020-03-13 17:31:24 UTC - Ebere Abanonu: out of four producers 3 fail ---- 2020-03-13 17:32:33 UTC - Ebere Abanonu: Does it have to do with Schema? ---- 2020-03-13 17:38:30 UTC - Ebere Abanonu: If I connect all four to same topic, pulsar only create two producers ---- 2020-03-13 17:48:11 UTC - Igor Zubchenok: @Matteo Merli @Sijie Guo could you review this? ---- 2020-03-13 18:08:22 UTC - Ebere Abanonu: this issue:<https://github.com/apache/pulsar/issues/4807> ---- 2020-03-13 18:08:37 UTC - Ebere Abanonu: was helpful ---- 2020-03-13 18:09:12 UTC - Ebere Abanonu: I retried creating producer when that error occurs and it worked ---- 2020-03-13 18:09:32 UTC - Ebere Abanonu: all producers created at second attempt +1 : David Kjerrumgaard ---- 2020-03-13 18:36:05 UTC - Nishu Goyal: @Nishu Goyal has joined the channel ---- 2020-03-13 19:27:10 UTC - Adam P.: Hey Pulsar friends, do any of you have good troubleshooting techniques for Connection Errors? I’m getting the `ERROR - Pulsar error: ConnectError` when trying to create a producer. I’ve verified I’ve got the right role, token and cert but I’m not really sure how to troubleshoot any further. ---- 2020-03-13 19:36:38 UTC - Matt Mitchell: Re. the Pulsar Java client, is it possible to use POJOs without having to resort to adding Jackson annotations to the POJOs? ---- 2020-03-13 19:37:18 UTC - Matt Mitchell: and I mean immutable POJOs … so all fields are final, and where 0 arg / default constructors don’t exist ---- 2020-03-13 19:37:35 UTC - Roman Popenov: Is it possible for function pods to claim a volume that is a hostPath? If it is, how would one proceed? ---- 2020-03-13 20:47:28 UTC - David Kjerrumgaard: @Adam P. You can start by looking in the broker log, which might have some useful messages. ---- 2020-03-13 20:48:56 UTC - Adam P.: I’ll see if I can make that happen. The cluster management is being done by another team but I’m sure they’ll take a look for me. +1 : David Kjerrumgaard ---- 2020-03-13 20:49:47 UTC - David Kjerrumgaard: @Matt Mitchell Any of the POJOs you use with a Pulsar client have to be serializable / de-serializable using one of the supported schema formats, e.g. Avro, JSON, or Protobuf. <https://pulsar.apache.org/docs/en/concepts-schema-registry/#supported-schema-formats> ---- 2020-03-13 22:14:19 UTC - Pedro Cardoso: The restart will first unsubscribe from previous topics that were matched by the old pattern and then subscribe to all topics of the new pattern right? ---- 2020-03-13 22:14:47 UTC - Pedro Cardoso: or is it smarter and adjust to only subscribing/unsubscribing the changes? ---- 2020-03-13 23:33:53 UTC - Joe Francis: Igor, lookup runs as a distributed service - any broker should be able to answer a lookup request for any topic. If a broker cannot answer a lookup using cached data, it will load the data from ZK to answer the request. ---- 2020-03-14 00:03:37 UTC - Ming: Sink connector is like a function. So it should have a subscription by default to consume messages. I think your scenario is covered. ---- 2020-03-14 01:00:26 UTC - xue: Running producer application error in karaf OSGi container ---- 2020-03-14 01:02:18 UTC - xue: ---- 2020-03-14 08:06:33 UTC - Igor Zubchenok: Hi @Joe Francis, thank you for your reply. After ZK restarts, all brokers still can perform all lookups somehow, but no watcher is set and no dramatic performance degradation until some broker is restarted/reconnected. Is it possible to explain it? ----
