2020-03-04 09:57:05 UTC - Vincent LE MAITRE: Hi, I have added this parameter on my bookkeeper instances. But I am still not able to use function state from my broker instances. I am using a Pulsar 2.5.0 deployed on a Kubernetes cluster using the provided helm chart. Please could you tell more about your setup ? ---- 2020-03-04 10:02:07 UTC - Vincent LE MAITRE: Did you configure "stateStorageServiceUrl" into the functions_worker.yml conf file ? If yes, which value ? Thanks ---- 2020-03-04 10:18:48 UTC - Viji: Hi, are there any restrictions on topic names in pulsar like the maximum length, allows characters etc ? ---- 2020-03-04 10:22:40 UTC - Viji: When we create a partitioned topic with name which is a substring of an already existing partitioned topic why do we see following error, is it a defect in pulsar ? ---- 2020-03-04 10:23:06 UTC - Viji: [3:50 PM] Sakthi Prasad K ./pulsar-admin topics create-partitioned-topic <persistent://public/default/create_substring_topic> -p 1
[3:50 PM] Sakthi Prasad K ./pulsar-admin topics create-partitioned-topic <persistent://public/default/substring_topic> -p 1 10:14:38.692 [AsyncHttpClient-5-1] WARN org.apache.pulsar.client.admin.internal.BaseResource - [<https://itomdipulsar-proxy:8443/admin/v2/persistent/public/default/substring_topic/partitions>] Failed to perform http put request: javax.ws.rs.ClientErrorException: HTTP 412 Precondition Failed Already have topic <persistent://public/default/create_substring_topic-partition-0> which contains partition suffix '-partition-' and end with numeric value, Creation of partitioned topic substring_topic could cause conflict. Reason: Already have topic <persistent://public/default/create_substring_topic-partition-0> which contains partition suffix '-partition-' and end with numeric value, Creation of partitioned topic substring_topic could cause conflict. ---- 2020-03-04 11:12:51 UTC - Viji: We raised an issue on git ( <https://github.com/apache/pulsar/issues/6468>) ---- 2020-03-04 12:04:29 UTC - eilonk: Did anyone try to use tls authentication and recieved this error? `sun.misc.Launcher$AppClassLoader@18769467] error Uncaught exception in thread main: Failed to load an authentication provider.` I set authenticationProviders as: `[conf/proxy.conf] Applying config authenticationProviders = org.apache.pulsar.broker.authentication.AuthenticationProviderTls,org.apache.pulsar.broker.authentication.AuthenticationProviderToken` (this is just an example from proxy, this is the same value I used everywhere) did anyone encounter this? ---- 2020-03-04 13:09:25 UTC - Yosi Attias: @Yosi Attias has joined the channel ---- 2020-03-04 13:32:06 UTC - Ryan Slominski: I'm using standalone mode of Pulsar (entire cluster on a single machine). I am not familiar with getting state working in containers, but I vaguely remember reading somewhere you need to set that URL in a YAML file. I think the parameter for standalone mode is something like: ```--state_storage_serviceurl <bk://localhost:4181>``` ---- 2020-03-04 13:45:15 UTC - Ryan Slominski: You might also need something like: `functionsWorkerEnabled=true` in your broker.conf or whatever is the equivalent in the containers. ---- 2020-03-04 13:49:57 UTC - Vincent LE MAITRE: OK. Thanks. Why are you using the port 4181 for bookkeeper ? The default port should be 3181 isn't it ? ---- 2020-03-04 13:51:14 UTC - Vincent LE MAITRE: functionsWorker is enabled. My functions without state are working fine. But functions with state dores not work ---- 2020-03-04 15:35:24 UTC - Luis Muniz: @Luis Muniz has joined the channel ---- 2020-03-04 16:08:57 UTC - Chris: @Chris has joined the channel ---- 2020-03-04 16:27:07 UTC - Vincent LE MAITRE: Hi, bookkeeper is configured by default on port 3181. In order to configure function state, I found this example into the conf : stateStorageServiceUrl: <bk://localhost:4181>. What is this 4181 port ? Another port of bookkeeper ? Thanks ---- 2020-03-04 16:27:43 UTC - Pavel Tishkevich: @Sijie Guo @Devin G. Bost Could you provide more details about tuning zookeeper to handle increased load? ---- 2020-03-04 16:28:29 UTC - Devin G. Bost: It’s all in here: <https://zookeeper.apache.org/doc/r3.4.13/zookeeperAdmin.pdf> ---- 2020-03-04 16:45:31 UTC - Ryan Slominski: I might have port wrong for bookkeeper. I can't find it now, but some sources on the Internet reference that alternate port number. The standalone.conf that ships with version 2.5.0 of Pulsar doesn't specify and the bookkeeper.conf says 3181 like you say. ---- 2020-03-04 16:50:47 UTC - Chris: How would I start diagnosing slow consumption from large backlogs? When trying to replay old data my consumers run at 1mb/s/partition instead of the 200 or so I see when consuming with no/very small backlog. It's equally slow when attaching `pulsar-perf` to that subscription name. I've checked my configs for rate limits on backlogged consumers, and they're all set to 0 which I assume is unlimited, so I'm not quite sure what's going on. ---- 2020-03-04 17:21:52 UTC - Sijie Guo: Ah I see. that seems to be a regression in 2.5.0. /cc @Penghui Li ---- 2020-03-04 17:25:54 UTC - Sijie Guo: If you have a huge backlog, try to check if it is because `dbStorage_rocksDB_blockCacheSize` is too small. ---- 2020-03-04 17:30:55 UTC - Mathieu Druart: @Vincent LE MAITRE we tried the same thing and ran into the same issue. Never succeeded to activate State into Pulsar Functions with the Helm deployment. ---- 2020-03-04 18:32:32 UTC - Chris: Is there a way to check the size of the index db? It looks like I missed a zero or two on that config though as it was set really small. ---- 2020-03-04 19:14:51 UTC - Chris: Changing that value didn't seem to help at all. Any other ideas? ---- 2020-03-04 19:21:08 UTC - Kevin Xu: @Kevin Xu has joined the channel ---- 2020-03-04 20:37:14 UTC - Sergii Zhevzhyk: Hi, I cannot find the release 2.5.1 among the milestones. When do you plan to release it? ---- 2020-03-04 20:43:34 UTC - Nick Poore: @Nick Poore has joined the channel ---- 2020-03-04 21:48:22 UTC - Sijie Guo: @Sergii Zhevzhyk we are not using milestone for minor releases any more. we are using label `release/2.5.1` instead. because a change can go with a major release (like 2.6.0) and a minor release like (2.5.1). Github doesn’t allow us tagging multiple milestones. We are in the process of cleaning up last batch of 2.5.1 fixes. expect to start releasing 2.5.1 early next week. You can subscribe to dev@ mailing list for these updates. ---- 2020-03-04 22:14:05 UTC - Sergii Zhevzhyk: @Sijie Guo thank you for the info! ---- 2020-03-04 23:32:09 UTC - Eric Simon: If I wasnt to override the ```instanceLivenessCheckFreqMs``` For a pulsar function. Is there a way to do this without updating the function_worker.yml? Can I pass it in as runtime flag? ---- 2020-03-05 01:42:26 UTC - Penghui Li: Ok, I will take a look. ---- 2020-03-05 02:19:29 UTC - Eugen: Thanks merlimat, `blockIfQueueFull=true` did the trick ---- 2020-03-05 03:31:52 UTC - Ken Huang: Hi, how do I understand processing guarantees of pulsar function? "At-most-once" means regardless of whether the consumer received the message "At-least-once" means promises consumer received the message "Effectively-once" means promises consumer received the "correct" message The above is my understanding, is it correct? ---- 2020-03-05 03:43:56 UTC - Prasad Reddy: I Managed to set up pulsar 2.5.0-x on kubernetes and doing some benchmark tests for producer and while doing tests, noticed that "Journal Queue Length(The total number of requests pending in the journal queue)" size is increasing gradually which I'm suspecting abnormal. Can anyone explain that why the Journal Queue Size is increasing and what might be the cause for this behavior? ---- 2020-03-05 05:02:11 UTC - Sijie Guo: Currently the setting is inherited from worker. Updating the function_worker.yml is the only way. Can you create a github issue for requesting a feature? We can improve the behavior. ---- 2020-03-05 05:03:06 UTC - Sijie Guo: I think these two metrics are gauge in bookie side. but they are probably not handled properly. ---- 2020-03-05 05:04:30 UTC - Sijie Guo: processing guarantees for functions reflect whether a function writes the result to the output topic. ---- 2020-03-05 05:05:06 UTC - Sijie Guo: ‘at-most-once’ means the function process the message “at-most-once” but there is no guarantee the result is written to the output topic. ---- 2020-03-05 05:05:57 UTC - Sijie Guo: ‘at-least-once’ guanrantee the the result is written to the output topic. but the function might be processed multiple times and multiple results might be written to the output topic. ---- 2020-03-05 05:06:27 UTC - Sijie Guo: ‘Effectively-once’ means the result is written exactly once to the output topic. but the function might still run multiple times. ---- 2020-03-05 05:17:45 UTC - Prasad Reddy: @Sijie Guo Thank you for the info. Ok. What might be the reason for not handling properly? How do resolve this issue? Is there any specific parameters at bookie side to play around to handle more data? ---- 2020-03-05 05:21:34 UTC - Ken Huang: thank you very much for reply ---- 2020-03-05 05:22:33 UTC - Sijie Guo: I think we need to fix the bookie code. Can you create a giithub issue? ---- 2020-03-05 05:23:58 UTC - Prasad Reddy: @Sijie Guo Sure.. thank you ! ---- 2020-03-05 06:01:47 UTC - Prasad Reddy: @Sijie Guo We raised an issue on git (<https://github.com/apache/pulsar/issues/6479>) *Describe the bug* Bookie Journal Queue Length is increasing gradually during performance executions *To Reproduce* Steps to reproduce the behavior: 1. Install Pulsar 2.5.0-08 on kubernetes cluster environment ( 3 Bookies, 3 brokers,3 proxy). 2. Create a topic with three partitions. 3. Run longevity test (48 hours) by passing 1 KB message size as input to send data to created topic in step 2 4. After completion of the few hours execution, Check size of "Journal Queue Length" bookie metric. *Expected behavior* Should not increase queue length of journal and bookies should be able to handle properly. *Screenshots* Attached screenshot of "Journal Queue Size Length" metric. *Desktop (please complete the following information):* • OS: CentOS 7.7 • Master Node: 1 • Worker Nodes : 3 *H/w Details*: • Master node: 12x16x200 (Not running any Pulsar pods on master node) • Worker nodes : 16x32x200 ( each worker node has additional 4 virtual disks for Pulsar pods) ---- 2020-03-05 06:39:57 UTC - Eric Simon: Sure thing. I will create that ticket in the morning. ---- 2020-03-05 08:29:44 UTC - Vincent LE MAITRE: Hi @Mathieu Druart, did you open an issue on Pulsar related to this ? Thanks for your answer. Do you find another solution to implement kind of state into your services / functions ? ----
