2018-12-13 10:05:20 UTC - B Senthil Kumar: @B Senthil Kumar has joined the channel ---- 2018-12-13 14:58:38 UTC - Grégory Guichard: Hello ! I have a cluster of 6 bookies, when I try to set the ensemble size of my namespace at 6, I get the message "Bookkeeper-Ensemble must be <= 5" with the pulsar-admin cli. Why ? ---- 2018-12-13 15:05:46 UTC - Sijie Guo: there is a setting `managedLedgerMaxEnsembleSize` in broker conf controlling the max bookkeeper ensemble size. by default it is 5, you can change it.
(it is probably missing from conf/broker.conf. you can manually add it `managedLedgerMaxEnsembleSize=<size>`) ---- 2018-12-13 15:14:09 UTC - Grégory Guichard: It works, thanks ! ---- 2018-12-13 19:16:43 UTC - Grant Wu: How can I get the last message published in a topic? The documentation seems to not be very clear on this. I immediately thought to reach for the Reader interface, however: <https://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ReaderBuilder.html#startMessageId-org.apache.pulsar.client.api.MessageId-> this states: > MessageId.latest : Start reading from the end topic, only getting messages published after the reader was created which is not quite what I want. Now <http://pulsar.apache.org/docs/en/concepts-clients/#reader-interface> has an image which states that readers read from the latest unacked message, which doesn’t seem to match up with the Javadoc. And neither appears to fit my ideal, which is something that always gets the latest message, whether or not it’s acked. ---- 2018-12-13 19:19:24 UTC - Matteo Merli: @Grant Wu the website doc is a bit imprecise in this case. The Javadoc reflects the real behavior. ---- 2018-12-13 19:19:58 UTC - Matteo Merli: There’s currently no straightforward way to just receive the last message published in the topic :confused: ---- 2018-12-13 19:20:05 UTC - Grant Wu: @Jonathan Budd sob : Jonathan Budd ---- 2018-12-13 19:20:11 UTC - Grant Wu: Okay, good to know ---- 2018-12-13 19:20:30 UTC - Matteo Merli: not that it’s impossible, or hard, to implement ---- 2018-12-13 19:20:52 UTC - Jonathan Budd: Right, I would imagine we would just need to store the message ID of the latest message somewhere ---- 2018-12-13 19:21:26 UTC - Grant Wu: There might be some place in the Pulsar Admin interface to get it too ---- 2018-12-13 19:21:51 UTC - Matteo Merli: For sure in the stats-internal ---- 2018-12-13 19:23:17 UTC - Matteo Merli: So, ideally in your case, you’d like to have a way to position the Reader on last published message and get that and then all the other messages after that. Correct? ---- 2018-12-13 19:23:37 UTC - Grant Wu: Uh, not sure if we need the second part ---- 2018-12-13 19:23:39 UTC - Matteo Merli: Something like `MessageId.latestIncluded` ? ---- 2018-12-13 19:23:45 UTC - Grant Wu: But something like that ---- 2018-12-13 19:24:00 UTC - Matteo Merli: well.. you can always just read one and close ---- 2018-12-13 19:24:07 UTC - Grant Wu: Sure ---- 2018-12-13 19:30:58 UTC - Tobias Gustafsson: Sounds more or less exactly like the use case I have. I wrote an issue for it a couple of days ago: <https://github.com/apache/pulsar/issues/3162> ---- 2018-12-13 19:37:29 UTC - Tobias Gustafsson: I have a question about the `Reader.HasNext()` method in the Go client. I'm not really sure how to interpret the documentation. If this method returns false, does it mean that there is no more published message on the topic or that there simply is no more message in the local buffer ready for immediate delivery? The javadoc for the corresponding method is the exact same btw. What I'm after is the former but it seems to me like its the latter since it sometimes returns false even though I know that there are messages published. ---- 2018-12-13 19:45:04 UTC - Matteo Merli: @Tobias Gustafsson `Reader.HasNext()` can be used to read all the messages until the reader is caught up with the producers. ---- 2018-12-13 19:45:20 UTC - Matteo Merli: Of course, producers can still publish messages after that ---- 2018-12-13 19:45:48 UTC - Matteo Merli: but when `HasNext() == False`, at least for a moment the reader will be have read everything ---- 2018-12-13 19:50:29 UTC - Tobias Gustafsson: OK, then I think I've found a bug in the Go client. I've created a test case where I repeatedly create a Reader instance starting from `pulsar.EarliestMessage`, and consume all messages on it until `HasNext()` returns false. After that the reader is closed. Every now and then, rougly 1 -5 times / 100 reads, `HasNext()` returns false on the first call. ---- 2018-12-13 19:51:34 UTC - Tobias Gustafsson: I know that there are messages on the topic since I'm reading from the exact same topic every time. There have not been any new messages published to the topic in the meantime, it's basically static. ---- 2018-12-13 19:52:48 UTC - Matteo Merli: Yes, looks like a bug ---- 2018-12-13 19:53:04 UTC - Matteo Merli: please open an issue ---- 2018-12-13 19:53:04 UTC - Tobias Gustafsson: OK, I'll file an issue, thanks ---- 2018-12-13 19:53:09 UTC - Matteo Merli: :+1: ---- 2018-12-13 21:06:24 UTC - Sundaram Surampudi: @Sundaram Surampudi has joined the channel ---- 2018-12-13 23:00:53 UTC - Craig Yoshioka: @Craig Yoshioka has joined the channel ---- 2018-12-14 00:17:45 UTC - Yang Cui: @Yang Cui has joined the channel ---- 2018-12-14 00:33:41 UTC - Craig Yoshioka: I have a newb question... I'm confused by the "path" syntax for creating topics in the python client. All the documentation I've found refers to topic urls being of the form: `<persistent://tenant/namespace/topic>`, but if I try to use that from python: `client.create_producer('<persistent://team/app1/events>')` I get a "Error Checking/Getting Partition Metadata while creating producer". ---- 2018-12-14 00:34:25 UTC - Sijie Guo: did you create tenant `team` and namespace `app1`? ---- 2018-12-14 00:34:37 UTC - Craig Yoshioka: yup ---- 2018-12-14 00:35:50 UTC - Craig Yoshioka: `pulsar-admin namespaces list team` returns: team/app1 ---- 2018-12-14 00:36:37 UTC - Sijie Guo: what is the version of python client are you using? ---- 2018-12-14 00:36:59 UTC - Craig Yoshioka: 2.2.0 ---- 2018-12-14 00:39:05 UTC - Sijie Guo: what is the error message after “while creating producer”? ---- 2018-12-14 00:40:43 UTC - Craig Yoshioka: 2018-12-13 16:40:15.663 INFO ConnectionPool:63 | Created connection for <pulsar://pncc-storage-1:6650> 2018-12-13 16:40:15.666 INFO ClientConnection:285 | [10.125.80.69:58762 -> 10.125.80.69:6650] Connected to broker 2018-12-13 16:40:15.674 ERROR ClientConnection:709 | [10.125.80.69:58762 -> 10.125.80.69:6650] Failed partition-metadata lookup req_id: 1 error: 1 2018-12-13 16:40:15.674 ERROR ClientImpl:178 | Error Checking/Getting Partition Metadata while creating producer on <persistent://team/app1/raw> -- 5 Traceback (most recent call last): File "./push-file-stream.py", line 64, in <module> main() File "./push-file-stream.py", line 55, in main writer = client.create_producer('<persistent://team/app1/raw>') File "/storage/sw/staging/conda-env/lib/python3.7/site-packages/pulsar/__init__.py", line 408, in create_producer p._producer = self._client.create_producer(topic, conf) Exception: Pulsar error: ConnectError ---- 2018-12-14 00:42:27 UTC - Craig Yoshioka: hmm, the python docs have an example using: ---- 2018-12-14 00:42:28 UTC - Craig Yoshioka: '<persistent://sample/standalone/ns/my-topic>' ---- 2018-12-14 00:42:34 UTC - Sijie Guo: `ConnectError` seems to indicate it can not connect. ---- 2018-12-14 00:42:46 UTC - Sijie Guo: that is the v1 name pattern ---- 2018-12-14 00:43:29 UTC - Craig Yoshioka: ok, that was one source of confusion ---- 2018-12-14 00:43:34 UTC - Sijie Guo: (fyi, topic name changes between v1 and v2 : <http://pulsar.apache.org/docs/en/pulsar-2.0/#topic-names>) ---- 2018-12-14 00:43:37 UTC - Craig Yoshioka: so don't use that ---- 2018-12-14 00:43:52 UTC - Sijie Guo: it also works ---- 2018-12-14 00:43:59 UTC - Sijie Guo: but it is just v1 naming pattern ---- 2018-12-14 00:46:04 UTC - Craig Yoshioka: hmmm, so I changed the python code to: <persistent://team/standalone/app1/raw> and now it seems to work... ---- 2018-12-14 00:46:29 UTC - Sijie Guo: are you using standalone? ---- 2018-12-14 00:47:09 UTC - Sijie Guo: how did you create the tenant? ---- 2018-12-14 00:47:54 UTC - Craig Yoshioka: i am using standalone ---- 2018-12-14 00:48:04 UTC - Craig Yoshioka: I created the tenant via pulsar-admin ---- 2018-12-14 00:48:12 UTC - Sijie Guo: what is your full command? ---- 2018-12-14 00:48:22 UTC - Sijie Guo: did you specify the cluster when creating tenant? ---- 2018-12-14 00:48:49 UTC - Craig Yoshioka: hmm, I didn't but when I checked the tenant config it listed standalone ---- 2018-12-14 00:48:53 UTC - Craig Yoshioka: pulsar-admin tenants create team ---- 2018-12-14 00:49:09 UTC - Craig Yoshioka: { "adminRoles" : [ ], "allowedClusters" : [ "standalone" ] } ---- 2018-12-14 00:49:21 UTC - Sijie Guo: okay ---- 2018-12-14 00:50:04 UTC - Craig Yoshioka: I'm going to try and build the python client from source... suspicious that the v1 pattern is working but the 2 is not ---- 2018-12-14 00:51:09 UTC - Sijie Guo: v2 should be working. let me double check ---- 2018-12-14 00:57:33 UTC - Nicolas Ha: on memory limits: <https://github.com/apache/pulsar/blob/master/deployment/kubernetes/helm/pulsar/values.yaml#L194-L204> this specifies 15G of memory and 3 replicas. Is it 3 * 15G or memory that I need, or will each replica get 5G? ---- 2018-12-14 00:58:33 UTC - Craig Yoshioka: thanks for the help. I gotta go commute, but I'll check back in later ---- 2018-12-14 00:58:44 UTC - Sijie Guo: the memory setting is per pod. so it is the former one slightly_smiling_face : Nicolas Ha ---- 2018-12-14 00:59:56 UTC - Nicolas Ha: thanks again :slightly_smiling_face: ---- 2018-12-14 01:03:48 UTC - Nicolas Ha: Also, is it expected to only have 1 bookie? <https://github.com/apache/pulsar/blob/master/deployment/kubernetes/generic/bookie.yaml> I do not see `replicaCount` in this file ---- 2018-12-14 01:08:04 UTC - Sijie Guo: @Craig Yoshioka: so I think you forgot to specify cluster when creating namespace. you should do `bin/pulsar-admin namespaces create --clusters standalone team/app`. ---- 2018-12-14 01:09:42 UTC - Sijie Guo: the generic one is using daemonset deployment. so it deploys to one pod per node. if you are deploying to a cloud environment, I would suggest you using helm chart or <https://github.com/apache/pulsar/tree/master/deployment/kubernetes/google-kubernetes-engine>, as it will use statefulset for deploying bookkeeper. ---- 2018-12-14 01:12:32 UTC - Nicolas Ha: ah I see :slightly_smiling_face: I am deploying using the `generic` because I am on “bare metal” kubernetes ---- 2018-12-14 01:12:49 UTC - Sijie Guo: oh gotcha ---- 2018-12-14 02:00:59 UTC - Craig Yoshioka: thanks, that worked, guess I assumed it would inherit from the parent tenant. thanks again! ----
