2019-10-07 09:36:12 UTC - Julien Lechalupé: Hi, thanks for the replies. On my use case, georeplication is not needed. Pulsar will be clustered and topics can be handled by a single node. Also, as pulsar is not deployed yet, there is no need for migration here. What is the best way to configure Pulsar in that case ? ---- 2019-10-07 10:00:51 UTC - Retardust: is there any reason for public tenant not to be accessible for ANY cluster? ---- 2019-10-07 10:17:03 UTC - Retardust: I solve this by change pulsarFunctionsNamespace - should it be added in workarounds list? ---- 2019-10-07 11:22:56 UTC - Shishir Pandey: Does this imply that deliverAt or deliverAfterSeconds is not available in C++ client? ---- 2019-10-07 11:47:32 UTC - Raman Gupta: @Julien Lechalupé Not sure what you mean... configure it as a normal cluster, size it per your requirements for ensemble size, write and ack quorum size. I recommend this article for an explanation: <https://jack-vanlightly.com/blog/2018/10/2/understanding-how-apache-pulsar-works>. +1 : Julien Lechalupé ---- 2019-10-07 12:31:34 UTC - Aki Väliaho: @Aki Väliaho has joined the channel ---- 2019-10-07 13:02:49 UTC - Joose Helle: Hi all, some help needed here... We have our Pulsar cluster handling some IoT data running in Azure. I'm still somewhat new to Pulsar but have familiarized with the main functionality and most of it makes a lot sense to me. However, after some days ago our bookies run out of disk space `/dev/sdc1 - /srv/bkledger - 100% of 260G` it has been difficult to trace down the root cause to this. Surprisingly, when we checked the stats of all the namespaces/topics (that is less than 10) we couldn't find any large backlogs (which would have been the most obvious explanation). One of the subscriptions had a small backlog (3000 small messages supposedly worth less than 0.2 G total) but a large storageSize of 55 G (`pulsar-admin persistent stats...`) which is hard to understand. Can anything else than message backlogs (e.g. some log files) take a lot of space in bookies' disks? If storageSize of a topic is large, does it necessarily mean that it's the messages in the backlog that are taking the space or could it possibly be something else? Can you share any ideas on how to debug what topics / messages are taking all the space in bookies (ledgers)? ---- 2019-10-07 13:14:56 UTC - Chris Bartholomew: @Joose Helle What is your message retention setting? ---- 2019-10-07 13:31:32 UTC - Joose Helle: @Chris Bartholomew "retentionTimeInMinutes" : 4320, "retentionSizeInMB" : 2048 ---- 2019-10-07 13:35:09 UTC - Raman Gupta: Can I specify multiple `advertisedAddress` values? Running inside k8s and want to advertise two addresses: one for clients internal to the cluster and one for clients external. I recognize I could use the proxy instead but I'm trying to keep this prototype installation as lightweight as possible. ---- 2019-10-07 13:51:49 UTC - Chris Bartholomew: @Joose Helle This means that you are storing up to 2048 MB of messages per topic in Bookkeeper even if the messages in the topic are all acknowledged. It doesn't quite add up if you have less than 10 topics. In any case, you can get an idea of how big the retention storage is using the internal stats of the topic: ```pulsar-admin topics stats-internal persistent://<tenant>/<namespace>/<topic>``` ---- 2019-10-07 14:05:51 UTC - Joose Helle: @Chris Bartholomew Thanks for helping! Indeed, our retention policies could be adjusted. 72 h (4320 min) message retention could cause problems in terms of the disk space, but it seems that the 2048 mb limit should prevent that (?). ---- 2019-10-07 14:36:30 UTC - Chris Bartholomew: @Joose Helle It could be a case where some ledgers can't be deleted because they are holding messages that are being retained for some topics that haven't hit their limit yet. Because of this fragmentation, the total storage required could be greater than the sum of the individual maximums. ---- 2019-10-07 16:02:55 UTC - Matteo Merli: I’ve opened a PR 3 days ago: <https://github.com/apache/pulsar/pull/5317> ---- 2019-10-07 16:05:00 UTC - Shishir Pandey: Thank you ---- 2019-10-07 22:57:41 UTC - BigSam: Is it possible to have 100k consumers on same topic? By default, one topic will be handled by one broker, so how many connections a broker may can support? Suppose an AWS xlarge EC2. Appreciated if somebody here have tested it. My initial try failed to subscribe when over about 600 connections, not sure if its configuration issue. ---- 2019-10-07 22:59:27 UTC - Matteo Merli: I’ve tested with several 10s of thousands of TCP connections. Typically, the hardest part to test this scenario is to setup the clients (since each client TCP connections requires 1 port assigned) ---- 2019-10-07 22:59:56 UTC - Matteo Merli: though, keep in mind that pulsar client will pool the TCP connections by default ---- 2019-10-07 23:00:18 UTC - Matteo Merli: So, unless you have 100K VMs, you won’t be needing 100K TCP connections ---- 2019-10-07 23:02:23 UTC - BigSam: Is there any limit for single connection to have few thousands of subscriptions ? ---- 2019-10-07 23:02:36 UTC - Matteo Merli: No hard limit ---- 2019-10-07 23:02:46 UTC - Addison Higham: I imagine it would start to get much harder if a significant set of those *aren't* doing tailing reads? ---- 2019-10-07 23:06:24 UTC - Chris Bartholomew: Note that if you want to set up 100K TCP connections on a single Linux box (for testing purposes), you are going to need multiple IP addresses. The TCP port range is only 65535, and by default Linux is going to reserve a chunk of them. ---- 2019-10-07 23:09:22 UTC - BigSam: I’m simulating 3000 clients that subscribe to same topic from single machine, so 100k will use multiple VMs. But it failed to connect when over 600. Wonder if it’s server configuration issue or client side. ---- 2019-10-07 23:16:57 UTC - BigSam: By the way, the design of Pulsar is really great and exciting. Thanks guys for making it! we are evaluating if it’s good for handling million+ of clients. ---- 2019-10-07 23:28:28 UTC - Nicolas Ha: If I have a topic that is: `<persistent://mytenant/standalone/subscriptions/myname>` (as shown by `getTopic` on my consumer ) Following the docs there I was expecting to see some data hitting `<http://localhost:8080/admin/v2/persistent/public/standalone>` but I get an empty vector `[]` instead? Am I misunderstanding the docs (<https://pulsar.apache.org/docs/en/admin-api-persistent-topics/#list-of-topics>)? ---- 2019-10-07 23:29:13 UTC - Chris Bartholomew: @BigSam what errors are you seeing? I just did a quick test and had no problem connecting 1K clients with 1K subscriptions to a single topic using the pulsar-perf tool. ---- 2019-10-07 23:57:54 UTC - BigSam: @Chris Bartholomew Thanks for the info. The error says”Pulsar error: ConnectError”. And there are other errors “Error Checking/Getting Partition Metadata while Subscribing on <persistent://public/>....”. ---- 2019-10-08 02:33:22 UTC - Joe Francis: This .. large storage size & small backlog.. generally means that you have very old messages that are not acked. Basically any message after the oldest unacked message is kept. The delete cursor will not move past the oldest unacked message. ---- 2019-10-08 06:08:06 UTC - Joose Helle: @Joe Francis Thanks! This is very interesting, seems that there are still some basics for us to learn @Aki Väliaho ---- 2019-10-08 06:09:32 UTC - Aki Väliaho: Thank you! ---- 2019-10-08 06:13:52 UTC - Joose Helle: @Chris Bartholomew Thank you! This mechanism doesn't sound very intuitive with respect to the parameters of the retention policy, but could explain our problem. @Aki Väliaho ----
