2020-08-05 11:04:53 UTC - Sean McClellan: @Sean McClellan has joined the channel ---- 2020-08-05 11:34:53 UTC - Yifan: thank you. ---- 2020-08-05 12:20:03 UTC - Manoj: Hi am getting an exception while connecting the consumer to the pulsar java.lang.NoClassDefFoundError: org/apache/pulsar/shaded/com/google/protobuf/v241/MessageLiteOrBuilder Note that i have used pulsar client and pulsar client api jars of 2.6.0 version ---- 2020-08-05 13:55:12 UTC - Juha Ristolainen: @Juha Ristolainen has joined the channel ---- 2020-08-05 15:08:28 UTC - Bre Gielissen: Hi everyone. I am trying to delete a namespace via the REST API and am getting the error "Cannot delete non empty namespace". I was trying to add the authoritative parameter to my url like `<http://example.com/admin/v2/{tenant}/{namespace}?authoritative=true>` but that is throwing the same error. Has anyone else tried to use this query parameter? Or is there a better way to empty the namespace and then delete? ---- 2020-08-05 16:09:19 UTC - Aaron: Hi any examples of producing JSON to a topic using AUTO_PRODUCE? ---- 2020-08-05 17:53:56 UTC - Addison Higham: @Rattanjot Singh What image are you using? You should be able to check a deployment with `kubectl get statefulset pulsar-mini-broker -o yaml`. Is it possible you may have overridden the image in the values? ---- 2020-08-05 17:55:52 UTC - Addison Higham: I am not sure I understand, but if you are talking about consuming cross region, it shouldn't be a problem other then you may need to tune some of the client timeout defaults (depending on latency, but may not really be an issue..) and also just the risks of a network partition, but as long as that is acceptable, then Pulsar should behave fine over higher latency connections ---- 2020-08-05 17:56:39 UTC - Daniel Ciocirlan: Thanks @Addison Higham ! ---- 2020-08-05 17:58:39 UTC - Caleb Epstein: This is a bit of an existential question, but is there any way for a Consumer to know that it is "up to date" on a Subscription and that it is not (still) getting replayed data? I'm wondering largely for the context of compacted topics. If I have a Consumer that is listening to a topic that has compaction enabled, I'd like to know if/how I can know that I've got a complete snapshot of everything and my state is fully consistent. ---- 2020-08-05 19:24:52 UTC - Rattanjot Singh: @Addison Higham it's cloned from the repo only. no changes. image is apachepulsar/pulsar-all:2.6.0 ---- 2020-08-05 19:26:43 UTC - Addison Higham: you can test that the image is valid just by doing `docker pull apachepulsar/pulsar-all:2.6.0`, if it still cann pull that way it may mean you are having an issue with your cluster. Can you pull other images with your cluster? ---- 2020-08-05 19:34:55 UTC - Addison Higham: pulsar doesn't have any API endpoints to empty a namespace, so you will need to script that yourself ---- 2020-08-05 19:35:49 UTC - Bre Gielissen: Ok. Thanks. ---- 2020-08-05 19:40:50 UTC - Addison Higham: hrm... I am thinking this is a bit of an edge case of sources with the functions API.
Effectively once in functions works by having the ack not happen on the input side of the function if the send fails. For sources, that probably doesn't make as much sense. But speaking more broadly, with the source API, it is going to be difficult to achieve effectively once unless you are doing a lot of custom work in your source. I would say in the case of most sources, you probably want to use at_least_once for sources and if you really need exactly once, then you may want to look into using flink to produce into pulsar instead of a source ---- 2020-08-05 20:06:31 UTC - Matt Mitchell: Hi. I’m currently experiencing the exact same issue as described here (delayed messages are sometimes stuck): <https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1596213427111400>. Is this a known issue? ---- 2020-08-05 20:24:38 UTC - Muljadi: Does pulsar provide ability to encrypt at rest / in storage, if so what’s the encryption keys it uses? ---- 2020-08-05 20:25:27 UTC - Matteo Merli: Take a look at <https://pulsar.apache.org/docs/en/2.6.0/security-encryption> ---- 2020-08-05 20:36:35 UTC - Muljadi: Thanks for the tips, I was looking at that one previously and planning on using that message encryption. However, it did not mentioned about how the messages are stored and whether they’re encrypted in storage / at rest. ---- 2020-08-05 20:36:46 UTC - Addison Higham: :thinking_face: There certainly is some internals that would be aware of being on a compacted topic and your topic stats could be compared against the `messageId` you can get from the message object, but I can't of anything really easy in the API ---- 2020-08-05 20:57:47 UTC - Matteo Merli: messages are encrypted by producers and decrypted by consumers. brokers and bookies are only seeing encrypted data and won't have the keys to decrypt them +1 : Muljadi ---- 2020-08-05 21:30:52 UTC - Caleb Epstein: Does what I'm asking make sense? I am looking at Pulsar as a possible replacement for some home-grown middleware at my work, where data is (mostly, not always) akin to a single Topic with N keys and when a client app starts, we want to know we have achieved a current / correct state of inputs before this client app begins its own processing. ---- 2020-08-05 21:37:22 UTC - Addison Higham: totally makes sense ---- 2020-08-06 05:06:12 UTC - Rattanjot Singh: I am able to pull these images using docker ---- 2020-08-06 07:36:15 UTC - Galen Suen: Hi everyone, how can I pass localrunner arguments (_e.g. --stateStorageServiceUrl_) when calling `pulsal-admin sinks localrun`? ---- 2020-08-06 07:45:30 UTC - charles: Hi all. I'm trying to reach high throughput figures on my local system that compare to what's published on the net. For example: • <https://www.splunk.com/en_us/blog/it/why-apache-pulsar-part-2.html#:~:text=Throughput%20can%20vary%20based%20on,of%201.8M%20messages%2Fsec.|Splunk.com - blog> (August 28, 2017): 120 MB/sec (of 1 KB messages) • <https://www.slideshare.net/merlimat/high-performance-messaging-with-apache-pulsar|Streamlio presentation> (Oct 4, 2018): 40 MB/sec (Slide 37: 40000 Messages/s of 1KB size) So far, I'm not reaching those figures. I'm measuring using the following setup: • Running a Java JUnit test from an IDE with parallel Producer and Reader threads as Executors in the java ExecutorService. • Using a 1000 message window for reporting: after it reaches 1000 messages, the window(both Producer and Reader) reports messages produced/consumed. This continues for 10000's of messages. • 1 persistent topic • 1KB messages • Pulsar on Docker Standalone (<https://pulsar.apache.org/docs/en/standalone-docker/|reference>) • Pulsar on Minikube (<https://pulsar.apache.org/docs/en/kubernetes-helm/|reference>) • Docker Desktop: CPU: 8, Memory 8GB, Swap 1GB, Disk Image: 10GB free My hardware configuration is: • 2,3 GHz Intel Core i9 • 16 GB 2400 MHz DDR4 • SSD The result for both Docker and Minikube setups are similar. On average, I get at cold start result of: • Producer: 300 messages/second: (300 KB/s) • Reader: 300 messages/second: (300 KB/s) As I leave the namespace retention to infinite (Using RetentionPolicies (-1, -1)) and use MessageId._*earliest*_ for the reader, the "cold" startup effect of the test is removed by consecutive runs : the earliest messages in the Broker service cache are retrieved first. After a couple of runs, the top result reached: • Producer: 300 messages/second: (300 KB/s) • Reader: 15000 messages/second: (15MB/s) (note: the average I see is: 13MB/s) Any similar experiences? I'm looking for examples to reach beyond my results. Thoughts? ----
