2019-02-26 13:42:20 UTC - Marc Le Labourier: Hi, We are trying to host pulsar on Kubernetes. We used the examples from the deployement directory and corrected minor errors. Every things seems to be running but when checking the monitoring on Grafana, most of the metrics are not working (except Prometheus Stats and Pulsar JVM - CPU). Any thoughts ? Does it requires to setup all the metrics in the prometheus config or something like this (in monitoring.yaml) ? ---- 2019-02-26 14:21:26 UTC - Guy Feldman: @Marc Le Labourier did you use the helm chart or the yaml files ---- 2019-02-26 14:25:04 UTC - Marc Le Labourier: No, we deployed directly to AWS EKS using kubectl and the YAML files. ---- 2019-02-26 14:27:09 UTC - Guy Feldman: I'm also using EKS and deployed using the helm chart. The helm chart is setup in such a way that the metrics populate. You can use helm template to get the kubernetes yaml files to see how it should be setup ---- 2019-02-26 14:28:08 UTC - Marc Le Labourier: I will look into it to see if i can find any information. Thanks. ---- 2019-02-26 14:29:37 UTC - Maarten Tielemans: Is there a quickguide somewhere on using the helm charts with kubernetes/EKS? ---- 2019-02-26 14:33:03 UTC - Guy Feldman: helm doesn't really care that it's eks. there's the general helm quickstart at <https://github.com/helm/helm/blob/master/docs/quickstart.md> ---- 2019-02-26 14:33:46 UTC - Maarten Tielemans: @Guy Feldman Thanks! ---- 2019-02-26 17:27:31 UTC - Sébastien de Melo: Hi, We have found the cause of our issue: it was related to RBAC. Prometheus didn't have the right to list the pods ---- 2019-02-26 17:46:20 UTC - Marc Le Labourier: However, it is still impossible to get information about topics yet. ---- 2019-02-26 18:12:09 UTC - David Kjerrumgaard: ^^^ what is ${DS_default} defined as? ---- 2019-02-26 18:26:24 UTC - Thor Sigurjonsson: Hi, I am having some trouble getting function_worker (in and out of broker) working with broker authentication (and also authorization turned on/off). I tried TLS client auth in 2.2.0/2.2.1 and am now trying it with token auth in 2.3.0. a) Is this known to work for somebody? And b) if so, I'm guessing I'm doing something wrong. I'm running on bare-metal and using docker, and I mount config/data dirs. I can go into details on configs/errors. Help appreciated as I've been banging my head against this for a while. So far, I've got a dev cluster and a prod cluster where I really need both broker auth for clients and functions. ---- 2019-02-26 18:51:59 UTC - Jerry Peng: @Thor Sigurjonsson did you get TLS to work with functions? or you couldn’t get TLS to worker with functions and now you are trying token based authentication? ---- 2019-02-26 18:57:34 UTC - Thor Sigurjonsson: @Jerry Peng I did not get it working with TLS. I am wondering if I omitted `useTls: true` which I think I got clued onto via looking at code -- in the `functions_worker.yml` config, But I'm now down the path of getting token auth working. ---- 2019-02-26 19:01:27 UTC - Jerry Peng: For TLS configurations for functions please take a look at this integration tests for configs to set: <https://github.com/apache/pulsar/blob/master/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarFunctionE2ETest.java> and yes you need to set ```useTls``` to be true ---- 2019-02-26 19:01:42 UTC - Jerry Peng: I can also help you with getting token auth to work ---- 2019-02-26 19:02:24 UTC - Thor Sigurjonsson: Thanks, I'll look at that. ---- 2019-02-26 19:02:31 UTC - Thor Sigurjonsson: Yes, that would be super helpful. ---- 2019-02-26 19:03:22 UTC - Thor Sigurjonsson: I followed the instructions in the docs. ---- 2019-02-26 19:03:57 UTC - Thor Sigurjonsson: I created a token for an admin role and put its name in super users as described. ---- 2019-02-26 19:05:31 UTC - Thor Sigurjonsson: One part of the error log may be relevant (I think that's where I'm at now in troubleshooting my setup). I'm seeing 401's and this: ``` 13:33:03.941 [main] ERROR org.apache.pulsar.functions.worker.WorkerService - Error Starting up in worker org.apache.pulsar.broker.PulsarServerException: Failed to load an authentication provider. at org.apache.pulsar.broker.authentication.AuthenticationService.<init>(AuthenticationService.java:64) ~[org.apache.pulsar-pulsar-broker-common-2.3.0.jar:2.3.0] at org.apache.pulsar.functions.worker.WorkerService.start(WorkerService.java:179) [org.apache.pulsar-pulsar-functions-worker-2.3.0.jar:2.3.0] at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1042) [org.apache.pulsar-pulsar-broker-2.3.0.jar:2.3.0] at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:441) [org.apache.pulsar-pulsar-broker-2.3.0.jar:2.3.0] at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:239) [org.apache.pulsar-pulsar-broker-2.3.0.jar:2.3.0] at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:293) [org.apache.pulsar-pulsar-broker-2.3.0.jar:2.3.0] Caused by: java.io.IOException: No secret key was provided for token authentication``` ---- 2019-02-26 19:06:19 UTC - Thor Sigurjonsson: I'm not sure I can pass the token params in `functions_worker.yml`. ---- 2019-02-26 19:06:31 UTC - Thor Sigurjonsson: Maybe I'm going about things wrong, but that's where I'm at now. ---- 2019-02-26 19:09:13 UTC - Thor Sigurjonsson: The Yaml in `functions_worker.yml` also is a bit different from `.conf` files, so I've had to adapt things there as the examples are focused on `.conf` setups. ---- 2019-02-26 19:09:40 UTC - Thor Sigurjonsson: Between looking at source code and hacking yaml I've made a little progress - up to this point. ---- 2019-02-26 19:10:08 UTC - Jerry Peng: @Thor Sigurjonsson in the functions_worker.yaml file please add: ``` # conf/functions_worker.yml
clientAuthenticationPlugin: org.apache.pulsar.client.impl.auth.AuthenticationToken clientAuthenticationParameters: "token:my-token" ``` ---- 2019-02-26 19:10:48 UTC - Thor Sigurjonsson: Yep, I got that. ---- 2019-02-26 19:11:15 UTC - Thor Sigurjonsson: Maybe I'm getting thrown by `# security settings for worker service` section. ---- 2019-02-26 19:11:49 UTC - Thor Sigurjonsson: I'm not sure if I need that or not. ---- 2019-02-26 19:12:15 UTC - Jerry Peng: @Thor Sigurjonsson you don’t really need to change any settings there. That is for running the worker service separately from the broker ---- 2019-02-26 19:12:37 UTC - Thor Sigurjonsson: Do I need to keep them at their defaults (for that use case)? ---- 2019-02-26 19:12:58 UTC - Thor Sigurjonsson: I'd like to run the workers separately if possible. ---- 2019-02-26 19:13:17 UTC - Thor Sigurjonsson: It would work better for the physical resources I have available to spread the load. ---- 2019-02-26 19:14:26 UTC - Jerry Peng: sure I can help you set that up as well but is that your current setup? ---- 2019-02-26 19:15:20 UTC - Thor Sigurjonsson: Yes, my configs reflect that approach right now, but I turned in-broker on to see if that works. Maybe that breaks because those settings are populated. ---- 2019-02-26 19:15:58 UTC - Thor Sigurjonsson: I can wrap that template for my config in a conditional whether the worker runs in the broker or not. ---- 2019-02-26 19:17:20 UTC - Jerry Peng: yup lets try to get one approach working first ---- 2019-02-26 19:18:06 UTC - Thor Sigurjonsson: I'm with you. ---- 2019-02-26 19:23:08 UTC - Jerry Peng: i would recommend getting running the function worker service as part of the broker first with auth and then we can easily transition to running the worker service separately +1 : Thor Sigurjonsson ---- 2019-02-26 19:23:35 UTC - Thor Sigurjonsson: Let me crank on that a little bit... Will get back to you. +1 : Jerry Peng ---- 2019-02-26 19:28:58 UTC - Jerry Peng: @Sijie Guo @Ivan Kelly @jia zhai can you please take a look at this issue: <https://github.com/apache/pulsar/issues/3686> ---- 2019-02-26 19:35:23 UTC - Thor Sigurjonsson: @Jerry Peng So far so good. Hello world function creation works and producer consumes fine. I put a conditional around the function_worker.yml security settings block and kept them defaults if it's running inside the broker. I'm also seeing `19:29:29.934 [bookkeeper-io-14-3] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xccb01458, L:/10.15.33.62:52742 - R:10.15.33.62/10.15.33.62:3181] authenticated as BookKeeperPrincipal{ANONYMOUS}` I made the anonymous role be supreuser previously. I'll turn that off. At least I know the token auth is turned on there. I'm also seeing `19:29:04.778 [pulsar-io-23-1] INFO org.apache.pulsar.broker.service.ServerCnx - [/127.0.0.1:50252] Client successfully authenticated with token role admin and originalPrincipal null` in the logs which leads me to believe the same thing. This is good. I'll remove the anonymous role from superusers on the broker and see what happens. +1 : Matteo Merli, Jerry Peng ---- 2019-02-26 19:40:31 UTC - Jerry Peng: @Thor Sigurjonsson Awesome good to here! ---- 2019-02-26 19:43:26 UTC - Thor Sigurjonsson: Still working after removing anonymous from superusers. ---- 2019-02-26 19:43:31 UTC - Thor Sigurjonsson: Perfect. Thanks! ---- 2019-02-26 19:44:34 UTC - Thor Sigurjonsson: That logic for what to fill out when in the `functions_worker.yml` broker security was what must have been breaking. ---- 2019-02-26 19:56:30 UTC - Thor Sigurjonsson: Does the same logic apply to any TLS config that would appear in the `functions_worker.yml`? [Keeping it to defaults if running in broker]. ---- 2019-02-26 19:57:39 UTC - Thor Sigurjonsson: I had turned TLS transport off earlier, now I'm turing it back on and I'm probably templating that file wrong for TLS settings as I'm seeing breakage. ---- 2019-02-26 19:58:53 UTC - Jerry Peng: yup when running function worker with broker you just need to modify the following two configs: ``` clientAuthenticationPlugin clientAuthenticationParameters ``` +1 : Thor Sigurjonsson ---- 2019-02-26 19:59:23 UTC - Thor Sigurjonsson: Will tweak my template. ---- 2019-02-26 20:00:34 UTC - Jerry Peng: for TLS auth its a little more involved because you also have to set the truststore. Please refer to the integration test I posted earlier ---- 2019-02-26 20:02:14 UTC - Jerry Peng: take look at this section: <https://github.com/apache/pulsar/blob/master/pulsar-broker/src/test/java/org/apache/pulsar/io/PulsarFunctionE2ETest.java#L251> +1 : Thor Sigurjonsson ---- 2019-02-27 00:44:03 UTC - Sijie Guo: sorry for the delay. replied on github issue. ---- 2019-02-27 04:23:47 UTC - Jacob O'Farrell: What settings did you have to change to get this all working happily? We're experienced the same issues with the AWS deployment on EKS ---- 2019-02-27 04:25:42 UTC - Jacob O'Farrell: Does anyone have any recommendations as to the best/recommended way to monitor/measure latency for topics etc? We've got a workflow that passes messages from one topic to another and so on, performing some basic processing on the way etc, and I'd love to get a feel for the latency end to end if possible ---- 2019-02-27 04:30:30 UTC - bossbaby: Hi @Matteo Merli, @Sijie Guo I have a question. After completing the pulsar performance test, this is the result of disk usage capacity: Start: Disk use: 28% End: Disk use: 66% After garbage collection: Disk use: 47% So my question is that after finishing checking the performance, the topic has been deleted but the bookkeeper still retains a part of the data, is it right? And if so right, how can I clean it? ---- 2019-02-27 06:23:59 UTC - Jacob O'Farrell: Any tips for optimising query performance when using Pulsar SQL? ---- 2019-02-27 06:29:37 UTC - Matteo Merli: You can use `publishTime` as a filter in your query to reduce the amount of data to be scanned ---- 2019-02-27 06:33:25 UTC - Matteo Merli: @bossbaby The ledgers might not have been deleted by the broker yet, and that could be due to a couple of reasons: 1. The data is still valid (either there’s a subscription retaining the data or there’s retention time set). 2. The ledgers were not rolled-over yet In BK we only delete a ledger when we roll-over to a new one and the old one only contains data that can be dropped. A quick trick to force a rollover of a ledger is to “unload” the topic so that it gets reloaded and a new ledger is created. `pulsar-admin topics unload $TOPIC` ---- 2019-02-27 06:38:29 UTC - Matteo Merli: @Jacob O'Farrell Typically it’s complicated to accurately measure end-to-end latency (producer -> broker -> consumer) because the timestamps are taken from different machines. Under normal conditions, the end-to-end latency is ~10-15 millis. It can be tuned to be < 1ms if needed (at the expense of some throughput) If you want to track the message as it flows through the system, you can use OpenTracing or similar. We had plans to integrate that into Pulsar though it’s not there at this point, although an application can use message properties and tag the messages on its own. ---- 2019-02-27 06:45:59 UTC - Jacob O'Farrell: Thanks @Matteo Merli - handy ---- 2019-02-27 06:46:07 UTC - Jacob O'Farrell: Anything else I should be aware of? ---- 2019-02-27 06:49:49 UTC - bossbaby: so i should unload topic before i delete it? ---- 2019-02-27 07:12:32 UTC - Matteo Merli: Sorry, I missed that you deleted the topic. that already takes care of it ---- 2019-02-27 07:17:30 UTC - bossbaby: is deletion included unload topic? ---- 2019-02-27 07:18:16 UTC - Matteo Merli: yes, after deletion all is gone. ---- 2019-02-27 07:18:39 UTC - Matteo Merli: then, do you have other topic’s data stored in the same bookies? ---- 2019-02-27 07:19:46 UTC - Matteo Merli: BK has its own GarbageCollection and compaction to get rid of the deleted data. If there’s other valid data there it might get delayed though at some the cleanup will happen. ---- 2019-02-27 07:21:03 UTC - bossbaby: I deleted all topics and after 1 day everything still hasn't changed. about 5.4G used for pulsar ---- 2019-02-27 07:23:28 UTC - Matteo Merli: Can you check the BK logs? It should print info logs when the GC/Compaction runs ---- 2019-02-27 07:26:23 UTC - bossbaby: this's log <https://gist.github.com/tuan6956/a144bc4cf0db728938684a217a6610ea> and report disk in VMs ``` 4.7G ./data/bookkeeper/journal/current 4.7G ./data/bookkeeper/journal 76M ./data/bookkeeper/ledgers/current/locations 75M ./data/bookkeeper/ledgers/current/ledgers 150M ./data/bookkeeper/ledgers/current 150M ./data/bookkeeper/ledgers 4.8G ./data/bookkeeper 4.8G ./data 5.4G . ``` ----
