2019-07-22 09:31:27 UTC - Norman Kabir: @Norman Kabir has joined the channel ---- 2019-07-22 10:50:03 UTC - divyasree: Hi ---- 2019-07-22 10:50:23 UTC - divyasree: When creating tenants using the below command ``` ./pulsar-admin tenants create geotest-tenant \ --admin-roles my-admin-role \ --allowed-clusters pulsar-prd-cluster-1,pulsar-prd-cluster-2 ``` ---- 2019-07-22 10:51:19 UTC - divyasree: wat does --admin-roles mean here? and how does it differ from "--role" when granting permission namespace level? ---- 2019-07-22 10:51:43 UTC - divyasree: what all values can be provided to --admin-roles parameters? ---- 2019-07-22 12:00:12 UTC - Heisenberg: @Heisenberg has joined the channel ---- 2019-07-22 15:51:32 UTC - Grant Wu: we’re running into a problem with Pulsar (standalone, default options) where we never ACK messages but _sometimes_ seeking to msgid earliest (with a reader) results in an error (presumably it has been forgotten). Any idea what might be going on? ---- 2019-07-22 15:53:07 UTC - Grant Wu: We’re both using the magic `MessageId.earliest` position and manually storing and seeking to a message ID ---- 2019-07-22 15:53:59 UTC - David Kjerrumgaard: @Grant Wu What is the error you are seeing? ---- 2019-07-22 16:48:32 UTC - Grant Wu: Apparently it’s “`failed to seek: UnknownError`” ---- 2019-07-22 17:10:44 UTC - David Kjerrumgaard: :smile: Not very useful ---- 2019-07-22 17:11:57 UTC - Grant Wu: > If you use the Reader interface (which is a wrapper around Consumer) the messages will be acked for you! Someone at my company just said this. Is this true? This doesn’t seem true ---- 2019-07-22 17:12:11 UTC - David Kjerrumgaard: My theory is that the message you are seeking (with the messageId) has been deleted due to retention policies, but then the msgId.earliest should be aware of that fact. ---- 2019-07-22 17:12:54 UTC - Grant Wu: Is it possible that the implementation details of readers impact message retention if there are no “normal” consumers of a topic? ---- 2019-07-22 17:14:12 UTC - Matteo Merli: Reader is implemented through an ephemeral subscription. A Reader will not retain the data in the topic, you’ll have to set time-based retention ---- 2019-07-22 17:15:21 UTC - Grant Wu: With respect to <https://pulsar.apache.org/docs/en/cookbooks-retention-expiry/>, does that mean that using a Reader will cause messages to be ack’ed? ---- 2019-07-22 17:15:53 UTC - Matteo Merli: In reader API there’s no concept of “acking” . ---- 2019-07-22 17:16:21 UTC - Matteo Merli: You’re “reading” though the messages in the topic. ---- 2019-07-22 17:16:59 UTC - Grant Wu: Right, I know that the interface doesn’t expose that, but I’m trying to figure out why we’re losing messages and was wondering if somehow the implementation detail of readers being implemented on top of a subscription (which may have the concept of acks) was leaking ---- 2019-07-22 17:17:09 UTC - Matteo Merli: internally, messages are auto-acked as the reader advances, to make sure the broker (and the stats) are aware where the reader is ---- 2019-07-22 17:17:45 UTC - Grant Wu: I guess taking a step back: Suppose I produce a bunch of messages on a topic. I never attach a consumer to this topic, but use readers to “read” the topic. Without configuring message retention, will we lose messages here? ---- 2019-07-22 17:18:36 UTC - Matteo Merli: Yes ---- 2019-07-22 17:18:44 UTC - Grant Wu: That’s unfortunate ---- 2019-07-22 17:20:04 UTC - Grant Wu: To me this feels like a leak of how readers are implemented. I would not expect this to be the case given the exposed reader interface ---- 2019-07-22 17:20:57 UTC - Matteo Merli: You might see some of the messages (given that ledgers are only rolled-over once in a while), but that’s not guaranteed.
You need to set time-based retention, either finite or indefinite (eg. -1) to make sure the messages are retained. You can also enable retention system wide: ``` # Default message retention time defaultRetentionTimeInMinutes=0 # Default retention size defaultRetentionSizeInMB=0 ``` ---- 2019-07-22 17:22:43 UTC - Matteo Merli: > I would not expect this to be the case given the exposed reader interface Reader is meant as low-level access to data with “manual” control of the MessageId. Since the positioning is manual, the broker doesn’t keep track of where the reader wants to read. This is very similar to Kafka model, where you can only set retention time/size to delete the data. ---- 2019-07-22 17:24:51 UTC - Grant Wu: > Reader is meant as low-level access to data with “manual” control of the MessageId. I understand this, but I don’t think that someone without knowledge that Readers are implemented internally with ephemeral subscriptions could read <https://pulsar.apache.org/docs/en/cookbooks-retention-expiry/> and understand that using exclusively readers would cause messages to be “acknowledged on every subscription” ---- 2019-07-22 17:28:14 UTC - Matteo Merli: > and understand that using exclusively readers would cause messages to be “acknowledged on every subscription” Uhm, but logically a Reader is not associated with a subscription. ---- 2019-07-22 17:28:58 UTC - Matteo Merli: Let me re-read that doc page, perhaps is not very clear. ---- 2019-07-22 17:29:39 UTC - Grant Wu: I think we’re not understanding each other very well ---- 2019-07-22 17:29:45 UTC - Matteo Merli: :slightly_smiling_face: ---- 2019-07-22 17:30:40 UTC - Grant Wu: Let me test my understanding: Suppose I produce some messages on a topic. Default settings. I don’t make any readers or consumers. Pulsar should store these messages in the backlog, right? ---- 2019-07-22 17:30:56 UTC - Matteo Merli: no ---- 2019-07-22 17:31:09 UTC - Grant Wu: Wait, why not? ---- 2019-07-22 17:31:19 UTC - Matteo Merli: Backlog is associated with a subscription ---- 2019-07-22 17:32:18 UTC - Grant Wu: <https://pulsar.apache.org/docs/en/cookbooks-retention-expiry> page doesn’t use subscription when talking about backlogs at all. It only mentions it in the “immediately delete all messages that have been acknowledged on every subscription, and” line ---- 2019-07-22 17:33:06 UTC - Grant Wu: > By default, when a Pulsar message arrives at a broker it will be stored until it has been acknowledged by a consumer, at which point it will be deleted. This sentence implies to me that a message should be stored until a consumer is first created ---- 2019-07-22 17:33:06 UTC - Matteo Merli: Yes, I guess that phrasing can be confusing :confused: ---- 2019-07-22 17:33:37 UTC - Matteo Merli: To clarify: * Subscriptions makes data to be retained (until acked) * Consumers are attached to subscription. * A reader doesn’t use a subscription (since it manages manually the position on a particular MessageId) ---- 2019-07-22 17:34:15 UTC - Matteo Merli: If you want to retain messages independently of subscriptions: * Use retention policy ---- 2019-07-22 17:36:13 UTC - Grant Wu: Okay. So, as another check on my understanding, even though I understand this would not be a particularly reasonable way of doing things, if I were to produce messages on a topic and then create a consumer on that topic, but never actually acknowledge any messages, this would effectively retain messages forever given default Pulsar settings? ---- 2019-07-22 17:43:57 UTC - Matteo Merli: Yes, though you will be subject to the backlog quota (eg: default 10GB). After that producer is blocked (by default) ---- 2019-07-22 17:44:05 UTC - Grant Wu: Got it. ---- 2019-07-22 18:23:55 UTC - Ping-Min Lin: Hi, we are seeing increasing jitter when we increase the write quorum while using the same ack quorum. Is this expected? How can we tune it? Specs: - Bookies: i3.4xl (single mount/raid0) - Brokers: c5.4xl - Persistency settings: ack = 1, ensemble = 5, write quorum = 1: ~30 ms ack = 1, ensemble = 5, write quorum = 3: ~300 ms max batch size is 2400, max pending size 2000, max batch interval 10ms cc: @Ambud Sharma ---- 2019-07-22 18:30:18 UTC - Matteo Merli: I’d keep the ensemble=write-quorum. In general this will increase the long-tail latency when “striping” to multiple bookies. This is because entries are ordered and write acks need to wait for all previous entries to be acked. ---- 2019-07-22 18:30:50 UTC - Matteo Merli: Also, what latency is the one reported? ---- 2019-07-22 18:46:22 UTC - Ambud Sharma: Latency above is time taken to write a single batch. We are trying to compare this to Kafka. ---- 2019-07-22 18:46:25 UTC - Ping-Min Lin: This is a latency we report on the producer application side. It is measured from the beginning of `producer.sendAsync` + `producer.flush` + get from all `sendAsync`s ---- 2019-07-22 18:46:54 UTC - Matteo Merli: is it avg, median or 99pct ? ---- 2019-07-22 18:47:01 UTC - Ambud Sharma: p99 ---- 2019-07-22 18:47:29 UTC - Ping-Min Lin: it's actually a ballpark average ---- 2019-07-22 18:48:04 UTC - Ambud Sharma: the issue is when the latency spikes; it is creating a massive backlog of data. ---- 2019-07-22 18:48:05 UTC - Matteo Merli: You could also track the latency for the individual messages (that would typically be lower) ---- 2019-07-22 18:48:58 UTC - Matteo Merli: In any case, can you try to get the baseline with `pulsar-perf` tool? ---- 2019-07-22 18:49:30 UTC - Matteo Merli: (just to get a comparative idea) ---- 2019-07-22 18:49:34 UTC - Ping-Min Lin: sure ---- 2019-07-22 18:50:35 UTC - Matteo Merli: eg. `pulsar-perf produce $TOPIC --rate 10000 --batch-time-window 10` ---- 2019-07-22 18:50:48 UTC - Matteo Merli: this will report the latencies every 10 sec ---- 2019-07-22 18:59:25 UTC - David Kjerrumgaard: I would like to announce the early access release of a new book on Apache Pulsar, *_Pulsar in Action_*. The book's page is here: <https://www.manning.com/books/pulsar-in-action> and there is a MEAP launch discount code for 50% off: mlkjerrumgaard tada : Matteo Merli, Ambud Sharma, Karthik Ramasamy, Grant Wu, tuteng, Géraud ---- 2019-07-22 19:09:47 UTC - Ping-Min Lin: sorry, that latency reported was max latency. I'll post the pulsar-perf results right away ---- 2019-07-22 19:11:10 UTC - Matteo Merli: Ok, in any case, my first suspicion is on the `ensemble=5` setting. :slightly_smiling_face: ---- 2019-07-22 19:14:03 UTC - Matteo Merli: 2nd thing to optimize would be the single disk mount on the bookies. To avoid any interference, it would be good to have the journal on a dedicated disk. Another option could be to turn off fsync()s on the bookie journal. In `bookkeeper.conf`, `journalSyncData=false`. ---- 2019-07-22 19:14:49 UTC - Ping-Min Lin: Thanks @Matteo Merli for the pointers. Could you elaborate more about how the persistence configs interact with each other and potential issues of setting it to 5 ( 6 is the amount of bookies we have) ---- 2019-07-22 19:16:51 UTC - Ping-Min Lin: This is a run for ack quorum = 1, write quorum = 3: ``` 19:08:28.329 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 990.7 msg/s --- 7.7 Mbit/s --- Latency: mean: 96.623 ms - med: 42.695 - 95pct: 380.845 - 99pct: 669.119 - 99.9pct: 752.039 - 99.99pct: 759.623 - Max: 760.631 19:08:38.369 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 1004.9 msg/s --- 7.9 Mbit/s --- Latency: mean: 71.864 ms - med: 49.357 - 95pct: 197.184 - 99pct: 302.795 - 99.9pct: 360.357 - 99.99pct: 369.893 - Max: 369.925 19:08:48.389 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 924.0 msg/s --- 7.2 Mbit/s --- Latency: mean: 154.954 ms - med: 108.287 - 95pct: 451.633 - 99pct: 657.827 - 99.9pct: 741.207 - 99.99pct: 749.667 - Max: 750.719 19:08:58.403 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 1074.2 msg/s --- 8.4 Mbit/s --- Latency: mean: 143.959 ms - med: 70.309 - 95pct: 510.987 - 99pct: 740.215 - 99.9pct: 837.159 - 99.99pct: 846.687 - Max: 847.735 19:09:08.415 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 1000.3 msg/s --- 7.8 Mbit/s --- Latency: mean: 61.905 ms - med: 37.210 - 95pct: 200.493 - 99pct: 292.985 - 99.9pct: 356.551 - 99.99pct: 365.019 - Max: 366.073 19:09:18.429 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 1001.5 msg/s --- 7.8 Mbit/s --- Latency: mean: 143.282 ms - med: 102.100 - 95pct: 406.213 - 99pct: 563.767 - 99.9pct: 652.747 - 99.99pct: 662.235 - Max: 663.291 19:09:28.442 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 995.3 msg/s --- 7.8 Mbit/s --- Latency: mean: 150.095 ms - med: 126.849 - 95pct: 352.591 - 99pct: 449.523 - 99.9pct: 495.753 - 99.99pct: 504.219 - Max: 505.273 19:09:38.454 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 996.4 msg/s --- 7.8 Mbit/s --- Latency: mean: 109.105 ms - med: 102.108 - 95pct: 238.332 - 99pct: 304.971 - 99.9pct: 345.075 - 99.99pct: 349.301 - Max: 350.353 19:09:48.466 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 1002.1 msg/s --- 7.8 Mbit/s --- Latency: mean: 102.036 ms - med: 95.397 - 95pct: 231.750 - 99pct: 285.731 - 99.9pct: 312.303 - 99.99pct: 321.365 - Max: 321.367 ^C19:09:53.246 [Thread-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Aggregated latency stats --- Latency: mean: 114.678 ms - med: 77.781 - 95pct: 333.477 - 99pct: 564.571 - 99.9pct: 757.447 - 99.99pct: 839.275 - 99.999pct: 846.687 - Max: 847.735 ``` ---- 2019-07-22 19:17:33 UTC - Ping-Min Lin: range of mean is around 60 to 150 ms ---- 2019-07-22 19:17:54 UTC - Ping-Min Lin: for ack = 1, write = 1: ``` 19:10:20.864 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 999.1 msg/s --- 7.8 Mbit/s --- Latency: mean: 40.625 ms - med: 16.877 - 95pct: 163.204 - 99pct: 203.415 - 99.9pct: 271.767 - 99.99pct: 280.243 - Max: 281.301 19:10:30.904 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 998.9 msg/s --- 7.8 Mbit/s --- Latency: mean: 32.665 ms - med: 15.912 - 95pct: 150.008 - 99pct: 216.919 - 99.9pct: 276.065 - 99.99pct: 285.639 - Max: 286.687 19:10:40.922 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 997.1 msg/s --- 7.8 Mbit/s --- Latency: mean: 47.094 ms - med: 20.307 - 95pct: 183.787 - 99pct: 271.529 - 99.9pct: 313.039 - 99.99pct: 322.559 - Max: 322.563 19:10:50.934 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 995.2 msg/s --- 7.8 Mbit/s --- Latency: mean: 81.261 ms - med: 68.212 - 95pct: 174.485 - 99pct: 281.167 - 99.9pct: 338.061 - 99.99pct: 347.579 - Max: 348.631 19:11:00.947 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 1008.3 msg/s --- 7.9 Mbit/s --- Latency: mean: 122.553 ms - med: 120.955 - 95pct: 229.293 - 99pct: 372.163 - 99.9pct: 458.717 - 99.99pct: 467.161 - Max: 468.215 19:11:10.958 [main] INFO org.apache.pulsar.testclient.PerformanceProducer - Throughput produced: 1000.6 msg/s --- 7.8 Mbit/s --- Latency: mean: 29.792 ms - med: 14.760 - 95pct: 111.315 - 99pct: 170.292 - 99.9pct: 241.416 - 99.99pct: 249.880 - Max: 250.934 ^C19:11:14.267 [Thread-1] INFO org.apache.pulsar.testclient.PerformanceProducer - Aggregated latency stats --- Latency: mean: 57.018 ms - med: 25.798 - 95pct: 175.907 - 99pct: 265.231 - 99.9pct: 405.967 - 99.99pct: 461.875 - 99.999pct: 467.161 - Max: 468.215 ``` ---- 2019-07-22 19:18:09 UTC - Matteo Merli: Both seem very high ---- 2019-07-22 19:18:23 UTC - Ping-Min Lin: mean falls in range [30, 120] ---- 2019-07-22 19:18:44 UTC - Matteo Merli: is this producing from within the same VPC? ---- 2019-07-22 19:20:02 UTC - Matteo Merli: I’d expect in both cases 99pct < 5millis ---- 2019-07-22 19:20:12 UTC - Ping-Min Lin: yes, they are in same VPC ---- 2019-07-22 19:20:37 UTC - Ping-Min Lin: test cmd : `bin/pulsar-perf produce -r 1000 -b 10 $TOPIC` ---- 2019-07-22 19:20:57 UTC - Ping-Min Lin: running on 2.3.2 ---- 2019-07-22 19:21:25 UTC - Matteo Merli: Can you double check on bookie nodes the IO stats? `iostat -xm 1` ---- 2019-07-22 19:23:18 UTC - Matteo Merli: That should show which device the bookie is writing into (it happened to me many times to see bad numbers and later discovered I was writing on the VM disk instead of the NVMe device) ---- 2019-07-22 19:24:03 UTC - Ping-Min Lin: ``` avg-cpu: %user %nice %system %iowait %steal %idle 1.56 0.00 2.63 2.82 0.31 92.68 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util xvda 0.00 349.00 0.00 1063.00 0.00 108.78 209.58 5.97 6.31 0.00 6.31 0.41 44.00 nvme0n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 nvme1n1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 ``` ---- 2019-07-22 19:24:32 UTC - Ping-Min Lin: hmm ---- 2019-07-22 19:24:41 UTC - Matteo Merli: :slightly_smiling_face: ---- 2019-07-22 19:25:06 UTC - Matteo Merli: looks like is not going to `md0` or the nvmes ---- 2019-07-22 19:28:19 UTC - Grant Wu: Thoughts on this? <https://github.com/apache/pulsar/pull/4780/files> ---- 2019-07-22 19:30:15 UTC - Ping-Min Lin: Thanks. Seems that we didn't get the bookie config migrated completely when we were upgrading from 2.2.1 to 2.3.2 :stuck_out_tongue: ---- 2019-07-22 19:35:40 UTC - Matteo Merli: That’s better than what it’s currently. I’d even make it more evident that it’s either: * a subscription that “retains” the data, * or a specific retention policy ---- 2019-07-22 19:35:54 UTC - Grant Wu: Good idea ---- 2019-07-22 20:23:51 UTC - Poule: is it possible to migrate data from standalone to a non-standalone? ---- 2019-07-22 20:25:44 UTC - Poule: you know, you start small, then you grow ---- 2019-07-22 20:27:30 UTC - David Kjerrumgaard: @Poule The short answer is yes, but it would require some work setting up the replication between the existing standalone cluster, and the newly added non-standalone cluster. ---- 2019-07-22 20:28:26 UTC - David Kjerrumgaard: so careful planning and testing would be recommended. Typically standalone is used for dev work and the data isn't usually meant to be retained, etc. ---- 2019-07-22 20:30:10 UTC - Poule: ok ---- 2019-07-22 20:31:28 UTC - Poule: lot of work i guess ---- 2019-07-22 20:38:57 UTC - David Kjerrumgaard: @Poule Not a lot of work, but it would need to be carefully planned out. The environments would need to be configured together as part of a larger "instance", so they are aware of one another. There would have to be network connectivity between them, etc. Then you would configure topic replication, etc. ---- 2019-07-22 20:43:17 UTC - Addison Higham: along those same lines... I have been wondering recently: how difficult/how open would it be to add an option so that any just-in-time created topics defaulted to a a partitioned topic with a parallelism of 1? Given that there isn't support for converting a non partitioned topic to a partitioned topic, that seems like it could be a good stop gap... ---- 2019-07-22 21:02:25 UTC - David Kjerrumgaard: @Addison Higham That would be a good feature request to create. I don't think it would require a lot of additional coding like you said. ---- 2019-07-22 21:53:23 UTC - Grant Wu: Wait, so, when you say that a backlog is per subscription, what does that entail exactly? Is there any instance where thinking of each subscription having its own individual backlog is a useful model? If we look at backlogs as per-subscription, I’d need to reword all of the “Backlog quota” text to refer to the “largest backlog on a topic” ---- 2019-07-22 22:18:53 UTC - Matteo Merli: > when you say that a backlog is per subscription, what does that entail exactly? Backlog means the number of message that one subscription has to go through > Is there any instance where thinking of each subscription having its own individual backlog is a useful model? It’s individual because the position of each subscription is independent > If we look at backlogs as per-subscription, I’d need to reword all of the “Backlog quota” text to refer to the “largest backlog on a topic” That’s correct ---- 2019-07-22 22:21:52 UTC - Grant Wu: er, and so when referring to size thresholds - what is the relevant size metric? ---- 2019-07-22 22:22:46 UTC - Grant Wu: It’s definitely not sum(all backlogs for a topic) because messages can be deduped, but it’s also not size(largest backlog) because you need overhead to store which messages are acknowledged and which aren’t ---- 2019-07-22 22:24:38 UTC - Grant Wu: is it a more complicated measure involving the size of all unack’d messages (after deduping) plus overhead per subscription? ---- 2019-07-23 00:05:53 UTC - Sijie Guo: @Addison Higham this was just supported in latest master. It will be released as 2.5. If you need, we can cherry pick that change to 2.4.1 +1 : David Kjerrumgaard ---- 2019-07-23 01:22:49 UTC - hopeqpy: @hopeqpy has joined the channel ---- 2019-07-23 02:23:02 UTC - myjavaday: @myjavaday has joined the channel ---- 2019-07-23 02:23:42 UTC - weili: @weili has joined the channel ---- 2019-07-23 02:41:46 UTC - Ambud Sharma: thanks @Matteo Merli that fixed the issue; the bookie journal directory was not on nvme ---- 2019-07-23 02:41:58 UTC - Matteo Merli: :+1: ---- 2019-07-23 03:17:38 UTC - Matteo Merli: @Sijie Guo I think the change in master allows to create topics with 1 partition, though not when topics are auto created ---- 2019-07-23 03:27:05 UTC - Sijie Guo: @Matteo Merli oh i see. we can add a flag to broker configuration to change the default behavior to create 1-partitioned topic by default. ---- 2019-07-23 08:52:12 UTC - divyasree: @Sijie Guo How can we delete the consumers for a topic? ---- 2019-07-23 08:54:51 UTC - Sijie Guo: @divyasree in administration way? you can use `pulsar-admin topics unsubscribe` ---- 2019-07-23 08:55:55 UTC - divyasree: I am trying to delete a topic, which required there should not be any active producers and subscriptions. ---- 2019-07-23 08:56:48 UTC - divyasree: When i am trying to unsubscribe the subscription, it is telling there are active consumers.. So i need to delete the consumers.... ---- 2019-07-23 08:57:08 UTC - Sijie Guo: You can specify “-f” when you delete a topic ---- 2019-07-23 08:57:20 UTC - divyasree: yes... still not alowing ---- 2019-07-23 08:57:22 UTC - Sijie Guo: to force close all active producers, consumers and delete the topic ---- 2019-07-23 08:57:40 UTC - divyasree: how to close all active producers and consumers ---- 2019-07-23 08:57:41 UTC - Sijie Guo: what is the error? ---- 2019-07-23 08:57:54 UTC - Sijie Guo: what is your command? ---- 2019-07-23 08:59:15 UTC - divyasree: ``` [root@pulsar-prd-bk1 bin]# ./pulsar-admin topics delete <persistent://public/functions/assignments> -f [root@pulsar-prd-bk1 bin]# ./pulsar-admin namespaces topics public/functions <persistent://public/functions/assignments> <persistent://public/functions/coordinate> <persistent://public/functions/metadata> ``` ---- 2019-07-23 09:02:16 UTC - Sijie Guo: @divyasree I see. assignments was actually deleted. but it was automatically created again because your function workers are still running. ---- 2019-07-23 09:02:47 UTC - Sijie Guo: Can you tell me a bit about what you are planning to do for deleting functions/assignments topic? ---- 2019-07-23 09:03:47 UTC - divyasree: oh ok.. Actually i am trying to grant permission for a namespace with specific role.... ---- 2019-07-23 09:04:29 UTC - divyasree: initial i have created a role "test-user" with have access to the namespaces " <persistent://public/functions/>" ---- 2019-07-23 09:05:19 UTC - divyasree: i revoked the role for that namespace, and removed the role from the 'superUserRoles' parameter of the broker.conf file also... ---- 2019-07-23 09:05:38 UTC - divyasree: after all these changes i restarted the broker ---- 2019-07-23 09:05:48 UTC - divyasree: and it got failed with the below error ---- 2019-07-23 09:06:28 UTC - divyasree: ``` Exception while at creating producer to topic <persistent://public/functions/assignments> java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.PulsarClientException$AuthorizationException: Authorization failed test-user on topic <persistent://public/functions/assignments> with error Don't have permission to administrate resources on this tenant ``` ----
