2020-08-27 09:43:22 UTC - SenthilM: @SenthilM has joined the channel ---- 2020-08-27 11:02:09 UTC - Julius S: Were you in the right database when you issued the grants? ---- 2020-08-27 11:15:36 UTC - Ali Ahmed: <https://github.com/cr-org/supernova> cool : Frank Kelly, Ali Ahmed ---- 2020-08-27 13:52:15 UTC - Frank Kelly: Is there anyway to make the Pulsar Java Client "fail fast" - I'm testing some AuthN/AuthZ scenarios and rather than wait for all the retries I'd rather it just stop on the first AuthN/AuthZ exception? ---- 2020-08-27 15:16:18 UTC - Thomas O'Neill: According to the documentation > Normal topics are served only by a single broker How do you handle a broker going down that has a topic you are writing or reading from? Can you reconnect through pulsar-proxy to another broker and continue executing, or do you need to wait for the broker to come back online? I am running pulsar in Kubernetes and there is a possibility of the node getting going down and am wondering how others have worked around this issue. All connections are made through pulsar-proxy to the brokers. ---- 2020-08-27 15:38:27 UTC - Raman Gupta: Not knowing exactly where that note is coming from in the docs, but I *think* that is referring more to performance i.e. as opposed to a partitioned topic that would be served by multiple brokers.
From a resilience perspective, I don't think it matters. Another broker should quickly take over the topic and clients will use that instead (transparently). ---- 2020-08-27 15:41:33 UTC - Raman Gupta: If the number of partitions for a partitioned topic is changed, is message ordering for past messages lost? Or does that change only affect future writes, thus maintaining order? ---- 2020-08-27 15:55:22 UTC - Thomas O'Neill: Thanks, I read it under the Partitioned topics section in the Messaging documentation. <https://pulsar.apache.org/docs/en/concepts-messaging/> ---- 2020-08-27 17:01:40 UTC - Austin Bennett: @Austin Bennett has joined the channel ---- 2020-08-27 17:19:28 UTC - Addison Higham: @Thomas O'Neill as @Raman Gupta mentioned, in the event that a broker fails, pretty quickly the brokers will notice that the broker has failed and a new broker will take control of that bundle of topics. During that period of time, the clients will buffer any messages and as soon as the new broker is ready it will flush those messages ---- 2020-08-27 18:43:50 UTC - Maksym Domarev: I want to TTL not the message but a topic e.g. to do a clean up the topics itself, I want to have a very dynamic configuration that will generate thousanfs of topics a day ---- 2020-08-27 18:44:27 UTC - Maksym Domarev: some of them will remain active some of them never re-use again, so in general if I will clean it once a week i will not have any issues, but if it will never clean it will be a problem ---- 2020-08-27 18:55:50 UTC - Evan Furman: Do we have to enable anything to take advantage of `pulsar_detector` in the grafana dashboards? It doesn’t look like `pulsar_detector_publish_latency_ms` or any other metrics are being scraped ---- 2020-08-27 19:17:48 UTC - Derek Moore: Schema algebras would allow the declarative definition of how aggregates are maintained. Has Pulsar talked much about those? ---- 2020-08-27 19:25:06 UTC - Derek Moore: What's the `journalSyncData=false` bug that Confluent mentions here: <https://www.confluent.io/blog/kafka-fastest-messaging-system/> ---- 2020-08-27 19:25:13 UTC - Addison Higham: I haven't seen much discussion in the community about it, but personally something I am very interested in :slightly_smiling_face: if you have any ideas of how it might be applicable to Pulsar writing up a small doc or blog post for discussion can be pretty useful ---- 2020-08-27 19:28:35 UTC - Addison Higham: Not sure, they asked one question here and we asked to open up an issue with more context and info about their configuration, but that never happened. ---- 2020-08-27 19:34:26 UTC - Addison Higham: (see <https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1594655381230800>) Also, just FYI, there is a change that is in progress currently which will be to bypass the journal (see <https://github.com/apache/bookkeeper/pull/2401>) that will allow bookkeeper to be configured to run more in line with how kafka works I don't think many people run with sync set to false, so I am not sure if there is enough data to know what happened there. If you have more questions about the article in general, there was some discussion in <#C5Z1W2BDY|random> +1 : Derek Moore ---- 2020-08-27 19:43:03 UTC - Addison Higham: any progress here? sorry meant to dig into this earlier. I believe this should be doable but it may depend on the way you are creating the client/product/etc Can you share a code example and also a test log if you have it? ---- 2020-08-27 19:47:11 UTC - Addison Higham: if you are using a key and a hash-partition router, then many keys will likely go to new partitions, but old messages will still stay in their older partitions. Compaction would change that however. Also, just to be clear, you would probably see some re-ordering around the period of the change. For example, with an exclusive consumer to a partitioned topic, it is actually multiple consumers internally that all may be at different offsets in each partition, which could lead to a consumer further ahead reading newly keyed data from new topics before another consumer finishes reading all the old keyed data ---- 2020-08-27 19:47:20 UTC - Frank Kelly: No success so far - I'm just trying to instantiate a pulsar client either without a token ``` return PulsarClient.builder() .allowTlsInsecureConnection(false) .serviceUrl("pulsar://" + SERVER + ":6650") .build();``` or with a bad JWT token ``` return PulsarClient.builder() .allowTlsInsecureConnection(false) .serviceUrl("pulsar://" + SERVER + ":6650") .authentication(new CogitoStreamAccessJWTAuthentication("HeyIamNotAValidJWTToken")) .build();``` then doing an `assertThrows` ``` assertThrows(PulsarClientException.class, () -> pulsarClient.newProducer() .topic(topicBeingRequested) .create());``` but I have to wait about 30 seconds for the retries ---- 2020-08-27 19:48:19 UTC - Frank Kelly: I see a lot of these in the Test debug logs ```2020-08-27 15:46:49,028 - WARN - [pulsar-client-io-8-1:BinaryProtoLookupService@197] - [<persistent://testTenant1/testNamespace1/my-topic-1598557604630>] failed to get Partitioned metadata : org.apache.pulsar.client.api.PulsarClientException: Disconnected from server at pulsar-broker.cogito.svc.cluster.local/10.100.43.26:6650 java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException$LookupException: org.apache.pulsar.client.api.PulsarClientException: Disconnected from server at pulsar-broker.cogito.svc.cluster.local/10.100.43.26:6650 at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) at java.base/java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:704) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) at org.apache.pulsar.client.impl.ClientCnx.handleLookupResponse(ClientCnx.java:474) at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:131) . . . at java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.apache.pulsar.client.api.PulsarClientException$LookupException: org.apache.pulsar.client.api.PulsarClientException: Disconnected from server at pulsar-broker.cogito.svc.cluster.local/10.100.43.26:6650 at org.apache.pulsar.client.impl.ClientCnx.getPulsarClientException(ClientCnx.java:991) at org.apache.pulsar.client.impl.ClientCnx.handleLookupResponse(ClientCnx.java:475) ... 22 more``` ---- 2020-08-27 19:49:32 UTC - Raman Gupta: That wouldn't be the case with Key_Shared right? Since a given key should only be assigned to one consumer? ---- 2020-08-27 19:58:47 UTC - Addison Higham: I am not 100% sure on that, it *should* but I don't know how extensively it has been tested ---- 2020-08-27 19:59:08 UTC - Addison Higham: I will ask around and try an update here ---- 2020-08-27 19:59:16 UTC - Addison Higham: ah is this via a proxy? ---- 2020-08-27 19:59:51 UTC - Raman Gupta: It surprises me that the docs mention nothing about these caveats. Proper ordering can be very important for many use cases. ---- 2020-08-27 19:59:58 UTC - Raman Gupta: Thanks for checking... ---- 2020-08-27 20:01:16 UTC - Frank Kelly: Yep ---- 2020-08-27 20:04:52 UTC - Enrico Olivelli: @Viktor did you create the issue on BK? ---- 2020-08-27 20:10:50 UTC - Addison Higham: `maxNumberOfRejectedRequestPerConnection` is what controls that but I am wondering if it is working as it should. Some requests, such as subscribing, we properly throw an exception right away ---- 2020-08-27 20:11:03 UTC - Addison Higham: that seems like we should be consistent for lookups as well ---- 2020-08-27 20:12:00 UTC - Frank Kelly: Yeah I tried that but it didn't seem to impact anything for my testing ---- 2020-08-27 20:12:41 UTC - Frank Kelly: Again it's kinda weird what I'm trying to do - but having some level of control over the number of retries would be nice if people would like to "fail fast" ---- 2020-08-27 20:14:23 UTC - Addison Higham: how low did you set the value? ---- 2020-08-27 20:14:44 UTC - Addison Higham: it is within a 30 second window, so depending on your backoff values, you would need to set it quite low ---- 2020-08-27 20:17:26 UTC - Addison Higham: I searched around a bit and couldn't find anything :shrug: ---- 2020-08-27 20:17:31 UTC - Frank Kelly: I set it to 1 :slightly_smiling_face: ---- 2020-08-27 20:17:58 UTC - Raman Gupta: Also, a follow-up on compaction: will compaction cause messages, both old and new, to be placed into partitions in a consistent way? i.e. will a consumer re-reading the topic from the beginning be guaranteed a properly ordered read? ---- 2020-08-27 20:20:41 UTC - Addison Higham: two things: 1. if you have a way to try this direct against a broker (perhaps a port forward?), I would be very curious to see if you get a different result 2. Could you make a github issue for this? I expect there are some issues here ---- 2020-08-27 20:21:45 UTC - Frank Kelly: 2. Will do 1. OK and this maybe just me but with a port-forward to my broker (all Pulsar components in my local minikube) it would refuse the connection - again this is probably something silly I set up but not sure ---- 2020-08-27 20:25:38 UTC - Addison Higham: ah if running with the helm chart there should be a service for pulsar brokers as well ---- 2020-08-27 20:25:49 UTC - Addison Higham: not sure why port forward wouldn't work, but you could try that ---- 2020-08-27 20:26:29 UTC - Frank Kelly: Yeah I was doing port forward in all test cases (proxy and broker) and there was a service for broker with those ports (8080, 6650) ---- 2020-08-27 20:27:35 UTC - Frank Kelly: Hold on might be have figured it out. . . . . ---- 2020-08-27 20:29:12 UTC - Frank Kelly: Nope. Here is the service setup ```$ kubectl get services | grep pulsar pulsar-bookkeeper ClusterIP None <none> 3181/TCP 86m pulsar-broker ClusterIP 10.100.43.26 <none> 8080/TCP,6650/TCP,8443/TCP,6651/TCP 86m pulsar-proxy NodePort 10.97.122.80 <none> 8080:31074/TCP,6650:30002/TCP,8000:32173/TCP 86m pulsar-zookeeper ClusterIP None <none> 2888/TCP,3888/TCP,2181/TCP 86m``` ---- 2020-08-27 20:29:38 UTC - Addison Higham: ah, it might be because of NodePort vs ClusterIP, I have not use minikube much ---- 2020-08-27 20:31:21 UTC - Frank Kelly: OK ---- 2020-08-27 20:32:02 UTC - Frank Kelly: Port-forward to the proxy works great - and the port-forward to the broker at least shows up on the Broker logs - just I get a bunch of connection errors ```2020-08-27 16:30:33,120 - WARN - [pulsar-client-io-8-1:ConnectionPool@212] - Failed to open connection to 172.17.0.3:6650 : org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.17.0.3:6650 2020-08-27 16:30:33,120 - WARN - [pulsar-client-io-8-1:ConnectionHandler@77] - [<persistent://testtenant/wav/topic1>] [null] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.17.0.3:6650 2020-08-27 16:30:33,121 - WARN - [pulsar-client-io-8-1:ConnectionHandler@101] - [<persistent://testtenant/wav/topic1>] [null] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: org.apache.pulsar.shade.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /172.17.0.3:6650 -- Will try again in 0.187 s 2020-08-27 16:30:33,309 - INFO - [pulsar-timer-10-1:ConnectionHandler@105] - [<persistent://testtenant/wav/topic1>] [null] Reconnecting after connection was closed``` ---- 2020-08-27 20:34:27 UTC - Addison Higham: I assume you mean with the same key? Compaction does re-order, as it only keeps the most recent version of every key and that can inherently be a different order that the original event ordering. However anything after the compacted part of the topic (the tail that hasn't yet been compacted) has the some order properties If you are just talking about the same key, that is another thing I am not sure about and have wondered as well. It may need to be explicitly handled by compaction, if compaction happens topic by topic and it doesn't "re-hash" the message by the key, then it would just keep the most recent version in each topic ---- 2020-08-27 20:35:01 UTC - Raman Gupta: Sorry I meant different keys that hash to the same partition. ---- 2020-08-27 20:35:56 UTC - Frank Kelly: (BTW on the Github issue - would you classify this as a bug or enhancement) ---- 2020-08-27 20:36:04 UTC - Raman Gupta: Or I suppose really messages that have the same `orderingKey` ---- 2020-08-27 20:40:11 UTC - Raman Gupta: I here <https://github.com/apache/pulsar/pull/7745> some discussion of `partitionKey` and `orderingKey` -- it seems `partitionKey` is used on compaction, but what's unclear is how this interacts with partition count changes. ---- 2020-08-27 20:41:04 UTC - Addison Higham: orderingKey isn't (currently) considered by compaction ---- 2020-08-27 20:45:03 UTC - Raman Gupta: Generally I would set partitionKey to be the same as orderingKey, because I'd want to ensure that messages with the same ordering key were in the same partition. Though I would think in the presence of orderingKey, partitionKey should be ignored anyway, because otherwise how could Pulsar guarantee ordering for a given ordering key if the messages with that orderingKey are splashed across multiple partitions? ---- 2020-08-27 20:46:43 UTC - Raman Gupta: Though I suppose there might be a use case for correct ordering by orderingKey, but the user only cares about that within the scope of a partition, so never mind my last msg. ---- 2020-08-27 20:50:07 UTC - Raman Gupta: Yet another confusion and/or omission of detail: the section of the docs discussing ordering gurantee for partitioned topics just talks about "key", not "partition key": <https://pulsar.apache.org/docs/en/concepts-messaging/#ordering-guarantee>. ---- 2020-08-28 09:08:24 UTC - Tymm: Hello, let's say I have a subscriber went offline and back online after 2 hours, and when it is back online, I want it to only get the latest message instead of all the messages it had missed. How can I do this? What should I configure? Thanks ----
