2020-01-04 10:00:14 UTC - Amulya S Kanwar: @Amulya S Kanwar has joined the channel ---- 2020-01-04 14:14:53 UTC - markg: @markg has joined the channel ---- 2020-01-04 15:36:49 UTC - DBDB: @DBDB has joined the channel ---- 2020-01-04 16:32:32 UTC - David Kjerrumgaard: That warning appears to be unrelated to the issue you are seeing. ---- 2020-01-04 17:19:59 UTC - juraj: okay, but is it normal to see these warnings/errors every time upon connecting a client? ---- 2020-01-04 17:38:03 UTC - David Kjerrumgaard: No, it looks like there is some sort of issue with the Netty library. Which version JDK are you using? ---- 2020-01-04 17:39:17 UTC - juraj: java 12.0.1 - but i doubt this can produce "No active brokers available" since this is in the cluster which runs off the official Pulsar 2.4.2 Docker images on Java 8 ---- 2020-01-04 17:40:35 UTC - David Kjerrumgaard: There error seems to indicate an DNS error, so if you fail to resolve a hostname to an IP address, you could end up in a state where the brokers can't be reached ---- 2020-01-04 17:40:42 UTC - juraj: first it says "No active brokers available" and then it finds a broker - I'm thinking, maybe the first Proxy it connects to is not initialized properly ---- 2020-01-04 17:40:54 UTC - David Kjerrumgaard: perhaps. ---- 2020-01-04 17:41:01 UTC - juraj: there's multiple Proxies and also multiple Brokers under the services in the K8S ---- 2020-01-04 17:41:41 UTC - juraj: is there a way how to directly query a specific Proxy via the cli admin tools or so ? ---- 2020-01-04 17:42:00 UTC - juraj: i can get local bash access to the given Proxy pod ---- 2020-01-04 17:45:42 UTC - juraj: ok i've got 403 forbidden on the proxies 0 and 1, and it went fine on the proxies 2 and 3 ---- 2020-01-04 17:50:45 UTC - David Kjerrumgaard: Interesting. ---- 2020-01-04 18:36:32 UTC - Stéphane Derosiaux: @Stéphane Derosiaux has joined the channel ---- 2020-01-04 20:18:04 UTC - juraj: i'm trying to figure out why would the Proxy be getting a 403 response ---- 2020-01-04 20:20:53 UTC - juraj: i assume that is coming from the zookeeper ? ---- 2020-01-04 20:28:26 UTC - juraj: `bin/pulsar zookeeper-shell -server pulsar-dev-zookeeper ls /admin/clusters` works from all the proxy pods ---- 2020-01-04 20:52:27 UTC - David Kjerrumgaard: A 403 indicates that the server understood the request, but will not fulfill it due to client related issues. Are those two proxies configured differently than the others that are reachable? Are those proxies on a different network segment? Do you have security enabled? ---- 2020-01-04 21:02:17 UTC - juraj: i'm using the pulsar helm chart, and all the proxies etc. have the same config, no reason for differences that i can think of, apart perhaps for init timing issues ---- 2020-01-04 21:02:44 UTC - juraj: the network segment - this is AWS EKS and all pods should be reachable by each other ---- 2020-01-04 21:03:12 UTC - juraj: if they weren't reachable, we wouldn't be getting the 403 i think ---- 2020-01-04 21:03:35 UTC - juraj: i can query the zookeeper but i don't know what to look for ---- 2020-01-04 21:05:20 UTC - David Kjerrumgaard: Are there any errors in the proxy logs on the pods that are generating the 403's? ---- 2020-01-04 21:43:26 UTC - Laran Evans: Is it possible to read a message with a specific key from a topic? ---- 2020-01-04 21:44:14 UTC - Laran Evans: Alternatively, is it possible to know when a particular message is the latest message in the topic? ---- 2020-01-04 21:51:37 UTC - David Kjerrumgaard: @Laran Evans You can use the reader interface to navigate to the end of the topic or provide a specific messageID ---- 2020-01-04 21:53:26 UTC - David Kjerrumgaard: <https://pulsar.apache.org/api/client/org/apache/pulsar/client/api/ReaderBuilder.html#startMessageId-org.apache.pulsar.client.api.MessageId-> ---- 2020-01-05 00:54:41 UTC - markg: Fairly new to Pulsar - looking to get some understanding on Pulsar compared to Kafka. Anyone willing to comment on the pulsar cons in this article (in terms of true / not true) <https://medium.com/@philipfeng/modern-open-source-messaging-apache-kafka-rabbitmq-nats-pulsar-and-nsq-ca3bf7422db5>. ---- 2020-01-05 01:05:13 UTC - Matteo Merli: > Java client has little to no Javadoc False. <https://pulsar.apache.org/api/client/>
> Small community — 8 stackoverflow questions currently <https://pulsar.apache.org/powered-by/> > MessageId concept tied to BookKeeper — consumers cannot easily position itself on the topic compared to Kafka offset which is continuous sequence of numbers. MessageId is opaque (on purpose) and not tied to BookKeeper. You can position on a topic if you know the message id. Message ids can be easily serialized, stored and restored. > A reader cannot easily read the last message on the topic — need to skim through all the messages to the end. You can position a reader on the latest message (included). > No transactions Transaction support is planned for 2.6 release > Higher operational complexity — Zookeeper + Broker nodes + BookKeeper — all clustered Broker and bookies can be deployed in 1 single JVM process (of course you lose independent scalability) > Latency questionable — there is one extra remote call between Broker node and BookKeeper (compared to Kafka) Completely false. There's no extra remote call compared to Kafka. In Kafka, with replication, the hops are: producer -> leader -> follower. In any case 1 network hop adds 100micros of latency. This is utterly irrelevant if you consider the latency introduced by the disk. eg: <https://www.slideshare.net/merlimat/high-performance-messaging-with-apache-pulsar#39> ---- 2020-01-05 01:40:19 UTC - juraj: just ```01:38:44.728 [pulsar-external-web-4-6] WARN org.apache.pulsar.proxy.server.AdminProxyHandler - [127.0.0.1:51186] Failed to get next active broker No active broker is available org.apache.pulsar.broker.PulsarServerException: No active broker is available at org.apache.pulsar.proxy.server.BrokerDiscoveryProvider.nextBroker(BrokerDiscoveryProvider.java:94) ~[org.apache.pulsar-pulsar-proxy-2.4.2.jar:2.4.2]``` ---- 2020-01-05 01:44:17 UTC - juraj: i can see using `lsof` that the proxy process (which reports 403) talks to zookeeper: ```java 10 root 247u IPv4 41983422 0t0 TCP pulsar-dev-proxy-0.pulsar-dev-proxy.pulsar-dev.svc.cluster.local:35030->pulsar-dev-zookeeper-0.pulsar-dev-zookeeper.pulsar-dev.svc.cluster.local:2181 (ESTABLISHED)``` ---- 2020-01-05 01:47:41 UTC - juraj: zookepers logs show nothing when i fire the 403 request, but are full of ```01:47:09.852 [Thread-19565] INFO org.apache.zookeeper.server.NIOServerCnxn - Closed socket connection for client /127.0.0.1:60906 (no session established for client)``` ---- 2020-01-05 02:11:39 UTC - juraj: i've noticed that in the values.yaml the zookeeper 2181 port is named `stats`, while in the statefulset it's named `client`: <https://github.com/apache/pulsar/blob/master/deployment/kubernetes/helm/pulsar/values.yaml#L111> vs <https://github.com/apache/pulsar/blob/master/deployment/kubernetes/helm/pulsar/templates/zookeeper-statefulset.yaml#L96> i've renamed it to `client` in the values.yaml and recreated the cluster and it seems to work okay now ----
