2019-02-24 12:16:52 UTC - Vincent Ngan: In release notes of Pulsar 2.3.0, it says >Support specifying multiple hosts in pulsar service url and web url #3249 Does it mean that we can specify multiple broker urls in serviceUrl so that whenever any one of them is available the client can connect to it? ---- 2019-02-24 12:21:48 UTC - Vincent Ngan: Can we use this feature to replace the use of pulsar-proxy because we can specify the urls of all the brokers in the client configuration and let the client figure out which broker to connect to without the need of a pulsar-proxy? ---- 2019-02-24 12:28:57 UTC - Sijie Guo: yes. but this only for client for now. it doesn’t support admin client yet. because admin client is using jersey, still havent’ figured out an way to do so. ---- 2019-02-24 12:30:25 UTC - Vincent Ngan: How about replacing the use of pulsar-proxy? ---- 2019-02-24 12:30:56 UTC - Sijie Guo: the purpose of proxy is different from multi-hosts service url. the proxy is used for hiding the details of brokers. because in some environment, people doesn’t want to expose their brokers’ address for security reason.
the `multi-hosts` service url is for people who isn’t able to configure a DNS or load balancer for their brokers. ---- 2019-02-24 12:31:18 UTC - Sijie Guo: replied in the question ---- 2019-02-24 12:34:35 UTC - Vincent Ngan: Yes, I understand the difference. But does it work correctly if I specify the actual broker urls directly in the client serviceUrl without going through the proxy? ---- 2019-02-24 12:38:41 UTC - Sijie Guo: proxy is confusing the discussion here. let’s remove proxy from the conversation. if you have 3 brokers, you can specify any broker’s hostname as service url. you are able to connect to the whole cluster. the service url is the entrypoint for you to “bootstrap”. but if you just directly specify the hostname of one broker, if the broker is down, you will have trouble when you reconnect. so in that case, you will need a DNS name or a load balancer for these 3 brokers. so that any broker down will not impact your client reconnecting. but if you don’t have a DNS or a load balancer, in 2.3.0, you can configure multiple hosts as the service url. client will “load balancing” the hosts in the service url. ---- 2019-02-24 12:38:55 UTC - Sijie Guo: hope this make it clear ---- 2019-02-24 12:51:37 UTC - Vincent Ngan: Thanks. I think you have answered my question. But, I actually was using proxy as a single entry point to the cluster rather than using DNS or load balancer. Does this mean that with this new `multi-hosts` feature, I can connect to any one of the brokers without the use of DNS, load-balancer or proxy? ---- 2019-02-24 12:53:05 UTC - Sijie Guo: > I actually was using proxy as a single entry point to the cluster rather than using DNS or load balancer. hmm but you only have one proxy? isn’t that proxy a single-point-of-failure? > Does this mean that with this new `multi-hosts` feature, I can connect to any one of the brokers without the use of DNS, load-balancer or proxy? yes. ---- 2019-02-24 12:56:44 UTC - Vincent Ngan: Yes, it will be a single-point-of-failure if I only have one proxy without load balancer. So, that’s why the `multi-hosts` feature sounds more appealing to me than using proxy. ---- 2019-02-24 12:59:02 UTC - Sijie Guo: > it will be a single-point-of-failure if I only have one proxy without load balancer. my point is if it is `single-point-of-failure` already in your proxy, you don’t even have to use proxy before. you can just pick any broker’s hostname as the service url. but yes, multi-hosts feature will help you ---- 2019-02-24 13:04:26 UTC - Vincent Ngan: Thanks a lot for you help ---- 2019-02-24 13:07:02 UTC - Sijie Guo: no problem :slightly_smiling_face: ---- 2019-02-24 13:14:15 UTC - Vincent Ngan: Yes, I understand what you mean now. In fact any one of the brokers can be a proxy in the way I use the proxy because any one of the brokers connects me to the whole cluster. I didn't really need to use a proxy for this purpose. ---- 2019-02-24 14:27:11 UTC - Sijie Guo: correct ---- 2019-02-24 14:50:43 UTC - Matteo Merli: @Vincent Ngan proxy is only needed when clients cannot connect directly to a specific broker, irrespective of whether the service discovery is done through dns, load balancer or multi-host list. ---- 2019-02-24 18:57:57 UTC - bhagesharora: yeah, Got it, Thanks :+1: ---- 2019-02-24 18:59:30 UTC - bhagesharora: yeah, Got it. Thanks :+1: ---- 2019-02-24 19:39:59 UTC - bhagesharora: Is there any example which help me to understand usecase of failover subscription, shared subscription or multiple consumer/producer in Java ? ---- 2019-02-24 23:44:22 UTC - Jacob O'Farrell: Upgrading to 2.3 - are there any docs / recommendations for upgrade process? ---- 2019-02-25 03:01:55 UTC - jia zhai: @bhagesharora As the doc for shared and failover subscription: <http://pulsar.apache.org/docs/en/concepts-messaging.html#shared> <http://pulsar.apache.org/docs/en/concepts-messaging.html#failover> Shared: In shared or round robin mode, multiple consumers can attach to the same subscription. Messages are delivered in a round robin distribution across consumers, and any given message is delivered to only one consumer. FailOver: In failover mode, multiple consumers can attach to the same subscription. But only one consumer(master consumer) is active and could receive messages, other consumers is standby for the failover. > multiple consumer/producer in Java what do you mean for this? ---- 2019-02-25 03:08:13 UTC - jia zhai: @Jacob O'Farrell Sorry there is no such doc currently, We will prepare one for it. You could search the channel for ‘upgrade’, there was some discussion for upgrade. ---- 2019-02-25 03:11:15 UTC - jia zhai: One thing that may need to pay attention is bookkeeper autorecovery. <http://bookkeeper.apache.org/docs/latest/admin/autorecovery/> ---- 2019-02-25 03:12:17 UTC - jia zhai: you may want to disable it before upgrade, and enable it after upgrade. ``` $ bin/bookkeeper shell autorecovery -disable $ bin/bookkeeper shell autorecovery -enable ``` ---- 2019-02-25 03:12:33 UTC - jia zhai: ``` Disable AutoRecovery You can disable AutoRecovery at any time, for example during maintenance. Disabling AutoRecovery ensures that bookies' data isn't unnecessarily rereplicated when the bookie is only taken down for a short period of time, for example when the bookie is being updated or the configuration if being changed. ``` ---- 2019-02-25 03:15:53 UTC - bossbaby: I have tried to install pulsar with tls authentication and token authentication but an error has occurred, can someone explain and help me? <https://gist.github.com/tuan6956/297b4b50e2341530cd4843180d54b51a> ---- 2019-02-25 05:21:57 UTC - Jacob O'Farrell: Hey all - loving the new Pulsar Python Client, being able to use schemas is great! However, we seem to be running into an issue when querying this data via presto, all queries are failing and I can see the following in the coordinator logs Would posting the schema as well perhaps be useful? +1 : Khoa Tran ---- 2019-02-25 05:35:52 UTC - jia zhai: @Jacob O'Farrell This looks like a leak of ByteBuf reference ---- 2019-02-25 05:36:55 UTC - jia zhai: @Sijie Guo @Jerry Peng Do you have any clue for this stack above? ---- 2019-02-25 05:37:51 UTC - Jacob O'Farrell: Submitting issue for it now as per @Sijie Guo request - will link in a second ---- 2019-02-25 05:38:01 UTC - Sijie Guo: :slightly_smiling_face: ---- 2019-02-25 05:38:25 UTC - Sijie Guo: was typing .. @Jacob O'Farrell is faster than me ---- 2019-02-25 05:38:39 UTC - jia zhai: :+1: ---- 2019-02-25 05:39:23 UTC - jia zhai: ``` 03:13:03.134 [main] ERROR org.apache.pulsar.broker.PulsarService - MultiException[java.lang.IllegalStateException: Insufficient configured threads: required=2 < max=2 for WebExecutorThreadPool[etp1799831668]@6b474074{STARTED,2<=2<=2,i=2,q=0,ReservedThreadExecutor@7a814310{s=0/1,p=0}}, java.lang.IllegalStateException: Insufficient configured threads: required=2 < max=2 for WebExecutorThreadPool[etp1799831668]@6b474074{STARTED,2<=2<=2,i=2,q=0,ReservedThreadExecutor@7a814310{s=0/1,p=0}}] ``` ---- 2019-02-25 05:39:35 UTC - jia zhai: @bossbaby It looks like a config issue ---- 2019-02-25 05:41:24 UTC - Matteo Merli: Looks more a “use after freed” than a leak ---- 2019-02-25 05:45:17 UTC - Jerry Peng: happens during close ---- 2019-02-25 05:48:25 UTC - Jerry Peng: we do another release here <https://github.com/apache/pulsar/blob/master/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarRecordCursor.java#L373> ---- 2019-02-25 05:49:00 UTC - Jerry Peng: in the close maybe we can just do a try catch in case currentMessage has already been released ---- 2019-02-25 06:05:50 UTC - Jacob O'Farrell: Issue created here: <https://github.com/apache/pulsar/issues/3679> ---- 2019-02-25 06:10:22 UTC - Matteo Merli: Can edit? ---- 2019-02-25 06:50:39 UTC - Jacob O'Farrell: Is this issue related to anything that we would be doing in particular, or is this a bigger issue with Pulsar SQL? ---- 2019-02-25 07:52:47 UTC - jia zhai: @Jacob O'Farrell as @Jerry Peng point out, it is related with current code. ---- 2019-02-25 07:54:29 UTC - Jacob O'Farrell: thanks @jia zhai ---- 2019-02-25 07:55:34 UTC - jia zhai: welcome. Thank you for the report. and thanks @Jerry Peng find the root cause. ---- 2019-02-25 07:56:39 UTC - Jacob O'Farrell: Any workarounds or similar worth trying? Or would it be expected that this issue is present there as well as 2.3 ---- 2019-02-25 08:01:22 UTC - jia zhai: right, I think it is present there in 2.3, maybe we could cut release 2.3.1 soon for the fix. @Sijie Guo @Matteo Merli @Jerry Peng ---- 2019-02-25 08:01:51 UTC - Shivji Kumar Jha: @Shivji Kumar Jha has joined the channel ---- 2019-02-25 08:57:01 UTC - bhagesharora: @jia zhai Multiple producer/consumer means, I am trying to implement one usecase in java, here we have 2 producer and 3 cosumer and producing a messages from two producer and consuming messages from 3 consumer with the same topic subscription ---- 2019-02-25 09:10:16 UTC - jia zhai: @bhagesharora 3 consumers could be with same topic subscription, and with shared mode ----
