Yes Kafka and Flink connect to that zookeeper only. Not sure why it is not listing the consumer
Regards, Vinay Patil On Tue, Nov 8, 2016 at 5:36 PM, Daniel Santos [via Apache Flink User Mailing List archive.] <ml-node+s2336050n9982...@n4.nabble.com> wrote: > Hi, > > brokerPath is just optional. > > Used if you want to have multile kafka clusters. > > Each kafka cluster would connect to the same brokerPath. > > Since I have multiple clusters I use the brokerPath. > > From the looks of it you dont. So never mind it doesn't matter. > > You only have one zookeeper correct ? > > And kafka and flink connects to that only zookeeper ? > > Best Regards, > > Daniel Santos > > On 11/08/2016 11:18 AM, vinay patil wrote: > > Hi Daniel, > > Yes I have specified the zookeeper host in server.properties file , so the > broker is connected to zookeeper. > > https://kafka.apache.org/documentation#brokerconfigs -> according to > this link, I guess all these configs are done in server.prop , so from > where did you get kafka09 as brokerPath ? > > this is my entry in server.prop file -> zookeeper.connect=localhost:2181 > Have you set this as zkhost:2181/kafka09 ? > > > Regards, > Vinay Patil > > On Tue, Nov 8, 2016 at 4:27 PM, Daniel Santos [via Apache Flink User > Mailing List archive.] <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=9980&i=0>> wrote: > >> Hi, >> >> Your kafka broker is connected to zookeeper I believe. >> >> I am using kafka 0.9.0.1 my self too. >> >> On kafka broker 0.9.0.1 I have configured the zookeeper connect to a >> path, for instances : >> >> zk1:2181,zk2:2181,zk3:2181/kafka09 >> >> https://kafka.apache.org/documentation#brokerconfigs >> >> Now on the flink side I would configure >> "props.setProperty("zookeeper.connect", >> zkHosts)" the same resulting in : >> >> >> props.setProperty("zookeeperconnect", "zk1:2181,zk2:2181,zk3:2181/ka >> fka09") >> >> >> That is what I mean by broker's path. >> >> Best Regards, >> >> Daniel Santos >> >> On 11/08/2016 10:49 AM, vinay patil wrote: >> >> Hi Daniel, >> >> I have the same properties set for the consumer and the same code >> >> *brokerspath only needed if you have set it on kafka config* -> I did >> not get this, do you mean to check the brokerspath in >> conf/server.properties file ? I have even tried by setting offset.storage >> property to zookeeeper, but still not getting the consumers listed >> >> I am using Kafka 0.9.0.1 >> >> >> Regards, >> Vinay Patil >> >> On Tue, Nov 8, 2016 at 3:59 PM, Daniel Santos [via Apache Flink User >> Mailing List archive.] <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=9978&i=0>> wrote: >> >>> Hello, >>> >>> This is my config. >>> >>> On kafka props : >>> >>> val props = new Properties() >>> >>> props.setProperty("zookeeper.connect", zkHosts) >>> props.setProperty("bootstrap.servers", kafHosts) >>> props.setProperty("group.id", "prod") >>> props.setProperty("auto.offset.reset", "earliest") >>> >>> Now for zkHosts beware that all your hosts quorum has to be included. >>> >>> For instances you have zk1 and zk2 and zk3 to form a quorum. >>> >>> Then it will result in zkHosts being -> >>> zk1:2181,zk2:2181,zk3:2181/[brokerspath] >>> . >>> >>> brokerspath only needed if you have set it on kafka config. Ignore it >>> otherwise, resulting in "zk1:2181,zk2:2181,zk3:2181" . >>> >>> After that -> val source = env.addSource(new >>> FlinkKafkaConsumer09[String](KAFKA_TOPIC, new SimpleStringSchema(), >>> props)) >>> >>> Then on kafkamanager -> consumers I have the groupID prod. >>> >>> Hope it helps. >>> >>> Best Regards, >>> >>> Daniel Santos >>> On 11/08/2016 08:45 AM, vinay patil wrote: >>> >>> Hi Limbo, >>> >>> I am using 0.9, I am not able to see updated results even after >>> refreshing. >>> There is some property that we have to set in order to make this work >>> >>> Regards, >>> Vinay Patil >>> >>> On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache Flink User Mailing >>> List archive.] <[hidden email] >>> <http:///user/SendEmail.jtp?type=node&node=9969&i=0>> wrote: >>> >>>> I am using kafka 0.8, just refresh the page and you will see the >>>> updated results, it’s not auto update. >>>> >>>> There is our logstash consumer: >>>> >>>> >>>> >>>> and this is our flink consumer: >>>> >>>> >>>> I find the flink consumer just write the offset of the kafka partition >>>> to zookeeper without owner and ids, >>>> so we can’t find the consumer in the manager page, we can only find the >>>> offset info. >>>> >>>> 在 2016年11月8日，下午12:02，vinay patil <[hidden email] >>>> <http:///user/SendEmail.jtp?type=node&node=9965&i=0>> 写道： >>>> >>>> Hi Limbo, >>>> >>>> I can see the lag by using that URL, but the Lag there is not showing >>>> updated results, it does not change, also if you try to to change the >>>> consumer group value it will still show you the same value instead of >>>> saying consumer group does not exist or similar kind of error :) >>>> >>>> According to documentation of 0.9.x the offsets are stored in Kafka, >>>> but we can set offset.storage property to zookeeper. >>>> Even by setting this I am not getting the consumer listed. >>>> >>>> Kafka cli command also does not show this consumer >>>> >>>> Regards, >>>> Vinay Patil >>>> >>>> On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink User Mailing >>>> List archive.] <<a href="x-msg://4/user/SendEmail >>>> .jtp?type=node&node=9964&i=0" target="_top" rel="nofollow" >>>> link="external" class="">[hidden email]> wrote: >>>> >>>>> Hi, >>>>> >>>>> I have the same problem, I think the reason is that the consumer of >>>>> flink use the low level API, >>>>> and when I type the group name in manager url I can get the lag of the >>>>> flink consumer, like this: >>>>> >>>>> http://your_manager_url/clusters/<kafka_name>/consumers/<con >>>>> sumer_name>/topic/<topic_name>/type/ZK >>>>> >>>>> 在 2016年11月8日，上午5:12，Daniel Santos <[hidden email] >>>>> <http://user/SendEmail.jtp?type=node&node=9962&i=0>> 写道： >>>>> >>>>> Hello, >>>>> >>>>> I have been using that setup. >>>>> From my understanding, if one desires to see the offset being consumed >>>>> by Flink on KafkaManger, one has to set it up with zookeeper. On 0.9 it >>>>> will only serve as a view of progress. >>>>> >>>>> Basically what's mandatory on 0.8 is optional on 0.9, and for viewing >>>>> purposes only. >>>>> >>>>> Best Regards, >>>>> Daniel Santos >>>>> >>>>> On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil <[hidden email] >>>>> <http://user/SendEmail.jtp?type=node&node=9962&i=1>> wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> I am monitoring Kafka using KafkaManager for checking offset lag and >>>>>> other Kafka metrics, however I am not able to see the consumers when I >>>>>> use >>>>>> FlinkKafkaConsumer , for console-consumer it shows them in the Consumers >>>>>> list. >>>>>> >>>>>> I have set the required parameters for the kafka consumer while >>>>>> running the application. >>>>>> >>>>>> Has anyone faced this issue ? >>>>>> I am using Kafka 0.9.0.1 >>>>>> >>>>>> Regards, >>>>>> Vinay Patil >>>>>> >>>>> >>>>> >>>>> >>>>> ------------------------------ >>>>> If you reply to this email, your message will be added to the >>>>> discussion below: >>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab >>>>> ble.com/Kafka-Monitoring-tp9957p9962.html >>>>> To start a new topic under Apache Flink User Mailing List archive., >>>>> email <a >>>>> href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=1" >>>>> target="_top" rel="nofollow" link="external" class="">[hidden email] >>>>> To unsubscribe from Apache Flink User Mailing List archive., click >>>>> here. >>>>> NAML >>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>>>> >>>> >>>> >>>> ------------------------------ >>>> View this message in context: Re: Kafka Monitoring >>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html> >>>> Sent from the Apache Flink User Mailing List archive. mailing list >>>> archive >>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> >>>> at Nabble.com <http://nabble.com/>. >>>> >>>> >>>> >>>> >>>> ------------------------------ >>>> If you reply to this email, your message will be added to the >>>> discussion below: >>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab >>>> ble.com/Kafka-Monitoring-tp9957p9965.html >>>> To start a new topic under Apache Flink User Mailing List archive., >>>> email [hidden email] >>>> <http:///user/SendEmail.jtp?type=node&node=9969&i=1> >>>> To unsubscribe from Apache Flink User Mailing List archive., click here >>>> . >>>> NAML >>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>>> >>> >>> >>> ------------------------------ >>> View this message in context: Re: Kafka Monitoring >>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9969.html> >>> Sent from the Apache Flink User Mailing List archive. mailing list >>> archive >>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> >>> at Nabble.com. >>> >>> >>> >>> >>> ------------------------------ >>> If you reply to this email, your message will be added to the discussion >>> below: >>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab >>> ble.com/Kafka-Monitoring-tp9957p9976.html >>> To start a new topic under Apache Flink User Mailing List archive., >>> email [hidden email] >>> <http:///user/SendEmail.jtp?type=node&node=9978&i=1> >>> To unsubscribe from Apache Flink User Mailing List archive., click here. >>> NAML >>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>> >> >> >> ------------------------------ >> View this message in context: Re: Kafka Monitoring >> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9978.html> >> Sent from the Apache Flink User Mailing List archive. mailing list >> archive >> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> >> at Nabble.com. >> >> >> >> >> ------------------------------ >> If you reply to this email, your message will be added to the discussion >> below: >> http://apache-flink-user-mailing-list-archive.2336050.n4. >> nabble.com/Kafka-Monitoring-tp9957p9979.html >> To start a new topic under Apache Flink User Mailing List archive., email >> [hidden >> email] <http:///user/SendEmail.jtp?type=node&node=9980&i=1> >> To unsubscribe from Apache Flink User Mailing List archive., click here. >> NAML >> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> > > > ------------------------------ > View this message in context: Re: Kafka Monitoring > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9980.html> > Sent from the Apache Flink User Mailing List archive. mailing list archive > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at > Nabble.com. > > > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka- > Monitoring-tp9957p9982.html > To start a new topic under Apache Flink User Mailing List archive., email > ml-node+s2336050n1...@n4.nabble.com > To unsubscribe from Apache Flink User Mailing List archive., click here > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx> > . > NAML > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9983.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
