Hi Daniel, Yes Now I am able to see it , this was just a dummy code I was running on local VM.
However on cluster , I had enabled checkpointing, still I was not able to see the consumers, I guess I have to put the brokerPath as you have provided in the properties. >From where did you get the brokerPath ? I mean for which property have you set it Regards, Vinay Patil On Wed, Nov 9, 2016 at 12:32 AM, Daniel Santos [via Apache Flink User Mailing List archive.] <ml-node+s2336050n9996...@n4.nabble.com> wrote: > Hello, > > On flink do you have the checkpoint enabled ? > > env.enableCheckpointing(interval = CHKPOINT_INTERVAL) > > Regards, > > Daniel Santos > > On 11/08/2016 12:30 PM, vinay patil wrote: > > Yes Kafka and Flink connect to that zookeeper only. > > Not sure why it is not listing the consumer > > Regards, > Vinay Patil > > On Tue, Nov 8, 2016 at 5:36 PM, Daniel Santos [via Apache Flink User > Mailing List archive.] <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=9983&i=0>> wrote: > >> Hi, >> >> brokerPath is just optional. >> >> Used if you want to have multile kafka clusters. >> >> Each kafka cluster would connect to the same brokerPath. >> >> Since I have multiple clusters I use the brokerPath. >> >> From the looks of it you dont. So never mind it doesn't matter. >> >> You only have one zookeeper correct ? >> >> And kafka and flink connects to that only zookeeper ? >> >> Best Regards, >> >> Daniel Santos >> >> On 11/08/2016 11:18 AM, vinay patil wrote: >> >> Hi Daniel, >> >> Yes I have specified the zookeeper host in server.properties file , so >> the broker is connected to zookeeper. >> >> https://kafka.apache.org/documentation#brokerconfigs -> according to >> this link, I guess all these configs are done in server.prop , so from >> where did you get kafka09 as brokerPath ? >> >> this is my entry in server.prop file -> zookeeper.connect=localhost:2181 >> Have you set this as zkhost:2181/kafka09 ? >> >> >> Regards, >> Vinay Patil >> >> On Tue, Nov 8, 2016 at 4:27 PM, Daniel Santos [via Apache Flink User >> Mailing List archive.] <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=9980&i=0>> wrote: >> >>> Hi, >>> >>> Your kafka broker is connected to zookeeper I believe. >>> >>> I am using kafka 0.9.0.1 my self too. >>> >>> On kafka broker 0.9.0.1 I have configured the zookeeper connect to a >>> path, for instances : >>> >>> zk1:2181,zk2:2181,zk3:2181/kafka09 >>> >>> https://kafka.apache.org/documentation#brokerconfigs >>> >>> Now on the flink side I would configure >>> "props.setProperty("zookeeper.connect", >>> zkHosts)" the same resulting in : >>> >>> >>> props.setProperty("zookeeperconnect", "zk1:2181,zk2:2181,zk3:2181/ka >>> fka09") >>> >>> >>> That is what I mean by broker's path. >>> >>> Best Regards, >>> >>> Daniel Santos >>> >>> On 11/08/2016 10:49 AM, vinay patil wrote: >>> >>> Hi Daniel, >>> >>> I have the same properties set for the consumer and the same code >>> >>> *brokerspath only needed if you have set it on kafka config* -> I did >>> not get this, do you mean to check the brokerspath in >>> conf/server.properties file ? I have even tried by setting offset.storage >>> property to zookeeeper, but still not getting the consumers listed >>> >>> I am using Kafka 0.9.0.1 >>> >>> >>> Regards, >>> Vinay Patil >>> >>> On Tue, Nov 8, 2016 at 3:59 PM, Daniel Santos [via Apache Flink User >>> Mailing List archive.] <[hidden email] >>> <http:///user/SendEmail.jtp?type=node&node=9978&i=0>> wrote: >>> >>>> Hello, >>>> >>>> This is my config. >>>> >>>> On kafka props : >>>> >>>> val props = new Properties() >>>> >>>> props.setProperty("zookeeper.connect", zkHosts) >>>> props.setProperty("bootstrap.servers", kafHosts) >>>> props.setProperty("group.id", "prod") >>>> props.setProperty("auto.offset.reset", "earliest") >>>> >>>> Now for zkHosts beware that all your hosts quorum has to be included. >>>> >>>> For instances you have zk1 and zk2 and zk3 to form a quorum. >>>> >>>> Then it will result in zkHosts being -> >>>> zk1:2181,zk2:2181,zk3:2181/[brokerspath] >>>> . >>>> >>>> brokerspath only needed if you have set it on kafka config. Ignore it >>>> otherwise, resulting in "zk1:2181,zk2:2181,zk3:2181" . >>>> >>>> After that -> val source = env.addSource(new >>>> FlinkKafkaConsumer09[String](KAFKA_TOPIC, new SimpleStringSchema(), >>>> props)) >>>> >>>> Then on kafkamanager -> consumers I have the groupID prod. >>>> >>>> Hope it helps. >>>> >>>> Best Regards, >>>> >>>> Daniel Santos >>>> On 11/08/2016 08:45 AM, vinay patil wrote: >>>> >>>> Hi Limbo, >>>> >>>> I am using 0.9, I am not able to see updated results even after >>>> refreshing. >>>> There is some property that we have to set in order to make this work >>>> >>>> Regards, >>>> Vinay Patil >>>> >>>> On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache Flink User Mailing >>>> List archive.] <[hidden email] >>>> <http:///user/SendEmail.jtp?type=node&node=9969&i=0>> wrote: >>>> >>>>> I am using kafka 0.8, just refresh the page and you will see the >>>>> updated results, it’s not auto update. >>>>> >>>>> There is our logstash consumer: >>>>> >>>>> >>>>> >>>>> and this is our flink consumer: >>>>> >>>>> >>>>> I find the flink consumer just write the offset of the kafka partition >>>>> to zookeeper without owner and ids, >>>>> so we can’t find the consumer in the manager page, we can only find >>>>> the offset info. >>>>> >>>>> 在 2016年11月8日，下午12:02，vinay patil <[hidden email] >>>>> <http:///user/SendEmail.jtp?type=node&node=9965&i=0>> 写道： >>>>> >>>>> Hi Limbo, >>>>> >>>>> I can see the lag by using that URL, but the Lag there is not showing >>>>> updated results, it does not change, also if you try to to change the >>>>> consumer group value it will still show you the same value instead of >>>>> saying consumer group does not exist or similar kind of error :) >>>>> >>>>> According to documentation of 0.9.x the offsets are stored in Kafka, >>>>> but we can set offset.storage property to zookeeper. >>>>> Even by setting this I am not getting the consumer listed. >>>>> >>>>> Kafka cli command also does not show this consumer >>>>> >>>>> Regards, >>>>> Vinay Patil >>>>> >>>>> On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink User Mailing >>>>> List archive.] <<a href="x-msg://4/user/SendEmail >>>>> .jtp?type=node&node=9964&i=0" target="_top" rel="nofollow" >>>>> link="external" class="">[hidden email]> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I have the same problem, I think the reason is that the consumer of >>>>>> flink use the low level API, >>>>>> and when I type the group name in manager url I can get the lag of >>>>>> the flink consumer, like this: >>>>>> >>>>>> http://your_manager_url/clusters/<kafka_name>/consumers/<con >>>>>> sumer_name>/topic/<topic_name>/type/ZK >>>>>> >>>>>> 在 2016年11月8日，上午5:12，Daniel Santos <[hidden email] >>>>>> <http://user/SendEmail.jtp?type=node&node=9962&i=0>> 写道： >>>>>> >>>>>> Hello, >>>>>> >>>>>> I have been using that setup. >>>>>> From my understanding, if one desires to see the offset being >>>>>> consumed by Flink on KafkaManger, one has to set it up with zookeeper. On >>>>>> 0.9 it will only serve as a view of progress. >>>>>> >>>>>> Basically what's mandatory on 0.8 is optional on 0.9, and for viewing >>>>>> purposes only. >>>>>> >>>>>> Best Regards, >>>>>> Daniel Santos >>>>>> >>>>>> On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil <[hidden email] >>>>>> <http://user/SendEmail.jtp?type=node&node=9962&i=1>> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am monitoring Kafka using KafkaManager for checking offset lag and >>>>>>> other Kafka metrics, however I am not able to see the consumers when I >>>>>>> use >>>>>>> FlinkKafkaConsumer , for console-consumer it shows them in the Consumers >>>>>>> list. >>>>>>> >>>>>>> I have set the required parameters for the kafka consumer while >>>>>>> running the application. >>>>>>> >>>>>>> Has anyone faced this issue ? >>>>>>> I am using Kafka 0.9.0.1 >>>>>>> >>>>>>> Regards, >>>>>>> Vinay Patil >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> ------------------------------ >>>>>> If you reply to this email, your message will be added to the >>>>>> discussion below: >>>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab >>>>>> ble.com/Kafka-Monitoring-tp9957p9962.html >>>>>> To start a new topic under Apache Flink User Mailing List archive., >>>>>> email <a >>>>>> href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=1" >>>>>> target="_top" rel="nofollow" link="external" class="">[hidden email] >>>>>> To unsubscribe from Apache Flink User Mailing List archive., click >>>>>> here. >>>>>> NAML >>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>>>>> >>>>> >>>>> >>>>> ------------------------------ >>>>> View this message in context: Re: Kafka Monitoring >>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html> >>>>> Sent from the Apache Flink User Mailing List archive. mailing list >>>>> archive >>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> >>>>> at Nabble.com <http://nabble.com/>. >>>>> >>>>> >>>>> >>>>> >>>>> ------------------------------ >>>>> If you reply to this email, your message will be added to the >>>>> discussion below: >>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab >>>>> ble.com/Kafka-Monitoring-tp9957p9965.html >>>>> To start a new topic under Apache Flink User Mailing List archive., >>>>> email [hidden email] >>>>> <http:///user/SendEmail.jtp?type=node&node=9969&i=1> >>>>> To unsubscribe from Apache Flink User Mailing List archive., click >>>>> here. >>>>> NAML >>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>>>> >>>> >>>> >>>> ------------------------------ >>>> View this message in context: Re: Kafka Monitoring >>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9969.html> >>>> Sent from the Apache Flink User Mailing List archive. mailing list >>>> archive >>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> >>>> at Nabble.com. >>>> >>>> >>>> >>>> >>>> ------------------------------ >>>> If you reply to this email, your message will be added to the >>>> discussion below: >>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab >>>> ble.com/Kafka-Monitoring-tp9957p9976.html >>>> To start a new topic under Apache Flink User Mailing List archive., >>>> email [hidden email] >>>> <http:///user/SendEmail.jtp?type=node&node=9978&i=1> >>>> To unsubscribe from Apache Flink User Mailing List archive., click here >>>> . >>>> NAML >>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>>> >>> >>> >>> ------------------------------ >>> View this message in context: Re: Kafka Monitoring >>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9978.html> >>> Sent from the Apache Flink User Mailing List archive. mailing list >>> archive >>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> >>> at Nabble.com. >>> >>> >>> >>> >>> ------------------------------ >>> If you reply to this email, your message will be added to the discussion >>> below: >>> http://apache-flink-user-mailing-list-archive.2336050.n4.nab >>> ble.com/Kafka-Monitoring-tp9957p9979.html >>> To start a new topic under Apache Flink User Mailing List archive., >>> email [hidden email] >>> <http:///user/SendEmail.jtp?type=node&node=9980&i=1> >>> To unsubscribe from Apache Flink User Mailing List archive., click here. >>> NAML >>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>> >> >> >> ------------------------------ >> View this message in context: Re: Kafka Monitoring >> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9980.html> >> Sent from the Apache Flink User Mailing List archive. mailing list >> archive >> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> >> at Nabble.com. >> >> >> >> >> ------------------------------ >> If you reply to this email, your message will be added to the discussion >> below: >> http://apache-flink-user-mailing-list-archive.2336050.n4. >> nabble.com/Kafka-Monitoring-tp9957p9982.html >> To start a new topic under Apache Flink User Mailing List archive., email >> [hidden >> email] <http:///user/SendEmail.jtp?type=node&node=9983&i=1> >> To unsubscribe from Apache Flink User Mailing List archive., click here. >> NAML >> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> > > > ------------------------------ > View this message in context: Re: Kafka Monitoring > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9983.html> > Sent from the Apache Flink User Mailing List archive. mailing list archive > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at > Nabble.com. > > > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka- > Monitoring-tp9957p9996.html > To start a new topic under Apache Flink User Mailing List archive., email > ml-node+s2336050n1...@n4.nabble.com > To unsubscribe from Apache Flink User Mailing List archive., click here > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx> > . > NAML > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p10006.html Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
