Hello,
This is my config.
On kafka props :
val props = new Properties()
props.setProperty("zookeeper.connect", zkHosts)
props.setProperty("bootstrap.servers", kafHosts)
props.setProperty("group.id", "prod")
props.setProperty("auto.offset.reset", "earliest")
Now for zkHosts beware that all your hosts quorum has to be included.
For instances you have zk1 and zk2 and zk3 to form a quorum.
Then it will result in zkHosts being ->
zk1:2181,zk2:2181,zk3:2181/[brokerspath] .
brokerspath only needed if you have set it on kafka config. Ignore it
otherwise, resulting in "zk1:2181,zk2:2181,zk3:2181" .
After that -> val source = env.addSource(new
FlinkKafkaConsumer09[String](KAFKA_TOPIC, new SimpleStringSchema(), props))
Then on kafkamanager -> consumers I have the groupID prod.
Hope it helps.
Best Regards,
Daniel Santos
On 11/08/2016 08:45 AM, vinay patil wrote:
Hi Limbo,
I am using 0.9, I am not able to see updated results even after
refreshing.
There is some property that we have to set in order to make this work
Regards,
Vinay Patil
On Tue, Nov 8, 2016 at 12:32 PM, limbo [via Apache Flink User Mailing
List archive.] <[hidden email]
</user/SendEmail.jtp?type=node&node=9969&i=0>> wrote:
I am using kafka 0.8, just refresh the page and you will see the
updated results, it’s not auto update.
There is our logstash consumer:
and this is our flink consumer:
I find the flink consumer just write the offset of the kafka
partition to zookeeper without owner and ids,
so we can’t find the consumer in the manager page, we can only
find the offset info.
在 2016年11月8日，下午12:02，vinay patil <[hidden email]
<http:///user/SendEmail.jtp?type=node&node=9965&i=0>> 写道：
Hi Limbo,
I can see the lag by using that URL, but the Lag there is not
showing updated results, it does not change, also if you try to
to change the consumer group value it will still show you the
same value instead of saying consumer group does not exist or
similar kind of error :)
According to documentation of 0.9.x the offsets are stored in
Kafka, but we can set offset.storage property to zookeeper.
Even by setting this I am not getting the consumer listed.
Kafka cli command also does not show this consumer
Regards,
Vinay Patil
On Tue, Nov 8, 2016 at 8:56 AM, limbo [via Apache Flink User
Mailing List archive.]<<a
href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=0"
target="_top" rel="nofollow" link="external" class="">[hidden
email]>wrote:
Hi,
I have the same problem, I think the reason is that the
consumer of flink use the low level API,
and when I type the group name in manager url I can get the
lag of the flink consumer, like this:
http://your_manager_url/clusters/<kafka_name>/consumers/<consumer_name>/topic/<topic_name>/type/ZK
<http://your_manager_url/clusters/%3Ckafka_name%3E/consumers/%3Cconsumer_name%3E/topic/%3Ctopic_name%3E/type/ZK>
在 2016年11月8日，上午5:12，Daniel Santos <[hidden email]
<http://user/SendEmail.jtp?type=node&node=9962&i=0>> 写道：
Hello,
I have been using that setup.
From my understanding, if one desires to see the offset
being consumed by Flink on KafkaManger, one has to set it up
with zookeeper. On 0.9 it will only serve as a view of progress.
Basically what's mandatory on 0.8 is optional on 0.9, and
for viewing purposes only.
Best Regards,
Daniel Santos
On November 7, 2016 7:13:54 PM GMT+00:00, Vinay Patil
<[hidden email]
<http://user/SendEmail.jtp?type=node&node=9962&i=1>> wrote:
Hi,
I am monitoring Kafka using KafkaManager for checking
offset lag and other Kafka metrics, however I am not
able to see the consumers when I use FlinkKafkaConsumer
, for console-consumer it shows them in the Consumers list.
I have set the required parameters for the kafka
consumer while running the application.
Has anyone faced this issue ?
I am using Kafka 0.9.0.1
Regards,
Vinay Patil
------------------------------------------------------------------------
If you reply to this email, your message will be added to the
discussion below:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9962.html>
To start a new topic under Apache Flink User Mailing List
archive., email<a
href="x-msg://4/user/SendEmail.jtp?type=node&node=9964&i=1"
target="_top" rel="nofollow" link="external" class="">[hidden
email]
To unsubscribe from Apache Flink User Mailing List
archive.,click here.
NAML
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
------------------------------------------------------------------------
View this message in context:Re: Kafka Monitoring
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9964.html>
Sent from theApache Flink User Mailing List archive. mailing list
archive
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>atNabble.com
<http://nabble.com/>.
------------------------------------------------------------------------
If you reply to this email, your message will be added to the
discussion below:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9965.html
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9965.html>
To start a new topic under Apache Flink User Mailing List
archive., email [hidden email]
</user/SendEmail.jtp?type=node&node=9969&i=1>
To unsubscribe from Apache Flink User Mailing List archive., click
here.
NAML
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
------------------------------------------------------------------------
View this message in context: Re: Kafka Monitoring
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Monitoring-tp9957p9969.html>
Sent from the Apache Flink User Mailing List archive. mailing list
archive
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/>
at Nabble.com.
