Hi,
You should setup a metric reporter to collect Flink's metrics.
https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/metrics.html
There's a lot of useful information in the metrics, including the
consumer lags.
I'm using the Graphite reporter with InfluxDB for storage + Grafana to
display the metrics,
it's been working great so far.
Regards,
Kien
On 7/9/17 1:16 PM, Karthik Deivasigamani wrote:
Hi,
I'm using Flink 1.2.1 and FlinkKafkaConsumer09 to read data from
Kafka Server (0.10.1.0). My consumer is able to read and everything
works fine.
But when I use the kafka_consumer_groups.sh command to find the lag
metrics it does not return the lag. Upon reading online I found that
Flink Kafka Consumer uses the low level API and manages the assignment
of partition itself.
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-kafka-group-question-td8185.html
What is the best way to monitor my flink kafka consumer's lag? Does
Flink expose the lag metrics per partition somewhere?
~
Karthik
