The latest Flink has consumers for Kafka 0.8, 0.9, 0.10 - which one are you using?
I would assume you use Flink with Kafka 0.8.x, because as far as I know, starting from Kafka 0.9, offsets are not handled by ZooKeeper any more... On Mon, Apr 24, 2017 at 12:18 AM, Meghashyam Sandeep V < vr1meghash...@gmail.com> wrote: > Hi All, > > Sorry for the miscommunication. I'm not using 0.8. I'm using latest > available flink-kafka client. I don't see my app registered as a consumer > group. I wanted to know if there is a way to monitor Kafka offsets. > > Thanks, > Sandeep > > On Apr 23, 2017 9:38 AM, "Stephan Ewen" <se...@apache.org> wrote: > >> Since it is something special to Kafka 0.8, it could be implemented in a >> simple addition to the ZooKeeperOffsetHandler used by the >> FlinkKafkaConsumer08. >> >> Would you be willing to contribute this? That would certainly help >> speeding up the resolution of the issue... >> >> >> On Fri, Apr 21, 2017 at 2:33 AM, Tzu-Li (Gordon) Tai <tzuli...@apache.org >> > wrote: >> >>> One additional note: >>> >>> In FlinkKafkaConsumer 0.9+, the current read offset should already exist >>> in Flink metrics. >>> See https://issues.apache.org/jira/browse/FLINK-4186. >>> >>> But yes, this is still missing for 0.8, so you need to directly query ZK >>> for this. >>> >>> Cheers, >>> Gordon >>> >>> >>> On 21 April 2017 at 8:28:09 AM, Tzu-Li (Gordon) Tai (tzuli...@apache.org) >>> wrote: >>> >>> Hi Sandeep, >>> >>> It isn’t fixed yet, so I think external tools like the Kafka offset >>> checker still won’t work. >>> If you’re using 08 and is currently stuck with this issue, you can still >>> directly query ZK to get the offsets. >>> >>> I think for FlinkKafkaConsumer09 the offset is exposed to Flink's metric >>> system using Kafka’s own returned metrics, but for 08 this is still missing. >>> >>> There is this JIRA [1] that aims at exposing consumer lag across all >>> Kafka consumer versions to Flink metrics. Perhaps it would make sense to >>> also generally expose the offset for all Kafka consumer versions to Flink >>> metrics as well. >>> >>> - Gordon >>> >>> [1] https://issues.apache.org/jira/browse/FLINK-6109 >>> >>> >>> On 19 April 2017 at 5:11:11 AM, sandeep6 (vr1meghash...@gmail.com) >>> wrote: >>> >>> Is this fixed now? If not, is there any way to monitor kafka offset that >>> is >>> being processed by Flink? This should be a use case for everyone who uses >>> Flink with Kafka. >>> >>> >>> >>> -- >>> View this message in context: http://apache-flink-user-maili >>> ng-list-archive.2336050.n4.nabble.com/Flink-Kafka-Consumer- >>> Behaviour-tp8257p12663.html >>> Sent from the Apache Flink User Mailing List archive. mailing list >>> archive at Nabble.com. >>> >>> >>
