Hi Renjie, 1. For what I could grasp from Kafka docs, you can subscribe and still use poll() to capture a specific offset. But I just read the starting point of it and didn't go deep into it.
2. Currently, Flink 1.4.2, Kafka 0.10.1 and the FlinkKafkaConsumer010. On Tue, Sep 4, 2018 at 12:47 AM, Renjie Liu <liurenjie2...@gmail.com> wrote: > Hi, Julio: > 1. Flink doesn't use subscribe because it needs to control partition > assignment itself, which is important for implementing exactly once. > 2. Can you share the versions you are using, including kafka, kafka > client, flink? We are also use flink kafka consumer and we can monitor it > correctly. > > On Tue, Sep 4, 2018 at 3:09 AM Julio Biason <julio.bia...@azion.com> > wrote: > >> Hey guys, >> >> We are trying to add external monitoring to our system, but we can only >> get the lag in kafka topics while the Flink job is running -- if, for some >> reason, the Flink job fails, we get no visibility on how big the lag is. >> >> (Besides that, the way Flink reports is not accurate and produces a lot >> of -Inf, which I already discussed before.) >> >> While looking at the problem, we noticed that the FlinkKafkaConsumer >> never uses `subscribe` to subscribe to the topics and that's why the values >> are never stored back into Kafka, even when the driver itself does >> `commitAsync`. >> >> Is there any reason for not subscribing to topics that I may have missed? >> >> -- >> *Julio Biason*, Sofware Engineer >> *AZION* | Deliver. Accelerate. Protect. >> Office: +55 51 3083 8101 <callto:+555130838101> | Mobile: +55 51 >> <callto:+5551996209291>*99907 0554* >> > -- > Liu, Renjie > Software Engineer, MVAD > -- *Julio Biason*, Sofware Engineer *AZION* | Deliver. Accelerate. Protect. Office: +55 51 3083 8101 <callto:+555130838101> | Mobile: +55 51 <callto:+5551996209291>*99907 0554*
