Hey Jins, our current back pressure tracking mechanism does not work with Kafka sources. To gather back pressure indicators we sample the main task thread of a subtask. For most tasks, this is the thread that emits records downstream (e.g. if you have a map function) and everything works as expected. In case of the Kafka source though there is a separate thread that consumes from Kafka and emits the records. Therefore we sample the "wrong" thread and don't observe any indicators for back pressure. :-( Unfortunately, this was not taking into account when back pressure sampling was implemented.
There is this old issue to track this: https://issues.apache.org/jira/browse/FLINK-3456 I'm not aware of any other way to track this situation. Maybe others can chime in here... – Ufuk On Mon, Jan 8, 2018 at 8:16 AM, Jins George <jins.geo...@aeris.net> wrote: > I have a Beam Pipeline consuming records from Kafka doing some > transformations and writing it to Hbase. I faced an issue in which records > were writing to Hbase at a slower rate than the incoming messages to Kafka > due to a temporary surge in the incoming traffic. > > From the flink UI, if I check the back pressure status, it shows OK. I have > one task which has all the operators including source. > > Any idea why backpressure indicator would show OK, but messages are backed > up in Kafka. > > Is there any other mechanism/metrics by which I can identify this situation > ? > > I'm running Flink 1.2/w beam 2.0. > > Thanks, > Jins George
