I have a Beam Pipeline consuming records from Kafka doing some transformations and writing it to Hbase. I faced an issue in which records were writing to Hbase at a slower rate than the incoming messages to Kafka due to a temporary surge in the incoming traffic.

From the flink UI, if I check the back pressure status, it shows OK. I have one task which has all the operators including source.

Any idea why backpressure indicator would show OK, but messages are backed up in Kafka.

Is there any other mechanism/metrics by which I can identify this situation ?

I'm running Flink 1.2/w beam 2.0.

Thanks,
Jins George

Reply via email to