I have a Beam Pipeline consuming records from Kafka doing some
transformations and writing it to Hbase. I faced an issue in which
records were writing to Hbase at a slower rate than the incoming
messages to Kafka due to a temporary surge in the incoming traffic.
From the flink UI, if I check the back pressure status, it shows OK. I
have one task which has all the operators including source.
Any idea why backpressure indicator would show OK, but messages are
backed up in Kafka.
Is there any other mechanism/metrics by which I can identify this
situation ?
I'm running Flink 1.2/w beam 2.0.
Thanks,
Jins George