IIRC, we don’t expose any Kafka Consumer metrics, so I’m afraid, that there is no easy way to get them in a Beam pipeline.
— Alexey > On 18 Jan 2023, at 21:43, Lydian <[email protected]> wrote: > > Hi, > I know that Beam KafkaIO doesn't use the native kafka offset, and therefore I > cannot use kafka metrics directly. > > Wondering what would be the right way to expose those metrics of my KafkaIO > pipeline? > Things I am interested includes: > bytes-consumed-rate > fetch-latency-avg > records-lag > commit-rate > consumer lagWondering how people get these metrics or instead of doing this? > or we should just enable `commit_offset_in_finalize` and then use the Kafka > metrics directly? > > also wondering if there's anything to notice when enabling the > commit_offset_in_finalize? Thanks! > > Sincerely, > Lydian Lee >
