Re: Tracking topic consumers

2021-10-07 Thread Murilo Tavares
Oh cool. I had the wrong impression that ACL would not control read/writes, but just topic operations like CREATE and DELETE. Thanks for pointing that out. On Thu, 7 Oct 2021 at 16:09, Boyang Chen wrote: > Thanks, for standalone consumers I do not have any useful monitoring tool > on top of my

Re: Tracking topic consumers

2021-10-07 Thread Boyang Chen
Thanks, for standalone consumers I do not have any useful monitoring tool on top of my head, maybe this KIP looks promising to you, but it's still WIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability Also regarding ACL setup, I think that's also a good

Re: Tracking topic consumers

2021-10-07 Thread Murilo Tavares
Hi Boyang Thanks for your response. Yes, I'm talking about that framework. goka has a few different things. What they call `views` are similar to Global KTables in KafkaStreams. It's a full snapshot of the Kafka topic, held in memory. So every time you spin up your service, the goka view will read

Re: Tracking topic consumers

2021-10-07 Thread Boyang Chen
Hey Murilo, could you explain what you mean by `goka views`? If you are talking about https://github.com/lovoo/goka, they should use consumer groups as well IIUC. Boyang On Thu, Oct 7, 2021 at 11:55 AM Murilo Tavares wrote: > Hi. Looking for some insights here. > We use Kafka at a large scale,

Tracking topic consumers

2021-10-07 Thread Murilo Tavares
Hi. Looking for some insights here. We use Kafka at a large scale, and have lots of microservices using Kafka for all sorts of things. Our biggest challenge nowadays is to track which topics are used and which are not. I have considered looking at consumer groups to identify which applications