Re: Kafka metrics - Requests.

2023-02-24 Thread Luke Chen
Hi David, It did look like a bug. Could you file a bug in JIRA? And if you have time, welcome to investigate and submit PR for it. (My guess is there are some internal topics are included in `TotalFetchRequestsPerSec`, but not included in another one, but not sure) Thank you. Luke On Sat, Feb 25

Re: Kafka metrics - Requests.

2023-02-24 Thread David Ballano Fernandez
Hi Pere, Thanks for replying. so what i can see is: *kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec* and *kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce}* show the same/close number of producer requests and that makes sense. but that is not the sam

Re: Monitoring Kafka

2023-02-24 Thread lirop kaykov
I have a system which gathers logs from several clients. The client logs are produced with a key specifying which client it is. That way processing logs is guaranteed to be ordered by time per user. Getting metrics per the topic is good and all, but getting metrics per key allows you to analyze how

Re: Kafka for IoT ingestion pipeline

2023-02-24 Thread Pere Urbón Bayes
Hi Paolo, for use cases, high-level view you can read https://www.confluent.io/blog/iot-streaming-use-cases-with-kafka-mqtt-confluent-and-waterstream/ (disclaimer, this is marketing material from a vendor, but would be a good high level view of real cases). from your other question: > I'm wonde

Re: Monitoring Kafka

2023-02-24 Thread Pere Urbón Bayes
Hi Iirop, question: > If not, what is the best way to know how many messages with a specific key got inside a topic? why do require this information? is this a business metric? or are you looking at it from an operational point of view? I'm not aware of any metric that will give you the per-k

Monitoring Kafka

2023-02-24 Thread lirop kaykov
Hi, I have a question about monitoring kafka. When using jmx exporter and kafka exporter, you get key metrics like topic metrics (bytes in per second, messages in per second etc). Is there a way to get bytes/messages per second per message key? Meaning, getting the distribution of messages by key.

Re: kafka-producer-perf-test.sh maximum throughput limits for broker on single node

2023-02-24 Thread Luke Chen
Hi Tomasz, There are some configurations needed to be tuned in producer side. Try searching kafka producer tuning, you should find many good articles describing it. Good luck. Luke On Fri, Feb 24, 2023 at 6:16 AM Tomasz Sąciński < tomasz.sacin...@linuxpolska.pl> wrote: > Hello users. > > I tes