Re: Best way for reading all messages and close

Specifically, you can monitor the "records-lag-max" ( https://docs.confluent.io/current/kafka/monitoring.html#fetch-metrics) metric. (or the more granular one per partition). Once this metric goes to 0, you know that you've caught up with the tail of the log. Hope this helps, -John On Fri, Sep

Re: Best way for reading all messages and close

Using Kafka Streams this is a little tricky. The API itself has no built-in mechanism to do this. You would need to monitor the lag of the application, and if the lag is zero (assuming you don't write new data into the topic in parallel), terminate the application. -Matthias On 9/14/18 4:19

Re: Understanding default.deserialization.exception.handler

Your observation is correct. It's a known bug: https://issues.apache.org/jira/browse/KAFKA-6502 In practice, it should not be a big issue though. - you would only hit this bug if you don't process a "good message" afterwards - even if you hit this bug, you would just skip the message again

Re: How many consumers can subscribe to the same topic

- With same consumer group, you can't have consumers more than the number of partitions. Even if you do, extra consumers are just idle. Each partition can be read only by one consumer. - With different consumer groups, you can have as many consumer groups as you want for a single

Kafka UI

Which are best/recommended user interfaces for kafka to see topics, brokers, partitions, consumers etc - preferably one which can be deployed without docker? It must be open source and must work with Kafka 2.0+ Thanks in advance.

How many consumers can subscribe to the same topic

Hi y’all, I’m pretty new to Kafka and wondering how many consumers I can have on a topic. Can I have 500+ users subscribe to particular topic ? — Bobai (https://www.bobaikato.io/)

RE: Best way for reading all messages and close

Spin up a consumer, subscribe to EOF events, assign all partitions from the beginning, and keep polling until all partitions has reached EOF. Though, if you have concurrent writers, new messages may be appended after you observe EOF on a partition, so you are never guaranteed to have read all

RE: Need info

Robin, Thanks for writing back. I am aware of this Kafka Connector. Do you have any info on Kafka connecting to Mainframe? Regards Chanchal -Original Message- From: Robin Moffatt Sent: Friday, September 14, 2018 3:06 PM To: users@kafka.apache.org Subject: Re: Need info As a side

Re: Need info

As a side note from your question, I'd recommend looking into Kafka Connect. It is another API with Apache Kafka, and it simplifies the building of pipelines with Kafka such as the one you are describing There are pre-built connectors, including for S3 (

Re: Kafka UI

Hi Majjd, Does kafka-manager support 2.0+? In requirement section, it mentions versions only upto 0.11 On Fri, Sep 14, 2018 at 2:23 PM, Majid Golshadi wrote: > Hello > I recommend yahoo Kafka manager: https://github.com/yahoo/kafka-manager > You can monitor and manager Kafka services from a

Re: Kafka UI

Hello I recommend yahoo Kafka manager: https://github.com/yahoo/kafka-manager You can monitor and manager Kafka services from a central point It gives you some REST API to fetch what you can see in panels from and use in your monitoring application like central Zabbix or what ever On Fri, Sep

Re: KAFKA-7093 - Warn Messages in Kafka 1.1.0

Anyone on any thoughts on this? On Mon, Sep 3, 2018 at 11:28 PM Debraj Manna wrote: > Hi > > I am also observing lot of logs as discussed in > KAFKA-7093 > . Anyone any thoughs? > What does this

Re: Kafka UI

Shameless plug: https://github.com/sourcelaborg/kafka-webview The project is definitely more geared around viewing data within topics but you can view details about your clusters as well. You can view things such as node membership, configuration of the nodes, topics defined on the cluster,

Kafka UI

What are best/recommended UI for kafka to check topics, brokers, partitions, brokers, etc - preferably ones which can be deployed without docker? It must be open source and should work with kafka 2.0+ version.

Best way for reading all messages and close

Hi all, Although the usage of Kafka is stream oriented, for a concrete use case I need to read all the messages existing in a topic and once all them has been read then closing the consumer. What's the best way or framework for doing this? Thanks in advance, David,