Re: Best way for reading all messages and close

[John Roesler]

Specifically, you can monitor the "records-lag-max" (
https://docs.confluent.io/current/kafka/monitoring.html#fetch-metrics)
metric. (or the more granular one per partition).

Once this metric goes to 0, you know that you've caught up with the tail of
the log.

Hope this helps,
-John

On Fri, Sep 14, 2018 at 2:02 PM Matthias J. Sax 
wrote:

> Using Kafka Streams this is a little tricky.
>
> The API itself has no built-in mechanism to do this. You would need to
> monitor the lag of the application, and if the lag is zero (assuming you
> don't write new data into the topic in parallel), terminate the
> application.
>
>
> -Matthias
>
> On 9/14/18 4:19 AM, Henning Røigaard-Petersen wrote:
> > Spin up a consumer, subscribe to EOF events, assign all partitions from
> the beginning, and keep polling until all partitions has reached EOF.
> > Though, if you have concurrent writers, new messages may be appended
> after you observe EOF on a partition, so you are never guaranteed to have
> read all messages at the time you choose to close the consumer.
> >
> > /Henning Røigaard-Petersen
> >
> > -Original Message-
> > From: David Espinosa 
> > Sent: 14. september 2018 09:46
> > To: users@kafka.apache.org
> > Subject: Best way for reading all messages and close
> >
> > Hi all,
> >
> > Although the usage of Kafka is stream oriented, for a concrete use case
> I need to read all the messages existing in a topic and once all them has
> been read then closing the consumer.
> >
> > What's the best way or framework for doing this?
> >
> > Thanks in advance,
> > David,
> >
>
>

Re: Best way for reading all messages and close

[Matthias J. Sax]

Using Kafka Streams this is a little tricky.

The API itself has no built-in mechanism to do this. You would need to
monitor the lag of the application, and if the lag is zero (assuming you
don't write new data into the topic in parallel), terminate the application.


-Matthias

On 9/14/18 4:19 AM, Henning Røigaard-Petersen wrote:
> Spin up a consumer, subscribe to EOF events, assign all partitions from the 
> beginning, and keep polling until all partitions has reached EOF.
> Though, if you have concurrent writers, new messages may be appended after 
> you observe EOF on a partition, so you are never guaranteed to have read all 
> messages at the time you choose to close the consumer.
> 
> /Henning Røigaard-Petersen
> 
> -Original Message-
> From: David Espinosa  
> Sent: 14. september 2018 09:46
> To: users@kafka.apache.org
> Subject: Best way for reading all messages and close
> 
> Hi all,
> 
> Although the usage of Kafka is stream oriented, for a concrete use case I 
> need to read all the messages existing in a topic and once all them has been 
> read then closing the consumer.
> 
> What's the best way or framework for doing this?
> 
> Thanks in advance,
> David,
> 



signature.asc
Description: OpenPGP digital signature

Re: Understanding default.deserialization.exception.handler

[Matthias J. Sax]

Your observation is correct. It's a known bug:
https://issues.apache.org/jira/browse/KAFKA-6502

In practice, it should not be a big issue though.
 - you would only hit this bug if you don't process a "good message"
afterwards
 - even if you hit this bug, you would just skip the message again

Thus, the only drawback I see is an additional log message. As long as
you don't have many _consecutive_ corrupted messages it should be impact
you much.

Hope this helps.


-Matthias

On 9/13/18 6:09 AM, Tim Ward wrote:
> With
> 
> 
> props.put(StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG,
>  LogAndContinueExceptionHandler.class);
> 
> Scenario A:
> 
> Run application. Feed a message into the topic that will fail deserialization.
> Application logs exception and keeps running.
> Shut down application. Restart application.
> Application re-reads broken message, logs exception again (and keeps running).
> 
> Scenario B:
> 
> Run application. Feed a message into the topic that will fail deserialization.
> Application logs exception and keeps running.
> Feed a good message into deserialization.
> Application processes it normally.
> Shut down application. Restart application.
> Application does *not* re-reads broken message.
> 
> So it looks like LogAndContinueExceptionHandler does not seem to commit() the 
> incoming "poison pill" message(s), and these will be re-read if the 
> application is restarted without any good messages having been read after the 
> bad ones.
> 
> Have I understood this correctly? If so, is this correct behaviour as 
> designed? Is it documented to that level of detail?
> 
> Tim Ward
> 
> The contents of this email and any attachment are confidential to the 
> intended recipient(s). If you are not an intended recipient: (i) do not use, 
> disclose, distribute, copy or publish this email or its contents; (ii) please 
> contact the sender immediately; and (iii) delete this email. Our privacy 
> policy is available here: https://origamienergy.com/privacy-policy/. Origami 
> Energy Limited (company number 8619644); Origami Storage Limited (company 
> number 10436515) and OSSPV001 Limited (company number 10933403), each 
> registered in England and each with a registered office at: Ashcombe Court, 
> Woolsack Way, Godalming, GU7 1LQ.
> 



signature.asc
Description: OpenPGP digital signature

Re: How many consumers can subscribe to the same topic

[Suman B N]

   - With same consumer group, you can't have consumers more than the
   number of partitions. Even if you do, extra consumers are just idle. Each
   partition can be read only by one consumer.
   - With different consumer groups, you can have as many consumer groups
   as you want for a single topic. Each consumer group will have its own
   offset management. Each consumer group will get all messages - meaning 2
   different consumer groups will receive the same number of messages if
   subscribed to the same topic.

Hope that answers your doubt.

Thanks,
Suman

On Fri, Sep 14, 2018 at 9:37 PM Bobai Kato  wrote:

> Hi y’all,
>
> I’m pretty new to Kafka and wondering how many consumers I can have on a
> topic.
>
> Can I have 500+ users subscribe to particular topic ?
>
> — Bobai (https://www.bobaikato.io/)
>
>

-- 
*Suman*
*OlaCabs*

Kafka UI

[mohits2287]

Which are best/recommended user interfaces for kafka to see topics, brokers, 
partitions, consumers etc - preferably one which can be deployed without 
docker? It must be open source and must work with Kafka 2.0+

Thanks in advance.

How many consumers can subscribe to the same topic

[Bobai Kato]

Hi y’all,

I’m pretty new to Kafka and wondering how many consumers I can have on a topic.

Can I have 500+ users subscribe to particular topic ?

— Bobai (https://www.bobaikato.io/)

RE: Best way for reading all messages and close

[Henning Røigaard-Petersen]

Spin up a consumer, subscribe to EOF events, assign all partitions from the 
beginning, and keep polling until all partitions has reached EOF.
Though, if you have concurrent writers, new messages may be appended after you 
observe EOF on a partition, so you are never guaranteed to have read all 
messages at the time you choose to close the consumer.

/Henning Røigaard-Petersen

-Original Message-
From: David Espinosa  
Sent: 14. september 2018 09:46
To: users@kafka.apache.org
Subject: Best way for reading all messages and close

Hi all,

Although the usage of Kafka is stream oriented, for a concrete use case I need 
to read all the messages existing in a topic and once all them has been read 
then closing the consumer.

What's the best way or framework for doing this?

Thanks in advance,
David,

RE: Need info

[Chanchal Chatterji]

Robin,

Thanks for writing back.  I am aware of this Kafka Connector.  Do you have any 
info on Kafka connecting to Mainframe?

Regards
Chanchal

-Original Message-
From: Robin Moffatt  
Sent: Friday, September 14, 2018 3:06 PM
To: users@kafka.apache.org
Subject: Re: Need info

As a side note from your question, I'd recommend looking into Kafka Connect. It 
is another API with Apache Kafka, and it simplifies the building of pipelines 
with Kafka such as the one you are describing There are pre-built connectors, 
including for S3 ( 
https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.confluent.io%2Fconnector%2Fkafka-connect-s3%2F&data=01%7C01%7Cchanchal.chatterji%40infosys.com%7C7382ed64a0214c778c8c08d61a25b0df%7C63ce7d592f3e42cda8ccbe764cff5eb6%7C1&sdata=yc3Mo3h5JFD%2BpoGBDthgAgT7GxW52tLS4mZJxO4HKXs%3D&reserved=0).
 There are also options for pulling from the mainframe, depending on specific 
platform and system.


-- 

Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff


On Wed, 12 Sep 2018 at 08:32, Chanchal Chatterji < 
chanchal.chatte...@infosys.com> wrote:

> Hi,
>
> In the process of mainframe modernization, we are attempting to stream 
> Mainframe data to AWS Cloud , using Kafka.  We are planning to use 
> Kafka 'Producer API' at mainframe side and 'Connector API' on the cloud side.
> Since our data is processed by a module called 'Central dispatch' 
> located in Mainframe and is sent to Kafka.  We want to know what is 
> rate of volume Kafka can handle.  The other end of Kafka is connected 
> to an AWS S3 Bucket As sink.  Please help us to provide this info or 
> else please help to connect with relevant person who can help us to 
> understand this.
>
> Thanks and Regards
>
> Chanchal Chatterji
> Principal Consultant,
> Infosys Ltd.
> Electronic city Phase-1,
> Bangalore-560100
> Contacts : 9731141606/ 8105120766
>
>

Re: Need info

[Robin Moffatt]

As a side note from your question, I'd recommend looking into Kafka
Connect. It is another API with Apache Kafka, and it simplifies the
building of pipelines with Kafka such as the one you are describing
There are pre-built connectors, including for S3 (
https://www.confluent.io/connector/kafka-connect-s3/). There are also
options for pulling from the mainframe, depending on specific platform and
system.


-- 

Robin Moffatt | Developer Advocate | ro...@confluent.io | @rmoff


On Wed, 12 Sep 2018 at 08:32, Chanchal Chatterji <
chanchal.chatte...@infosys.com> wrote:

> Hi,
>
> In the process of mainframe modernization, we are attempting to stream
> Mainframe data to AWS Cloud , using Kafka.  We are planning to use Kafka
> 'Producer API' at mainframe side and 'Connector API' on the cloud side.
> Since our data is processed by a module called 'Central dispatch' located
> in Mainframe and is sent to Kafka.  We want to know what is rate of volume
> Kafka can handle.  The other end of Kafka is connected to an AWS S3 Bucket
> As sink.  Please help us to provide this info or else please help to
> connect with relevant person who can help us to understand this.
>
> Thanks and Regards
>
> Chanchal Chatterji
> Principal Consultant,
> Infosys Ltd.
> Electronic city Phase-1,
> Bangalore-560100
> Contacts : 9731141606/ 8105120766
>
>

Re: Kafka UI

[Mohit Srivastav]

Hi Majjd,

Does kafka-manager support 2.0+? In requirement section, it mentions
versions only upto 0.11

On Fri, Sep 14, 2018 at 2:23 PM, Majid Golshadi 
wrote:

> Hello
> I recommend yahoo Kafka manager: https://github.com/yahoo/kafka-manager
> You can monitor and manager Kafka services from a central point
> It gives you some REST API to fetch what you can see in panels from and use
> in your monitoring application like central Zabbix or what ever
>
>
> On Fri, Sep 14, 2018 at 1:12 PM Stephen Powis 
> wrote:
>
> > Shameless plug: https://github.com/sourcelaborg/kafka-webview
> >
> > The project is definitely more geared around viewing data within
> topics
> > but you can view details about your clusters as well.  You can view
> things
> > such as node membership, configuration of the nodes, topics defined on
> the
> > cluster, partition distribution, ISR, etc..
> >
> > Stephen
> >
> > On Fri, Sep 14, 2018 at 5:22 PM, Mohit Srivastav 
> > wrote:
> >
> > > What are best/recommended UI for kafka to check topics, brokers,
> > > partitions, brokers, etc - preferably ones which can be deployed
> without
> > > docker? It must be open source and should work with kafka 2.0+ version.
> > >
> >
>

Re: Kafka UI

[Majid Golshadi]

Hello
I recommend yahoo Kafka manager: https://github.com/yahoo/kafka-manager
You can monitor and manager Kafka services from a central point
It gives you some REST API to fetch what you can see in panels from and use
in your monitoring application like central Zabbix or what ever


On Fri, Sep 14, 2018 at 1:12 PM Stephen Powis  wrote:

> Shameless plug: https://github.com/sourcelaborg/kafka-webview
>
> The project is definitely more geared around viewing data within topics
> but you can view details about your clusters as well.  You can view things
> such as node membership, configuration of the nodes, topics defined on the
> cluster, partition distribution, ISR, etc..
>
> Stephen
>
> On Fri, Sep 14, 2018 at 5:22 PM, Mohit Srivastav 
> wrote:
>
> > What are best/recommended UI for kafka to check topics, brokers,
> > partitions, brokers, etc - preferably ones which can be deployed without
> > docker? It must be open source and should work with kafka 2.0+ version.
> >
>

Re: KAFKA-7093 - Warn Messages in Kafka 1.1.0

[Debraj Manna]

Anyone on any thoughts on this?

On Mon, Sep 3, 2018 at 11:28 PM Debraj Manna 
wrote:

> Hi
>
> I am also observing lot of logs as discussed in
> KAFKA-7093
>  . Anyone any thoughs?
> What does this denote? What does it effect and how to recover from this?
>
> Thanks,
>

Re: Kafka UI

[Stephen Powis]

Shameless plug: https://github.com/sourcelaborg/kafka-webview

The project is definitely more geared around viewing data within topics
but you can view details about your clusters as well.  You can view things
such as node membership, configuration of the nodes, topics defined on the
cluster, partition distribution, ISR, etc..

Stephen

On Fri, Sep 14, 2018 at 5:22 PM, Mohit Srivastav 
wrote:

> What are best/recommended UI for kafka to check topics, brokers,
> partitions, brokers, etc - preferably ones which can be deployed without
> docker? It must be open source and should work with kafka 2.0+ version.
>

Kafka UI

[Mohit Srivastav]

What are best/recommended UI for kafka to check topics, brokers,
partitions, brokers, etc - preferably ones which can be deployed without
docker? It must be open source and should work with kafka 2.0+ version.

Best way for reading all messages and close

[David Espinosa]

Hi all,

Although the usage of Kafka is stream oriented, for a concrete use case I
need to read all the messages existing in a topic and once all them has
been read then closing the consumer.

What's the best way or framework for doing this?

Thanks in advance,
David,