Re: Reading from lagging replicas

[Michał Łowicki]

Thanks. I'm aware of ISR but looking for exact logic inside Java client how
is this handled so when follower replica, consumer is using, starts lagging
/ goes out-of-sync.

On Mon, Sep 20, 2021 at 2:52 PM Luke Chen  wrote:

> Hi Michał,
> I think you want to check the term in Kafka: In Sync Replicas (a.k.a ISR).
> You might want to check the "Replication" section in doc here
> .
>
> The configuration is this one: *replica.lag.time.max.ms
> *
>
> https://kafka.apache.org/documentation/#brokerconfigs_replica.lag.time.max.ms
>
> Hope that helps.
>
> Thank you.
> Luke
>
> On Mon, Sep 20, 2021 at 8:30 PM Michał Łowicki  wrote:
>
> > Hey,
> >
> > When Java client is reading from replica then how is handled the case
> when
> > such replica starts to lag? (struggles to replicate data from the
> leader).
> > Does consumer, coordinator or group leader have any logic to detect such
> > cases and try to switch over to other replica in those cases? (pointers
> to
> > code, configuration options would be awesome). Thanks in advance.
> >
> > --
> > BR,
> > Michał Łowicki
> >
>


-- 
BR,
Michał Łowicki

Re: Kafka possibilities

[Denis Nutiu]

Hi,

You could try to use a stream processing library that supports joining
multiple streams. Check out
https://www.confluent.io/blog/crossing-streams-joins-apache-kafka/

Best,
Denis

On Mon, Sep 20, 2021 at 6:46 PM Andrew Rakov 
wrote:

> Hello!
>
> I'm working with Kafka to send messages from one microservice to another.
> And I have a problem. It completely stops my work.
>
> I have a couple of topics with a few partitions. I want to round-robin read
> from each topic. I mean to take one message from a topic and go to the
> next. But I can't understand how to code this.
>
> Is it possible at all? Could you advise me on how to do it?
>
> P.S. In my case consumer reads all messages in a topic and goes to the
> next.
>
> Best regards,
> Andrew Rakov
>


-- 
Regards,
Denis Nutiu

Kafka possibilities

[Andrew Rakov]

Hello!

I'm working with Kafka to send messages from one microservice to another.
And I have a problem. It completely stops my work.

I have a couple of topics with a few partitions. I want to round-robin read
from each topic. I mean to take one message from a topic and go to the
next. But I can't understand how to code this.

Is it possible at all? Could you advise me on how to do it?

P.S. In my case consumer reads all messages in a topic and goes to the next.

Best regards,
Andrew Rakov

difference in disk utilization between replica and leader of a partition

[Dhirendra Singh]

Hi All,
I have a kafka cluster running in kubernetes with 3 brokers.
I have a topic with 3 partitions and replication factor of 3.
Topic is evenly distributed across brokers.
I have noticed that there is a huge difference in disk utilization of one
partition between replica and leader.
Ideally replica and leader partition should have same disk utilization if
there is no issue with replication.
In my case replication is running fine and under replicated partitions
count is 0.
what could be the reason of this anomaly ?

Thanks,
Dhirendra.

Re: [ANNOUNCE] Apache Kafka 2.8.1

[Randall Hauch]

Thank you, David, for serving as the release manager for 2.8.1. Great work!

And congratulations to the Apache Kafka community!

Best regards,

Randall

On Mon, Sep 20, 2021 at 2:48 AM David Jacot  wrote:

> The Apache Kafka community is pleased to announce the release for
> Apache Kafka 2.8.1
>
> Apache Kafka 2.8.1 is a bugfix release and fixes 49 issues since the 2.8.0
> release. Please see the release notes for more information.
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/2.8.1/RELEASE_NOTES.html
>
>
> You can download the source and binary release (Scala 2.12 and 2.13) from:
> https://kafka.apache.org/downloads#2.8.1
>
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 35 contributors to this release!
>
> A. Sophie Blee-Goldman, Alexander Iskuskov, Andras Katona, Bill Bejeck,
> Bruno Cadonna, Chris Egerton, Colin Patrick McCabe, David Arthur, David
> Jacot,
> Davor Poldrugo, Dejan Stojadinović, Geordie, Guozhang Wang, Ismael Juma,
> Jason Gustafson, Jeff Kim, John Gray, John Roesler, Justine Olshan,
> Konstantine Karantasis, Lee Dongjin, Luke Chen, Matthias J. Sax, Michael
> Carter,
> Mickael Maison, Phil Hardwick, Rajini Sivaram, Randall Hauch, Shay Elkin,
> Stanislav Vodetskyi, Tom Bentley, vamossagar12, wenbingshen, YiDing-Duke
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
>
> Regards,
> David
>

Re: MM2 - Overriding MirrorSourceConnector Consumer & Producer values

[Men Lim]

did you try setting these values in the distributor.properties file?

On Mon, Sep 20, 2021 at 12:43 AM Jamie  wrote:

> Hi All,
> Has anyone been able to override the producer and consumer values of the
> MM2 MirrorSourceConnector as described below?
> Many Thanks,
> Jamie
>
>
> -Original Message-
> From: Jamie 
> To: users@kafka.apache.org 
> Sent: Thu, 16 Sep 2021 15:52
> Subject: MM2 - Overriding MirrorSourceConnector Consumer & Producer values
>
> Hi All,
> I've trying to override the properties of the consumer and producer in MM2
> to tune them for high throughput. For example, I'm trying to set the
> consumers fetch.min.bytes to 10.
> I'm running MM2 in a dedicated mirror maker cluster (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP382:MirrorMaker2.0-RunningadedicatedMirrorMakercluster)
> and using version 2.7.1 of Kafka.
> I have the equivalent of the following in my mirror maker properties file:
> clusters = A, B
> A.bootstrap.servers = A_host1:9092, A_host2:9092, A_host3:9092
>  B.bootstrap.servers = B_host1:9092, B_host2:9092, B_host3:9092
> # enable and configure individual replication flowsA->B.enabled =
> true
> # regex which defines which topics gets replicated. For eg "foo-.*"
>  A->B.topics = .test-topic
>
> I'm trying to override the properties of the consumer which fetches
> records from cluster "A" and the producer that sends records to cluster
> "B".
> I've tried the following in the config file:
> A.consumer.fetch.min.bytes = 10
> A->B.consumer.fetch.min.bytes = 10
> A.fetch.min.bytes = 10
> B.consumer.fetch.min.bytes = 10
> B.fetch.min.bytes = 10
> None of which seem to work, when I start MM2 and go into the logs and look
> at the value using the the MirrorSourceConnector tasks consumer and
> producer config I still see the default value for fetch.min.bytes (1) being
> used.
> Am I trying to override the values of the consumer incorrectly or do I
> need to set these in a different place?
> Many Thanks,
> Jamie
>
>
>
>

ApacheCon starts tomorrow!

[Rich Bowen]

ApacheCon @Home starts tomorrow! Details at 
https://www.apachecon.com/acah2021/index.html


(Note: You're receiving this because you are subscribed to one or more 
user lists for Apache Software Foundation projects.)


We've got three days of great content lined up for you, spanning 14 
project communities. And we're very excited about the keynotes, with 
presentations from David Nalley, Ashley Wolfe, Mark Cox, Alison Parker, 
and Michael Weinberg. And we'll be hearing from our Platinum sponsors in 
their keynotes as well! (Schedule is at 
https://www.apachecon.com/acah2021/tracks/)


You can still register today, at 
https://www.apachecon.com/acah2021/register.html


We especially want to thank our sponsors, who have made this event possible:

Strategic sponsor: Google
Platinum sponsors: Huawei, Tencent, Instaclustr, and Apple
Gold sponsors: AWS, Aiven, Gradle, Replicated, Red 
Hat, Baidu, Fiter, Cerner, Dremio, and Didi
Silver sponsors: Bamboo, SpereEx, Microsoft, Imply, Securonix, DataStax, 
and Crafter Software

Bronze sponsor: Technical Arts

Please join us on our Slack for discussion before, during, and after the 
event! http://s.apache.org/apachecon-slack


And follow us on Twitter - https://twitter.com/apachecon - for tips and 
announcements during the event.


See you tomorrow!


--
Rich Bowen, VP Conferences
The Apache Software Foundation
https://apachecon.com/
@apachecon

Re: Reading from lagging replicas

[Luke Chen]

Hi Michał,
I think you want to check the term in Kafka: In Sync Replicas (a.k.a ISR).
You might want to check the "Replication" section in doc here
.

The configuration is this one: *replica.lag.time.max.ms
*
https://kafka.apache.org/documentation/#brokerconfigs_replica.lag.time.max.ms

Hope that helps.

Thank you.
Luke

On Mon, Sep 20, 2021 at 8:30 PM Michał Łowicki  wrote:

> Hey,
>
> When Java client is reading from replica then how is handled the case when
> such replica starts to lag? (struggles to replicate data from the leader).
> Does consumer, coordinator or group leader have any logic to detect such
> cases and try to switch over to other replica in those cases? (pointers to
> code, configuration options would be awesome). Thanks in advance.
>
> --
> BR,
> Michał Łowicki
>

Reading from lagging replicas

[Michał Łowicki]

Hey,

When Java client is reading from replica then how is handled the case when
such replica starts to lag? (struggles to replicate data from the leader).
Does consumer, coordinator or group leader have any logic to detect such
cases and try to switch over to other replica in those cases? (pointers to
code, configuration options would be awesome). Thanks in advance.

-- 
BR,
Michał Łowicki

[ANNOUNCE] Apache Kafka 2.8.1

[David Jacot]

The Apache Kafka community is pleased to announce the release for
Apache Kafka 2.8.1

Apache Kafka 2.8.1 is a bugfix release and fixes 49 issues since the 2.8.0
release. Please see the release notes for more information.

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/2.8.1/RELEASE_NOTES.html


You can download the source and binary release (Scala 2.12 and 2.13) from:
https://kafka.apache.org/downloads#2.8.1

---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.


Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following 35 contributors to this release!

A. Sophie Blee-Goldman, Alexander Iskuskov, Andras Katona, Bill Bejeck,
Bruno Cadonna, Chris Egerton, Colin Patrick McCabe, David Arthur, David Jacot,
Davor Poldrugo, Dejan Stojadinović, Geordie, Guozhang Wang, Ismael Juma,
Jason Gustafson, Jeff Kim, John Gray, John Roesler, Justine Olshan,
Konstantine Karantasis, Lee Dongjin, Luke Chen, Matthias J. Sax, Michael Carter,
Mickael Maison, Phil Hardwick, Rajini Sivaram, Randall Hauch, Shay Elkin,
Stanislav Vodetskyi, Tom Bentley, vamossagar12, wenbingshen, YiDing-Duke

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!


Regards,
David

Re: MM2 - Overriding MirrorSourceConnector Consumer & Producer values

[Jamie]

Hi All, 
Has anyone been able to override the producer and consumer values of the MM2 
MirrorSourceConnector as described below?
Many Thanks, 
Jamie


-Original Message-
From: Jamie 
To: users@kafka.apache.org 
Sent: Thu, 16 Sep 2021 15:52
Subject: MM2 - Overriding MirrorSourceConnector Consumer & Producer values

Hi All, 
I've trying to override the properties of the consumer and producer in MM2 to 
tune them for high throughput. For example, I'm trying to set the consumers 
fetch.min.bytes to 10. 
I'm running MM2 in a dedicated mirror maker cluster 
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP382:MirrorMaker2.0-RunningadedicatedMirrorMakercluster)
 and using version 2.7.1 of Kafka. 
I have the equivalent of the following in my mirror maker properties file:
    clusters = A, B
    A.bootstrap.servers = A_host1:9092, A_host2:9092, A_host3:9092    
B.bootstrap.servers = B_host1:9092, B_host2:9092, B_host3:9092
    # enable and configure individual replication flows    A->B.enabled = true
    # regex which defines which topics gets replicated. For eg "foo-.*"    
A->B.topics = .test-topic

I'm trying to override the properties of the consumer which fetches records 
from cluster "A" and the producer that sends records to cluster "B". 
I've tried the following in the config file:
    A.consumer.fetch.min.bytes = 10
    A->B.consumer.fetch.min.bytes = 10
    A.fetch.min.bytes = 10
    B.consumer.fetch.min.bytes = 10
    B.fetch.min.bytes = 10
None of which seem to work, when I start MM2 and go into the logs and look at 
the value using the the MirrorSourceConnector tasks consumer and producer 
config I still see the default value for fetch.min.bytes (1) being used.
Am I trying to override the values of the consumer incorrectly or do I need to 
set these in a different place?
Many Thanks, 
Jamie