Just want to make a correction, Bruno - My understanding is that Kafka Streams
3.5.1 uses Incremental Cooperative Rebalancing which seems to help reduce the
impact of rebalancing caused by autoscaling etc.:
https://www.confluent.io/blog/incremental-cooperative-rebalancing-in-kafka/
Static
Apologies for the delay in responding to you, Bruno. Thank you very much for
your important inputs.
Just searched for log messages in the MSK broker logs pertaining to rebalancing
and updating of metadata for the consumer group and found 412 occurrences in a
13 hour period. During this time, a
Hi.
> By setting max.in.flight.requests.per.connection to 1, I'm concerned that
this could become a performance bottleneck
As Greg pointed out, this is a trade-off between the ordering-guarantee and
the throughput.
So you should first measure the throughput of
Hi.
By default, Kafka returns ack without waiting fsync to the disk. But you
can change this behavior by log.flush.interval.messages config.
For data durability, Kafka mainly relies on replication instead.
> then there is potential for message loss if the node crashes before
On the crashed
I am trying to understand when does Kafka signal to the producer that the
message was successfully accepted into Kafka.
Does Kafka:
1) Write to the pagecache of the node's OS and then return back an ACK ?
If so, then there is potential for message loss if the node crashes before
fsync to disk.
