Fwd: [Question] About Kafka producer design decision making

[Sean Sin]

Dear Apache Kakfa Developers,

I'm 4-year SWE in South Korea.
I have some questions while watching Kafka Producer API.

*Why Use "Future" and Not "CompletableFuture"?*

In the case of "Future", blocking occurs when calling "*get()*", so I
thought "Computable Future" would be better when doing more asynchronous
operations.

I looked at the Java API document

based on the latest version, version 3.6.x.

If you look at that version, you can see that the Future object provides
the "toCompletionStage() "method, which can convert "KafkaFuture" to
"ComputableFuture".

In response to this, I think that in the initial design decision process,
we considered compatibility issues under JDK 1.8 and the level of knowledge
of the learning curve or developer when introducing ComputableFuture, but I
wonder if this is correct.

In addition, I wonder if it is recommended to use the "toCompletionStage()"
method to produce more non-blocking if we assume JDK 1.8 or higher.

Thanks.
Su-Ung Shin.

Re: [Question] About Kafka producer design decision making

[Haruki Okada]

Hi.

I also guess the main reason for using Future was for JDK1.7 support which
is no longer necessary in the current Kafka version.
Actually, there's a KIP about this:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100829459
but it seems it's not active now.

> I wonder if it is recommended to use

For now, the practical way for asynchronous produce is to use producer
callback. We can easily create CompletableFuture using it:

CompletableFuture future = new CompletableFuture<>();
producer.send(record, (r, e) -> {
if (e != null) {
future.completeExceptionally(e);
} else {
future.complete(r);
}
});

2023年11月14日(火) 18:30 신수웅(Sean Sin) :

> Dear Apache Kakfa Developers,
>
> I'm 4-year SWE in South Korea.
> I have some questions while watching Kafka Producer API.
>
> *Why Use "Future" and Not "CompletableFuture"?*
>
> In the case of "Future", blocking occurs when calling "*get()*", so I
> thought "Computable Future" would be better when doing more asynchronous
> operations.
>
> I looked at the Java API document
> <
> https://kafka.apache.org/36/javadoc/org/apache/kafka/common/KafkaFuture.html#thenApply(org.apache.kafka.common.KafkaFuture.BaseFunction)
> >
> based on the latest version, version 3.6.x.
>
> If you look at that version, you can see that the Future object provides
> the "toCompletionStage() "method, which can convert "KafkaFuture" to
> "ComputableFuture".
>
> In response to this, I think that in the initial design decision process,
> we considered compatibility issues under JDK 1.8 and the level of knowledge
> of the learning curve or developer when introducing ComputableFuture, but I
> wonder if this is correct.
>
> In addition, I wonder if it is recommended to use the "toCompletionStage()"
> method to produce more non-blocking if we assume JDK 1.8 or higher.
>
> Thanks.
> Su-Ung Shin.
>


-- 

Okada Haruki
ocadar...@gmail.com

Re: About Kafka Java Client Producer Retry And Callback

[王有胜]

Thanks for your answer, I set a large min.insync.replicas for the
topic and I can see in the console log that Kafka Sender keeps
retrying.

Haruki Okada  于2023年11月13日周一 16:50写道：
>
> > will the callback be executed for each retry
>
> The callback will be triggered only once when the produce is finally ended
> up with succeeded or failed after retries.
>
> > is there any way to make Kafka producers retry locally
>
> Easiest way would be to make produce failing artificially. it can be done
> by e.g.:
> - set acks=all and set topic's min.insync.replicas to impossibly large value
> - use iptables between producer and broker to block the network connectivity
>
> 2023年11月12日(日) 18:10 王有胜 :
>
> > Hi Community, I use Kafka Java Client to send messages asynchronously.
> > I wonder if the producer fails to send a message, during the retry
> > period, will the callback be executed for each retry?
> > I debugged the source code
> > org.apache.kafka.clients.producer.internals.Sender#canRetry and
> > org.apache.kafka.clients.producer.internals.Sender#reenqueueBatch
> > parts, and I'm still not sure.
> > Can anyone help answer this question?
> > In addition, is there any way to make Kafka producers retry locally? I
> > want to completely understand the code logic.
> > Thanks!
> >
>
>
> --
> 
> Okada Haruki
> ocadar...@gmail.com
> 

Important Security Notice for Apache Kafka Users

[Divij Vaidya]

Dear Apache Kafka Users,

We want to bring to your attention a security vulnerability affecting all
released versions of Apache Kafka that have a dependency on Zookeeper. The
vulnerability, identified as CVE-2023-44981 [1], specifically impacts users
utilizing SASL Quorum Peer authentication in Zookeeper.

Vulnerability Details:
- Affected Versions: All released versions of Apache Kafka with Zookeeper
dependency.
- CVE Identifier: CVE-2023-44981 [1]
- Impact: Limited to users employing SASL Quorum Peer authentication in
Zookeeper (quorum.auth.enableSasl=true)

Action Required:
Upcoming Apache Kafka versions, 3.6.1 (release date - tentative Dec '23)
and 3.7.0 (release date - Jan'23 [3]), will depend on Zookeeper versions
containing fixes for the vulnerability. In the interim, we highly advise
taking proactive steps to safeguard Zookeeper ensemble election/quorum
communication by implementing a firewall [2].

Future Updates:
We are diligently working on addressing this vulnerability in our upcoming
releases. We will keep you updated on any changes to our recommendations
and promptly inform you of the release dates for Apache Kafka versions
3.6.1 and 3.7.0.

If you have any further questions regarding this, please don't hesitate to
reach out to us at secur...@kafka.apache.org or post a comment at
https://issues.apache.org/jira/browse/KAFKA-15658

Best Regards,

Divij Vaidya
On behalf of Apache Kafka PMC

[1] https://zookeeper.apache.org/security.html#CVE-2023-44981
[2] https://lists.apache.org/thread/wf0yrk84dg1942z1o74kd8nycg6pgm5b
[3] https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.7.0


--

Kafka Broker Backups

[Upesh Desai]

Hello all,

I have been doing some testing around backup solutions for Kafka disk volumes, 
and was wondering if anyone has used point-in-time disk snapshots, such as 
Valero with AWS EBS volumes. We’re worried about timing inconsistencies since 
there can be N brokers and EBS volume snapshots may not always be exactly at 
the same time.

Thanks in advance!
Upesh

Upesh Desai|
Senior Software Developer
ude...@itrsgroup.com
ITRS
Internet communications are not secure and therefore the ITRS Group does not 
accept legal responsibility for the contents of this message. Any view or 
opinions presented are solely those of the author and do not necessarily 
represent those of the ITRS Group unless otherwise specifically stated.
[itrs.email.signature]