[jira] [Resolved] (KAFKA-15462) Add group type filter to the admin client

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-15462.
-
Fix Version/s: 3.8.0
   Resolution: Fixed

> Add group type filter to the admin client
> -
>
> Key: KAFKA-15462
> URL: https://issues.apache.org/jira/browse/KAFKA-15462
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: David Jacot
>Assignee: Ritika Reddy
>Priority: Major
> Fix For: 3.8.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] MINOR: Use archive repository for older releases [kafka-site]

[via GitHub]

mimaison merged PR #589:
URL: https://github.com/apache/kafka-site/pull/589


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Permissions to contribute to Apache Kafka project

[Paul Amar]

Hi there,

My Jira ID is  "paul_amar" and I'd like to ask permissions to contribute to 
Apache Kafka Project.
Meanwhile, I haven't been able to create a wiki ID as it does not allow me to 
create an account.

Greetings and let's collaborate soon.

Paul


PAUL AMAR
Groupe Michelin - Information Technology
Delivery Manager Orchestration - CBS/CORP/SI/BS/SC

Clermont-Ferrand / Les Carmes / A17 4ème étage
paul.a...@michelin.com
Classification : D3
Conservation : 90 jours

Ce courriel peut contenir des informations de nature confidentielle et 
destinées uniquement à l'usage du destinataire. Toute divulgation des 
informations contenues dans ce courriel est strictement interdite et peut être 
contraire à la législation applicable. Si vous avez reçu ce courriel par 
erreur, merci de nous le notifier immédiatement et de détruire ce courriel. Par 
avance merci.

This e-mail may contain confidential information and is intended solely for the 
use of the addressee. Any disclosure of this information is strictly prohibited 
and may be unlawful. If you have received this e-mail by mistake, please notify 
us immediately and delete this e-mail. Thank you in advance.

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2685

[Apache Jenkins Server]

See 

Re: Permissions to contribute to Apache Kafka project

[Bruno Cadonna]

Hi Paul,

I gave you the permission in JIRA.

Currently, all Apache projects have this issue with the wiki account 
creation. There is a ongoing INFRA ticket about it: 
https://issues.apache.org/jira/browse/INFRA-25451


Sorry about that!

In any case, thank you for your interest in Apache Kafka!

Best,
Bruno

On 2/29/24 10:52 AM, Paul Amar wrote:

Hi there,

My Jira ID is  "paul_amar" and I'd like to ask permissions to contribute to 
Apache Kafka Project.
Meanwhile, I haven't been able to create a wiki ID as it does not allow me to 
create an account.

Greetings and let's collaborate soon.

Paul


PAUL AMAR
Groupe Michelin - Information Technology
Delivery Manager Orchestration - CBS/CORP/SI/BS/SC

Clermont-Ferrand / Les Carmes / A17 4ème étage
paul.a...@michelin.com
Classification : D3
Conservation : 90 jours

Ce courriel peut contenir des informations de nature confidentielle et 
destinées uniquement à l'usage du destinataire. Toute divulgation des 
informations contenues dans ce courriel est strictement interdite et peut être 
contraire à la législation applicable. Si vous avez reçu ce courriel par 
erreur, merci de nous le notifier immédiatement et de détruire ce courriel. Par 
avance merci.

This e-mail may contain confidential information and is intended solely for the 
use of the addressee. Any disclosure of this information is strictly prohibited 
and may be unlawful. If you have received this e-mail by mistake, please notify 
us immediately and delete this e-mail. Thank you in advance.

Re: [ANNOUNCE] Apache Kafka 3.7.0

[Guozhang Wang]

Thanks Stan for running the release!

On Thu, Feb 29, 2024 at 5:39 AM Boudjelda Mohamed Said
 wrote:
>
> Thanks Stanislav for running the release!
>
> On Wed, Feb 28, 2024 at 10:36 PM Kirk True  wrote:
>
> > Thanks Stanislav
> >
> > > On Feb 27, 2024, at 10:01 AM, Stanislav Kozlovski <
> > stanislavkozlov...@apache.org> wrote:
> > >
> > > The Apache Kafka community is pleased to announce the release of
> > > Apache Kafka 3.7.0
> > >
> > > This is a minor release that includes new features, fixes, and
> > > improvements from 296 JIRAs
> > >
> > > An overview of the release and its notable changes can be found in the
> > > release blog post:
> > > https://kafka.apache.org/blog#apache_kafka_370_release_announcement
> > >
> > > All of the changes in this release can be found in the release notes:
> > > https://www.apache.org/dist/kafka/3.7.0/RELEASE_NOTES.html
> > >
> > > You can download the source and binary release (Scala 2.12, 2.13) from:
> > > https://kafka.apache.org/downloads#3.7.0
> > >
> > >
> > ---
> > >
> > >
> > > Apache Kafka is a distributed streaming platform with four core APIs:
> > >
> > >
> > > ** The Producer API allows an application to publish a stream of records
> > to
> > > one or more Kafka topics.
> > >
> > > ** The Consumer API allows an application to subscribe to one or more
> > > topics and process the stream of records produced to them.
> > >
> > > ** The Streams API allows an application to act as a stream processor,
> > > consuming an input stream from one or more topics and producing an
> > > output stream to one or more output topics, effectively transforming the
> > > input streams to output streams.
> > >
> > > ** The Connector API allows building and running reusable producers or
> > > consumers that connect Kafka topics to existing applications or data
> > > systems. For example, a connector to a relational database might
> > > capture every change to a table.
> > >
> > >
> > > With these APIs, Kafka can be used for two broad classes of application:
> > >
> > > ** Building real-time streaming data pipelines that reliably get data
> > > between systems or applications.
> > >
> > > ** Building real-time streaming applications that transform or react
> > > to the streams of data.
> > >
> > >
> > > Apache Kafka is in use at large and small companies worldwide, including
> > > Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> > > Target, The New York Times, Uber, Yelp, and Zalando, among others.
> > >
> > > A big thank you to the following 146 contributors to this release!
> > > (Please report an unintended omission)
> > >
> > > Abhijeet Kumar, Akhilesh Chaganti, Alieh, Alieh Saeedi, Almog Gavra,
> > > Alok Thatikunta, Alyssa Huang, Aman Singh, Andras Katona, Andrew
> > > Schofield, Anna Sophie Blee-Goldman, Anton Agestam, Apoorv Mittal,
> > > Arnout Engelen, Arpit Goyal, Artem Livshits, Ashwin Pankaj,
> > > ashwinpankaj, atu-sharm, bachmanity1, Bob Barrett, Bruno Cadonna,
> > > Calvin Liu, Cerchie, chern, Chris Egerton, Christo Lolov, Colin
> > > Patrick McCabe, Colt McNealy, Crispin Bernier, David Arthur, David
> > > Jacot, David Mao, Deqi Hu, Dimitar Dimitrov, Divij Vaidya, Dongnuo
> > > Lyu, Eaugene Thomas, Eduwer Camacaro, Eike Thaden, Federico Valeri,
> > > Florin Akermann, Gantigmaa Selenge, Gaurav Narula, gongzhongqiang,
> > > Greg Harris, Guozhang Wang, Gyeongwon, Do, Hailey Ni, Hanyu Zheng, Hao
> > > Li, Hector Geraldino, hudeqi, Ian McDonald, Iblis Lin, Igor Soarez,
> > > iit2009060, Ismael Juma, Jakub Scholz, James Cheng, Jason Gustafson,
> > > Jay Wang, Jeff Kim, Jim Galasyn, John Roesler, Jorge Esteban Quilcate
> > > Otoya, Josep Prat, José Armando García Sancio, Jotaniya Jeel, Jouni
> > > Tenhunen, Jun Rao, Justine Olshan, Kamal Chandraprakash, Kirk True,
> > > kpatelatwork, kumarpritam863, Laglangyue, Levani Kokhreidze, Lianet
> > > Magrans, Liu Zeyu, Lucas Brutschy, Lucia Cerchie, Luke Chen, maniekes,
> > > Manikumar Reddy, mannoopj, Maros Orsak, Matthew de Detrich, Matthias
> > > J. Sax, Max Riedel, Mayank Shekhar Narula, Mehari Beyene, Michael
> > > Westerby, Mickael Maison, Nick Telford, Nikhil Ramakrishnan, Nikolay,
> > > Okada Haruki, olalamichelle, Omnia G.H Ibrahim, Owen Leung, Paolo
> > > Patierno, Philip Nee, Phuc-Hong-Tran, Proven Provenzano, Purshotam
> > > Chauhan, Qichao Chu, Matthias J. Sax, Rajini Sivaram, Renaldo Baur
> > > Filho, Ritika Reddy, Robert Wagner, Rohan, Ron Dagostino, Roon, runom,
> > > Ruslan Krivoshein, rykovsi, Sagar Rao, Said Boudjelda, Satish Duggana,
> > > shuoer86, Stanislav Kozlovski, Taher Ghaleb, Tang Yunzi, TapDang,
> > > Taras Ledkov, tkuramoto33, Tyler Bertrand, vamossagar12, Vedarth
> > > Sharma, Viktor Somogyi-Vass, Vincent Jiang, Walker Carlson,
> > > Wuzhengyu97, Xavier Léauté, Xiaobing Fang, yangy, Ritika Reddy,
> > > Yanming Zhou, Yash Mayya, yuyli, zhaohaidao, Zihao Lin, Ziming Deng
> >

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2686

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-16315) Investigate propagating metadata updates via queues

[Kirk True (Jira)]

Kirk True created KAFKA-16315:
-

 Summary: Investigate propagating metadata updates via queues
 Key: KAFKA-16315
 URL: https://issues.apache.org/jira/browse/KAFKA-16315
 Project: Kafka
  Issue Type: Task
  Components: clients, consumer
Affects Versions: 3.7.0
Reporter: Kirk True
 Fix For: 4.0.0


Some of the new {{AsyncKafkaConsumer}} logic enqueues events for the network 
I/O thread then issues a call to update the {{ConsumerMetadata}} via 
{{requestUpdate()}}. If the event ends up stuck in the queue for a long time, 
it is possible that the metadata is not updated at the correct time.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

[Andrew Schofield]

Hi David,
Thanks for the summary. It seems to me that the Flow-like option is best because
it can easily handle cancellations and exceptions, returning the topic 
partition information
and signalling when the last of the results have been returned. I think it’s 
also equally
applicable to any of the other KafkaAdminClient methods which could benefit from
returning results progressively from the broker such as describing all of the 
consumer groups
on a massive cluster. That would of course be another KIP :)

Thanks,
Andrew

> On 28 Feb 2024, at 22:30, David Arthur  
> wrote:
>
> Andrew/Jose, I like the suggested Flow API. It's also similar to the stream
> observers in GPRC. I'm not sure we should expose something as complex as
> the Flow API directly in KafkaAdminClient, but certainly we can provide a
> similar interface.
>
> ---
> Cancellations:
>
> Another thing not yet discussed is how to cancel in-flight requests. For
> other calls in KafkaAdminClient, we use KafkaFuture which has a "cancel"
> method. With the callback approach, we need to be able to cancel the
> request from within the callback as well as externally. Looking to the Flow
> API again for inspiration, we could have the admin client pass an object to
> the callback which can be used for cancellation. In the simple case, users
> can ignore this object. In the advanced case, they can create a concrete
> class for the callback and cache the cancellation object so it can be
> accessed externally. This would be similar to the Subscription in the Flow
> API.
>
> ---
> Topics / Partitions:
>
> For the case of topic descriptions, we actually have two data types
> interleaved in one stream (topics and partitions). This means if we go with
> TopicDescription in the "onNext" method, we will have a partial set of
> topics in some cases. Also, we will end up calling "onNext" more than once
> for each RPC in the case that a single RPC response spans multiple topics.
>
> One alternative to a single "onNext" would be an interface more tailored to
> the RPC like:
>
> interface DescribeTopicsStreamObserver {
>  // Called for each topic in the result stream.
>  void onTopic(TopicInfo topic);
>
>  // Called for each partition of the topic last handled by onTopic
>  void onPartition(TopicPartitionInfo partition);
>
>  // Called once the broker has finished streaming results to the admin
> client. This marks the end of the stream.
>  void onComplete();
>
>  // Called if an error occurs on the underlying stream. This marks the end
> of the stream.
>  void onError(Throwable t);
> }
>
> ---
> Consumer API:
>
> Offline, there was some discussion about using a simple SAM consumer-like
> interface:
>
> interface AdminResultsConsumer {
>  void onNext(T next, Throwable t);
> }
>
> This has the benefit of being quite simple and letting callers supply a
> lambda instead of a full anonymous class definition. This would use
> nullable arguments like CompletableFuture#whenComplete. We could also use
> an Optional pattern here instead of nullables.
>
> ---
> Summary:
>
> So far, it seems like we are looking at these different options. The main
> difference in terms of API design is if the user will need to implement
> more than one method, or if a lambda can suffice.
>
> 1. Generic, Flow-like interface: AdminResultsSubscriber
> 2. DescribeTopicsStreamObserver (in this message above)
> 3. AdminResultsConsumer
> 4. AdminResultsConsumer with an Optional-like type instead of nullable
> arguments
>
>
>
> -David
>
>
>
>
> On Fri, Feb 23, 2024 at 4:00 PM José Armando García Sancio
>  wrote:
>
>> Hi Calvin
>>
>> On Fri, Feb 23, 2024 at 9:23 AM Calvin Liu 
>> wrote:
>>> As we agreed to implement the pagination for the new API
>>> DescribeTopicPartitions, the client side must also add a proper interface
>>> to handle the pagination.
>>> The current KafkaAdminClient.describeTopics returns
>>> the DescribeTopicsResult which is the future for querying all the topics.
>>> It is awkward to fit the pagination into it because
>>
>> I suggest taking a look at Java's Flow API:
>>
>> https://docs.oracle.com/en/java/javase/21/docs/api/java.base/java/util/concurrent/Flow.html
>> It was design for this specific use case and many libraries integrate with
>> it.
>>
>> If the Kafka client cannot be upgraded to support the Java 9 which
>> introduced that API, you can copy the same interface and semantics.
>> This would allow users to easily integrate with reactive libraries
>> since they all integrate with Java Flow.
>>
>> Thanks,
>> --
>> -José
>>
>
>
> --
> -David

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

[José Armando García Sancio]

Hi Justine and Jun,

Thanks for the KIP Justine. See my comments below.

On Wed, Feb 28, 2024 at 3:09 PM Jun Rao  wrote:
> 13. KIP-853 also extends the tools to support a new feature kraft.version.
> It would be useful to have alignment between that KIP and this one.

I agree. I took a look at this KIP and these are the differences that
I can spot.

1. KIP-853 uses --feature for kafka-storage instead of --features.
This is consistent with the use of --feature in the "kafka-feature.sh
upgrade" command.

2. I find it a bit inconsistent that --feature and --release-version
are mutually exclusive in the kafka-feature CLI but not in the
kafka-storage CLI. What is the reason for this decision?

3. KIP-853 deprecates --metadata in the kafka-features and makes it an
alias for --release-version. In KIP-1022, what happens if the user
specified both --metadata and --feature?

4. There is an example: "kafka-storage format --features
metadata.version=16,transaction.version=2,group.coordinator.version=1".
This is different from the --feature flag in kafka-features which is
an append flag. Meaning that multiple features are specified as
"--feature metadata.version=16 --feature transaction.version=2
--feature group.coordinator.version=1". I would suggest keeping this
consistent with kafka-features. It would avoid having to implement one
more parser in Kafka.

5. In the section "A Note on Feature Interdependence", you mention "an
error should be thrown if trying to format with X version 13 and Y
version 12". How would the user discover (or describe) these
dependencies? As currently described, trial and error seem to be the
only mechanism. Should the Kafka documentation describe these
dependencies? Is that good enough?

6. In "3.8-IV2 that could map to TV1, 3.8-IV3 could also be TV1, and
3.8-IV3 could be TV2" did you mean that 3.8-IV4 would map to TV2? If
not, 3.8-IV3 would map to two different TV values.

7. For --release-version in "kafka-features upgrade", does
--release-version mean that all of the feature versions will either
upgrade or stay the same? Meaning that downgrades will be rejected by
the Kafka controller. How about when --release-version is used with
"kafka-features downgrade"?

-José

Re: [DISCUSS] KIP-939: Support Participation in 2PC

[Artem Livshits]

Hi Jun,

>  So, it doesn't provide the same guarantees as 2PC either.

I think the key point is that we don't claim 2PC guarantees in that case.
Maybe it's splitting hairs from the technical perspective (in the end of
the day if the operator doesn't let the user use 2PC, it's going to be a
"works until timeout" solution), but from user model perspective it
provides a clear structure:

- if 2PC is possible then all guarantees are in place and there is no gray
area where we sort of provide guarantees but not fully
- if 2PC is not possible, then it's a well-informed constrain / decision
with well-known characteristics and the user can choose whether this is
acceptable or not for them

Maybe we can look at it from a slightly different perspective: we are not
making a choice between allowing or not allowing using keepPrepareTxn=true
when 2PC=false (even though that's exactly how it looks from the KIP).  In
fact, we're making a choice is whether Flink will be able to use an
official API when 2PC is not possible (and I think we've converged to agree
that sometimes it won't be) or keep using a reflection hack.  In other
words, we already have a hacky implementation for the case of
keepPrepareTxn=true + 2PC=false, our choice is only whether we provide an
official API for that or not.

In general, if someone goes and implements a reflection-based solution
that's an indication that there is a gap in public APIs.  And we can debate
whether keepPreparedTxn=true + 2PC=false is the right API or not; and if we
think it's not, then we should provide an alternative.  Right now the
alternative is to just keep using the reflection and I think it's always
worse than using a public API.

-Artem

On Wed, Feb 28, 2024 at 2:23 PM Jun Rao  wrote:

> Hi, Artem,
>
> Thanks for the reply.
>
> I understand your concern on having a timeout breaking the 2PC guarantees.
> However, the fallback plan to disable 2PC with an independent
> keepPreparedTxn is subject to the timeout too. So, it doesn't provide the
> same guarantees as 2PC either.
>
> To me, if we provide a new functionality, we should make it easy such that
> the application developer only needs to implement it in one way, which is
> always correct. Then, we can consider what additional things are needed to
> make the operator comfortable enabling it.
>
> Jun
>
> On Tue, Feb 27, 2024 at 4:45 PM Artem Livshits
>  wrote:
>
> > Hi Jun,
> >
> > Thank you for the discussion.
> >
> > > For 3b, it would be useful to understand the reason why an admin
> doesn't
> > authorize 2PC for self-hosted Flink
> >
> > I think the nuance here is that for cloud, there is a cloud admin
> > (operator) and there is cluster admin (who, for example could manage acls
> > on topics or etc.).  The 2PC functionality can affect cloud operations,
> > because a long running transaction can block the last stable offset and
> > prevent compaction or data tiering.  In a multi-tenant environment, a
> long
> > running transaction that involves consumer offset may affect data that is
> > shared by multiple tenants (Flink transactions don't use consumer
> offsets,
> > so this is not an issue for Flink, but we'd need a separate ACL or some
> > other way to express this permission if we wanted to go in that
> direction).
> >
> > For that reason, I expect 2PC to be controlled by the cloud operator and
> it
> > just may not be scalable for the cloud operator to manage all potential
> > interactions required to resolve in-doubt transactions (communicate to
> the
> > end users, etc.).  In general, we make no assumptions about Kafka
> > applications -- they may come and go, they may abandon transactional ids
> > and generate new ones.  For 2PC we need to make sure that the application
> > is highly available and wouldn't easily abandon an open transaction in
> > Kafka.
> >
> > > If so, another way to address that is to allow the admin to set a
> timeout
> > even for the 2PC case.
> >
> > This effectively abandons the 2PC guarantee because it creates a case for
> > Kafka to unilaterally make an automatic decision on a prepared
> > transaction.  I think it's fundamental for 2PC to abandon this ability
> and
> > wait for the external coordinator for the decision, after all the
> > coordinator may legitimately be unavailable for an arbitrary amount of
> > time.  Also, we already have a timeout on regular Kafka transactions,
> > having another "special" timeout could be confusing, and a large enough
> > timeout could still produce the undesirable effects for the cloud
> > operations (so we kind of get worst of both options -- we don't provide
> > guarantees and still have impact on operations).
> >
> > -Artem
> >
> > On Fri, Feb 23, 2024 at 8:55 AM Jun Rao 
> wrote:
> >
> > > Hi, Artem,
> > >
> > > Thanks for the reply.
> > >
> > > For 3b, it would be useful to understand the reason why an admin
> doesn't
> > > authorize 2PC for self-hosted Flink. Is the main reason that 2PC has
> > > unbounded timeout that could lead to unbound

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

[José Armando García Sancio]

Hi Jun and Luke,

Jun,

I added a section the documents the process for upgrading the
controller listeners endpoints:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes#KIP853:KRaftControllerMembershipChanges-Upgradingcontrollerlistener

Jun and Luke,

I am going to address your latest comments next.

Thanks!
-- 
-José

[DISCUSS] KIP-1024: Make the restore behavior of GlobalKTables with custom processors configureable

[Walker Carlson]

Hello everybody,

I wanted to propose a change to our addGlobalStore methods so that the
restore behavior can be controlled on a preprocessor level. This should
help Kafka Stream users to better tune Global stores added with the
processor API to better fit their needs.

Details are in the kip here: https://cwiki.apache.org/confluence/x/E4t3EQ

Thanks,
Walker

[jira] [Created] (KAFKA-16316) Make the restore behavior of GlobalKTables with custom processors configureable

[Walker Carlson (Jira)]

Walker Carlson created KAFKA-16316:
--

 Summary: Make the restore behavior of GlobalKTables with custom 
processors configureable
 Key: KAFKA-16316
 URL: https://issues.apache.org/jira/browse/KAFKA-16316
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Walker Carlson
Assignee: Walker Carlson


Take the change implemented in https://issues.apache.org/jira/browse/KAFKA-7663 
and make it optional through adding a couple methods to the API



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2687

[Apache Jenkins Server]

See 

Re: [DISCUSS] Uninitialized metadata cache during broker startup

[Chris Wildman]

I up PR 15441  with the
immediate fix for the KafkaAdminClient to help get this conversation
started. I would appreciate some feedback on that.

Thanks,
Chris

On Fri, Feb 23, 2024 at 3:41 PM Chris Wildman  wrote:

> Hi All,
>
> I recently discovered the race condition where Kafka clients may request
> metadata from brokers who have not yet received a snapshot of the
> cluster metadata. At scale with ZK managed clusters we see this happen
> frequently when brokers are restarted, possibly due to the admin client's
> preference for the least loaded node. This results in unexpected behavior
> for some metadata requests on the Admin api. For example describeTopics
> will fail with an UnknownTopicOrPartition exception for topics which do
> exist in the cluster.
>
> I also learned that consumers and producers ignore uninitialized metadata
> by detecting if the set of brokers in a metadata response is empty:
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L1187-L1191
>
> The only Admin API that has this protection seems to be listConsumerGroups
> which throws a StaleMetadataException for the empty brokers condition:
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L3369-L3371
>
> I'm interested in making it clearer to users of the Admin api when they
> should retry a metadata request because it was handled by a broker with
> uninitialized metadata. I think the proper fix here would be to have
> brokers respond with a UninitializedMetadataException when handling
> metadata requests if they haven't yet received a metadata snapshot. That is
> a big change that would need to be handled appropriately in all clients. A
> more immediate fix would be to change the KafkaAdminClient to *always*
> detect the empty brokers condition when getting a MetadataResponse and
> throw a StaleMetadataException or some other RetriableException.
>
> Some questions I have:
>
> 1. Does the likelihood of a broker responding with stale metadata decrease
> significantly or entirely when using KRAFT? I can understand not fixing
> this if that is the case. I tried but could not reproduce this behavior
> using kafka integration tests for both ZK and KRAFT.
>
> 2. Do we want to go for the proper fix or the more immediate one?
>
> 3. Would the immediate fix mentioned above, patching the KafkaAdminClient,
> require a KIP or should I just PR that?
>
> 4. Is StaleMetadataException the exception we want to use for the
> unitialized metadata case? From the docs for both StaleMetadataException
> and InvalidMetadataException it seems more geared toward old data, not
> uninitialized data.
>
> Thanks for your time and I hope this is an appropriate discussion!
>
> Chris Wildman
>

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

[Justine Olshan]

Hey folks,

Thanks for the discussion. Let me try to cover everyone's comments.

Artem --
I can add the examples you mentioned. As for naming, right now the feature
is called "transaction version" or "TV". Are you suggesting it should be
called "transaction protocol version" or "TPV"? I don't mind that, but just
wanted to clarify if we want to include protocol or if simply "transaction
version" is enough.

Jun --

10.  *With **more features, would each of those be controlled by a separate
feature or*

*multiple features. For example, is the new transaction record format*

*controlled only by MV with TV having a dependency on MV or is it
controlled*

*by both MV and TV.*


I think this will need to be decided on a case by case basis. There should
be a mechanism to set dependencies among features.
For transaction version specifically, I have no metadata version
dependencies besides requiring 3.3 to write the feature records and use the
feature tools. I would suspect all new features would have this requirement.


11. *Basically, if **--release-version is not used, the command will just
use the latest*

*production version of every feature. Should we apply that logic to both*

*tools?*


How would this work with the upgrade tool? I think we want a way to set a
new feature version for one feature and not touch any of the others.


*12. Should we remove --metadata METADATA from kafka-features? It does the*

*same thing as --release-version.*


When I previously discussed with Colin McCabe offline about this tool, he
was strongly against deprecation or changing flags. I personally think it
could good

to unify and not support a ton of flags, but I would want to make sure he
is aligned.


*13. KIP-853 also extends the tools to support a new feature kraft.version.*

*It would be useful to have alignment between that KIP and this one.*


Sounds good. Looks like Jose is in on the discussion so we can continue
here. :)



Jose --


*1. KIP-853 uses --feature for kafka-storage instead of --features.*

*This is consistent with the use of --feature in the "kafka-feature.sh*

*upgrade" command.*


I wanted to include multiple features in one command, so it seems like
features is a better name. I discuss more below about why I think we should
allow setting multiple features at once.


*2. I find it a bit inconsistent that --feature and --release-version*

*are mutually exclusive in the kafka-feature CLI but not in the*

*kafka-storage CLI. What is the reason for this decision?*


For the storage tool, we are setting all the features for the cluster. By
default, all are set. For the upgrade tool, the default is to set one
feature. In the storage tool, it is natural for the --release-version to
set the remaining features that --features didn't cover since otherwise we
would need to set them all

If we use the flag. In the feature upgrade case, it is less necessary for
all the features to be set at once and the tool can be run multiple times.
I'm not opposed to allowing both flags, but it is less necessary in my
opinion.


*3. KIP-853 deprecates --metadata in the kafka-features and makes it an*

*alias for --release-version. In KIP-1022, what happens if the user*

*specified both --metadata and --feature?*


See my note above (Jun's comment 12) about deprecating old commands. I
would think as the KIP stands now, we would not accept both commands.


*4. I would suggest keeping this*

*consistent with kafka-features. It would avoid having to implement one*

*more parser in Kafka.*


I sort of already implemented it as such, so I don't think it is too
tricky. I'm not sure of an alternative. Kafka features currently only
supports one feature at a time.
I would like to support more than one for the storage tool. Do you have
another suggestion for multiple features in the storage tool?


*5. As currently described, trial and error seem to be the*

*only mechanism. Should the Kafka documentation describe these*

*dependencies? Is that good enough?*


The plan so far is documentation. The idea is that this is an advanced
feature, so I think it is reasonable to ask folks use documentation


*6. Did you mean that 3.8-IV4 would map to TV2? If*

*not, 3.8-IV3 would map to two different TV values.*


It was a typo. Each MV maps to a single other feature version.


*7. For --release-version in "kafka-features upgrade", does*

*--release-version mean that all of the feature versions will either*

*upgrade or stay the same? Meaning that downgrades will be rejected by*

*the Kafka controller. How about when --release-version is used with*

*"kafka-features downgrade"?*


The idea I had was that the cluster will check if all the versions are
supported. If any version is not supported the tool will throw an error. We
can also issue a warning if the given command (if supported by the cluster)
will downgrade a feature. One option is also to require a yes/no prompt or
flag to allow downgrades. The downgrade tool would allow the same.
Alternativel

[jira] [Created] (KAFKA-16317) Add event rate in GroupCoordinatorRuntimeMetrics

[Jeff Kim (Jira)]

Jeff Kim created KAFKA-16317:


 Summary: Add event rate in GroupCoordinatorRuntimeMetrics
 Key: KAFKA-16317
 URL: https://issues.apache.org/jira/browse/KAFKA-16317
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jeff Kim
Assignee: Jeff Kim


We want a sensor to record every time we process a new event in the coordinator 
runtime.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

[José Armando García Sancio]

Thanks for the reply Justine. See my comments below:

On Thu, Feb 29, 2024 at 3:39 PM Justine Olshan
 wrote:
> I wanted to include multiple features in one command, so it seems like
> features is a better name. I discuss more below about why I think we should
> allow setting multiple features at once.

--feature in kafka-features allows for multiple features to be
specified. Please see the implementation of the tool and
UpdateFeatures RPC.

For example, you can execute this command kafka-features.sh upgrade
--feature metadata.version=15 --feature kraft.version=1. You should be
able to support the same UI in kafka-storage.sh. It is what KIP-853
suggests in the interest of making it consistent across CLI.

> For the storage tool, we are setting all the features for the cluster. By
> default, all are set. For the upgrade tool, the default is to set one
> feature. In the storage tool, it is natural for the --release-version to
> set the remaining features that --features didn't cover since otherwise we
> would need to set them all
>
> If we use the flag. In the feature upgrade case, it is less necessary for
> all the features to be set at once and the tool can be run multiple times.
> I'm not opposed to allowing both flags, but it is less necessary in my
> opinion.

I was thinking of making them mutually exclusive in both cases. Much
easier to explain and document. If the user wants to use --feature in
kafka-storage, they need to specify all of the supported features.

For the "kafka-feature upgrade" case, they can enumerate all of the
--feature versions that they want to upgrade in one command.

> See my note above (Jun's comment 12) about deprecating old commands. I
> would think as the KIP stands now, we would not accept both commands.

If you are not going to deprecate or alias --metadata what happens if
the user uses these flags in one command: "kafka-features upgrade
--metadata 3.8 --feature kraft.version=1"

One of the problems I am having is that I can't seem to find the KIP
that introduces --metadata so I can only speculate as to the intent of
this flag from the CLI help and implementation. Do you know which KIP
added that flag?

> I sort of already implemented it as such, so I don't think it is too
> tricky. I'm not sure of an alternative. Kafka features currently only
> supports one feature at a time.
> I would like to support more than one for the storage tool. Do you have
> another suggestion for multiple features in the storage tool?

"kafka-features.sh upgrade" supports multiple --feature flags. Please
see the tools implementation and the UpdateFeatures RPC.

You can specify multiple features in the storage tool with:
"kafka-storage format --feature kraft.version=1 --feature
metadata.version=15". The command line parser used by both tools
support flags that append values to a list. This is how the
kafka-features CLI works already.

> The plan so far is documentation. The idea is that this is an advanced
> feature, so I think it is reasonable to ask folks use documentation

Are you going to generate the documentation of these dependencies
automatically from the released code like we do for Kafka
configuration properties? Or do Kafka developers need to remember to
update the documentation with these dependencies?

> The idea I had was that the cluster will check if all the versions are
> supported. If any version is not supported the tool will throw an error. We
> can also issue a warning if the given command (if supported by the cluster)
> will downgrade a feature. One option is also to require a yes/no prompt or
> flag to allow downgrades. The downgrade tool would allow the same.
> Alternatively we could just fail the command if a feature is downgraded on
> upgrade command or vice versa. I don't have a strong preference.

"kafka-features upgrade" should only upgrade and "kafka-features
downgrade" command should only downgrade. It should be an error if any
of the UpdateFeatures requests violates this. What we need to do is
define if an error in one feature results in an error for the entire
request. The UpdateFeatures RPC seems to allow individual errors but
that gets confusing and difficult to enforce once you introduce
dependencies between feature versions.

Thanks!
-- 
-José

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

[Justine Olshan]

Thanks José --

Let me answer a few quick things

If you are not going to deprecate or alias --metadata what happens if
the user uses these flags in one command: "kafka-features upgrade
--metadata 3.8 --feature kraft.version=1"

> I  would think as the KIP stands now, we would not accept both commands

Metadata flag was added as part of KIP-778. See
https://cwiki.apache.org/confluence/display/KAFKA/KIP-778%3A+KRaft+to+KRaft+Upgrades
and
https://github.com/apache/kafka/commit/28d5a059438634db6fdecdbb816e2584715884d6


I think the usage of --feature multiple times is a little bit clunky, but
we can use it for consistency with the existing tool. I was hoping it could
be simple to set one feature differently and take the default for
everything else. But alas.
For the storage tool, if we have to set all the features, do you expect to
throw an error when one is missing? This seems acceptable, but just wanted
to clarify.

For documentation, we can have an autogenerated doc. I haven't thought too
much how this would look yet.

I am ok with throwing an error if a feature isn't moving in the correct
direction (upgrade or downgrade) for the command that is given.

I will continue to think about everyone's comments and update the KIP over
the next few days.

Thanks,
Justine

On Thu, Feb 29, 2024 at 4:31 PM José Armando García Sancio
 wrote:

> Thanks for the reply Justine. See my comments below:
>
> On Thu, Feb 29, 2024 at 3:39 PM Justine Olshan
>  wrote:
> > I wanted to include multiple features in one command, so it seems like
> > features is a better name. I discuss more below about why I think we
> should
> > allow setting multiple features at once.
>
> --feature in kafka-features allows for multiple features to be
> specified. Please see the implementation of the tool and
> UpdateFeatures RPC.
>
> For example, you can execute this command kafka-features.sh upgrade
> --feature metadata.version=15 --feature kraft.version=1. You should be
> able to support the same UI in kafka-storage.sh. It is what KIP-853
> suggests in the interest of making it consistent across CLI.
>
> > For the storage tool, we are setting all the features for the cluster. By
> > default, all are set. For the upgrade tool, the default is to set one
> > feature. In the storage tool, it is natural for the --release-version to
> > set the remaining features that --features didn't cover since otherwise
> we
> > would need to set them all
> >
> > If we use the flag. In the feature upgrade case, it is less necessary for
> > all the features to be set at once and the tool can be run multiple
> times.
> > I'm not opposed to allowing both flags, but it is less necessary in my
> > opinion.
>
> I was thinking of making them mutually exclusive in both cases. Much
> easier to explain and document. If the user wants to use --feature in
> kafka-storage, they need to specify all of the supported features.
>
> For the "kafka-feature upgrade" case, they can enumerate all of the
> --feature versions that they want to upgrade in one command.
>
> > See my note above (Jun's comment 12) about deprecating old commands. I
> > would think as the KIP stands now, we would not accept both commands.
>
> If you are not going to deprecate or alias --metadata what happens if
> the user uses these flags in one command: "kafka-features upgrade
> --metadata 3.8 --feature kraft.version=1"
>
> One of the problems I am having is that I can't seem to find the KIP
> that introduces --metadata so I can only speculate as to the intent of
> this flag from the CLI help and implementation. Do you know which KIP
> added that flag?
>
> > I sort of already implemented it as such, so I don't think it is too
> > tricky. I'm not sure of an alternative. Kafka features currently only
> > supports one feature at a time.
> > I would like to support more than one for the storage tool. Do you have
> > another suggestion for multiple features in the storage tool?
>
> "kafka-features.sh upgrade" supports multiple --feature flags. Please
> see the tools implementation and the UpdateFeatures RPC.
>
> You can specify multiple features in the storage tool with:
> "kafka-storage format --feature kraft.version=1 --feature
> metadata.version=15". The command line parser used by both tools
> support flags that append values to a list. This is how the
> kafka-features CLI works already.
>
> > The plan so far is documentation. The idea is that this is an advanced
> > feature, so I think it is reasonable to ask folks use documentation
>
> Are you going to generate the documentation of these dependencies
> automatically from the released code like we do for Kafka
> configuration properties? Or do Kafka developers need to remember to
> update the documentation with these dependencies?
>
> > The idea I had was that the cluster will check if all the versions are
> > supported. If any version is not supported the tool will throw an error.
> We
> > can also issue a warning if the given command (if supported by the
> clu

[PR] PayPal powered by Apache Kafka section Update [kafka-site]

[via GitHub]

parvase opened a new pull request, #590:
URL: https://github.com/apache/kafka-site/pull/590

   Hi Team,
   Please review the changes made to powered-by-page to update the PayPal Kafka 
usage description and link.
   
   Thanks
   Parvase


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [ANNOUNCE] Apache Kafka 3.7.0

[Kamal Chandraprakash]

Thanks Stanislav for running the release!

On Thu, Feb 29, 2024, 21:03 Guozhang Wang 
wrote:

> Thanks Stan for running the release!
>
> On Thu, Feb 29, 2024 at 5:39 AM Boudjelda Mohamed Said
>  wrote:
> >
> > Thanks Stanislav for running the release!
> >
> > On Wed, Feb 28, 2024 at 10:36 PM Kirk True  wrote:
> >
> > > Thanks Stanislav
> > >
> > > > On Feb 27, 2024, at 10:01 AM, Stanislav Kozlovski <
> > > stanislavkozlov...@apache.org> wrote:
> > > >
> > > > The Apache Kafka community is pleased to announce the release of
> > > > Apache Kafka 3.7.0
> > > >
> > > > This is a minor release that includes new features, fixes, and
> > > > improvements from 296 JIRAs
> > > >
> > > > An overview of the release and its notable changes can be found in
> the
> > > > release blog post:
> > > > https://kafka.apache.org/blog#apache_kafka_370_release_announcement
> > > >
> > > > All of the changes in this release can be found in the release notes:
> > > > https://www.apache.org/dist/kafka/3.7.0/RELEASE_NOTES.html
> > > >
> > > > You can download the source and binary release (Scala 2.12, 2.13)
> from:
> > > > https://kafka.apache.org/downloads#3.7.0
> > > >
> > > >
> > >
> ---
> > > >
> > > >
> > > > Apache Kafka is a distributed streaming platform with four core APIs:
> > > >
> > > >
> > > > ** The Producer API allows an application to publish a stream of
> records
> > > to
> > > > one or more Kafka topics.
> > > >
> > > > ** The Consumer API allows an application to subscribe to one or more
> > > > topics and process the stream of records produced to them.
> > > >
> > > > ** The Streams API allows an application to act as a stream
> processor,
> > > > consuming an input stream from one or more topics and producing an
> > > > output stream to one or more output topics, effectively transforming
> the
> > > > input streams to output streams.
> > > >
> > > > ** The Connector API allows building and running reusable producers
> or
> > > > consumers that connect Kafka topics to existing applications or data
> > > > systems. For example, a connector to a relational database might
> > > > capture every change to a table.
> > > >
> > > >
> > > > With these APIs, Kafka can be used for two broad classes of
> application:
> > > >
> > > > ** Building real-time streaming data pipelines that reliably get data
> > > > between systems or applications.
> > > >
> > > > ** Building real-time streaming applications that transform or react
> > > > to the streams of data.
> > > >
> > > >
> > > > Apache Kafka is in use at large and small companies worldwide,
> including
> > > > Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest,
> Rabobank,
> > > > Target, The New York Times, Uber, Yelp, and Zalando, among others.
> > > >
> > > > A big thank you to the following 146 contributors to this release!
> > > > (Please report an unintended omission)
> > > >
> > > > Abhijeet Kumar, Akhilesh Chaganti, Alieh, Alieh Saeedi, Almog Gavra,
> > > > Alok Thatikunta, Alyssa Huang, Aman Singh, Andras Katona, Andrew
> > > > Schofield, Anna Sophie Blee-Goldman, Anton Agestam, Apoorv Mittal,
> > > > Arnout Engelen, Arpit Goyal, Artem Livshits, Ashwin Pankaj,
> > > > ashwinpankaj, atu-sharm, bachmanity1, Bob Barrett, Bruno Cadonna,
> > > > Calvin Liu, Cerchie, chern, Chris Egerton, Christo Lolov, Colin
> > > > Patrick McCabe, Colt McNealy, Crispin Bernier, David Arthur, David
> > > > Jacot, David Mao, Deqi Hu, Dimitar Dimitrov, Divij Vaidya, Dongnuo
> > > > Lyu, Eaugene Thomas, Eduwer Camacaro, Eike Thaden, Federico Valeri,
> > > > Florin Akermann, Gantigmaa Selenge, Gaurav Narula, gongzhongqiang,
> > > > Greg Harris, Guozhang Wang, Gyeongwon, Do, Hailey Ni, Hanyu Zheng,
> Hao
> > > > Li, Hector Geraldino, hudeqi, Ian McDonald, Iblis Lin, Igor Soarez,
> > > > iit2009060, Ismael Juma, Jakub Scholz, James Cheng, Jason Gustafson,
> > > > Jay Wang, Jeff Kim, Jim Galasyn, John Roesler, Jorge Esteban Quilcate
> > > > Otoya, Josep Prat, José Armando García Sancio, Jotaniya Jeel, Jouni
> > > > Tenhunen, Jun Rao, Justine Olshan, Kamal Chandraprakash, Kirk True,
> > > > kpatelatwork, kumarpritam863, Laglangyue, Levani Kokhreidze, Lianet
> > > > Magrans, Liu Zeyu, Lucas Brutschy, Lucia Cerchie, Luke Chen,
> maniekes,
> > > > Manikumar Reddy, mannoopj, Maros Orsak, Matthew de Detrich, Matthias
> > > > J. Sax, Max Riedel, Mayank Shekhar Narula, Mehari Beyene, Michael
> > > > Westerby, Mickael Maison, Nick Telford, Nikhil Ramakrishnan, Nikolay,
> > > > Okada Haruki, olalamichelle, Omnia G.H Ibrahim, Owen Leung, Paolo
> > > > Patierno, Philip Nee, Phuc-Hong-Tran, Proven Provenzano, Purshotam
> > > > Chauhan, Qichao Chu, Matthias J. Sax, Rajini Sivaram, Renaldo Baur
> > > > Filho, Ritika Reddy, Robert Wagner, Rohan, Ron Dagostino, Roon,
> runom,
> > > > Ruslan Krivoshein, rykovsi, Sagar Rao, Said Boudjelda, Satish
> Duggana,
> > > > shuoer86, Stanislav Kozlovski, Taher