Re: [DISCUSS] KIP-1050: Consistent error handling for Transactions

[Lianet M.]

Hello, thanks for the KIP! After going through the KIP and discussion here
are some initial comments.

107 - I understand we’re  proposing a new
ProducerRetriableTransactionException, and changing existing exceptions to
inherit from it (the ones on the table below it).  The existing exceptions
inherit from RetriableException today, but with this KIP, they would
inherit from ProducerRetriableTransactionException which is not a
RetriableException ("*ProducerRetriableTransactionException extends
ApiException"*). Is my understanding correct? Wouldn’t this break
applications that could be handling/expecting RetriableExceptions today?
(Ie. apps dealing with TimeoutException on send , if they have
catch(RetriableException) or checks in the form of instanceOf
RetriableException, would need to change to the new
 ProducerRetriableTransactionException or the specific TimeoutException,
right?). I get this wouldn’t bring a problem for most of the retriable
exceptions on the table given that they end up being handled/retried
internally, but TimeoutException is tricky.


108 -  Regarding how we limit the scope of the change to the
producer/transactional API. TimeoutException is not only used in the
transactional API, but also in the consumer API, propagated to the user in
multiple api calls. Not clear to me how with this proposal we wouldn’t end
up with a consumer throwing a TimeoutException instanceOf
ProducerRetriableTransactionException? (Instead of instanceOf
RetriableException like it is today)? Again, potentially breaking apps but
also with a conceptually wrong consumer error?


109 - Similar to above, for exceptions like
UnknownTopicOrPartitionException, which are today instanceOf
RetriableException, if we’re saying they will be subclass of
ProducerRefreshRetriableTransactionException (ApiException) that will
affect the consumer logic too, where we do handle RetriableExceptions like
the unknownTopic, expecting RetriableException. This is all internal logic
and could be updated as needed of course, but without leaking
producer-specific groupings into the consumer I would expect.


110 - The KIP refers to the existing TransactionAbortableException (from
KIP-890), but on the public changes it refers to class
AbortableTransactionException extends ApiException. So are we proposing a
new exception type for this or reusing the existing one?

111 - I notice the proposed new exceptions, even though they seem like
abstract groupings, are not defined as "abstract". Is it intentional to
allow creation of instances of those?

Best!

Lianet


On Thu, Sep 12, 2024 at 6:26 AM Kaushik Raina 
wrote:

> Thanks for review Matthias
>
> 100/101 - Updated in KIP
>
> 104 - Added explicitly
> `For Producer-Retriable errors, the producer handles retries internally,
> keeping the failure details hidden from the application. Conversely, other
> types of exceptions will be surfaced to the application code for handling.`
>
> 105 - Grouped default exceptions explicitly
> `We will handle all default exceptions as generic unknown errors, which
> will be application recoverable. Below are few such exceptions:`
>
>
> On Sat, Aug 31, 2024 at 4:27 AM Matthias J. Sax  wrote:
>
> > Thanks for updating the KIP. It's much clearer now what you propose. I
> > have a bunch of question about the proposal:
> >
> >
> >
> > (100) nit (typo / missing word?):
> >
> > > We would new error types
> >
> >
> >
> > (101) `TransactionAbortableException`, `ProducerFencedException`, and
> > `UnknownProducerIdException` are listed twice in the tables.
> >
> >
> >
> > (102) You introduce a new exception `AbortableTransactionException`
> > which will only be extended by `TransactionAbortableException`. Given
> > that the existing TransactionAbortableException is not thrown by the
> > producer right now (no passed into the `Callback`), it seem if the
> > producer starts to throw/return the exiting
> > `TransactionAbortableException` or the new
> > `AbortableTransactionException` is would be an incompatible change?
> >
> >
> >
> > (103) It's unclear which method would throw the new
> > `AbortableTransactionException` and/or if this new exception might be
> > passe into the producer's send `Callback`.
> >
> >
> >
> > Btw: KIP-890 does not mention `TransactionAbortableException`... Does
> > KIP-890 need an update? KIP-890 only mentions a new error code
> > TRANSACTION_ABORTABLE -- or is this an implicit introduction of
> > TransactionAbortableException -- I am not familiar with the details how
> > core KIPs are written?
> >
> >
> >
> > (104) The KIP does not explicitly say, which of the new exceptions are
> > actually user facing? It seems only AbortableTransactionException,
> > ApplicationRecoverableTransactionException, and
> > InvalidConfiguationTransactionException are exception which user will be
> > able to catch (or handle vie the `Callback`), while
> > ProducerRetriableTransactionException and
> > ProducerRefreshRetriableTransactionException won't be thrown/return by
> >

Re: [VOTE] KIP-1082: Require Client-Generated IDs over the ConsumerGroupHeartbeat RPC

[Lianet M.]

Hello TengYao, just one last minor comment. The KIP refers in multiple
places to bumping the heartbeat RPC from version 0 to version 1. We should
update that given that the RPC is already in version 1.

With that I'm +1 (binding)

Thanks!
Lianet

On Thu, Oct 3, 2024 at 5:01 AM David Jacot 
wrote:

> +1 (binding)
>
> Thanks for the KIP!
>
> Best,
> David
>
> On Thu, Oct 3, 2024 at 10:49 AM TengYao Chi  wrote:
>
> > Hello everyone,
> > As the vote has been pending for two weeks, I would like to push it
> > manually.
> > Thank you for your attention.
> >
> > Best,
> > TengYao
> >
> > Chris Egerton  於 2024年9月19日 週四 上午1:42寫道：
> >
> > > Thanks for the KIP! +1 (binding)
> > >
> > > On Tue, Sep 17, 2024 at 12:45 PM Andrew Schofield <
> > > andrew_schofield_j...@outlook.com> wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > 
> > > > From: 吳岱儒 
> > > > Sent: 17 September 2024 04:24
> > > > To: dev@kafka.apache.org 
> > > > Subject: Re: [VOTE] KIP-1082: Require Client-Generated IDs over the
> > > > ConsumerGroupHeartbeat RPC
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Best,
> > > > TaiJuWu
> > > >
> > > > Chia-Ping Tsai  於 2024年9月16日 週一 下午11:41寫道：
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > thanks for reaching out to the corner case.
> > > > >
> > > > > Best,
> > > > > Chia-Ping
> > > > >
> > > > > Kirk True  於 2024年9月16日 週一 下午11:36寫道：
> > > > >
> > > > > > Hi TengYao,
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > Thanks for all the hard work with the tricky edge cases :)
> > > > > >
> > > > > > Kirk
> > > > > >
> > > > > > > On Sep 16, 2024, at 6:47 AM, TengYao Chi 
> > > > wrote:
> > > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > Based on our discussion
> > > > > > > <
> > https://lists.apache.org/thread/vvytypk3l8cvv8yltrckfg6yf7ovd371>
> > > > > > > regarding KIP-1082 <
> https://cwiki.apache.org/confluence/x/XBDOEg
> > >,
> > > I
> > > > > > > believe this KIP is now ready for a vote.
> > > > > > >
> > > > > > > Sincerely,
> > > > > > > TengYao
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

[Lianet M.]

Congratulations Kamal!!!



On Mon, Sep 30, 2024, 9:22 a.m. Chia-Ping Tsai  wrote:

> Congratulations Kamal!!!
>
> Satish Duggana  於 2024年9月30日 週一 下午9:01寫道：
>
> > Congratulations Kamal!
> >
> > On Mon, 30 Sept 2024 at 18:18, Josep Prat 
> > wrote:
> >
> > > Congrats Kamal!
> > >
> > > On Mon, Sep 30, 2024 at 2:46 PM  wrote:
> > >
> > > > Congrats Kamal!
> > > >
> > > > -
> > > > Gaurav
> > > >
> > > > > On 30 Sep 2024, at 18:10, Mickael Maison  >
> > > > wrote:
> > > > >
> > > > > Congratulations Kamal!
> > > > >
> > > > > On Mon, Sep 30, 2024 at 2:37 PM Luke Chen 
> wrote:
> > > > >>
> > > > >> Hi all,
> > > > >>
> > > > >> The PMC of Apache Kafka is pleased to announce a new Kafka
> > committer,
> > > > Kamal
> > > > >> Chandraprakash.
> > > > >>
> > > > >> Kamal has been a Kafka contributor since May 2017. He has made
> > > > significant
> > > > >> contributions to the tiered storage feature (KIP-405). He authored
> > > > KIP-1018
> > > > >> and KIP-1075 which improved tiered storage operation. He also
> > > > contributed
> > > > >> to discussing and reviewing many KIPs.
> > > > >>
> > > > >> Congratulations, Kamal!
> > > > >>
> > > > >> Thanks,
> > > > >> Luke (on behalf of the Apache Kafka PMC)
> > > >
> > > >
> > >
> > > --
> > > [image: Aiven] 
> > >
> > > *Josep Prat*
> > > Open Source Engineering Director, *Aiven
> 
> *
> > > josep.p...@aiven.io   |   +491715557497
> > > aiven.io    |   <
> > https://www.facebook.com/aivencloud
> > > >
> > >      <
> > > https://twitter.com/aiven_io>
> > > *Aiven Deutschland GmbH*
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > Anna Richardson, Kenneth Chen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> >
>

Re: [VOTE] KIP-1090 Flaky Test Management

[Lianet M.]

+1.

Thanks David!


On Thu, Sep 26, 2024, 7:47 p.m. Matthias J. Sax  wrote:

> +1
>
> On 9/26/24 3:38 PM, Chia-Ping Tsai wrote:
> > +1
> >
> > nit: Could you please add the KIP link to KAFKA-17629
> >
> > David Arthur  於 2024年9月27日 週五 上午6:31寫道：
> >
> >> I would like to call a vote for KIP-1090. Please take a moment to review
> >> the proposal and cast your vote.
> >>
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1090+Flaky+Test+Management
> >>
> >> Thanks!
> >> David A
> >>
> >
>

Re: [DISCUSS] Apache Kafka 4.0.0 release

[Lianet M.]

+1

Thanks David!

On Mon, Sep 23, 2024, 9:54 a.m. Justine Olshan 
wrote:

> +1 and thanks!
>
> On Mon, Sep 23, 2024 at 6:36 AM Chia-Ping Tsai  wrote:
>
> > +1 and thanks David for volunteering!
> >
> > Best,
> > Chia-Ping
> >
> > Josep Prat  於 2024年9月23日 週一 下午9:13寫道：
> >
> > > Thanks David for volunteering!
> > > +1
> > >
> > > On Mon, Sep 23, 2024 at 3:11 PM David Jacot
>  > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I would like to volunteer to be the release manager driving the next
> > > > release - Apache Kafka 4.0.0.
> > > >
> > > > If there are no objections, I'll start building a release plan in the
> > > wiki
> > > > in the next couple of days.
> > > >
> > > > Best,
> > > > David
> > > >
> > >
> > >
> > > --
> > > [image: Aiven] 
> > >
> > > *Josep Prat*
> > > Open Source Engineering Director, *Aiven*
> > > josep.p...@aiven.io   |   +491715557497
> > > aiven.io    |   <
> > https://www.facebook.com/aivencloud
> > > >
> > >      <
> > > https://twitter.com/aiven_io>
> > > *Aiven Deutschland GmbH*
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > Anna Richardson, Kenneth Chen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> >
>

Re: [VOTE] KIP-1043: Administration of groups

[Lianet M.]

Hello Andrew,

Thanks for the KIP.

+1 (binding)

Lianet

On Mon, Sep 23, 2024, 3:37 a.m. Lucas Brutschy
 wrote:

> Hi Andrew,
>
> thanks for the KIP!
>
> +1 (binding)
>
> Cheers,
> Lucas
>
> On Mon, Sep 23, 2024 at 9:27 AM Andrew Schofield
>  wrote:
> >
> > Hi,
> > I would like to start voting for KIP-1043: Administration of groups.
> This KIP enhances the command-line tools to make it easier to administer
> groups on clusters with a variety of types of groups.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1043%3A+Administration+of+groups
> >
> > Thanks.
> > Andrew
>

Re: [VOTE] KIP-1079: Deprecate `delete-config` of TopicCommand

[Lianet M.]

Thanks for the update! Looks good to me now.

+1 (binding)

Cheers,

Lianet

On Mon, Sep 16, 2024 at 9:31 AM TengYao Chi  wrote:

> Hi Lianet,
>
> Thanks for comment.
> It is very helpful and I have applied it to the content of this KIP.
> Please take a look.
>
> Sincerely,
> TengYao
>
> Lianet M.  於 2024年9月16日 週一 下午7:16寫道：
>
> > Hello TengYao.
> >
> > Thanks for the KIP, nice cleanup. Just one question left. I wonder how we
> > make sure that we don't just bring in this KIP, deprecate the arg and
> > forget about it?. First we could maybe mention in the KIP that is with
> the
> > intention of removing the option in 5.0 (just for reference), and also
> > create and include here the jira for it already (maybe in the
> > "Compatibility, Deprecation, and Migration Plan" section)?
> >
> > Thanks!
> >
> > Lianet
> >
> > On Mon, Sep 16, 2024 at 1:01 AM TengYao Chi  wrote:
> >
> > > Hello everyone,
> > > I hope this message finds you well.
> > > As the vote has been pending for two weeks, I would like to push it
> > > manually.
> > >
> > > Thank you for your attention.
> > >
> > > Sincerely,
> > > TengYao
> > >
> > > Andrew Schofield  於 2024年8月27日 週二
> 下午10:20寫道：
> > >
> > > > Thanks for the KIP.
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Andrew
> > > >
> > > > > On 27 Aug 2024, at 10:31, Federico Valeri 
> > > wrote:
> > > > >
> > > > > +1 non binding
> > > > >
> > > > > Should we also create a Jira to track future removal?
> > > > >
> > > > > On Tue, Aug 27, 2024 at 11:02 AM Chia-Ping Tsai <
> chia7...@apache.org
> > >
> > > > wrote:
> > > > >>
> > > > >> +1 (binding)
> > > > >>
> > > > >> On 2024/08/27 06:15:57 TengYao Chi wrote:
> > > > >>> Hi everyone,
> > > > >>>
> > > > >>> The discussion
> > > > >>> <
> https://lists.apache.org/thread/p4nz31zh6lokw4c1nld9dm0740oxw98j>
> > > of
> > > > >>> KIP-1079
> > > > >>> <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1079:+Deprecate+%60delete-config%60+of+TopicCommand
> > > > >
> > > > >>> has
> > > > >>> been silent for a long time, I believe this KIP is now ready for
> a
> > > > vote.
> > > > >>>
> > > > >>> Sincerely,
> > > > >>> TengYao
> > > > >>>
> > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-1079: Deprecate `delete-config` of TopicCommand

[Lianet M.]

Hello TengYao.

Thanks for the KIP, nice cleanup. Just one question left. I wonder how we
make sure that we don't just bring in this KIP, deprecate the arg and
forget about it?. First we could maybe mention in the KIP that is with the
intention of removing the option in 5.0 (just for reference), and also
create and include here the jira for it already (maybe in the
"Compatibility, Deprecation, and Migration Plan" section)?

Thanks!

Lianet

On Mon, Sep 16, 2024 at 1:01 AM TengYao Chi  wrote:

> Hello everyone,
> I hope this message finds you well.
> As the vote has been pending for two weeks, I would like to push it
> manually.
>
> Thank you for your attention.
>
> Sincerely,
> TengYao
>
> Andrew Schofield  於 2024年8月27日 週二 下午10:20寫道：
>
> > Thanks for the KIP.
> >
> > +1 (non-binding)
> >
> > Andrew
> >
> > > On 27 Aug 2024, at 10:31, Federico Valeri 
> wrote:
> > >
> > > +1 non binding
> > >
> > > Should we also create a Jira to track future removal?
> > >
> > > On Tue, Aug 27, 2024 at 11:02 AM Chia-Ping Tsai 
> > wrote:
> > >>
> > >> +1 (binding)
> > >>
> > >> On 2024/08/27 06:15:57 TengYao Chi wrote:
> > >>> Hi everyone,
> > >>>
> > >>> The discussion
> > >>> 
> of
> > >>> KIP-1079
> > >>> <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1079:+Deprecate+%60delete-config%60+of+TopicCommand
> > >
> > >>> has
> > >>> been silent for a long time, I believe this KIP is now ready for a
> > vote.
> > >>>
> > >>> Sincerely,
> > >>> TengYao
> > >>>
> >
> >
>

Re: [DISCUSS] KIP-1043: Administration of groups

[Lianet M.]

Hi Andrew,

Took another full pass here after the latest changes (thanks for addressing
all the feedback!).

Only minor comment left is to add to the reject alternatives the
INCONSISTENT_GROUP_PROTOCOL error that we discussed and discarded (see
LM9). Also agree with the 2 minor points made by Lucas.

With that, I have no other comments and agree we could proceed to voting.

Thanks!

Lianet

On Wed, Sep 11, 2024 at 10:17 AM Lucas Brutschy
 wrote:

> Hi Andrew,
>
> thanks for the KIP!
>
> It is looking good from my side! I like the simplification, and that
> we added the new error but only of the Describe RPCs. It's a good
> pragmatic improvement of the current state of things.
>
> I only have very minor comments:
>  - nit: In `GroupListing`, you seem to import `ShareGroupState` and
> it's not clear why.
>  - The documentation for `--consumer` in the table is not enough. We
> should make sure that the comment below the table is also included in
> the command-line help of the CLI tool -- I was confused by this at
> first. Possibly just explain it in terms of the equivalent sequence of
> commands.
>
> From my point of view, this is ready for a vote.
>
> Cheers,
> Lucas
>
>
>
> On Tue, Sep 3, 2024 at 2:56 PM Andrew Schofield
>  wrote:
> >
> > Hi,
> > I’ve spent some time working with clusters containing groups of multiple
> > types, fixing problems and improving error handling.
> >
> > I’ve simplified the KIP so that it just adds kafka-groups.sh and improves
> > the error handling for describing groups of the wrong type. With the
> other
> > improvements I’ve already made, it seems to me that this is sufficient to
> > make working with groups of multiple types work nicely.
> >
> > I’d like to ask for another round of reviews before hopefully opening up
> > a vote soon.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1043%3A+Administration+of+groups
> >
> > Thanks,
> > Andrew
> >
> > 
> > From: Andrew Schofield 
> > Sent: 02 August 2024 15:00
> > To: dev@kafka.apache.org 
> > Subject: Re: [DISCUSS] KIP-1043: Administration of groups
> >
> > Hi Lianet,
> > Thanks for your comment.
> >
> > I’ve been digging more into the situation with describing groups in a
> > broker with groups of multiple types. It’s a bit fiddly because of the
> > introduction of the modern consumer groups by KIP-848 and the
> > need for the admin client to cope with both kinds of consumer groups
> > and older brokers.
> >
> > If you use `kafka-consumer-groups.sh --describe --group MYSHARE`
> > the output is:
> >
> >   Error: Consumer group ‘MYSHARE’ does not exist.
> >
> > How does it get there? AdminClient.describeConsumerGroups
> > is complicated.
> >
> > First, it uses the ConsumerGroupDescribe RPC which responds
> > with GROUP_ID_NOT_FOUND (69) and an empty error message.
> > The broker *could* fill in the error message to help with this situation
> > but I don’t like that as a solution. Seems quite brittle.
> >
> > Then, it uses the DescribeGroups RPC in case it’s a classic consumer
> > group. This responds with error code NONE (0) and makes the group
> > look like a Dead consumer group. There is no error message field
> > in that RPC at all, so we don’t have the option of using an error
> > message to disambiguate.
> >
> > So, `kafka-consumer-groups.sh` thinks that it’s dealing with a dead
> > consumer group and its output makes sense.
> >
> > My preferred course of action here is as you suggest to introduce
> > the new error code, INVALID_GROUP_TYPE. If you use any of the following
> > RPCs with the wrong type of group, you get this response:
> >
> > * ConsumerGroupDescribe
> > * ShareGroupDescribe
> > * ConsumerGroupHeartbeat
> > * ShareGroupHeartbeat
> >
> > The remaining RPCs for consumer groups, such as ListOffsets and
> > TxnOffsetCommit continue to use `GROUP_ID_NOT_FOUND`.
> >
> > Does that make sense? Any further comments?
> >
> > Thanks,
> > Andrew
> >
> > > On 23 Jul 2024, at 17:26, Lianet M.  wrote:
> > >
> > > Hello Andrew,
> > >
> > > Bringing here the point I surfaced on the KIP-1071 thread:
> > >
> > > I wonder if at this point, where we're getting several new group types
> > >> added, each with RPCs that are supposed to include groupId of a
> certain
> > >> type, we should be more explicit about this situation. Maybe a kind of
> > >> INVALI

Re: [VOTE] KIP-1080: Fix the typo: `maxlifeTimeMs` in CreateDelegationTokenOptions

[Lianet M.]

Hello,

Thanks for updating the KIP. Looks good to me now.

+1 (binding)

Lianet


On Thu, Sep 5, 2024, 8:29 p.m. 黃竣陽  wrote:

> It make sense to me, I will change to
> `CreateDelegationTokenRequestData.setMaxLifetimeMs`
>
> > Chia-Ping Tsai  於 2024年9月5日 晚上11:22 寫道：
> >
> >> it aligns better with the RPC field (max_lifetime_ms)
> >
> > That makes sense to me. +1 to align it
> >
> > Best,
> > Chia-Ping
> >
> > Lianet M.  於 2024年9月5日 週四 下午11:13寫道：
> >
> >> Hey all, just one small comment: shouldn't the new method be named "
> >> maxLifetimeMs()" (Lifetime vs LifeTime)? I believe it's the right
> spelling,
> >> but mostly, it aligns better with the RPC field (max_lifetime_ms), that
> >> generates a CreateDelegationTokenRequestData.maxLifetimeMs() and
> >> CreateDelegationTokenRequestData.setMaxLifetimeMs(...)
> >>
> >> Thanks!
> >>
> >> Lianet
> >>
> >> On Thu, Sep 5, 2024 at 9:36 AM Chia-Ping Tsai 
> wrote:
> >>
> >>> +1 (binding)
> >>>
> >>> 黃竣陽  於 2024年9月5日 週四 下午9:08寫道：
> >>>
> >>>> Hello everyone,
> >>>>
> >>>> I would like to call for a vote on KIP-1080: <
> >>>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1080%3A+Fix+the+typo%3A+%60maxlifeTimeMs%60+in+CreateDelegationTokenOptions
> >>>>>
> >>>>
> >>>> The discussion can be found here:
> >>>> <https://lists.apache.org/thread/00jvqxoo6d59k5z5cjm9wtj6dd075vro>
> >>>>
> >>>> Best regards,
> >>>> Jiunn-Yang
> >>>
> >>
>
>

Re: [VOTE] KIP-1080: Fix the typo: `maxlifeTimeMs` in CreateDelegationTokenOptions

[Lianet M.]

Hey all, just one small comment: shouldn't the new method be named "
maxLifetimeMs()" (Lifetime vs LifeTime)? I believe it's the right spelling,
but mostly, it aligns better with the RPC field (max_lifetime_ms), that
generates a CreateDelegationTokenRequestData.maxLifetimeMs() and
CreateDelegationTokenRequestData.setMaxLifetimeMs(...)

Thanks!

Lianet

On Thu, Sep 5, 2024 at 9:36 AM Chia-Ping Tsai  wrote:

> +1 (binding)
>
> 黃竣陽  於 2024年9月5日 週四 下午9:08寫道：
>
> > Hello everyone,
> >
> > I would like to call for a vote on KIP-1080: <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1080%3A+Fix+the+typo%3A+%60maxlifeTimeMs%60+in+CreateDelegationTokenOptions
> > >
> >
> > The discussion can be found here:
> > 
> >
> > Best regards,
> > Jiunn-Yang
>

Re: [VOTE] KIP-512: make Record Headers available in onAcknowledgement and onComplete

[Lianet M.]

Hi all,

I'll re-cast my vote here :)

+1 (binding)

Thanks Rich!


On Fri, Aug 23, 2024, 10:32 p.m. Chia-Ping Tsai  wrote:

> +1 (binding)
>
> On 2024/08/10 17:28:45 "Rich C." wrote:
> > Hi all,
> >
> > After incorporating the feedback received
> > , I
> have
> > updated KIP-512 to reflect the majority preference for the new interface
> > approach. The KIP is now in its final form and ready for a vote.
> >
> > Regards,
> > Rich
> >
>

Re: [ANNOUNCE] New committer: Lianet Magrans

[Lianet M.]

Thank you very much everyone! It truly takes the great shared knowledge you
all put out there with amazing reviews and discussions.

Looking forward to continuing working with you all!

Lianet

On Wed, Aug 28, 2024, 7:12 p.m. Sophie Blee-Goldman 
wrote:

> Congrats Lianet! Well deserved
>
> On Wed, Aug 28, 2024 at 2:27 PM Jiashen Zhang 
> wrote:
>
> > Congratulations Lianet! Awesome!
> >
> > On Wed, Aug 28, 2024 at 2:10 PM Andrew Schofield <
> > andrew_schofi...@live.com>
> > wrote:
> >
> > > Awesome news. Well done, Lianet.
> > >
> > > Andrew
> > >
> > > > On 28 Aug 2024, at 17:47, Bruno Cadonna  wrote:
> > > >
> > > > Well deserved!
> > > >
> > > > Best,
> > > > Bruno
> > > >
> > > > On 8/28/24 6:37 PM, Bill Bejeck wrote:
> > > >> Congrats Lianet!
> > > >> On Wed, Aug 28, 2024 at 12:32 PM Matthias J. Sax 
> > > wrote:
> > > >>> Congrats! Very well deserved!
> > > >>>
> > > >>> On 8/28/24 9:18 AM, Justine Olshan wrote:
> > >  Congratulations Lianet!!
> > >  Justine
> > > 
> > >  On Wed, Aug 28, 2024 at 8:58 AM David Arthur 
> > > wrote:
> > > 
> > > > Congrats, Lianet!
> > > >
> > > > On Wed, Aug 28, 2024 at 11:48 AM Mickael Maison <
> > > >>> mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > >> Congratulations Lianet!
> > > >>
> > > >> On Wed, Aug 28, 2024 at 5:40 PM Josep Prat
> > >  > > 
> > > >> wrote:
> > > >>>
> > > >>> Congrats Lianet!
> > > >>>
> > > >>> On Wed, Aug 28, 2024 at 5:38 PM Chia-Ping Tsai <
> > > chia7...@apache.org>
> > > >> wrote:
> > > >>>
> > >  Congratulations, Lianet!!!
> > > 
> > >  On 2024/08/28 15:35:23 David Jacot wrote:
> > > > Hi all,
> > > >
> > > > The PMC of Apache Kafka is pleased to announce a new Kafka
> > > > committer,
> > > > Lianet Magrans.
> > > >
> > > > Lianet has been a Kafka contributor since June 2023. In
> > addition
> > > to
> > > > being a regular contributor and reviewer, she has made
> > > significant
> > > > contributions to the next generation of the consumer
> rebalance
> > > > protocol (KIP-848) and to the new consumer. She has also
> > > > contributed
> > > > to discussing and reviewing many KIPs.
> > > >
> > > > Congratulations, Lianet!
> > > >
> > > > Thanks,
> > > > David (on behalf of the Apache Kafka PMC)
> > > >
> > > 
> > > >>>
> > > >>>
> > > >>> --
> > > >>> [image: Aiven] 
> > > >>>
> > > >>> *Josep Prat*
> > > >>> Open Source Engineering Director, *Aiven*
> > > >>> josep.p...@aiven.io   |   +491715557497
> > > >>> aiven.io    |   <
> > > >> https://www.facebook.com/aivencloud>
> > > >>>   <
> > > >> https://twitter.com/aiven_io>
> > > >>> *Aiven Deutschland GmbH*
> > > >>> Alexanderufer 3-7, 10117 Berlin
> > > >>> Geschäftsführer: Oskari Saarenmaa, Hannu Valtonen,
> > > >>> Anna Richardson, Kenneth Chen
> > > >>> Amtsgericht Charlottenburg, HRB 209739 B
> > > >>
> > > >
> > > >
> > > > --
> > > > David Arthur
> > > >
> > > 
> > > >>>
> > >
> > >
> >
> > --
> > Thanks,
> > Jiashen
> >
>

Re: [DISCUSS] KIP-1082: Enable ID Generation for Clients over the ConsumerGroupHeartbeat RPC

[Lianet M.]

Hello TengYao! Thanks for taking on this issue, we've been going around it
for a while.

LM1: About the title of the KIP: "Enable ID Generation for Clients over the
ConsumerGroupHeartbeat RPC". I find it confusing because it hints that
we're adding it as an alternative (which was discussed and discarded, in
favour of really enforcing it). It's also missing the core change imo,
which is "where" the generation happens. So, maybe more to the point with
something along the lines of "Client-side generated ID for clients over
ConsumerGroupHeartbeat RPC"?

LM2: On the public interfaces section, the KIP states that "the server will
reject the request", but we should agree on the specific error type. I
expect it should fail with an InvalidRequestException, is that the
intention? (This was also suggested in the discussion thread before but is
not in the KIP).

LM3. Related to my previous point, I find that to be the true public-facing
change (member ID mandatory at the protocol level), but it's only at the
end of the Public interfaces changes, kind of lost among details of how
we're going to do it. Should we rephrase that section with the actual
change first, and the hows after (ex. Bumping the version is not the
public-facing change in this case, it's just the mechanism to properly
introduce our change)

LM4. Regarding the lifetime of the UUID: the KIP states we will "Verify
that the UUID remains consistent across all subsequent heartbeats during
the session". What is this "session" referring to here? I would expect that
the UUID is associated to a consumer instance (generated for the consumer
the first time it needs to send a HB if it doesn't have the UUID yet. From
there on, every time it needs to send a "first HB" again, it will reuse its
UUID, is that the intention? Note that we should consider that the same
consumer instance may have many "first heartbeats", meaning heartbeats to
join the group when it's not part of it (ex. consumer unsubscribe +
subscribe, fenced, stale). Is this the intention or are you considering the
lifetime differently? We should clarify it in the KIP.

Thanks!

Lianet

On Tue, Aug 27, 2024 at 2:27 AM TengYao Chi  wrote:

> Hi everyone,
>
> I have revised this KIP multiple times based on the feedback from our
> discussions.
> I would greatly appreciate it if you could review it when you have the
> time.
> If there are no further comments or suggestions, I plan to proceed with
> initiating a vote soon.
>
> Best regards,
> TengYao
>
> TengYao Chi  於 2024年8月23日 週五 下午2:43寫道：
>
> > Hi Andrew,
> > Thank you for your previous feedback and insights.
> > Your contributions have added valuable perspectives to the discussions.
> > And we also benefit from the comparison of different solutions.
> > I’m also looking forward to seeing an initial version in KIP-932, as it
> > will provide a good reference for future implementations.
> >
> > Regarding your comment on AS2, I wanted to clarify that my specification
> > references org.apache.kafka.common.Uuid.
> > I believe we’re referring to the same class, and it might just be a small
> > oversight due to the busy schedule.
> >
> > I want to express my gratitude once again for your many insightful
> > comments, which have helped the discussion progress smoothly.
> >
> > Best regards,
> > TengYao
> >
> >
> > Andrew Schofield  於 2024年8月22日 週四 下午11:28寫道：
> >
> >> Hi TengYao,
> >> I’ve been reading through the comments and I’m happy that the lobby
> >> approach has not gained support.
> >>
> >> Assuming that this KIP is voted, I will be happy to change KIP-932 so
> >> that it only supports client-side member ID generation. Because that KIP
> >> is still
> >> under development, I can do this in the first version of
> >> ShareGroupHeartbeat.
> >>
> >> AS2: For the encoding section, I suppose the specific encoding which
> >> is used is what org.apache.kafka.utils.Uuid uses.
> >>
> >> Thanks,
> >> Andrew
> >>
> >> > On 14 Aug 2024, at 17:00, TengYao Chi  wrote:
> >> >
> >> > Hello Apoorv,
> >> > Thank you for your feedback.
> >> > Regarding the questions you raised, unfortunately, this KIP cannot
> >> > guarantee the order of heartbeats. As with many classic distributed
> >> system
> >> > challenges, what we can do is make our best effort to ensure that
> there
> >> are
> >> > no idle members or stale assignments under normal circumstances.
> >> >
> >> > As for the lobby approach, I’m not a fan of it because it requires
> >> adding a
> >> > mechanism to maintain client state within the ConsumerGroup, which, in
> >> my
> >> > view, resembles something like a two-phase commit. This would
> introduce
> >> > more complexity than the proposal in this KIP, which is something we
> >> want
> >> > to avoid. KIP-848 aims to simplify the existing protocol, and while
> the
> >> > lobby approach is a good one, I believe it is not the right fit for
> this
> >> > particular situation.
> >> >
> >> > Best regards,
> >> > TengYao
> >> >
> >> > TengYao Chi  於 2024年8月14日 週三 下午11:45

Re: [VOTE] KIP-512: make Record Headers available in onAcknowledgement and onComplete

[Lianet M.]

Hello Rich,

Sorry for the late reply, but thanks for your answers on the discussion
thread and updates, it makes sense to me and the final proposal looks right.

+1 (non-binding)

Thanks!

Lianet

On Mon, Aug 12, 2024 at 10:29 AM Andrew Schofield 
wrote:

> Hi Rich,
> Thanks for the KIP. Looks good to me.
>
> +1 (non-binding)
>
> Thanks,
> Andrew
>
> > On 10 Aug 2024, at 18:53, Rich C.  wrote:
> >
> > Sorry, forgot the link to KIP-512
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-512%3A+make+Record+Headers+available+in+onAcknowledgement+and+onComplete
> >
> > Regards,
> > Rich
> >
> >
> > On Sat, Aug 10, 2024 at 1:28 PM Rich C.  wrote:
> >
> >> Hi all,
> >>
> >> After incorporating the feedback received
> >> , I
> >> have updated KIP-512 to reflect the majority preference for the new
> >> interface approach. The KIP is now in its final form and ready for a
> vote.
> >>
> >> Regards,
> >> Rich
> >>
>
>

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

[Lianet M.]

Hi Bruno, answering your questions:

About the full heartbeat (LM1): I just wanted to confirm that you'll be
sending full HBs in case of errors in general. It's not clear from the KIP,
since it referred to sending Id/epoch and whatever had changed since the
last HB only. Sending full HB on error is key to ensure fresh rejoins after
fencing for instance, and retries with all relevant info.

About the instanceId (LM2): The instanceId is needed on every HB to be able
to identify a member using one that is already taken. On every HB, the
broker uses the instance id (if any) to retrieve the member ID associated
with it, and checks it against the memberId received in the HB
(throwing UnreleasedInstance exception if needed). So similar to my
previous point, just wanted to confirm that we are considering that here
too.

Now some other thoughts:

LM9: Definitely interesting imo if we can avoid the dependency between the
StreamsGroupInitialize and the StreamsGroupHeartbeat. I totally get that
the initial client implementation will do a HB first, and that's fine, but
not having the flow enforced at the protocol level would allow for further
improvement in the future (that initialize via admin idea you mentioned,
for instance). Actually, I may be missing something about the HB, but if we
are at the point where HB requires that the topology has been initialized,
and the topology init requires the group, why is it the heartbeat RPC the
one responsible for the group creation? (vs. StreamsGroupInitialize creates
group if needed + HB just fails if topology not initialized)

Thanks!

Lianet
(I didn't miss your answer on my INVALID_GROUP_TYPE proposal, just still
thinking about it in sync with the same discussion we're having on the
KIP-1043 thread...I'll come back on that)

On Thu, Aug 1, 2024 at 10:55 AM Andrew Schofield 
wrote:

> Hi Bruno,
> Thanks for adding the detail on the schemas on records written to
> __consumer_offsets.
> I’ve reviewed them in detail and they look good to me. I have one naive
> question.
>
> AS11: I notice that an assignment is essentially a set of partition
> indices for
> subtopologies. Since a subtopology can be defined by a source topic regex,
> does
> this mean that an assignment gives the same set of partition indices for
> all topics
> which happen to match the regex? So, a subtopology reading from A* that
> matches
> AB and AC would give the same set of partitions to each task for both
> topics, and
> is not able to give AB:0 to one task and AC:0 to a different task. Is this
> correct?
>
> Thanks,
> Andrew
>
> > On 23 Jul 2024, at 16:16, Bruno Cadonna  wrote:
> >
> > Hi Lianet,
> >
> > Thanks for the review!
> >
> > Here my answers:
> >
> > LM1. Is your question whether we need to send a full heartbeat each time
> the member re-joins the group even if the information in the RPC did not
> change since the last heartbeat?
> >
> > LM2. Is the reason for sending the instance ID each time that a member
> could shutdown, change the instance ID and then start and heartbeat again,
> but the group coordinator would never notice that the instance ID changed?
> >
> > LM3. I see your point. I am wondering whether this additional
> information is worth the dependency between the group types. To return
> INVALID_GROUP_TYPE, the group coordinator needs to know that a group ID
> exists with a different group type. With a group coordinator as we have it
> now in Apache Kafka that manages all group types, that is not a big deal,
> but imagine if we (or some implementation of the Apache Kafka protocol)
> decide to have a separate group coordinator for each group type.
> >
> > LM4. Using INVALID_GROUP_ID if the group ID is empty makes sense to me.
> I going to change that.
> >
> > LM5. I think there is a dependency from the StreamsGroupInitialize RPC
> to the heartbeat. The group must exist when the initialize RPC is received
> by the group coordinator. The group is created by the heartbeat RPC. I
> would be in favor of making the initialize RPC independent from the
> heartbeat RPC. That would allow to initialize a streams group explicitly
> with an admin tool.
> >
> > LM6. I think it affects streams and streams should behave as the
> consumer group.
> >
> > LM7. Good point that we will consider.
> >
> > LM8. Fixed! Thanks!
> >
> >
> > Best,
> > Bruno
> >
> >
> >
> >
> > On 7/19/24 9:53 PM, Lianet M. wrote:
> >> Hi Lucas/Bruno, thanks for the great KIP! First comments:
> >> LM1. Related to where the KIP says:  *“Group ID, member ID, member epoch
> >> are sent with each heartbeat request. Any other information that has not
> >> changed 

Re: [DISCUSS] KIP-512: make Record Headers available in onAcknowledgement

[Lianet M.]

Hello Rich, thanks for resurrecting the KIP, seems to fill a gap indeed.

LM1. Specifically related to motivation#1. ProducerRecord already has a
timestamp, passed into the RecordMetadata, that represents the creation
time provided on new ProducerRecord, so couldn't we reuse it to avoid the
extra complexity of having to "include a timestamp in the header when the
message is sent" to be able to compute latency properly. The challenge of
course is that that timestamp may be overwritten (and this is the root
cause of the gap), but that could be resolved just by keeping the original
time and making it available.
RecordMetadata would keep a timestamp (passed from the record creation,
never mutated), and the "effectiveTimestamp" (the one it currently has,
updated with the broker result based on configs). Main advantage would be
not having to add a header for calculating latency. The user simply creates
the record with a timestamp (known existing concept), and we make that
value accessible in the RecordMetadata (where it exists already at some
point, but it's mutated). Thoughts?

LM2. Regardless of the point above, if we think having the headers
available on the onAcknowledgement would be helpful, I definitely see the
case for both alternatives (headers in RecordMetadata and as param). I
share Andrew's feeling because Headers are indeed part of the
ProducerRecord. But then headers will in practice simply contain info
related to the record, so it seems sensible to expect to find headers in
the RecordMetadata as you suggest, ok with me.

Thanks!

Lianet

On Mon, Jul 29, 2024 at 9:41 PM Rich C.  wrote:

> Hi Andrew,
>
> Thanks for the feedback. I have updated KIP-512 and addressed AS2, AS3, and
> AS4. For AS1, let's wait for further responses from the community.
>
>
> Regards,
> Rich
>
>
> On Mon, Jul 29, 2024 at 5:59 AM Andrew Schofield <
> andrew_schofi...@live.com>
> wrote:
>
> > Hi,
> > Thanks for adding the detail. It seems quite straightforward to
> > implement in the producer code.
> >
> > AS1: Personally, and of course this is a matter of taste and just one
> > opinion, I don’t like adding Headers to RecordMetadata. It seems to me
> > that RecordMetadata is information about the record that’s been produced
> > whereas the Headers are really part of the record itself. So, I prefer
> the
> > alternative which overloads ProducerInterceptor.onAcknowledgement.
> >
> > AS2: ProducerBatch and FutureRecordMetadata are both internal classes
> > and do not need to be documented in the KIP.
> >
> > AS3: This KIP is adding rather than replcaing the constructor for
> > RecordMetadata.
> > You should define the value for the Headers if an existing constructor
> > without headers is used.
> >
> > AS4: You should add a method `Headers headers()` to RecordMetadata.
> >
> >
> > I wonder what other community members think about whether it’s a good
> > idea to extend RecordMetadata with the headers.
> >
> > Thanks,
> > Andrew
> >
> > > On 29 Jul 2024, at 05:36, Rich C.  wrote:
> > >
> > > Hi all,
> > >
> > > Thank you for the positive feedback. I added proposal changes to
> KIP-512
> > > and included a FAQ section to address some concerns.
> > >
> > > Hi Andrew, yes, this KIP focuses on
> > > `ProducerInterceptor.onAcknowledgement`. I added FAQ#3 to explain that.
> > >
> > > Hi Matthias, for your question about "RecordMetadata being Kafka
> > metadata" in
> > > this thread
> > > <
> >
> https://lists.apache.org/list?dev@kafka.apache.org:lte=1M:make%20Record%20Headers%20available%20in%20onAcknowledgement
> > >,
> > > I added FAQ#2 to explain that. If I have missed any documentation
> > regarding
> > > the design of RecordMetadata, please let me know.
> > >
> > > Regards,
> > > Rich
> > >
> > >
> > > On Fri, Jul 26, 2024 at 4:00 PM Andrew Schofield <
> > andrew_schofi...@live.com>
> > > wrote:
> > >
> > >> Hi Rich,
> > >> Thanks for resurrecting this KIP. It seems like a useful idea to me
> and
> > >> I’d be interested in seeing the proposed public interfaces.
> > >>
> > >> I note that you specifically called out the
> > >> ProducerInterceptor.onAcknowledgement
> > >> method, as opposed to the producer Callback.onCompletion method.
> > >>
> > >> Thanks,
> > >> Andrew
> > >>
> > >>> On 26 Jul 2024, at 04:54, Rich C.  wrote:
> > >>>
> > >>> Hi Kevin,
> > >>>
> > >>> Thanks for your support.
> > >>>
> > >>> Hi Matthias,
> > >>>
> > >>> I apologize for the confusion. I've deleted the Public Interface
> > sections
> > >>> for now. I think we should focus on discussing its necessity with the
> > >>> community. I'll let it sit for a few more days, and if there are no
> > >>> objections, I will propose changes over the weekend and share them
> here
> > >>> again.
> > >>>
> > >>> Regards,
> > >>> Rich
> > >>>
> > >>>
> > >>> On Thu, Jul 25, 2024 at 5:51 PM Matthias J. Sax 
> > >> wrote:
> > >>>
> >  Rich,
> > 
> >  thanks for resurrecting this KIP. I was not part of the original
> >  discussion back in the day, but persona

Re: [VOTE] KIP-1068: New metrics for the new KafkaConsumer

[Lianet M.]

Hi Brenden,

+1 (non-binding) from me.

Thanks for the KIP!

Lianet

On Thu, Jul 25, 2024 at 3:18 AM Bruno Cadonna  wrote:

> Hi Brenden,
>
> Thanks for the KIP!
>
> +1 (binding)
>
> Best,
> Bruno
>
> On 7/24/24 11:55 PM, Brenden Deluna wrote:
> > Hello everyone,
> >
> > I would like to start the vote on KIP-1068:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1068%3A+New+metrics+for+the+new+KafkaConsumer
> >
> > This KIP introduces new metrics for the new KafkaConsumer implementation
> > for broader metrics coverage.
> >
> > Thanks,
> > Brenden
> >
>

Re: [DISCUSS] KIP-1068: KIP-1068: New JMX Metrics for AsyncKafkaConsumer

[Lianet M.]

gt; >>
> > >> AM4: Is the telemetry name for `unsent-requests-queue-size` intended
> > >> as `org.apache.kafka.consumer.unsent.requests.size`,
> > >> or it should be corrected to `
> > >> org.apache.kafka.consumer.unsent.requests.queue.size`?
> > >>
> > >> AM2:
> > >> Regards,
> > >> Apoorv Mittal
> > >> +44 7721681581
> > >>
> > >>
> > >> On Mon, Jul 15, 2024 at 2:45 PM Andrew Schofield <
> > >> andrew_schofi...@live.com>
> > >> wrote:
> > >>
> > >>> Hi Brenden,
> > >>> Thanks for the updates.
> > >>>
> > >>> AS4. I see that you’ve added `.ms` to a bunch of the metrics
> reflecting
> > >> the
> > >>> fact that they’re measured in milliseconds. However, I observe that
> > most
> > >>> metrics
> > >>> in Kafka that are measured in milliseconds, with some exceptions in
> > Kafka
> > >>> Connect
> > >>> and MirrorMaker do not follow this convention. I would tend to err on
> > the
> > >>> side of
> > >>> consistency with the existing metrics and not use `.ms`. However,
> > that’s
> > >>> just my
> > >>> opinion, so I’d be interested to know what other reviewers of the KIP
> > >>> think.
> > >>>
> > >>> Thanks,
> > >>> Andrew
> > >>>
> > >>>> On 12 Jul 2024, at 20:11, Brenden Deluna
>  > >>>
> > >>> wrote:
> > >>>>
> > >>>> Hey Lianet,
> > >>>>
> > >>>> Thank you for your suggestions and feedback!
> > >>>>
> > >>>>
> > >>>> LM1. This has now been addressed.
> > >>>>
> > >>>>
> > >>>> LM2. I think that would be a valuable addition to the current set of
> > >>>> metrics, I will get that added.
> > >>>>
> > >>>>
> > >>>> LM3. Again great idea, that would certainly be helpful. Will add
> that
> > >> as
> > >>>> well.
> > >>>>
> > >>>>
> > >>>> Let me know if you have any more suggestions!
> > >>>>
> > >>>>
> > >>>> Thanks,
> > >>>>
> > >>>> Brenden
> > >>>>
> > >>>> On Fri, Jul 12, 2024 at 2:11 PM Brenden Deluna <
> bdel...@confluent.io>
> > >>> wrote:
> > >>>>
> > >>>>> Hi Lucas,
> > >>>>>
> > >>>>> Thank you for the feedback! I have addressed your comments:
> > >>>>>
> > >>>>>
> > >>>>> LB1. Good catch there, I will update the names as needed.
> > >>>>>
> > >>>>>
> > >>>>> LB2. Good catch again! I will update the name to be more
> consistent.
> > >>>>>
> > >>>>>
> > >>>>> LB3. Thank you for pointing this out, I realized that all metric
> > >> values
> > >>>>> will actually be set to 0. I will specifiy this and explain why
> they
> > >>> will
> > >>>>> be 0.
> > >>>>>
> > >>>>>
> > >>>>> Nit: This metric is referring to the queue of unsent requests in
> the
> > >>>>> NetworkClientDelegate. For the metric descriptions I am trying to
> not
> > >>>>> include too much of the implementation details, hence the reason
> that
> > >>>>> description is quite short. I cannot think of other ways to
> describe
> > >> the
> > >>>>> metric without going deeper into the implementation, but please do
> > let
> > >>> me
> > >>>>> know if you have any ideas.
> > >>>>>
> > >>>>>
> > >>>>> Thank you,
> > >>>>>
> > >>>>> Brenden
> > >>>>>
> > >>>>> On Fri, Jul 12, 2024 at 1:27 PM Lianet M. 
> > wrote:
> > >>>>>
> > >>>>>> Hey Brenden, thanks for the KIP! Great to get more visibility into
> > >> the
> > >>> new
> > >>>>>> consumer.
&g

Re: [DISCUSS] KIP-1043: Administration of groups

[Lianet M.]

 `--consumer-property group.protocol=consumer` to enable
> > modern consumer group. But the listing for `classic` "type" also defines
> > "protocol" as `consumer` in some scenarios. Is it intended or `classic`
> > type should different protocol?
> >
> > AM3: The KIP adds behaviour for  `kafka-share-groups.sh` which defines
> the
> > `--create` option. Though it simplifies group creation, what should be
> the
> > error behaviour when the group with the same name exists but not of
> "share"
> > group type?
> >
> > AM4: The GroupMetadataManager.java stores all groups in the same data
> > structure which means the name has to be unique across different group
> > types. Do you think we should also change the error message for existing
> > kafka-consumer-groups.sh and kafka-share-groups.sh to recommend using new
> > kafka-groups.sh for extensive groups list? As currently the individual
> > scripts will result in "Group already exists" while creating new groups
> but
> > listing with respective utility will not yield the group.
> >
> > AM5: The KIP defines compatibility considerations for the ListGroups RPC.
> > But it's unclear to me why it's needed as the client and server
> supporting
> > `kafka-groups.sh` will be post ListGroups v5 API anyways hence
> TypesFilter
> > will be supported in both client and server. Am I missing something here?
> >
> > Regards,
> > Apoorv Mittal
> > +44 7721681581
> >
> >
> > On Wed, Jul 17, 2024 at 6:26 PM Andrew Schofield <
> andrew_schofi...@live.com>
> > wrote:
> >
> >> Hi Lianet,
> >> Thanks for your comments.
> >>
> >> LM5. Unfortunately, the protocol type has to be a string rather than
> >> an enumeration. This is because when people have created their own
> >> extensions of the classic consumer group protocol, they have chosen
> >> their own protocol strings. For example, the Confluent schema registry
> >> uses “sr” and there are other examples in the wild.
> >>
> >> LM6.1. It’s because of the difference between a parameterised
> >> type and a raw type.
> >>
> >> If I use:
> >> public class ListGroupsResult
> >> public class ListConsumerGroupsResult
> >>
> >> then ListGroupsResult (no type variable) is a raw type which does
> >> not provide a type for the type variable. This causes compiler warnings
> >> when the type is used, unless it’s used as
> ListGroupsResult.
> >>
> >> However, this works better.
> >> public class AbstractListGroupsResult
> >> public class ListGroupsResult extends
> >> AbstractListGroupsResult
> >> public class ListConsumerGroupsResult extends
> >> AbstractListGroupsResult
> >>
> >> I’ll change the KIP to use this.
> >>
> >> LM6.2. I was just pointing out a difference and you’re happy
> >> with it. That’s good.
> >>
> >> LM7. If you have a cluster with a mixture of classic and modern
> >> consumer groups, to list them all you could use this:
> >>
> >> bin/kafka-groups.sh --protocol consumer
> >>
> >> When there are no classic consumer groups, you could do:
> >>
> >> bin/kafka-groups.sh --group-type consumer
> >>
> >> However, this only gives a complete list if you don’t have any classic
> >> consumer groups.
> >>
> >> As a result, I suggested --consumer so you don’t need to know
> >> or care about the existence of classic and modern consumer groups.
> >> I think it helps, but you aren’t convinced I think, which tells me
> >> more thinking needed here.
> >>
> >> Maybe adding --share would help, so only power users would
> >> use --group-type or --protocol to deal with the more complicated
> >> cases.
> >>
> >> LM8. It’s just not clear. I was trying to make the output the same
> >> whether the group was created, or whether it already existed. In
> >> either case, there’s a share group in existence. The
> >> AlterShareGroupOffsets RPC doesn’t distinguish between the
> >> two cases in its response.
> >>
> >> Thanks,
> >> Andrew
> >>
> >>> On 16 Jul 2024, at 21:24, Lianet M.  wrote:
> >>>
> >>> Hello Andrew, thanks for considering the feedback. Some follow-ups and
> >>> other comments:
> >>>
> >>> LM4. Good point about the older RPC versions and therefore the
> >>> Option

Re: [DISCUSS] KIP-1071: Streams Rebalance Protocol

[Lianet M.]

Hi Lucas/Bruno, thanks for the great KIP! First comments:

LM1. Related to where the KIP says:  *“Group ID, member ID, member epoch
are sent with each heartbeat request. Any other information that has not
changed since the last heartbeat can be omitted.”. *I expect all the other
info also needs to be sent whenever a full heartbeat is required (even if
it didn’t change from the last heartbeat), ex. on fencing scenarios,
correct?

LM2. For consumer groups we always send the groupInstanceId (if any) as
part of every heartbeat, along with memberId, epoch and groupId. Should we
consider that too here?

LM3. We’re proposing returning a GROUP_ID_NOT_FOUND error in response to
the stream-specific RPCs if the groupId is associated with a group type
that is not streams (ie. consumer group or share group). I wonder if at
this point, where we're getting several new group types added, each with
RPCs that are supposed to include groupId of a certain type, we should be
more explicit about this situation. Maybe a kind of INVALID_GROUP_TYPE
(group exists but not with a valid type for this RPC) vs a
GROUP_ID_NOT_FOUND (group does not exist).  Those errors would be
consistently used across consumer, share, and streams RPCs whenever the
group id is not of the expected type.
This is truly not specific to this KIP, and should be addressed with all
group types and their RPCs in mind. I just wanted to bring out my concern
and get thoughts around it.

LM4. On a related note, StreamsGroupDescribe returns INVALID_REQUEST if
groupId is empty. There is already an INVALID_GROUP_ID error, that seems
more specific to this situation. Error handling of specific errors would
definitely be easier than having to deal with a generic INVALID_REQUEST
(and probably its custom message). I know that for KIP-848 we have
INVALID_REQUEST for similar situations, so if ever we take down this path
we should review it there too for consistency. Thoughts?

LM5. The dependency between the StreamsGroupHeartbeat RPC and the
StreamsGroupInitialize RPC is one-way only right? HB requires a previous
StreamsGroupInitialize request, but StreamsGroupInitialize processing is
totally independent of heartbeats (and could perfectly be processed without
a previous HB, even though the client implementation we’re proposing won’t
go down that path). Is my understanding correct? Just to double check,
seems sensible like that at the protocol level.

LM6. With KIP-848, there is an important improvement that brings a
difference in behaviour around the static membership: with the classic
protocol, if a static member joins with a group instance already in use, it
makes the initial member fail with a FENCED_INSTANCED_ID exception, vs.
with the new consumer group protocol, the second member trying to join
fails with an UNRELEASED_INSTANCE_ID. Does this change need to be
considered in any way for the streams app? (I'm not familiar with KS yet,
but thought it was worth asking. If it doesn't affect in any way, still
maybe helpful to call it out on a section for static membership)

LM7. Regarding the admin tool to manage streams groups. We can discuss
whether to have it here or separately, but I think we should aim for some
basic admin capabilities from the start, mainly because I believe it will
be very helpful/needed in practice during the impl of the KIP. From
experience with KIP-848, we felt a bit blindfolded in the initial phase
where we still didn't have kafka-consumer-groups dealing with the new
groups (and then it was very helpful and used when we were able to easily
inspect them from the console)

LM8. nit: the links the KIP-848 are not quite right (pointing to an
unrelated “Future work section” at the end of KIP-848)

Thanks!

Lianet


On Fri, Jul 19, 2024 at 11:13 AM Lucas Brutschy
 wrote:

> Hi Andrew,
>
> AS2: I added a note for now. If others feel strongly about it, we can
> still add more administrative tools to the KIP - it should not change
> the overall story significantly.
>
> AS8: "streams.group.assignor.name" sounds good to me to distinguish
> the config from class names. Not sure if I like the "default". To be
> consistent, we'd then have to call it
> `group.streams.default.session.timeout.ms` as well. I only added the
> `.name` on both broker and group level for now.
>
> AS10: Ah, I misread your comment, now I know what you meant. Good
> point, fixed (by Bruno).
>
> Cheers,
> Lucas
>
> On Fri, Jul 19, 2024 at 4:44 PM Andrew Schofield
>  wrote:
> >
> > Hi Lucas,
> > I see that I hit send too quickly. One more comment:
> >
> > AS2: I think stating that there will be a `kafka-streams-group.sh` in a
> > future KIP is fine to keep this KIP focused. Personally, I would probably
> > put all of the gory details in this KIP, but then it’s not my KIP. A
> future
> > pointer is fine too.
> >
> > Thanks,
> > Andrew
> >
> >
> > > On 19 Jul 2024, at 13:46, Lucas Brutschy 
> wrote:
> > >
> > > Hi Andrew,
> > >
> > > thanks for getting the discussion going! Here are my responses

Re: [DISCUSS] KIP-1043: Administration of groups

[Lianet M.]

Hello Andrew, thanks for considering the feedback. Some follow-ups and
other comments:

LM4. Good point about the older RPC versions and therefore the
Optional, agreed.

LM5. In GroupListing, should we use the existing
org.apache.kafka.clients.ProtocolType to represent the protocol (instead of
String). I don’t quite like the fact that the enum is inside the
GroupRebalanceConfig though, feels like it should be a first level citizen.


LM6. Regarding the changes around ListGroupResults with generics.
 - LM6.1. What’s the need for keeping both, a base
AbstractListGroupsResult and the ListGroupsResult
extends AbstractListGroupsResult? Would it work if instead we
simply have a single ListGroupsResult from which
specific groups would inherit? I'm thinking of this:

public class *ListGroupsResult* -> this would
probably end up containing the implementation that currently exists in
ListConsumerGroupsResult for #all, #errors and #valid, that all group types
would be able to reuse if we use a generic T extends GroupListing


public class *ListConsumerGroupsResult extends
ListGroupsResult* -> slim impl, agreed

 - LM6.2. Related to the concern of the javadoc for
ListConsumerGroupsResult. This class will inherit 3 funcs (all, valid,
error), that have a common behaviour (and javadoc) regardless of the
generic type, so I expect it won’t be confusing in practice? We will end up
with the java doc for, let’s say, ListConsumerGroupsResult#all showing the
parent javadoc that aligns perfectly with what the #all does. If ever we
need a different behaviour/javadoc for any of the functions in the child
classes, we would have the alternative of overriding the func and javadoc.
Makes sense? Not sure if I’m missing other readability issues with the
javadocs you’re seeing.

LM7. Looks better to me now with the added filter on the kafka-group.sh for
the protocol. But then, the new –consumer filter achieves the same as
–protocol CONSUMER right? If so, I wonder if it would just be simpler to
support the --protocol as a way to achieve this? (sharing your struggle on
how to get this right, but feels easier to discover and reason about the
more we have filters based on the output, and not made up of
combinationslet's keep iterating and we'll get there :) )

LM8. Is the output wrong (or just not clear) in this example? (It seemed to
me this was referring to the successful case where we create a new share
group, so I was expecting a "successfully created" kind of output)

  $ bin/kafka-share-groups.sh --bootstrap-server localhost:9092
--create --group NewShareGroup
  Share group 'NewShareGroup' exists.

Thanks!

Lianet

On Mon, Jul 15, 2024 at 6:00 AM Andrew Schofield 
wrote:

> Hi Lianet,
> Thanks for your comments.
>
> LM1. Admin.listGroups() in principle needs to be able to return
> results from any version of the ListGroups RPC. The older versions do
> not contain the group type, so I think it’s reasonable to have
> Optional. I think there’s a difference between
> Optional.empty (I don’t know the group type) and
> GroupType.UNKNOWN (I know and do not understand the group type).
> As a result, I’ve changed the KIP to use Optional.
>
> I think that changing ConsumerGroupListing to extend
> GroupListing, and to do the same for ShareGroupListing makes sense.
> This does require that the overridden methods such as type() have
> signatures that match today’s definition of ConsumerGroupListing but
> that’s fine with the change I made to use Optional above.
>
> LM2. I think it’s possible to do something with generics along the
> lines you described.
>
> * public abstract class AbstractListGroupsResult
> * public class ListGroupsResult extends
> AbstractListGroupsResult
> * public class ListConsumerGroupsResult extends
> AbstractListGroupsResult
>
> This does make the javadoc for ListConsumerGroupsResult less
> readable because its methods are now all inherited. The classes
> such as ListConsumerGroupsResult of course still have to exist
> but the implementation is very slim.
>
> What do you think of this? I haven’t yet updated the KIP in this
> case.
>
> LM3. I have been kicking around the syntax for kafka-group.sh
> for a while now and I too am not happy with the filters yet. I absolutely
> want to be able to display all consumer groups with a simple option,
> but history means that not a single filter under the covers.
>
> I suggest the following:
> --group-type which supports all group types
> --protocol which supports any string for protocol (there’s no enumeration)
> --consumer which matches all classic and modern consumer groups
>  (and is thus a confection made by filtering on both group type and
> protocol).
>
> I’ve changed the KIP accordingly. Let me know what you think.
>
> Thanks,
> Andrew
>
>
> > On 12 

Re: [DISCUSS] KIP-1043: Administration of groups

[Lianet M.]

Hey Andrew, thanks for the KIP, we definitely need visibility from a higher
level now that groups are growing.

LM1. Should we have the existing
org.apache.kafka.clients.admin.ConsumerGroupListing extend the GroupListing
you’re proposing? ConsumerGroupListing already exists with a very similar
shape, and this would allow to set a common ground for the existing group
types, and the ones that are coming up (share groups and KS groups). Side
note, the existing ConsumerGroupListing has the type as Optional, but given
that the GroupType enum has an UNKNOWN type, I don’t quite get the need for
Optional and seems ok to me as you’re proposing it.

LM2: if the point above makes sense, it would allow us to consider changing
the new ListGroupResult you’re proposing to make it generic and potentially
reused by all group types:


public class ListGroupsResult {

public KafkaFuture> all()

public KafkaFuture> valid() {}

public KafkaFuture> errors() {}

}

Makes sense? With this, maybe we won’t need specific result classes for
each group (like the existing ListConsumerGroupsResult), given that in the
end it’s just a wrapper around the GroupListing (which is what each group
type would redefine).


LM3. I get how you're playing with the filters for group types and
protocol, but then I find it confusing how we end up with filters that do
not match the output ( --group-type that matches the protocol from the
output and not the type for "consumer"  example). What about having the
–group-type filter on the actual GroupType field of the RPC response (shown
in the cmd line output as TYPE); and add a –protocol-type that would filter
on the ProtocolType field of  RPC response (shown in the cmd line output as
PROTOCOL). We would have the filters aligned with the output for all cases,
which seems more consistent.

Thanks!

Lianet



On Thu, Jun 6, 2024 at 8:16 AM Andrew Schofield 
wrote:

> Hi Kirk,
> Thanks for your comments.
>
> 1. I’m a big fan of consistency in these things and the method signatures
> match
> ListConsumerGroupsResult and ListShareGroupsResult.
>
> 2. Yes, client-side filtering.
>
> 3. I didn’t offer “classic” as an option for --group-type. I’ve kicked the
> options
> around in my mind for a while and I decided that using --group-type as a
> way of
> filtering types in a way that a normal user would understand them was a
> good
> place to start. For example, I didn’t have `--protocol consumer` for
> consumer groups
> and `--group-type share` for share groups, even though that’s technically
> more
> correct.
>
> Since KIP-848, the set of consumer groups is actually formed from those
> which
> use the classic protocol and those which use the modern protocol. This tool
> gives you both together when you use `--group-type consumer`, which is
> exactly
> what kafka-consumer-groups.sh does.
>
> Do you think - -group-type classic is helpful? It would give a list of all
> groups using
> any variant of the classic group protocol. I can easily add it.
>
> 4, 5. Yes, maybe the wording of the message could improve. These things
> are always
> tricky. I went with “Group CG1 is not a share group.” because it doesn’t
> require the tool
> to interpret the group type in order to generate the message.
>
> Imagine this scenario. You are using kafka-share-groups.sh --describe and
> you’ve
> used the group ID of a consumer group. Here are some options:
>
> a) “Group CG1 is not a share group.”
> b) “Incorrect group type (Consumer). Group CG1 is not a share group.”
> c) “Group CG1 has the wrong type for this operation. It is not a share
> group."
>
> I don’t think “There is already a (consumer) group named ‘CG1’” is quite
> right.
>
> Any preference?
>
> 6. Yes, it is a change in behaviour which is why I mention it in the KIP.
> Personally, I think that’s OK because the existing message is misleading
> and could definitely cause frustration. Let’s see what other reviewers
> think.
>
> Thanks,
> Andrew
>
> > On 6 Jun 2024, at 00:44, Kirk True  wrote:
> >
> > Hi Andrew,
> >
> > Thanks for the KIP! I don’t have much experience as a Kafka operator,
> but this seems like a very sane proposal.
> >
> > Questions & comments:
> >
> > 1. Do you think the ListGroupsResult.all() method is a bit of a
> potential ‘foot gun’? I can imagine cases where developers reach for that
> without understanding its potential of throwing errors. It could lead to
> cases where all() works in development but not in production.
> >
> > 2. Based on the LIST_GROUPS RPC, it appears that filtering is all
> performed client side, correct? (I know that’s not specific to this KIP,
> but just want to make sure I understand.)
> >
> > 3. For kafka-groups.sh --list, is ‘classic’ valid for --group-type? If
> so, should we allow users of kafka-groups.sh --list to provide multiple
> --group-type arguments?
> >
> > 4. In the last kafka-share-groups.sh --create example (“ConsumerGroup”),
> the error simply states that “Group 'ConsumerGroup’ is not a share group.”
> I’m as

Re: [DISCUSS] KIP-1068: KIP-1068: New JMX Metrics for AsyncKafkaConsumer

[Lianet M.]

Hey Brenden, thanks for the KIP! Great to get more visibility into the new
consumer.

LM1. +1 on Lucas's suggestion for including the unit in the name, seems
clearer and consistent (I do see several time metrics including ms)

LM2. What about a new metric for application-event-queue-time-ms. It would
be a complement to the application-event-queue-size you're proposing, and
it will tell us how long the events sit in the queue, waiting to be
processed (from the time the API call adds the event to the queue, to the
time it's processed in the background thread). I find it would be very
interesting.

LM3. Thinking about the actual usage of
"time-between-network-thread-poll-xxx" metric, I imagine it would be
helpful to know more about what could be impacting it. As I see it, the
network thread cadence could be mainly impacted by: 1- app event processing
(generate requests), 2- network client poll (actual send/receive). For 2,
the new consumer reuses the same component as the legacy one, but 1 is
specific to the new consumer, so what about a metric
for application-event-processing-time-ms (we could consider avg I would
say). It would be the time that the network thread takes to process all
available events on each run.

Cheers!
Lianet

On Fri, Jul 12, 2024 at 1:57 PM Lucas Brutschy
 wrote:

> Hey Brenden,
>
> thanks for the KIP! These will be great to observe and debug the
> background thread of the new consumer.
>
> LB1. `time-between-network-thread-poll-max` → I see several similar
> metrics including the unit in the metric name (ms or us). We could
> consider this, although it's probably not strictly required. However,
> at least the description should state the unit. Same for the `avg`
> version.
> LB2. `unsent-requests-size` → Naming sounds a bit like it's referring
> to the size of the request. How about `unsent-request-queue-size` or
> `unsent-request-count` or simply `unsent-requests`?
> LB3. "the proposed metrics below will be set to null or 0." → which
> one will be set to null and which ones will be set to 0, and why?
>
> nit: "The current number of unsent requests in the consumer network" →
> Seems to be missing something?
>
> Cheers,
> Lucas
>
> On Fri, Jul 12, 2024 at 7:28 PM Brenden Deluna
>  wrote:
> >
> > Hi Andrew,
> > Thank you for the feedback and your question.
> >
> > AS1. Great idea, I will get that added.
> >
> > AS2. For unsent-events-age-max, age will be calculated once the event is
> > sent, so you are correct.
> >
> > AS3. I agree, I think that would be a helpful metric to add, thank you! I
> > will get that added.
> >
> > Please let me know if you have any additional feedback, suggestions, or
> > questions.
> >
> > Thanks,
> > Brenden
> >
> > On Fri, Jul 12, 2024 at 11:45 AM Andrew Schofield <
> andrew_schofi...@live.com>
> > wrote:
> >
> > > Hi Brenden,
> > > Thanks for the KIP. It fills a gap in the metrics for the new consumer
> > > nicely.
> > >
> > > AS1. If using the CLASSIC protocol, and thus the LegacyKafkaConsumer,
> > > I would expect that the metrics do not exist at all. Maybe say
> something
> > > like
> > > “These metrics are for the new consumer implementation using the
> > > CONSUMER protocol”.
> > >
> > > AS2. For unsent-events-age-max, when is the age calculated? For
> example,
> > > is it calculated at the time that the unsent event is removed from the
> > > list and sent, or does the metric reflect unsent events which are still
> > > enqueued? I suspect the former, but thought I’d check.
> > >
> > > AS3. I think that unsent-events-age-avg would also be interesting to
> > > get an idea of how long unsent events tend to sit around before
> sending.
> > > Of course, the question from AS2 would also apply to the average.
> > >
> > > Thanks,
> > > Andrew
> > >
> > > > On 10 Jul 2024, at 17:44, Philip Nee 
> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > This is the link to the KIP document.
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1068%3A+New+JMX+metrics+for+the+new+KafkaConsumer
> > > >
> > > > Any comment is appreciated,
> > > >
> > > >
> > > > On Tue, Jul 9, 2024 at 10:14 AM Brenden Deluna
> > > 
> > > > wrote:
> > > >
> > > >> Hello everyone,
> > > >>
> > > >> I would like to start the discussion thread for KIP-1068. This is a
> > > >> relatively small KIP, only proposing to add a couple of new metrics.
> > > >>
> > > >> If you have any suggestions or feedback, let me know, it will be
> much
> > > >> appreciated.
> > > >>
> > >
> > >
>

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

[Lianet M.]

Hi all, thanks for the KIP Alieh!

LM1. Totally agree with Artem's point about the config not being the most
explicit/flexible way to express this capability. Getting then to Matthias
4 options, what I don't like about 3 and 4 is that it seems they might not
age very well? Aren't we going to be wanting some other twist to the flush
semantics that will have us adding yet another param to it, or another
overloaded method? I truly don't have the context to answer that, but if it
feels like a realistic future maybe adding some kind FlushOptions params to
the flush would be better from an extensibility point of view. It would
only have the clearErrors option available for now but could accept any
other we may need. I find that this would remove the "ugliness" Matthias
pointed out for 3. and 4.

LM2. No matter how we end up expressing the different semantics for flush,
let's make sure we update the KIP on the flush and commitTransaction java
docs. It currently states that  flush "clears the last exception" and
commitTransaction "will NOT throw" if called after flush, but it really all
depends on the config/options/method used.

LM3. I find it would be helpful to include an example to show the new flow
that we're unblocking (I see this as the great gain here): flush with clear
error option enabled -> catch and do whatever error handling we want ->
commitTransaction successfully

Thanks!

Lianet

On Wed, Jun 19, 2024 at 11:26 PM Chris Egerton 
wrote:

> Hi Matthias,
>
> I like the alternatives you've listed. One more that might help is if,
> instead of overloading flush(), we overloaded commitTransaction() to
> something like commitTransaction(boolean tolerateRecordErrors). This seems
> slightly cleaner in that it takes the behavioral change we want, which only
> applies to transactional producers, to an API method that is only used for
> transactional producers. It would also avoid the issue of whether or not
> flush() (or a new variant of it with altered semantics) should throw or
> not. Thoughts?
>
> Hi Alieh,
>
> Thanks for the KIP, I like this direction a lot more than the pluggable
> handler!
>
> I share Artem's concerns that enabling this behavior via configuration
> doesn't seem like a great fit. It's likely that application code will be
> written in a style that only works with one type of behavior from
> transactional producers, so requiring that application code to declare its
> expectations for the behavior of its producer seems more appropriate than,
> e.g., allowing users deploying that application to tweak a configuration
> file that gets fed to producers spun up inside it.
>
> Cheers,
>
> Chris
>
> On Wed, Jun 19, 2024 at 10:32 PM Matthias J. Sax  wrote:
>
> > Thanks for the KIP Alieh. I actually like the KIP as-is, but think
> > Arthem raises very good points...
> >
> > Seems we have four options on how to move forward?
> >
> >   1. add config to allow "silent error clearance" as the KIP proposes
> >   2. change flush() to clear error and let it throw
> >   3. add new flushAndThrow()` (or better name) which clears error and
> > throws
> >   4. add `flush(boolean clearAndThrow)` and let user pick (and deprecate
> > existing `flush()`)
> >
> > For (2), given that it would be a behavior change, we might also need a
> > public "feature flag" config.
> >
> > It seems, both (1) and (2) have the issue Artem mentioned. (3) and (4)
> > would be safer to this end, however, for both we kinda get an ugly API?
> >
> > Not sure right now if I have any preference. Seems we need to pick some
> > evil and that there is no clear best solution? Would be good to her from
> > others what they think
> >
> >
> > -Matthias
> >
> >
> > On 6/18/24 8:39 PM, Artem Livshits wrote:
> > > Hi Alieh,
> > >
> > > Thank you for the KIP.  I have a couple of suggestions:
> > >
> > > AL1.  We should throw an error from flush after we clear it.  This
> would
> > > make it so that both "send + commit" and "send + flush + commit" (the
> > > latter looks like just a more verbose way to express the former, and it
> > > would be intuitive if it behaves the same) would throw if the
> transaction
> > > has an error (so if the code is written either way it's going be
> > correct).
> > > At the same time, the latter could be extended by the caller to
> intercept
> > > exceptions from flush, ignore as needed, and commit the transaction.
> > This
> > > solution would keep basic things simple (if someone has code that
> doesn't
> > > require advanced error handling, then basic "send + flush + commit"
> would
> > > do the right thing) and advanced things possible, an application can
> add
> > > try + catch around flush and ignore some errors.
> > >
> > > AL2.  I'm not sure if config is the best way to express the
> modification
> > of
> > > the "flush" semantics -- the application logic that calls "flush" needs
> > to
> > > match the "flush" semantics and configuring semantics in a detached
> place
> > > creates a room for bugs due to discrepancies

Re: [VOTE] KIP-1033: Add Kafka Streams exception handler for exceptions occurring during processing

[Lianet M.]

+1 (non-binding)

Thanks for the KIP and updates!

Lianet

On Tue, May 14, 2024, 12:03 a.m. Matthias J. Sax  wrote:

> +1 (binding)
>
> On 5/13/24 5:54 PM, Sophie Blee-Goldman wrote:
> > Thanks for the KIP guys!
> >
> > +1 (binding)
> >
> > On Mon, May 13, 2024 at 6:02 AM Bill Bejeck  wrote:
> >
> >> Thanks for the KIP, this will be a great addition!
> >>
> >> +1(binding)
> >>
> >> -Bill
> >>
> >> On Fri, May 3, 2024 at 4:48 AM Bruno Cadonna 
> wrote:
> >>
> >>> Hi Damien, Sébastien, and Loïc,
> >>>
> >>> Thanks for the KIP!
> >>>
> >>> +1 (binding)
> >>>
> >>> Best,
> >>> Bruno
> >>>
> >>>
> >>> On 4/26/24 4:00 PM, Damien Gasparina wrote:
>  Hi all,
> 
>  We would like to start a vote for KIP-1033: Add Kafka Streams
>  exception handler for exceptions occurring during processing
> 
>  The KIP is available on
> 
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1033%3A+Add+Kafka+Streams+exception+handler+for+exceptions+occurring+during+processing
> 
>  If you have any suggestions or feedback, feel free to participate to
>  the discussion thread:
>  https://lists.apache.org/thread/1nhhsrogmmv15o7mk9nj4kvkb5k2bx9s
> 
>  Best regards,
>  Damien Sebastien and Loic
> >>>
> >>
> >
>

Re: [DISCUSS] KIP-1033: Add Kafka Streams exception handler for exceptions occuring during processing

[Lianet M.]

Hi all, thanks Damien for the KIP!

After looking into the KIP and comments, my only concern is aligned with
one of Matthias comments, around the ImmutableHeaders introduction, with
the motivation not being clear enough. The existing handlers already expose
the headers (indirectly). Ex.
ProductionExceptionHandler.handleSerializationException provides the
ProducerRecord as an argument, so they are already exposed in those
callbacks through record.headers(). Is there a reason to think that it
would be a problem to expose the headers in the
new ProcessingExceptionHandler, but that it's not a problem for the
existing handler?

If there is no real concern about the KS engine requiring those headers, it
feels hard to mentally justify the complexity we transfer to the user by
exposing a new concept into the callbacks to represent the headers. In the
end, it strays aways from the simple/consistent representation of Headers
used all over. Even if eventually the KS engine needs to use the headers
after the callbacks with certainty that they were not altered, still feels
like it's something we could attempt to solve internally, without having to
transfer "new concepts" into the user (ex. the deep-copy as it was
suggested, seems like the kind of trade-off that would maybe be acceptable
here to gain simplicity and consistency among the handlers with a single
existing representation of Headers).

Best!

Lianet



On Tue, Apr 30, 2024 at 9:36 PM Matthias J. Sax  wrote:

> Thanks for the update.
>
> I am wondering if we should use `ReadOnlyHeaders` instead of
> `ImmutableHeaders` as interface name?
>
> Also, the returned `Header` interface is technically not immutable
> either, because `Header#key()` returns a mutable byte-array... Would we
> need a `ReadOnlyHeader` interface?
>
> If yes, it seems that `ReadOnlyHeaders` should not be a super-interface
> of `Headers` but it would rather be a standalone interface, and a
> wrapper for a `Headers` instance? And `ReadOnlyHeader` would return some
> immutable type instead of `byte[]` for the value()?
>
> An alternative would be to deep-copy the value byte-array what would not
> be free, but given that we are talking about exception handling, it
> would not be on the hot code path, and thus might be acceptable?
>
>
> The above seems to increase the complexity significantly though. Hence,
> I have seconds thoughts on the immutability question:
>
> Do we really need to worry about mutability after all, because in the
> end, KS runtime won't read the Headers instance after the handler was
> called, and if a user modifies the passed in headers, there won't be any
> actual damage (ie, no side effects)? For this case, it might even be ok
> to also not add `ImmutableHeaders` to begin with?
>
>
>
> Sorry for the forth and back (yes, forth and back, because back and
> forth does not make sense -- it's not logical -- just trying to fix
> English :D) as I did bring up the immutability question in the first
> place...
>
>
>
> -Matthias
>
> On 4/25/24 5:56 AM, Loic Greffier wrote:
> > Hi Matthias,
> >
> > I have updated the KIP regarding points 103 and 108.
> >
> > 103.
> > I have suggested a new `ImmutableHeaders` interface to deal with the
> > immutability concern of the headers, which is basically the `Headers`
> > interface without the write accesses.
> >
> > public interface ImmutableHeaders {
> >  Header lastHeader(String key);
> >  Iterable headers(String key);
> >  Header[] toArray();
> > }
> >
> > The `Headers` interface can be updated accordingly:
> >
> > public interface Headers extends ImmutableHeaders, Iterable {
> >  //…
> > }
> >
> > Loïc
>

Re: [VOTE] 3.6.2 RC2

[Lianet M.]

Hi Manikumar, I did the following checks:

- downloaded and built from src
- ran all unit test and integration test for clients
- ran quickstart with Kraft mode
- ran simple workloads with the console consumer/producer
- checked all links

All looks good to me with this.

+1 (non-binding)

Thanks!



On Wed, Apr 3, 2024, 2:19 a.m. Manikumar  wrote:

> Gentle reminder. Please download, test and vote for the release.
>
> Thanks,
>
> On Fri, Mar 29, 2024 at 4:57 PM Manikumar  wrote:
>
> > Hi All,
> >
> > System test runs are green. There were 13 test failures in the first run.
> > All the failed tests passed in the second run.
> >
> > System test results:
> > https://gist.github.com/omkreddy/17d23d3eb36ef840011f2494d65bbd4f
> >
> > Thanks,
> >
> > On Thu, Mar 28, 2024 at 3:21 PM Manikumar  wrote:
> >
> >> Hello Kafka users, developers and client-developers,
> >>
> >> This is the second candidate we are considering for the release of
> Apache
> >> Kafka 3.6.2.
> >>
> >> This is a bugfix release with several fixes, including dependency
> >> version bumps for CVEs.
> >>
> >> Release notes for the 3.6.2 release:
> >> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/RELEASE_NOTES.html
> >>
> >> *** Please download, test and vote by by Wednesday, April 3rd
> >>
> >> Kafka's KEYS file containing PGP keys we use to sign the release:
> >> https://kafka.apache.org/KEYS
> >>
> >> * Release artifacts to be voted upon (source and binary):
> >> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/
> >>
> >>
> >> * Maven artifacts to be voted upon:
> >> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >>
> >> * Javadoc:
> >> https://home.apache.org/~manikumar/kafka-3.6.2-rc2/javadoc/
> >>
> >> * Tag to be voted upon (off 3.6 branch) is the 3.6.2 tag:
> >> https://github.com/apache/kafka/releases/tag/3.6.2-rc2
> >>
> >> * Documentation:
> >> https://kafka.apache.org/36/documentation.html
> >>
> >> * Protocol:
> >> https://kafka.apache.org/36/protocol.html
> >>
> >> * Successful Jenkins builds for the 3.6 branch:
> >> Unit/integration tests:
> >> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/167/ (test
> >> failures passed in local runs)
> >> System tests: I will update system test results soon.
> >>
> >>
> >> Thanks,
> >> Manikumar
> >>
> >
>

Re: [VOTE] KIP-932: Queues for Kafka

[Lianet M.]

Hi Andrew,

Happy to see this KIP unleashing the capabilities of Kafka, from the core
up, to elegantly bring in a new, long wanted consumption pattern.

+1 (non-binding)

Lianet


On Sun, Mar 17, 2024, 7:06 p.m. Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:

> Hi,
> I’ve been working to complete KIP-932 over the past few months and
> discussions have quietened down.
>
> I’d like to open the voting for KIP-932:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
>
> Thanks,
> Andrew

Re: [ANNOUNCE] New committer: Lucas Brutschy

[Lianet M.]

Congratulations Lucas!

On Thu, Sept 21, 2023, 11:45 a.m. Bruno Cadonna  wrote:

> Hi all,
>
> The PMC of Apache Kafka is pleased to announce a new Kafka committer
> Lucas Brutschy.
>
> Lucas' major contributions are around Kafka Streams.
>
> Lucas' significantly contributed to the state updater
> (https://issues.apache.org/jira/browse/KAFKA-10199) and he drives the
> implementation of the new threading model for Kafka Streams
> (https://issues.apache.org/jira/browse/KAFKA-15326).
>
> Lucas' contributions to KIP discussions and PR reviews are very thoughtful.
>
> Congratulations, Lucas!
>
> Thanks,
>
> Bruno (on behalf of the Apache Kafka PMC)
>

Request to contribute to Kafka

[Lianet M.]

Wiki ID: lianetmr

Jira ID: lianetm

Both on email liane...@gmail.com


Thanks!

Lianet Magrans