Re: [DISCUSS] KIP-926: introducing acks=min.insync.replicas config

[Luke Chen]

Hi Haruki,

Yes, this scenario could happen.
I'm thinking we can fix it in step 6, when controller tried to get LEO from
B,C replicas, the B,C replica should stop fetcher for this partition
immediately, before returning the LEO.
About if we need quorum-based or not, We can discuss in another KIP. I'm
still thinking about it.

Thank you.
Luke


On Fri, May 12, 2023 at 3:59 PM Luke Chen  wrote:

> Hi David,
>
> > It can't be in another KIP as it is required for your proposal to work.
> This is also an important part to discuss as it requires the controller to
> do more operations on leader changes.
>
> Yes, I know this is a requirement for this KIP to work, and need a lot of
> discussion.
> So that's why I think it'd be better to have a separate KIP to write the
> content and discussion.
> I've put the status of this KIP as "pending" and added a note on the top
> of this KIP:
>
> Note: This KIP requires leader election change, which will be proposed in
> another KIP.
>
> Thanks.
> Luke
>
> On Thu, May 11, 2023 at 11:43 PM Alexandre Dupriez <
> alexandre.dupr...@gmail.com> wrote:
>
>> Hi, Luke,
>>
>> Thanks for your reply.
>>
>> 102. Whether such a replica could become a leader depends on what the
>> end-user wants to use it for and what tradeoffs they wish to make down
>> the line.
>>
>> There are cases, for instance with heterogeneous or interregional
>> networks, where the difference in latency between subsets of brokers
>> can be high enough for the "slow replicas" to have a detrimental
>> impact on the ISR traffic they take part in. This can justify
>> permanently segregating them from ISR traffic by design. And, an
>> end-user could still prefer to have these "slow replicas" versus
>> alternative approaches such as mirroring for the benefits they can
>> bring, for instance: a) they belong to the same cluster with no added
>> admin and ops, b) benefit from a direct, simpler replication path, c)
>> require less infrastructure than a mirrored solution, d) could become
>> unclean leaders for failovers under disaster scenarios such as a
>> regional service outages.
>>
>> Thanks,
>> Alexandre
>>
>> Le jeu. 11 mai 2023 à 14:57, Haruki Okada  a écrit :
>> >
>> > Hi, Luke.
>> >
>> > Though this proposal definitely looks interesting, as others pointed
>> out,
>> > the leader election implementation would be the hard part.
>> >
>> > And I think even LEO-based-election is not safe, which could cause
>> silent
>> > committed-data loss easily.
>> >
>> > Let's say we have replicas A,B,C and A is the leader initially, and
>> > min.insync.replicas = 2.
>> >
>> > - 1. Initial
>> > * A(leo=0), B(leo=0), C(leo=0)
>> > - 2. Produce a message to A
>> > * A(leo=1), B(leo=0), C(leo=0)
>> > - 3. Another producer produces a message to A (i.e. as the different
>> batch)
>> > * A(leo=2), B(leo=0), C(leo=0)
>> > - 4. C replicates the first batch. offset=1 is committed (by
>> > acks=min.insync.replicas)
>> > * A(leo=2), B(leo=0), C(leo=1)
>> > - 5. A loses ZK session (or broker session timeout in KRaft)
>> > - 6. Controller (regardless ZK/KRaft) doesn't store LEO in itself, so it
>> > needs to interact with each replica. It detects C has the largest LEO
>> and
>> > decided to elect C as the new leader
>> > - 7. Before leader-election is performed, B replicates offset=1,2 from
>> A.
>> > offset=2 is committed
>> > * This is possible because even if A lost ZK session, A could handle
>> > fetch requests for a while.
>> > - 8. Controller elects C as the new leader. B truncates its offset.
>> > offset=2 is lost silently.
>> >
>> > I have a feeling that we need quorum-based data replication? as Divij
>> > pointed out.
>> >
>> >
>> > 2023年5月11日(木) 22:33 David Jacot :
>> >
>> > > Hi Luke,
>> > >
>> > > > Yes, on second thought, I think the new leader election is required
>> to
>> > > work
>> > > for this new acks option. I'll think about it and open another KIP
>> for it.
>> > >
>> > > It can't be in another KIP as it is required for your proposal to
>> work.
>> > > This is also an important part to discuss as it requires the
>> controller to
>> > > do more operations on leader changes.
>> > >
>> >

Re: [DISCUSS] KIP-926: introducing acks=min.insync.replicas config

[Luke Chen]

Hi David,

> It can't be in another KIP as it is required for your proposal to work.
This is also an important part to discuss as it requires the controller to
do more operations on leader changes.

Yes, I know this is a requirement for this KIP to work, and need a lot of
discussion.
So that's why I think it'd be better to have a separate KIP to write the
content and discussion.
I've put the status of this KIP as "pending" and added a note on the top of
this KIP:

Note: This KIP requires leader election change, which will be proposed in
another KIP.

Thanks.
Luke

On Thu, May 11, 2023 at 11:43 PM Alexandre Dupriez <
alexandre.dupr...@gmail.com> wrote:

> Hi, Luke,
>
> Thanks for your reply.
>
> 102. Whether such a replica could become a leader depends on what the
> end-user wants to use it for and what tradeoffs they wish to make down
> the line.
>
> There are cases, for instance with heterogeneous or interregional
> networks, where the difference in latency between subsets of brokers
> can be high enough for the "slow replicas" to have a detrimental
> impact on the ISR traffic they take part in. This can justify
> permanently segregating them from ISR traffic by design. And, an
> end-user could still prefer to have these "slow replicas" versus
> alternative approaches such as mirroring for the benefits they can
> bring, for instance: a) they belong to the same cluster with no added
> admin and ops, b) benefit from a direct, simpler replication path, c)
> require less infrastructure than a mirrored solution, d) could become
> unclean leaders for failovers under disaster scenarios such as a
> regional service outages.
>
> Thanks,
> Alexandre
>
> Le jeu. 11 mai 2023 à 14:57, Haruki Okada  a écrit :
> >
> > Hi, Luke.
> >
> > Though this proposal definitely looks interesting, as others pointed out,
> > the leader election implementation would be the hard part.
> >
> > And I think even LEO-based-election is not safe, which could cause silent
> > committed-data loss easily.
> >
> > Let's say we have replicas A,B,C and A is the leader initially, and
> > min.insync.replicas = 2.
> >
> > - 1. Initial
> > * A(leo=0), B(leo=0), C(leo=0)
> > - 2. Produce a message to A
> > * A(leo=1), B(leo=0), C(leo=0)
> > - 3. Another producer produces a message to A (i.e. as the different
> batch)
> > * A(leo=2), B(leo=0), C(leo=0)
> > - 4. C replicates the first batch. offset=1 is committed (by
> > acks=min.insync.replicas)
> > * A(leo=2), B(leo=0), C(leo=1)
> > - 5. A loses ZK session (or broker session timeout in KRaft)
> > - 6. Controller (regardless ZK/KRaft) doesn't store LEO in itself, so it
> > needs to interact with each replica. It detects C has the largest LEO and
> > decided to elect C as the new leader
> > - 7. Before leader-election is performed, B replicates offset=1,2 from A.
> > offset=2 is committed
> > * This is possible because even if A lost ZK session, A could handle
> > fetch requests for a while.
> > - 8. Controller elects C as the new leader. B truncates its offset.
> > offset=2 is lost silently.
> >
> > I have a feeling that we need quorum-based data replication? as Divij
> > pointed out.
> >
> >
> > 2023年5月11日(木) 22:33 David Jacot :
> >
> > > Hi Luke,
> > >
> > > > Yes, on second thought, I think the new leader election is required
> to
> > > work
> > > for this new acks option. I'll think about it and open another KIP for
> it.
> > >
> > > It can't be in another KIP as it is required for your proposal to work.
> > > This is also an important part to discuss as it requires the
> controller to
> > > do more operations on leader changes.
> > >
> > > Cheers,
> > > David
> > >
> > > On Thu, May 11, 2023 at 2:44 PM Luke Chen  wrote:
> > >
> > > > Hi Ismael,
> > > > Yes, on second thought, I think the new leader election is required
> to
> > > work
> > > > for this new acks option. I'll think about it and open another KIP
> for
> > > it.
> > > >
> > > > Hi Divij,
> > > > Yes, I agree with all of them. I'll think about it and let you know
> how
> > > we
> > > > can work together.
> > > >
> > > > Hi Alexandre,
> > > > > 100. The KIP makes one statement which may be considered critical:
> > > > "Note that in acks=min.insync.replicas case, the slow follower might
> > > > be 

[DISCUSS] KIP-931: Flag to ignore unused message attribute field

[Luke Chen]

Hi all,

I'd like to start a discussion for the KIP-931: Flag to ignore unused
message attribute field. This KIP is to add a flag in the batch header to
indicate if messages inside the batch have attribute field or not, to
reduce the message size, thus, save network traffic and storage size (and
money, of course).

Please check the link for more detail:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-931%3A+Flag+to+ignore+unused+message+attribute+field

Any feedback is welcome.

Thank you.
Luke

Re: Requesting permissions to contribute to Apache Kafka.

[Luke Chen]

Hi MJ,

Thanks for the interest in Apache kafka.
I've granted your JIRA and wiki account.
Looking forward to seeing your contribution!

Thanks.
Luke

On Fri, May 12, 2023 at 11:29 AM Mahesh Jayawardena 
wrote:

> Dear Apache Kafka Development Team,
>
> My name is Mahesh and I am interested in contributing to the Apache Kafka
> project. I have been following the development of Kafka for some time and I
> am impressed with the work that has been done by the community.
>
> I would like to request permission to contribute to the project. My Wiki ID
> is" Mahesh Madushan Jayawardhana   and my Jira ID is  mahesh98  . I have
> signed the Contributor License Agreement and I am ready to begin
> contributing to the project.
>
> I am particularly interested in working on KAFKA-13349, but I am also open
> to other areas where I can be of assistance.I am excited to apply my skills
> to contribute to the Kafka community.
>
> Please let me know what steps I need to take to become a contributor to the
> Apache Kafka project. I am looking forward to being part of this vibrant
> and dynamic community.
>
> Thank you for considering my request. I appreciate your time and attention.
>
> Best regards,
>
> MJ.
>

Re: Unable to access JIRA

[Luke Chen]

Hi Ram,

Please submit an account creation request and we'll approve it.
https://selfserve.apache.org/jira-account.html

Thanks.
Luke

On Thu, May 11, 2023 at 9:50 PM Ramakrishnan Subramanian <
ramkris.subraman...@gmail.com> wrote:

> Hi Team,
>
>I am unable to access JIRA. Also, I would like to contribute to kafka.
> Please help me on this.
>
> JIRA ID ramkris.subramanian
> mail id : ramkris.subraman...@gmail.com
>
>
> Thanks & Regards
> Ram
>

Re: [DISCUSS] KIP-926: introducing acks=min.insync.replicas config

[Luke Chen]

Hi Ismael,
Yes, on second thought, I think the new leader election is required to work
for this new acks option. I'll think about it and open another KIP for it.

Hi Divij,
Yes, I agree with all of them. I'll think about it and let you know how we
can work together.

Hi Alexandre,
> 100. The KIP makes one statement which may be considered critical:
"Note that in acks=min.insync.replicas case, the slow follower might
be easier to become out of sync than acks=all.". Would you have some
data on that behaviour when using the new ack semantic? It would be
interesting to analyse and especially look at the percentage of time
the number of replicas in ISR is reduced to the configured
min.insync.replicas.

The comparison data would be interesting. I can have a test when available.
But this KIP will be deprioritized because there should be a pre-requisite
KIP for it.

> A (perhaps naive) hypothesis would be that the
new ack semantic indeed provides better produce latency, but at the
cost of precipitating the slowest replica(s) out of the ISR?

Yes, it could be.

> 101. I understand the impact on produce latency, but I am not sure
about the impact on durability. Is your durability model built against
the replication factor or the number of min insync replicas?

Yes, and also the new LEO-based leader election (not proposed yet).

> 102. Could a new type of replica which would not be allowed to enter
the ISR be an alternative? Such replica could attempt replication on a
best-effort basis and would provide the permanent guarantee not to
interfere with foreground traffic.

You mean a backup replica, which will never become leader (in-sync), right?
That's an interesting thought and might be able to become a workaround with
the existing leader election. Let me think about it.

Hi qiangLiu,

> It's a good point that add this config and get better P99 latency, but is
this changing the meaning of "in sync replicas"? consider a situation with
"replica=3 acks=2", when two broker fail and left only the broker that
does't have the message, it is in sync, so will be elected as leader, will
it cause a NOT NOTICED lost of acked messages?

Yes, it will, so the `min.insync.replicas` config in the broker/topic level
should be set correctly. In your example, it should be set to 2, so that
when 2 replicas down, no new message write will be successful.


Thank you.
Luke


On Thu, May 11, 2023 at 12:16 PM 67 <6...@gd67.com> wrote:

> Hi Luke,
>
>
> It's a good point that add this config and get better P99 latency, but is
> this changing the meaning of "in sync replicas"? consider a situation with
> "replica=3 acks=2", when two broker fail and left only the broker that
> does't have the message, it is in sync, so will be elected as leader, will
> it cause a *NOT NOTICED* lost of acked messages?
>
> qiangLiu
>
>
> 在2023年05月10 12时58分，"Ismael Juma"写道：
>
>
> Hi Luke,
>
> As discussed in the other KIP, there are some subtleties when it comes to
> the semantics of the system if we don't wait for all members of the isr
> before we ack. I don't understand why you say the leader election question
> is out of scope - it seems to be a core aspect to me.
>
> Ismael
>
>
> On Wed, May 10, 2023, 8:50 AM Luke Chen  wrote:
>
> > Hi Ismael,
> >
> > No, I didn't know about this similar KIP! I hope I've known that so that I
> > don't need to spend time to write it again! :(
> > I checked the KIP and all the discussions (here
> > <https://lists.apache.org/list?dev@kafka.apache.org:gte=100d:KIP-250>). I
> > think the consensus is that adding a client config to `acks=quorum` is
> > fine.
> > This comment
> > <https://lists.apache.org/thread/p77pym5sxpn91r8j364kmmf3qp5g65rn> from
> > Guozhang pretty much concluded what I'm trying to do.
> >
> >
> >
> >
> >
> >
> >
> >
> > *1. Add one more value to client-side acks config:   0: no acks needed at
> > all.   1: ack from the leader.   all: ack from ALL the ISR replicas
> >  quorum: this is the new value, it requires ack from enough number of ISR
> > replicas no smaller than majority of the replicas AND no smaller
> > than{min.isr}.2. Clarify in the docs that if a user wants to tolerate X
> > failures, she needs to set client acks=all or acks=quorum (better tail
> > latency than "all") with broker {min.sir} to be X+1; however, "all" is not
> > necessarily stronger than "quorum".*
> >
> > Concerns from KIP-250 are:
> > 1. Introducing a new leader LEO based election method. This is not clear in
> > the KIP-250 and needs more discussion
> > 2. The KIP-250 also tried to

Re: [VOTE] KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory copying

[Luke Chen]

+1(binding) from me.
Thanks for the improvement!

Luke

On Sun, May 7, 2023 at 6:34 PM Divij Vaidya  wrote:

> Vote +1 (non binding)
>
> I think that this is a nice improvement as it prevents an unnecessary data
> copy for users who are using ByteBuffer serialization on the producer.
>
> --
> Divij Vaidya
>
>
>
> On Sun, May 7, 2023 at 9:24 AM ShunKang Lin 
> wrote:
>
> > Hi everyone,
> >
> > I'd like to open the vote for KIP-872, which proposes to add
> > Serializer#serializeToByteBuffer() to reduce memory copying.
> >
> > The proposal is here:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=228495828
> >
> > The pull request is here:
> > https://github.com/apache/kafka/pull/12685
> >
> > Thanks to all who reviewed the proposal, and thanks in advance for taking
> > the time to vote!
> >
> > Best,
> > ShunKang
> >
>

Re: [DISCUSS] KIP-927: Improve the kafka-metadata-quorum output

[Luke Chen]

Hi Fede,

Thanks for the KIP.
LGTM.

@Divij
I think the timestamp is timezone independent, so it should be fine.

Luke

On Thu, May 11, 2023 at 2:04 AM Divij Vaidya 
wrote:

> Thank you for the KIP.
>
> The current proposal has the limitation that it uses a duration syntax for
> representation of a timestamp. Also, the syntax relies on the locale and
> timezone of the caller machine. This makes it difficult to share the output
> with others. As an example, let's say you want to share the state of the
> quorum with me which was captured but "3s ago" gives me no information on
> when it was executed.
>
> Alternatively, may I suggest introducing a parameter called
> "--datetime-format=" which takes as value the ISO-8601 [1] format and
> prints the timestamp based on the provided format. It would solve the
> problem of readability (epochs are hard to read!) as well as the problem of
> portability of output across machines.
>
> What do you think?
>
> [1] https://en.wikipedia.org/wiki/ISO_8601
>
> --
> Divij Vaidya
>
>
>
> On Wed, May 10, 2023 at 6:10 PM Federico Valeri 
> wrote:
>
> > Hi all, I'd like to start a new discussion thread on KIP-927: Improve
> > the kafka-metadata-quorum output.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-927%3A+Improve+the+kafka-metadata-quorum+output
> >
> > This KIP is small and proposes to add a new optional flag to have a
> > human-readable timestamp output.
> >
> > Thank you!
> >
> > Br
> > Fede
> >
>

[jira] [Created] (KAFKA-14982) Improve the kafka-metadata-quorum output

[Luke Chen (Jira)]

Luke Chen created KAFKA-14982:
-

 Summary: Improve the kafka-metadata-quorum output
 Key: KAFKA-14982
 URL: https://issues.apache.org/jira/browse/KAFKA-14982
 Project: Kafka
  Issue Type: Improvement
Reporter: Luke Chen
Assignee: Federico Valeri


When running kafka-metadata-quorum script to get the quorum replication status, 
I found the LastFetchTimestamp and LastCaughtUpTimestamp output is not human 
readable. The timestamp 1683701749161 is just a random integer to me. We should 
convert it into date/time (ex: May 10, 08:00 UTC), or if possible, convert it 
into strings like "10 seconds ago", "5 minutes ago"...

 

 
{code:java}
 % ./bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe 
--replication
NodeId    LogEndOffset    Lag    LastFetchTimestamp    LastCaughtUpTimestamp    
Status      
1         166             0      1683701749161         1683701749161            
Leader      
6         166             0      1683701748776         1683701748776            
Follower    
7         166             0      1683701748773         1683701748773            
Follower    
2         166             0      1683701748766         1683701748766            
Observer {code}
 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-926: introducing acks=min.insync.replicas config

[Luke Chen]

Hi Ismael,

No, I didn't know about this similar KIP! I hope I've known that so that I
don't need to spend time to write it again! :(
I checked the KIP and all the discussions (here
<https://lists.apache.org/list?dev@kafka.apache.org:gte=100d:KIP-250>). I
think the consensus is that adding a client config to `acks=quorum` is
fine.
This comment
<https://lists.apache.org/thread/p77pym5sxpn91r8j364kmmf3qp5g65rn> from
Guozhang pretty much concluded what I'm trying to do.








*1. Add one more value to client-side acks config:   0: no acks needed at
all.   1: ack from the leader.   all: ack from ALL the ISR replicas
 quorum: this is the new value, it requires ack from enough number of ISR
replicas no smaller than majority of the replicas AND no smaller
than{min.isr}.2. Clarify in the docs that if a user wants to tolerate X
failures, she needs to set client acks=all or acks=quorum (better tail
latency than "all") with broker {min.sir} to be X+1; however, "all" is not
necessarily stronger than "quorum".*

Concerns from KIP-250 are:
1. Introducing a new leader LEO based election method. This is not clear in
the KIP-250 and needs more discussion
2. The KIP-250 also tried to optimize the consumer latency to read messages
beyond high watermark, which also has some discussion about how to achieve
that, and no conclusion

Both of the above 2 concerns are out of the scope of my current KIP.
So, I think it's good to provide this `acks=quorum` or
`acks=min.insync.replicas` option to users to give them another choice.


Thank you.
Luke


On Wed, May 10, 2023 at 8:54 AM Ismael Juma  wrote:

> Hi Luke,
>
> Are you aware of
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-250+Add+Support+for+Quorum-based+Producer+Acknowledgment
> ?
>
> Ismael
>
> On Tue, May 9, 2023 at 10:14 PM Luke Chen  wrote:
>
> > Hi all,
> >
> > I'd like to start a discussion for the KIP-926: introducing
> > acks=min.insync.replicas config. This KIP is to introduce
> > `acks=min.insync.replicas` config value in producer, to improve the write
> > throughput and still guarantee high durability.
> >
> > Please check the link for more detail:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-926%3A+introducing+acks%3Dmin.insync.replicas+config
> >
> > Any feedback is welcome.
> >
> > Thank you.
> > Luke
> >
>

Re: [DISCUSS] KIP-926: introducing acks=min.insync.replicas config

[Luke Chen]

Hi Justine,

Thanks for the comment. Yes, I know we have a configuration
min.insync.replicas to avoid this case. And actually, that's my following
paragraph is talking about.

Please let me know if you have other comments.

Thanks
Luke

Justine Olshan  於 2023年5月10日 週三 上午1:01 寫道：

> Hey Luke,
>
> I was taking a quick pass over the KIP and saw this line:
>
> >It looks perfect. But there's a caveat here. Like the doc said, acks=all
> will "wait for the *full set of in-sync replicas *to acknowledge the
> record", so if there's only 1 replica in in-sync replicas, it will have the
> same effect as acks=1 (even though we have replication-factor set to 3).
>
> I thought we had a configuration min.insync.replicas to avoid this case.
> (Typically it is set to two so we need a leader and one follower to
> respond.)
>
> I am curious to understand the proposal more, but just thought I'd share
> this as it came up.
>
> Justine
>
> On Tue, May 9, 2023 at 9:44 AM Luke Chen  wrote:
>
> > Hi all,
> >
> > I'd like to start a discussion for the KIP-926: introducing
> > acks=min.insync.replicas config. This KIP is to introduce
> > `acks=min.insync.replicas` config value in producer, to improve the write
> > throughput and still guarantee high durability.
> >
> > Please check the link for more detail:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-926%3A+introducing+acks%3Dmin.insync.replicas+config
> >
> > Any feedback is welcome.
> >
> > Thank you.
> > Luke
> >
>

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Luke Chen]

Hi Mickael,

Since we haven't had the CR created yet, I'm thinking we should backport
this doc improvement to v3.5.0 to make the doc complete.
https://github.com/apache/kafka/pull/13660

What do you think?

Luke

On Sat, May 6, 2023 at 11:31 PM David Arthur  wrote:

> I resolved these three:
> * KAFKA-14840 is merged to trunk and 3.5. I removed the 3.4.1 fix version
> * KAFKA-14805 is merged to trunk and 3.5
> * KAFKA-14918 is merged to trunk and 3.5
>
> KAFKA-14692 (docs issue) is still a not done
>
> Looks like KAFKA-14084 is now resolved as well (it's in trunk and 3.5).
>
> I'll try to find out about KAFKA-14698, I think it's likely a WONTFIX.
>
> -David
>
> On Fri, May 5, 2023 at 10:43 AM Mickael Maison 
> wrote:
>
> > Hi David,
> >
> > Thanks for the update!
> > You still own 4 other tickets targeting 3.5: KAFKA-14840, KAFKA-14805,
> > KAFKA-14918, KAFKA-14692. Should I move all of them to the next
> > release?
> > Also KAFKA-14698 and KAFKA-14084 are somewhat related to the
> > migration. Should I move them too?
> >
> > Thanks,
> > Mickael
> >
> > On Fri, May 5, 2023 at 4:27 PM David Arthur
> >  wrote:
> > >
> > > Hey Mickael, my two ZK migration fixes are in 3.5 now.
> > >
> > > Cheers,
> > > David
> > >
> > > On Fri, May 5, 2023 at 9:37 AM Mickael Maison <
> mickael.mai...@gmail.com>
> > > wrote:
> > >
> > > > Hi Divij,
> > > >
> > > > Some dependencies (ZooKeeper, Snappy, Swagger, zstd, etc) have been
> > > > updated since 3.4.
> > > > Regarding your PR, I would have been in favor of bringing this to 3.5
> > > > a couple of weeks ago, but we're now a week past code freeze for 3.5.
> > > > Apart if this fixes CVEs, or significant bugs, I think we should only
> > > > merge it in trunk.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Fri, May 5, 2023 at 1:49 PM Divij Vaidya  >
> > > > wrote:
> > > > >
> > > > > Hey Mickael
> > > > >
> > > > > Should we consider performing an update of the minor versions of
> the
> > > > > dependencies in 3.5 (per
> https://github.com/apache/kafka/pull/13673
> > )?
> > > > >
> > > > > --
> > > > > Divij Vaidya
> > > > >
> > > > >
> > > > >
> > > > > On Tue, May 2, 2023 at 5:48 PM Mickael Maison <
> > mickael.mai...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Luke,
> > > > > >
> > > > > > Yes I think it makes sense to backport both to 3.5.
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > > On Tue, May 2, 2023 at 11:38 AM Luke Chen 
> > wrote:
> > > > > > >
> > > > > > > Hi Mickael,
> > > > > > >
> > > > > > > There are 1 bug and 1 improvement that I'd like to backport to
> > 3.5.
> > > > > > > 1. A small improvement for ZK migration based on KAFKA-14840
> > > > (mentioned
> > > > > > > above in David's mail). PR is already merged to trunk.
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-14909
> > > > > > >
> > > > > > > 2. A bug will cause the KRaft controller node to shut down
> > > > unexpectedly.
> > > > > > PR
> > > > > > > is ready for review.
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-14946
> > > > > > > https://github.com/apache/kafka/pull/13653
> > > > > > >
> > > > > > > Thanks.
> > > > > > > Luke
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Apr 28, 2023 at 4:18 PM Mickael Maison <
> > > > mickael.mai...@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi David,
> > > > > > > >
> > > > > > > > Yes you can backport these to 3.5. Let me know when you are
> > done.
> > > > > > > >
> > > > > > > > Thanks,
&g

Re: apply for permission to contribute to Apache Kafka

[Luke Chen]

Done.

Thanks.
Luke

On Sat, May 6, 2023 at 9:38 PM Doe John  wrote:

> my Jira ID: johndoe
>
> on email zh2725284...@gmail.com
>
> Thanks!
>

[DISCUSS] KIP-926: introducing acks=min.insync.replicas config

[Luke Chen]

Hi all,

I'd like to start a discussion for the KIP-926: introducing
acks=min.insync.replicas config. This KIP is to introduce
`acks=min.insync.replicas` config value in producer, to improve the write
throughput and still guarantee high durability.

Please check the link for more detail:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-926%3A+introducing+acks%3Dmin.insync.replicas+config

Any feedback is welcome.

Thank you.
Luke

[jira] [Resolved] (KAFKA-14662) ACL listings in documentation are out of date

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14662.
---
Fix Version/s: 3.6.0
   Resolution: Fixed

> ACL listings in documentation are out of date
> -
>
> Key: KAFKA-14662
> URL: https://issues.apache.org/jira/browse/KAFKA-14662
> Project: Kafka
>  Issue Type: Bug
>  Components: core, docs
>Reporter: Mickael Maison
>Assignee: Gantigmaa Selenge
>Priority: Major
> Fix For: 3.6.0
>
>
> ACLs listed in 
> https://kafka.apache.org/documentation/#operations_resources_and_protocols 
> are out of date. They only cover API keys up to 47 (OffsetDelete) and don't 
> include DescribeClientQuotas, AlterClientQuotas, 
> DescribeUserScramCredentials, AlterUserScramCredentials, DescribeQuorum, 
> AlterPartition, UpdateFeatures, DescribeCluster, DescribeProducers, 
> UnregisterBroker, DescribeTransactions, ListTransactions, AllocateProducerIds.
> This is hard to keep up to date so we should consider whether this could be 
> generated automatically.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Request to contribute to Kafka

[Luke Chen]

Hi Doe,

I've added you into the contributor group.

Thanks.
Luke

On Sat, May 6, 2023 at 11:36 AM Doe John  wrote:

> my Jira ID: JohnDoe
>
> on email zh2725284...@gmail.com
>
> Thanks!
>

[jira] [Resolved] (KAFKA-14963) Incorrect partition count metrics for kraft controllers

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14963.
---
Resolution: Fixed

> Incorrect partition count metrics for kraft controllers
> ---
>
> Key: KAFKA-14963
> URL: https://issues.apache.org/jira/browse/KAFKA-14963
> Project: Kafka
>  Issue Type: Bug
>  Components: controller, kraft
>Affects Versions: 3.4.0
>Reporter: José Armando García Sancio
>Assignee: José Armando García Sancio
>Priority: Major
> Fix For: 3.4.1
>
>
> It is possible for the KRaft controller to report more partitions than are 
> available in the cluster. This is because the following test fail against 
> 3.4.0:
> {code:java}
>        @Test
>       public void testPartitionCountDecreased() {
>           ControllerMetrics metrics = new MockControllerMetrics();
>           ControllerMetricsManager manager = new 
> ControllerMetricsManager(metrics);          Uuid createTopicId = 
> Uuid.randomUuid();
>           Uuid createPartitionTopicId = new Uuid(
>               createTopicId.getMostSignificantBits(),
>               createTopicId.getLeastSignificantBits()
>           );
>           Uuid removeTopicId = new 
> Uuid(createTopicId.getMostSignificantBits(), 
> createTopicId.getLeastSignificantBits());
>           manager.replay(topicRecord("test", createTopicId));
>           manager.replay(partitionRecord(createPartitionTopicId, 0, 0, 
> Arrays.asList(0, 1, 2)));
>           manager.replay(partitionRecord(createPartitionTopicId, 1, 0, 
> Arrays.asList(0, 1, 2)));
>           manager.replay(removeTopicRecord(removeTopicId));
>           assertEquals(0, metrics.globalPartitionCount());
>       }
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Luke Chen]

t; > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > > > >> Mickael
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> On Fri, Apr 14, 2023 at 11:23 AM Satish
> Duggana
> > > > > > > > > > > > > > >>  wrote:
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> Thanks Luke for helping with the reviews and
> > > adding a
> > > > > > > few tests in
> > > > > > > > > > > > a
> > > > > > > > > > > > > > >>> couple of PRs.
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> Hi Mickael,
> > > > > > > > > > > > > > >>> I raised 3 PRs recently for tiered storage,
> one
> > > is
> > > > > > > merged. The
> > > > > > > > > > > > other 2
> > > > > > > > > > > > > > >>> PRs are in the critical path of non-tiered
> > > storage
> > > > > > > changes also.
> > > > > > > > > > > > > > >>> Especially, in consumer fetch and retention
> > > cleanup
> > > > > > > paths. These
> > > > > > > > > > > > need
> > > > > > > > > > > > > > >>> to be thoroughly reviewed and avoid any
> > > regressions in
> > > > > > > that area.
> > > > > > > > > > > > We
> > > > > > > > > > > > > > >>> should merge them to the trunk as soon as
> > > possible to
> > > > > > > make it
> > > > > > > > > > > > easier
> > > > > > > > > > > > > > >>> to work on follow-up PRs. IMO, we can avoid
> > > merging
> > > > > > > these PRs in
> > > > > > > > > > > > 3.5
> > > > > > > > > > > > > > >>> just before the release without baking for a
> > > longer
> > > > > > > duration. We
> > > > > > > > > > > > can
> > > > > > > > > > > > > > >>> take a call on this later after the reviews
> are
> > > done.
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> Many of the individual functionalities
> related to
> > > > > > tiered
> > > > > > > storage
> > > > > > > > > > > > like
> > > > > > > > > > > > > > >>> default topic based RLMM implementation,
> enhanced
> > > > > > > follower fetch
> > > > > > > > > > > > > > >>> protocol implementation for tiered storage,
> > > copying
> > > > > > > remote log
> > > > > > > > > > > > > > >>> segments are merged.
> > > > > > > > > > > > > > >>> There are 2 PRs for consumer fetch for remote
> > > record
> > > > > > > reads, remote
> > > > > > > > > > > > > > >>> retention cleanup and topic deletion
> > > functionality are
> > > > > > > under
> > > > > > > > > > > > review.
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> I do not think it can be considered as an
> early
> > > access
> > > > > > > review even
> > > > > > > > > > > > > > >>> with the 2 PRs in review. Luke and I synced
> up
> > > and
> > > > > > > agreed on the
> > > > > > > > > > > > same.
> > > > > > > > > > > > > > >>> Most o

Re: Access to create a KIP

[Luke Chen]

Hi Ravindra,

Your accounts are all set.

Thanks.
Luke

On Tue, May 2, 2023 at 12:30 AM Ravindra Nath Kakarla <
ravindhran...@gmail.com> wrote:

> Hi,
>
> I am trying to create a KIP as part of JIRA
> https://issues.apache.org/jira/browse/KAFKA-14952 and need access to the
> wiki to create a KIP. Below is my JIRA and wiki IDs.
>
> Wiki ID: ravindhranath
> JIRA ID: blacktooth
>
> Thank you
>

Re: [DISCUSS] Apache Kafka 3.4.1 release

[Luke Chen]

Hi,

I've created a release plan for 3.4.1 in the wiki:
https://cwiki.apache.org/confluence/display/KAFKA/Release+plan+3.4.1

The release date will be in May 2023.

Please take a look at the plan.

This date is approximate at best and subject to change. Bug fix releases
are driven both by the impact of bug fixes that have already been committed
and known outstanding issues. Please email the dev list if you think the
timeline needs to be adjusted.

Thanks,
Luke

Re: [DISCUSS] Adding non-committers as Github collaborators

[Luke Chen]

Hi David,

Thanks for the suggestion. I'm +1.
This is also a good way to show appreciation from Kafka community to some
great contributors who haven't qualified to become committers.

Luke

On Fri, Apr 28, 2023 at 2:45 AM David Arthur
 wrote:

> Hey folks,
>
> I stumbled across this wiki page from the infra team that describes the
> various features supported in the ".asf.yaml" file:
> https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features
>
> One section that looked particularly interesting was
>
> https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-AssigningexternalcollaboratorswiththetriageroleonGitHub
>
> github:
>   collaborators:
> - userA
> - userB
>
> This would allow us to define non-committers as collaborators on the Github
> project. Concretely, this means they would receive the "triage" Github role
> (defined here
>
> https://docs.github.com/en/organizations/managing-user-access-to-your-organizations-repositories/repository-roles-for-an-organization#permissions-for-each-role
> ).
> Practically, this means we could let non-committers do things like assign
> labels and reviewers on Pull Requests.
>
> I wanted to see what the committer group thought about this feature. I
> think it could be useful.
>
> Cheers,
> David
>

Re: Requesting permissions to contribute to Apache Kafka

[Luke Chen]

Hi Keith,

Your accounts are all set.

Thanks.
Luke

On Thu, Apr 27, 2023 at 8:44 PM Keith Wall  wrote:

> Hello,
>
> I'm following
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-GettingStarted
> and have reached step 4.
>
> wiki ID: kwall
> Jira ID: kwall
>
> Keith Wall.
>

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

[Luke Chen]

Hi Colin,

Some comments:
1. I agree we should set "top-level" errors for metadata response

2. In the "brokers" field of metadata response from controller, it'll
respond with "Controller endpoint information as given in
controller.quorum.voters", instead of the "alive" controllers(voters). That
will break the existing admin client because in admin client, we'll rely on
the metadata response to build the "current alive brokers" list, and choose
one from them to connect (either least load or other criteria). That means,
if now, we return the value in `controller.quorum.voters`, but one of them
is down. We might choose it to connect and get connection errors. Should we
return the "alive" controllers(voters) to client?

3. In the KIP, we list the command-line tools will get a new
--bootstrap-controllers argument, but without explaining why these tools
need to talk to controller directly. Could we add some explanation about
them? I tried but cannot know why some tools are listed here:
- kafka-acls.sh -> Allow clients to update ACLs via controller before
brokers up

- kafka-cluster.sh -> Reasonable to get/update cluster info via
controller

- kafka-configs.sh -> Allow clients to dynamically update
configs/describe configs from controller. But in this script, client can
still set quota for users/clients/topics... is client also able to update
via controllers? Or we only allow partial actions in the script to talk to
controllers?

- kafka-delegation-tokens.sh -> Reasonable to update delegation-tokens
via controllers

- kafka-features.sh -> Reasonable
- kafka-metadata-quorum.sh -> Reasonable
- kafka-metadata-shell.sh -> Reasonable

- kafka-reassign-partitions.sh -> Why should we allow clients to move
metadata log partitions in controller nodes? What's the use-case?


Thank you.
Luke

On Thu, Apr 27, 2023 at 8:04 AM Colin McCabe  wrote:

> On Tue, Apr 25, 2023, at 04:59, Divij Vaidya wrote:
> > Thank you for the KIP Colin.
> >
> > In general, I like the idea of having the ability to interact directly
> with
> > the controllers. I agree with your observation that it helps in
> situations
> > where you would want to get data directly from the controller instead of
> > going via a broker. I have some general comments but the main concern I
> > have is with the piggy-backing of error code with response of
> > __cluster_metadata topic.
> >
> > 1. With this change, how are we guarding against the possibility of
> > misbehaving client traffic from disrupting the controller (that you
> > mentioned as a motivation of earlier behaviour)? One solution could be to
> > have default values set for request throttling on the controller.
>
> Hi Divij,
>
> Thanks for the comments.
>
> Our guards against client misbehavior remain the same:
> 1. our recommendation to put the clients on a separate network
> 2. the fact that producers and consumers can't interact directly with the
> controller
> 3. the authorizer.
>
> Re: #3, I will spell out in the KIP that clients must have DESCRIBE on the
> CLUSTER resource to send a METADATA request to the controller.
>
> > 2. This KIP also increases the network attack surface area. Prior to this
> > KIP, it was possible to have firewall rules setup for controllers such
> that
> > only the brokers can talk to it. But now, the controller would require
> > access from other endpoints other than brokers as well. Can we add a
> > suggestion to the upgrade documentation and inform users
>
> There is no requirement for access from other endpoints. It is still
> possible to set up firewall rules such that only the brokers can talk to
> the controller. In fact I would even say this is desirable. Since this
> faculty is intended for infrequent manual administrative operations,
> needing to log into the broker to use it seems perfectly fine.
>
> > 3. In section KRaft Controller MetadataResponse, row 3, "There is no
> > top-level error code in MetadataResponse, so we use the
> __cluster_metadata
> > topic to send back our error.". This will definitely confuse the users.
> Can
> > we introduce a top level error field instead?
>
> Let me check how we're handling this in other places. I recall some other
> cases where we used the dummy topic approach, but I can't find them just
> now.
>
> > 4. As part of the KIP, could we please add some documentation for users
> > with the suggestion of when to get information directly from the
> controller
> > and when not to (and associated tradeoffs)?
>
> I think most users will not face this tradeoff because they simply won't
> have network access to the controller servers.
>
> For those who do want more information, we'll have command-line
> documentation for --boostrap-controllers and the bootstrap.controllers
> configuration key.
>
> > 5. Why do we need the `FromKRaftController` field in the response? What
> do
> > we expect the users to do with this information?
>
> The field is present so that we can throw an exception in the client if we

[jira] [Resolved] (KAFKA-14944) Reduce CompletedFetch#parseRecord() memory copy

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14944.
---
Fix Version/s: 3.6.0
   Resolution: Fixed

> Reduce CompletedFetch#parseRecord() memory copy
> ---
>
> Key: KAFKA-14944
> URL: https://issues.apache.org/jira/browse/KAFKA-14944
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: LinShunkang
>Assignee: LinShunkang
>Priority: Major
> Fix For: 3.6.0
>
>
> JIRA for KIP-863: 
> [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-14945) Add Serializer#serializeToByteBuffer() to reduce memory copying

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen reopened KAFKA-14945:
---

> Add Serializer#serializeToByteBuffer() to reduce memory copying
> ---
>
> Key: KAFKA-14945
> URL: https://issues.apache.org/jira/browse/KAFKA-14945
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: LinShunkang
>Assignee: LinShunkang
>Priority: Major
> Fix For: 3.6.0
>
>
> JIAR for KIP-872: 
> [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=228495828|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=228495828]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14945) Add Serializer#serializeToByteBuffer() to reduce memory copying

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14945.
---
Fix Version/s: 3.6.0
   Resolution: Fixed

> Add Serializer#serializeToByteBuffer() to reduce memory copying
> ---
>
> Key: KAFKA-14945
> URL: https://issues.apache.org/jira/browse/KAFKA-14945
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: LinShunkang
>Assignee: LinShunkang
>Priority: Major
> Fix For: 3.6.0
>
>
> JIAR for KIP-872: 
> [https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=228495828|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=228495828]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] Apache Kafka 3.4.1 release

[Luke Chen]

Hi all,

I'd like to volunteer as release manager for the Apache
Kafka 3.4.1 release.
If there are no objections, I'll start building a release plan in the
wiki in the next couple of days.

Thank you.
Luke

[jira] [Resolved] (KAFKA-14908) Sporadic "Address already in use" when starting kafka cluster embedded within tests

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14908.
---
Fix Version/s: 3.6.0
   Resolution: Fixed

> Sporadic "Address already in use" when starting kafka cluster embedded within 
> tests
> ---
>
> Key: KAFKA-14908
> URL: https://issues.apache.org/jira/browse/KAFKA-14908
> Project: Kafka
>  Issue Type: Bug
>Reporter: Keith Wall
>Priority: Minor
> Fix For: 3.6.0
>
>
> We have an integration test suite that starts/stops a kafka cluster 
> before/after each test.   Kafka is being started programmatically within the 
> same JVM that is running the tests.
> Sometimes we get sporadic failures from with Kafka as it tries to bind the 
> server socket.
> {code:java}
> org.apache.kafka.common.KafkaException: Socket server failed to bind to 
> 0.0.0.0:9092: Address already in use.
>     at kafka.network.Acceptor.openServerSocket(SocketServer.scala:684)
>     at kafka.network.Acceptor.(SocketServer.scala:576)
>     at kafka.network.DataPlaneAcceptor.(SocketServer.scala:433)
>     at 
> kafka.network.SocketServer.createDataPlaneAcceptor(SocketServer.scala:247)
>     at 
> kafka.network.SocketServer.createDataPlaneAcceptorAndProcessors(SocketServer.scala:226)
>     at kafka.network.SocketServer.$anonfun$new$31(SocketServer.scala:173)
>     at 
> kafka.network.SocketServer.$anonfun$new$31$adapted(SocketServer.scala:173)
>     at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:575)
>     at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:573)
>     at scala.collection.AbstractIterable.foreach(Iterable.scala:933)
>     at kafka.network.SocketServer.(SocketServer.scala:173)
>     at kafka.server.KafkaServer.startup(KafkaServer.scala:331) {code}
> Investigation has shown that the socket is in the timed_wait state from a 
> previous test.
> I know Kafka supports ephemeral ports, but this isn't convenient to our 
> use-case.  
> I'd like to suggest that Kafka is changed to set the SO_REUSEADDR on the 
> server socket.  I believe this is standard practice for server applications 
> that run on well known ports .
> I don't believe this change would introduce a backward compatibility 
> concerns. 
>  
> I will open a PR so that can be considered. Thank you.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.6.0 release

[Luke Chen]

Thanks for volunteering!

+1

Luke

On Mon, Apr 17, 2023 at 2:03 AM Ismael Juma  wrote:

> Thanks for volunteering Satish. +1.
>
> Ismael
>
> On Sun, Apr 16, 2023 at 10:08 AM Satish Duggana 
> wrote:
>
> > Hi,
> > I would like to volunteer as release manager for the next release,
> > which will be Apache Kafka 3.6.0.
> >
> > If there are no objections, I will start a release plan a week after
> > 3.5.0 release(around early May).
> >
> > Thanks,
> > Satish.
> >
>

[jira] [Resolved] (KAFKA-14883) Broker state should be "observer" in KRaft quorum

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14883.
---
Fix Version/s: 3.5.0
   Resolution: Fixed

> Broker state should be "observer" in KRaft quorum
> -
>
> Key: KAFKA-14883
> URL: https://issues.apache.org/jira/browse/KAFKA-14883
> Project: Kafka
>  Issue Type: Improvement
>  Components: kraft, metrics
>Affects Versions: 3.4.0
>Reporter: Paolo Patierno
>Assignee: Paolo Patierno
>Priority: Major
> Fix For: 3.5.0
>
>
> Currently, the `current-state` KRaft related metric reports `follower` state 
> for a broker while technically it should be reported as an `observer`  as the 
> `kafka-metadata-quorum` tool does.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-9550) RemoteLogManager - copying eligible log segments to remote storage implementation

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-9550.
--
Resolution: Fixed

> RemoteLogManager - copying eligible log segments to remote storage 
> implementation 
> --
>
> Key: KAFKA-9550
> URL: https://issues.apache.org/jira/browse/KAFKA-9550
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Satish Duggana
>Assignee: Satish Duggana
>Priority: Major
> Fix For: 3.5.0
>
>
> Implementation of RLM as mentioned in the HLD section of KIP-405, this JIRA 
> covers copying segments to remote storage. 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP-405:KafkaTieredStorage-High-leveldesign]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Luke Chen]

gt; > > > > >>>> Thanks!
> > > > > >>>>
> > > > > >>>> Ismael
> > > > > >>>>
> > > > > >>>> On Tue, Feb 14, 2023 at 1:07 PM Mickael Maison <
> > > > > mickael.mai...@gmail.com>
> > > > > >>>> wrote:
> > > > > >>>>
> > > > > >>>>> Hi Ismael,
> > > > > >>>>>
> > > > > >>>>> Good call. I shifted all dates by 2 weeks and moved them to
> > > > > Wednesdays.
> > > > > >>>>>
> > > > > >>>>> Thanks,
> > > > > >>>>> Mickael
> > > > > >>>>>
> > > > > >>>>> On Tue, Feb 14, 2023 at 6:01 PM Ismael Juma <
> ism...@juma.me.uk
> > >
> > > > > wrote:
> > > > > >>>>>>
> > > > > >>>>>> Thanks Mickael. A couple of notes:
> > > > > >>>>>>
> > > > > >>>>>> 1. We typically choose a Wednesday for the various freeze
> > dates -
> > > > > there
> > > > > >>>>> are
> > > > > >>>>>> often 1-2 day slips and it's better if that doesn't require
> > people
> > > > > >>>>>> working through the weekend.
> > > > > >>>>>> 2. Looks like we're over a month later compared to the
> > equivalent
> > > > > release
> > > > > >>>>>> last year (
> > > > > >>>>>>
> > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.2.0).
> > I
> > > > > >>>>>> understand that some of it is due to 3.4.0 slipping, but I
> > wonder
> > > > > if we
> > > > > >>>>>> could perhaps aim for the KIP freeze to be one or two weeks
> > > > earlier.
> > > > > >>>>>>
> > > > > >>>>>> Ismael
> > > > > >>>>>>
> > > > > >>>>>> On Tue, Feb 14, 2023 at 8:00 AM Mickael Maison <
> > > > > mickael.mai...@gmail.com
> > > > > >>>>>>
> > > > > >>>>>> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> Hi,
> > > > > >>>>>>>
> > > > > >>>>>>> I've created a release plan for 3.5.0 in the wiki:
> > > > > >>>>>>>
> > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.5.0
> > > > > >>>>>>>
> > > > > >>>>>>> Current dates are:
> > > > > >>>>>>> 1) KIP Freeze: 07 Apr 2023
> > > > > >>>>>>> 2) Feature Freeze: 27 Apr 2023
> > > > > >>>>>>> 3) Code Freeze: 11 May 2023
> > > > > >>>>>>>
> > > > > >>>>>>> Please take a look at the plan. Let me know if there are
> > other
> > > > KIPs
> > > > > >>>>>>> targeting 3.5.0.
> > > > > >>>>>>> Also if you are the author of one of the KIPs that's
> missing
> > a
> > > > > status
> > > > > >>>>>>> (or the status is incorrect) please update it and let me
> > know.
> > > > > >>>>>>>
> > > > > >>>>>>> Thanks,
> > > > > >>>>>>> Mickael
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> On Thu, Feb 9, 2023 at 9:23 AM Bruno Cadonna <
> > cado...@apache.org
> > > > >
> > > > > >>>>> wrote:
> > > > > >>>>>>>>
> > > > > >>>>>>>> Thanks, Mickael!
> > > > > >>>>>>>>
> > > > > >>>>>>>> Best,
> > > > > >>>>>>>> Bruno
> > > > > >>>>>>>>
> > > > > >>>>>>>> On 09.02.23 03:15, Luke Chen wrote:
> > > > > >>>>>>>>> Hi Mickael,
> > > > > >>>>>>>>> Thanks for volunteering!
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> Luke
> > > > > >>>>>>>>>
> > > > > >>>>>>>>> On Thu, Feb 9, 2023 at 6:23 AM Chris Egerton <
> > > > > >>>>> fearthecel...@gmail.com>
> > > > > >>>>>>>>> wrote:
> > > > > >>>>>>>>>
> > > > > >>>>>>>>>> Thanks for volunteering, Mickael!
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>> On Wed, Feb 8, 2023 at 1:12 PM José Armando García
> Sancio
> > > > > >>>>>>>>>>  wrote:
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>>> Thanks for volunteering Mickael.
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>> --
> > > > > >>>>>>>>>>> -José
> > > > > >>>>>>>>>>>
> > > > > >>>>>>>>>>
> > > > > >>>>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>
> > > > >
> > > > >
> > > >
> >
>

[jira] [Resolved] (KAFKA-14889) RemoteLogManager - allow consumer fetch records from remote storage implementation

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14889.
---
Resolution: Duplicate

> RemoteLogManager - allow consumer fetch records from remote storage 
> implementation 
> ---
>
> Key: KAFKA-14889
> URL: https://issues.apache.org/jira/browse/KAFKA-14889
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Luke Chen
>Assignee: Satish Duggana
>Priority: Major
>
> Implementation of RLM as mentioned in the HLD section of KIP-405, this JIRA 
> covers enabling consumers fetch records from remote storage
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP-405:KafkaTieredStorage-High-leveldesign]
> h4.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14889) RemoteLogManager - allow consumer fetch records from remote storage implementation

[Luke Chen (Jira)]

Luke Chen created KAFKA-14889:
-

 Summary: RemoteLogManager - allow consumer fetch records from 
remote storage implementation 
 Key: KAFKA-14889
 URL: https://issues.apache.org/jira/browse/KAFKA-14889
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen


Implementation of RLM as mentioned in the HLD section of KIP-405, this JIRA 
covers enabling consumers fetch records from remote storage

[https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP-405:KafkaTieredStorage-High-leveldesign]
h4.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14888) RemoteLogManager - deleting expired/size breached log segments to remote storage implementation

[Luke Chen (Jira)]

Luke Chen created KAFKA-14888:
-

 Summary: RemoteLogManager - deleting expired/size breached log 
segments to remote storage implementation 
 Key: KAFKA-14888
 URL: https://issues.apache.org/jira/browse/KAFKA-14888
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen


Implementation of RLM as mentioned in the HLD section of KIP-405, this JIRA 
covers deleting time/size breached log segments in remote storage.

[https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP-405:KafkaTieredStorage-High-leveldesign]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Failing tests in Kafka stream is under investigation now

[Luke Chen]

Hi all,

Just want to let you know, our current CI test results will contain
many(more than 600 in build 1738
) failed
tests. I'm already investigating them now, and have identified some errors
and a PR  is raised running the
CI tests now. Hopefully, it can bring the healthy CI tests back soon.

Stay tuned.

Luke

[jira] [Created] (KAFKA-14877) refactor

[Luke Chen (Jira)]

Luke Chen created KAFKA-14877:
-

 Summary: refactor 
 Key: KAFKA-14877
 URL: https://issues.apache.org/jira/browse/KAFKA-14877
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen


follow up with this comment: 
https://github.com/apache/kafka/pull/13456#discussion_r1154306477



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14850) Introduce InMemoryLeaderEpochCheckpoint to allow quick write/read

[Luke Chen (Jira)]

Luke Chen created KAFKA-14850:
-

 Summary: Introduce InMemoryLeaderEpochCheckpoint to allow quick 
write/read
 Key: KAFKA-14850
 URL: https://issues.apache.org/jira/browse/KAFKA-14850
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen
Assignee: Luke Chen






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-911: Add source tag to MirrorSourceConnector metrics

[Luke Chen]

+1 from me.

Thanks for the KIP.

Luke

On Fri, Mar 17, 2023 at 10:56 PM Hector Geraldino (BLOOMBERG/ 919 3RD A) <
hgerald...@bloomberg.net> wrote:

> +1 (non-binding)
>
> From: dev@kafka.apache.org At: 03/15/23 07:08:33 UTC-4:00To:
> dev@kafka.apache.org
> Subject: [VOTE] KIP-911: Add source tag to MirrorSourceConnector metrics
>
> Hi,
>
> I'd like to start the vote on KIP-911 to add the source cluster alias
> as a tag on the MirrorSourceConnector metrics:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-911%3A+Add+source+tag+to+M
> irrorSourceConnector+metrics
>
> Thanks,
> Mickael
>
>
>

Re: [VOTE] KIP-581: Value of optional null field which has default value

[Luke Chen]

+1 (binding) from me.

Thank you.
Luke

On Fri, Mar 17, 2023 at 11:07 PM Tom Bentley  wrote:

> +1 (binding), thanks again Mickael.
>
> On Fri, 17 Mar 2023 at 14:56, Chris Egerton 
> wrote:
>
> > +1 (binding), thanks Mickael!
> >
> > On Fri, Mar 17, 2023, 10:11 Mickael Maison 
> > wrote:
> >
> > > Hi all,
> > >
> > > I'd like to start a vote on KIP-581 to allow keeping null values
> > > instead of always replacing with the default values:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
> > >
> > > Thanks,
> > > Mickael
> > >
> >
>

Re: [Discuss] KIP-581: Value of optional null field which has default value

[Luke Chen]

Thanks Mickael and Cheng!
This KIP LGTM!

Thanks.
Luke

On Fri, Mar 17, 2023 at 7:27 PM Mickael Maison 
wrote:

> Thanks Tom and Chris for your feedback!
>
> I agree one configuration is enough. I've renamed it to
> 'replace.null.with.default'.
> Since you both seemed happy with the KIP overall, I'll start a vote later
> today.
>
> Thanks,
> Mickael
>
> On Thu, Mar 16, 2023 at 7:08 PM Chris Egerton 
> wrote:
> >
> > Hi Mickael,
> >
> > Tom got this part perfect so I'm just going to copy+paste: Thanks to
> Cheng
> > for the KIP and to you for picking it up.
> >
> > I'm wondering why we need separate properties for serialization vs.
> > deserialization? If the converter is being used by the Kafka Connect
> > runtime, a given instance of it will only ever be responsible for one or
> > the other (in other words, sink connectors' converters will only ever be
> > used for deserialization and source connectors' converters will only ever
> > be used for serialization). It seems like a single "use.optional.null"
> (or
> > "map.null.to.default") property would be simpler and easier to use, but I
> > might be missing something about why we'd want to add this kind of
> > granularity.
> >
> > I'm slightly in favor of the alternative name that Tom has proposed. I
> > think highlighting that this property deals with how to handle default
> > values is important, possibly more important than the fact that it deals
> > with null field values. I'm a little hesitant to use "map" since that may
> > be harder to remember and at first glance it might seem like it deals
> with
> > the map schema type. Maybe "replace.null.with.default.value"? (For the
> > record, I don't consider any of this worthy of blocking the KIP, so don't
> > feel the need to appease me on this front before starting a vote.)
> >
> > Cheers,
> >
> > Chris
> >
> > On Thu, Mar 16, 2023 at 11:38 AM Tom Bentley 
> wrote:
> >
> > > Hi Mickael,
> > >
> > > Thanks to Cheng for the KIP and to you for picking it up.
> > >
> > > My only comment (feel free to ignore) is about the names of the
> configs.
> > > Personally I don't think I'd correctly guess what
> > > "serialize.use.optional.null" meant. Something like
> > > "serialize.map.null.to.default" is much clearer to me, for the cost
> of one
> > > extra token.
> > >
> > > Otherwise LGTM.
> > >
> > > Thanks,
> > >
> > > Tom
> > >
> > > On Wed, 8 Mar 2023 at 15:55, Mickael Maison 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > This KIP has been staled for a long time. Since it would be a useful
> > > > feature, I pinged Cheng about a month ago asking if he was planning
> to
> > > > work on it again. I've not received a reply, so I've allowed myself
> to
> > > > update the KIP (hopefully preserving the initial requirements) and
> > > > would like to restart a discussion.
> > > >
> > > > The DISCUSS thread was split in two, you can find the other part in
> > > > https://lists.apache.org/thread/dc56k17zptzvbyc7gtscovzgzwf6yn9p
> > > >
> > > > Let me know if you have any feedback.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Tue, Apr 14, 2020 at 8:28 PM Christopher Egerton <
> chr...@confluent.io
> > > >
> > > > wrote:
> > > > >
> > > > > Hi Cheng,
> > > > >
> > > > > Thanks for the KIP! I really appreciate the care that was taken to
> > > ensure
> > > > > backwards compatibility for existing users, and the minimal
> changes to
> > > > > public interface that are suggested to address this.
> > > > >
> > > > > I have two quick requests for clarification:
> > > > >
> > > > > 1) Where is the proposed "accept.optional.null" property going to
> > > apply?
> > > > > It's hinted that it'll take effect on the JSON converter but not
> > > actually
> > > > > called out anywhere.
> > > > >
> > > > > 2) Assuming this takes effect on the JSON converter, is the intent
> to
> > > > alter
> > > > > the semantics for both serialization and deserialization? The code
> > > > snippet
> > > > > from the JSON converter that's included in the KIP comes from the
> > > > > "convertToJson" method, which is used for serialization. However,
> based
> > > > on
> > > > >
> > > >
> > >
> https://github.com/apache/kafka/blob/ea47a885b1fe47dfb87c1dc86db1b0e7eb8a273c/connect/json/src/main/java/org/apache/kafka/connect/json/JsonConverter.java#L712-L713
> > > > > it
> > > > > looks like the converter also inserts the default value for
> > > > > optional-but-null data during deserialization.
> > > > >
> > > > > Thanks again for the KIP!
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Wed, Mar 18, 2020 at 12:00 AM Cheng Pan <379377...@qq.com>
> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to use this thread to discuss KIP-581: Value of optional
> > > null
> > > > > > field which has default value, please see detail at:
> > > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
> > > > > >
> > > > > >
> > > > 

Re: [DISCUSSION] MirrorMaker2 offset translation for compacted, filtered, and transactional source topics

[Luke Chen]

Hi Greg,

Thanks for your summary.
I'm not very familiar with MM2, actually, but I'd like to give my answer to
some of the questions below:

Q1. Would an improvement to allow translation from earlier in the topic be
reasonable to propose in a KIP?
--> I checked the proposed 5 solutions in KAFKA-14666, I think all of them
are trying to fix a bug, not changing any public API.
So, IMO, no KIP is needed.

Q2. Is anyone relying on the current poor correctness and high availability
translation, such that making the availability worse is a
backwards-incompatible regression?
Q3. Should we prioritize correctness, even if it hurts availability?
--> This is obviously hard to choose. But I hope we can have a way to have
both of them.

Q4. Should we address correctness and availability of this feature in a
patch or only minor releases?
--> If it's a bug fix, patch or a new release can be included.

Q5. Is there some tactical improvement to availability we can make which
does not count as backwards-incompatible, allowing us to land the
correctness fix without a KIP?
--> Again, if it's a bug fix or an improvement without affecting public
API, I don't think a KIP is needed.


Q6. Do you have any suggestions on how to improve availability of offset
translation?



Thank you.
Luke

On Wed, Mar 15, 2023 at 4:39 AM Greg Harris 
wrote:

> Hey all!
>
> I realized that the information above is a bit in-the-weeds, and I think a
> re-framing of the situation might be necessary.
>
> Since the release of MM2, offsets translation has been limited to only
> performing translation ahead of the most recent offset sync. This
> limitation appears to have worked for existing use-cases where offset syncs
> are infrequent.
> For topics which emit offset syncs frequently, the window for offset
> translation becomes shorter, and may become unusable. In those unusable
> cases, offset translation may stop completely for an otherwise
> fully-functional steady-state MM2 instance.
> Recently, we have been interested in improving the correctness of offset
> translation to address data loss, and those fixes end up causing more
> offset syncs to be emitted, making the translation window smaller than
> before, and often unusable.
>
> Q1. Would an improvement to allow translation from earlier in the topic be
> reasonable to propose in a KIP?
> Q2. Is anyone relying on the current poor correctness and high availability
> translation, such that making the availability worse is a
> backwards-incompatible regression?
> Q3. Should we prioritize correctness, even if it hurts availability?
> Q4. Should we address correctness and availability of this feature in a
> patch or only minor releases?
> Q5. Is there some tactical improvement to availability we can make which
> does not count as backwards-incompatible, allowing us to land the
> correctness fix without a KIP?
> Q6. Do you have any suggestions on how to improve availability of offset
> translation?
>
> I'm interested in finding a tactical solution that we can backport, and a
> holistic solution for more future use-cases.
> I hope that the above is more clear.
>
> Thanks!
> Greg
>
> On Fri, Mar 10, 2023 at 12:16 PM Greg Harris  wrote:
>
> > Hi all,
> >
> > Recently, we've been experimenting with using MM2 to mirror topics that
> > were populated by transactional producers. We've noticed that MM2
> > replicates records but not transaction markers, causing certain offsets
> to
> > appear in the source topic but not destination topic. These behaviors can
> > also be seen when using Filter SMTs, or when replicating topics which
> have
> > undergone compaction, which cause the same concentration of offsets in
> the
> > target topic.
> >
> > This has the following negative effects with offset translation:
> > P1. When starting replication on an existing topic with existing consumer
> > groups, offsets are translated beyond the end of the topic, leading to
> > "negative lag" for the downstream consumer group
> > P2. When in a "negative lag" situation, and a consumer fail-over from
> > source to is triggered, downstream consumption will stall until the
> > downstream offsets exceed the "negative lag" offsets.
> > P3. When failing over from source to target, certain records may have
> been
> > ahead of the upstream consumer group and behind the downstream consumer
> > group, leading to records not being delivered at least once.
> >
> > We merged a solution the above by making a change to the translation
> logic
> > in https://issues.apache.org/jira/browse/KAFKA-12468 , and settled on a
> > strategy to make offset translation more conservative, effectively making
> > it such that the MirrorCheckpointTask only emits offsets at or
> immediately
> > after the latest offset sync. This has the effect that offsets are more
> > correct than previously, but that did not come without costs:
> >
> > P4. More offset syncs must be emitted to the offset syncs topic to
> enforce
> > the `offset.lag.max` config property, once per

Re: [VOTE] KIP-906 Tools migration guidelines

[Luke Chen]

+1 from me.

Luke

On Wed, Mar 15, 2023 at 4:11 PM Federico Valeri 
wrote:

> Hi everyone,
>
> I'd like to start the vote on KIP-906 Tools migration guidelines.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines
>
> Discussion thread:
> https://lists.apache.org/thread/o2ytmjj2tyc2xcy6xh5tco31yyjzvz8p
>
> Thanks
> Fede
>

Re: Request for Permission

[Luke Chen]

Hi Jeff.

Your accounts are all set.

Thank you.
Luke

On Wed, Mar 15, 2023 at 7:12 AM Jeff Kim 
wrote:

> Hi,
>
> I would like to request permission to contribute to the Apache Kafka wiki.
> For
>
> JIRA ID: jeffkbkim
> Wiki ID: jeff.kim
>
> Thanks,
> Jeff
>

Re: [DISCUSS] Migrating documentation to Markdown/Hugo

[Luke Chen]

Hi Jorge,

Maintaining the HTML doc is pretty troublesome.
So, I'm +1 to have this migration!

My thought is that we can have old version and new(Hugo) version co-existed
in kafka-site for a period of time, and then replace the old one.

Thank you.
Luke

On Wed, Mar 8, 2023 at 5:25 AM Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:

> Hi all,
>
> I wanted to share on the mailing list a follow up from
> https://issues.apache.org/jira/browse/KAFKA-2967 to gather feedback from a
> wider audience:
>
> I put together a static-site generation with Hugo, and migrated most of the
> documentation under `docs/` to Markdown using Pandoc:
> https://github.com/jeqo/ak-docs
>
> Hopefully the readme is good enough for anyone who wants to test it. Did
> some initial integration with kafka-site to check how that would look.
>
> There are some style minor details (e.g. URLs, image sizing, kafka streams
> navigation, generate JSON instead of HTML on Config/Metrics, etc) that
> could be discussed in a following issue, but I'd like to get feedback at
> this point to see if the migration seems promising and what the next steps
> could be.
>
> Original HTMLs are kept to diff with current repo and align any diff, and
> potentially the content of the repo could go under kafka/docs directory.
>
> Looking forward to your feedback!
>
> Jorge.
>

Re: [ANNOUNCE] New Kafka PMC Member: Chris Egerton

[Luke Chen]

Congratulations, Chris!

On Fri, Mar 10, 2023 at 9:57 AM Yash Mayya  wrote:

> Congratulations Chris!
>
> On Thu, Mar 9, 2023, 23:42 Jun Rao  wrote:
>
> > Hi, Everyone,
> >
> > Chris Egerton has been a Kafka committer since July 2022. He has been
> very
> > instrumental to the community since becoming a committer. It's my
> pleasure
> > to announce that Chris is now a member of Kafka PMC.
> >
> > Congratulations Chris!
> >
> > Jun
> > on behalf of Apache Kafka PMC
> >
>

Re: [ANNOUNCE] New Kafka PMC Member: David Arthur

[Luke Chen]

Congratulations, David!

On Fri, Mar 10, 2023 at 9:56 AM Yash Mayya  wrote:

> Congrats David!
>
> On Thu, Mar 9, 2023, 23:42 Jun Rao  wrote:
>
> > Hi, Everyone,
> >
> > David Arthur has been a Kafka committer since 2013. He has been very
> > instrumental to the community since becoming a committer. It's my
> pleasure
> > to announce that David is now a member of Kafka PMC.
> >
> > Congratulations David!
> >
> > Jun
> > on behalf of Apache Kafka PMC
> >
>

Re: Requesting permissions to contribute to Apache Kafka

[Luke Chen]

Hi Jian,

You wiki account is all set.
For the JIRA account, please create an account request via here:
https://selfserve.apache.org/jira-account.html

Thank you.
Luke

On Mon, Mar 6, 2023 at 10:09 AM jian fu  wrote:

> [image: image.png]
> I think the issue related to this request. Thanks for your help to solve
> it.
>
> My IDs:
> Wiki ID:  fujian1115
> Jira ID:  fujian1115
>
> jian fu  于2023年3月6日周一 09:43写道：
>
>> Hi Team:
>> Nice to meet you. I am writing this mail to requesting permissions to
>> contribute to Apache Kafka
>>
>> Refer to:
>> "Send an email to the dev mailing list (dev@kafka.apache.org) containing
>> your wiki ID and Jira ID requesting permissions to contribute to Apache
>> Kafka."
>>
>> My IDs:
>> Wiki ID:  fujian1115
>> Jira ID:  fujian1115
>> --
>> Regards  Fu.Jian
>> --
>> Cisco Communications, Inc.
>>
>>
>
> --
> Regards  Fu.Jian
> --
> Cisco Communications, Inc.
>
>

Re: Need Contributer Access (JIRAID- ganeshsahu)

[Luke Chen]

Hi Ganesh,

Your account is all set.

Thank you.
Luke

On Fri, Mar 3, 2023 at 12:26 PM Ganesh S  wrote:

> Hello Team,i have got my JIRA ID created.
> Would like to know if I need an additional contributor access to start
> contributing.
> I would appreciate it if you could inform me if yes.
>
> My Jira ID :  ganeshsahu
>
>
> Regards
> Ganesh
>

Re: [DISCUSS] KIP-902: Upgrade Zookeeper to 3.8.1

[Luke Chen]

Hi Christo,

Thanks for the KIP.
The motivation of upgrading ZK makes sense.
And thanks for the great analysis for the ZK upgrading.

One question:
Since we are going to remove ZK in v4.0, and we don't need the latest
feature in the "current release" ZK 3.8.1, why can't we choose the "stable
release" (3.7.1)?

Thank you.
Luke

On Wed, Feb 15, 2023 at 5:47 PM Christo Lolov 
wrote:

> Hello!
>
> I would like to start a discussion for KIP-902: Upgrade Zookeeper to
> 3.8.1. The Zookeeper version currently used in Kafka reached its end of
> life in December 2022. You can find the KIP at
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240882784
>
> Thanks in advance for the reviews.
>
> Christo

[jira] [Resolved] (KAFKA-14729) The kafakConsumer pollForFetches(timer) method takes up a lot of cpu due to the abnormal exit of the heartbeat thread

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14729.
---
Fix Version/s: 3.5.0
   Resolution: Fixed

> The kafakConsumer pollForFetches(timer) method takes up a lot of cpu due to 
> the abnormal exit of the heartbeat thread
> -
>
> Key: KAFKA-14729
> URL: https://issues.apache.org/jira/browse/KAFKA-14729
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 3.3.0
>Reporter: RivenSun
>Assignee: RivenSun
>Priority: Major
> Fix For: 3.5.0
>
> Attachments: image-2023-02-17-13-15-50-362.png, jstack_highCpu.txt
>
>
> h2. case situation:
> 1. The business program occupies a large amount of memory, causing the 
> `run()` method of HeartbeatThread of kafkaConsumer to exit abnormally.
> {code:java}
> 2023-02-14 06:55:57.771[][ERROR][AbstractCoordinator][kafka-coor][Consumer 
> clientId=consumer-5, groupId=*_dev_VA] Heartbeat thread failed due to 
> unexpected error java.lang.OutOfMemoryError: Java heap space {code}
> 2. The finally module of the heartbeat thread ` run()` method only prints the 
> log, but does not update the value of `AbstractCoordinator.state`.
> 3. For kafkaConsumer with the groupRebalance mechanism enabled, in the 
> `kafkaConsumer#pollForFetches(timer)` method, pollTimeout may eventually take 
> the value `timeToNextHeartbeat(now)`.
> 4. Since the heartbeat thread has exited, `heartbeatTimer.deadlineMs` will 
> never be updated again.
> And the `AbstractCoordinator.state` field value will always be {*}STABLE{*},
> So the `timeToNextHeartbeat(long now)` method will return 
> {color:#ff}0{color}.
> 0 will be passed to the underlying `networkClient#poll` method.
>  
> In the end, the user calls the `poll(duration)` method in an endless loop, 
> and the `kafkaConsumer#pollForFetches(timer)` method will always return very 
> quickly, taking up a lot of cpu.
>  
> h2. solution:
> 1. Refer to the note of `MemberState.STABLE` :
> {code:java}
> the client has joined and is sending heartbeats.{code}
> When the heartbeat thread exits, in `finally` module, we should add code:
> {code:java}
> state = MemberState.UNJOINED;
> closed = true;{code}
> 2. In the `AbstractCoordinator#timeToNextHeartbeat(now)` method, add a new 
> judgment condition: `heartbeatThread.hasFailed()`
> {code:java}
> if (state.hasNotJoinedGroup() || heartbeatThread.hasFailed())
> return Long.MAX_VALUE;
> return heartbeat.timeToNextHeartbeat(now);{code}
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14371) quorum-state file contains empty/unused clusterId field

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14371.
---
Fix Version/s: 3.5.0
   Resolution: Fixed

> quorum-state file contains empty/unused clusterId field
> ---
>
> Key: KAFKA-14371
> URL: https://issues.apache.org/jira/browse/KAFKA-14371
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ron Dagostino
>Assignee: Gantigmaa Selenge
>Priority: Minor
> Fix For: 3.5.0
>
>
> The KRaft controller's quorum-state file 
> `$LOG_DIR/__cluster_metadata-0/quorum-state` contains an empty clusterId 
> value.  This value is never non-empty, and it is never used after it is 
> written and then subsequently read.  This is a cosmetic issue; it would be 
> best if this value did not exist there.  The cluster ID already exists in the 
> `$LOG_DIR/meta.properties` file.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14743) MessageConversionsTimeMs for fetch request metric is not updated

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14743.
---
Fix Version/s: 3.5.0
   3.4.1
   Resolution: Fixed

> MessageConversionsTimeMs for fetch request metric is not updated
> 
>
> Key: KAFKA-14743
> URL: https://issues.apache.org/jira/browse/KAFKA-14743
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 3.4.0, 3.3.1
>    Reporter: Luke Chen
>Assignee: Luke Chen
>Priority: Major
> Fix For: 3.5.0, 3.4.1
>
> Attachments: image-2023-02-23-18-09-24-916.png
>
>
> During message conversion, there are 2 metrics we should update as doc 
> written:
> !image-2023-02-23-18-09-24-916.png|width=652,height=121!
>  
> In KAFKA-14295, it is addressing the issue in FetchMessageConversionsPerSec 
> metric. This ticket will address the issue in 
> *kafka.network:type=RequestMetrics,name=MessageConversionsTimeMs,request=fetch.*
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

[Luke Chen]

+1 from me.

Thank you
Luke

On Sat, Feb 25, 2023 at 8:39 AM Chia-Ping Tsai  wrote:

> Dear all,
>
> All comments are addressed , and so please take look at this KIP. It needs
> vote and love :)
>
> thanks.

[jira] [Created] (KAFKA-14756) improve exactly-once-demo example and ExactlyOnceMessageProcessor

[Luke Chen (Jira)]

Luke Chen created KAFKA-14756:
-

 Summary: improve exactly-once-demo example and 
ExactlyOnceMessageProcessor
 Key: KAFKA-14756
 URL: https://issues.apache.org/jira/browse/KAFKA-14756
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen


while running both the examples, I saw flood of logs output because we print 
one line for message sent, and one line for message received. In 
java-producer-consumer-demo, there will be 1 records sent/received, so > 
2 lines of logs output. Same for exactly-once-demo. Maybe we should 
consider to reduce the record number.

 

Also, we should add more comments to both example files

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14755) improve java-producer-consumer-demo

[Luke Chen (Jira)]

Luke Chen created KAFKA-14755:
-

 Summary: improve java-producer-consumer-demo
 Key: KAFKA-14755
 URL: https://issues.apache.org/jira/browse/KAFKA-14755
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen


while running both the examples, I saw flood of logs output because we print 
one line for message sent, and one line for message received. In 
java-producer-consumer-demo, there will be 1 records sent/received, so > 
2 lines of logs output. Same for exactly-once-demo. Maybe we should 
consider to reduce the record number.

 

One more thing, in exactly-once-demo.java, there are clear class java doc in 
the demo file, but there's nothing in java-producer-consumer-demo.java. We 
should also improve that.

 

Lastly, the main thread in java-producer-consumer-demo won't exit if timed out 
waiting for demo producer and consumer to finish. You'll see exception thrown, 
but the application keeps waiting there.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14753) improve producer under example package

[Luke Chen (Jira)]

Luke Chen created KAFKA-14753:
-

 Summary: improve producer under example package
 Key: KAFKA-14753
 URL: https://issues.apache.org/jira/browse/KAFKA-14753
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen


I found the producer and consumer example is not in a good form. For example:
 # Both consumer and producer doesn't gracefully close the resource after 
completed
 # The example doesn't provide a good example to handle different kind of 
exceptions. It's just a happy path example
 # No clear comment to instruct users why we should do this, and what it is 
doing for each operation.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14754) improve consumer under example package

[Luke Chen (Jira)]

Luke Chen created KAFKA-14754:
-

 Summary: improve consumer under example package
 Key: KAFKA-14754
 URL: https://issues.apache.org/jira/browse/KAFKA-14754
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen


I found the producer and consumer example is not in a good form. For example:
 # Both consumer and producer doesn't gracefully close the resource after 
completed
 # The example doesn't provide a good example to handle different kind of 
exceptions. It's just a happy path example
 # No clear comment to instruct users why we should do this, and what it is 
doing for each operation.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14752) improve kafka examples under examples package

[Luke Chen (Jira)]

Luke Chen created KAFKA-14752:
-

 Summary: improve kafka examples under examples package
 Key: KAFKA-14752
 URL: https://issues.apache.org/jira/browse/KAFKA-14752
 Project: Kafka
  Issue Type: Improvement
Reporter: Luke Chen


Kafka provided some examples under "examples" package. Currently we provided
 * java-producer-consumer-demo, which is to produce 1 records and then 
consume all of them
 * exactly-once-demo, which is to produce records -> consume -> process  -> 
consume.

Among them, the base component is producer and consumer. However, I found the 
producer and consumer example is not in a good form. For example:
 # Both consumer and producer doesn't gracefully close the resource after 
completed
 # The example doesn't provide a good example to handle different kind of 
exceptions. It's just a happy path example
 # No clear comment to instruct users why we should do this, and what it is 
doing for each operation.

 

Furthermore, while running both the examples, I saw flood of logs output 
because we print one line for message sent, and one line for message received. 
In java-producer-consumer-demo, there will be 1 records sent/received, so > 
2 lines of logs output. Same for exactly-once-demo. Maybe we should 
consider to reduce the record number.

 

One more thing, in exactly-once-demo.java, there are clear class java doc in 
the demo file, but there's nothing in java-producer-consumer-demo.java. We 
should also improve that.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14743) MessageConversionsTimeMs metric is not updated

[Luke Chen (Jira)]

Luke Chen created KAFKA-14743:
-

 Summary: MessageConversionsTimeMs metric is not updated
 Key: KAFKA-14743
 URL: https://issues.apache.org/jira/browse/KAFKA-14743
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 3.3.1, 3.4.0
Reporter: Luke Chen
Assignee: Luke Chen
 Attachments: image-2023-02-23-18-09-24-916.png

During message conversion, there are 2 metrics we should update as doc written:

!image-2023-02-23-18-09-24-916.png|width=652,height=121!

 

In KAFKA-14295, it is addressing the issue in FetchMessageConversionsPerSec 
metric. This ticket will address the issue in 
*kafka.network:type=RequestMetrics,name=MessageConversionsTimeMs,request=fetch.*

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14495) Improve the RemoteIndexCacheTest

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14495.
---
Fix Version/s: 3.5.0
   Resolution: Fixed

> Improve the RemoteIndexCacheTest
> 
>
> Key: KAFKA-14495
> URL: https://issues.apache.org/jira/browse/KAFKA-14495
> Project: Kafka
>  Issue Type: Sub-task
>    Reporter: Luke Chen
>Assignee: Luke Chen
>Priority: Major
> Fix For: 3.5.0
>
>
> https://github.com/apache/kafka/pull/11390/files#r1049392445
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-900: KRaft kafka-storage.sh API additions to support SCRAM

[Luke Chen]

Hi Proven,

Thanks for the KIP.

Some questions:
1. During ZK migrating to KRaft, should we format the KRaft controller with
the SCRAM info, or it'll sync from ZK?
2. In the KIP, you provided a link to a doc from Confluent (i.e. this line: See
Configuring SCRAM

for details), could you change it into official Kafka doc link? That is
this one: https://kafka.apache.org/documentation/#security_sasl_scram

Thank you.
Luke


On Wed, Feb 22, 2023 at 6:17 AM José Armando García Sancio
 wrote:

> Hi Proven,
>
> On Tue, Feb 21, 2023 at 1:37 PM Proven Provenzano
>  wrote:
> >
> > Hi Jose,
> >
> > 1. The SCRAM in SCRAM-SHA-256 is required as the mechanism name is
> > SCRAM-SHA-256.
> > I do realize there is a bit of redundancy here.
> >
> > 2. I'll add documentation for all the possible values. They are
> > SCRAM-SHA-256 and SCRAM-SHA-512.
> >
> > 3. I'd like to keep it with a capital letter as it is a specific type of
> > message we want to insert. I do agree that -A
> > is not the correct choice so I'll suggest -S for SCRAM and that
> leaves
> > -A for --add-acl and -a for a generic
> > --add-config in the future.
> >
>
> It makes sense to me. Thanks for the changes.
>
> --
> -José
>

Re: [VOTE] KIP-894: Use incrementalAlterConfigs API for syncing topic configurations

[Luke Chen]

+1 (binding) from me.

Thank you.
Luke

On Thu, Jan 26, 2023 at 11:24 PM Federico Valeri 
wrote:

> +1 (non binding)
>
> Thanks.
>
> On Thu, Jan 26, 2023 at 2:30 PM Gantigmaa Selenge 
> wrote:
> >
> > Hi
> >
> > I'd like to call for a vote on KIP-894, which updates MirrorMaker to use
> > IncrementalAlterConfigs API to sync topic configurations between
> clusters.
> >
> > The KIP:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-894%3A+Use+incrementalAlterConfigs+API+for+syncing+topic+configurations
> >
> > The discussion thread:
> > https://lists.apache.org/thread/4chr4s5vbd9rqhml2tk60fsghojwo6bb
> >
> > Thank you
> > Tina
>

Re: Request permissions to contribute to Kafka

[Luke Chen]

Hi,

I can't find your wiki ID.
Have you created the account in the confluence system

?

Luke


On Tue, Feb 21, 2023 at 12:05 AM ze zhang  wrote:

> Hi,
> I'd like to request permissions to contribute to Kafka to propose a KIP
>
> wiki iD : bo bo
> email: zhangze1...@gmail.com
>
> Thanks
>

Re: Requesting permissions to contribute to Apache Kafka.

[Luke Chen]

Hi Jakub,

Your wiki account is all set.

Thank you.
Luke

On Fri, Feb 17, 2023 at 4:23 AM Jakub Miroś 
wrote:

> Hello Everyone :)
>
> According to this guide:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> Could you give me permissions to contribute to Apache Kafka, please?
> WikiID: spacrocketos
> JiraID: SpacRocket ( I already have permissions, adding just in case )
>
> Thanks,
> Jakub

Re: [DISCUSS] KIP-641 An new java interface to replace 'kafka.common.MessageReader'

[Luke Chen]

Hi Chia-Ping,

Thanks for the KIP!

Overall LGTM, just one minor comment:
Could we log warning messages to users when using deprecated MessageReader?

Thank you.
Luke


On Fri, Feb 17, 2023 at 2:04 PM Chia-Ping Tsai  wrote:

> hi Bentley
>
> I have updated the KIP to make RecordReader extend Configurable. PTAL
>

Re: [ANNOUNCE] New committer: Lucas Bradstreet

[Luke Chen]

Congratulations, Lucas!

Luke

On Fri, Feb 17, 2023 at 8:57 AM Guozhang Wang 
wrote:

> Congratulations, Lucas!
>
> On Thu, Feb 16, 2023 at 3:18 PM Kowshik Prakasam 
> wrote:
> >
> > Congratulations, Lucas!
> >
> >
> > Cheers,
> > Kowshik
> >
> > On Thu, Feb 16, 2023, 2:07 PM Justine Olshan
> 
> > wrote:
> >
> > > Congratulations Lucas!
> > >
> > > Thanks for your mentorship on some of my KIPs as well :)
> > >
> > > On Thu, Feb 16, 2023 at 1:56 PM Jun Rao 
> wrote:
> > >
> > > > Hi, Everyone,
> > > >
> > > > The PMC of Apache Kafka is pleased to announce a new Kafka committer
> > > Lucas
> > > > Bradstreet.
> > > >
> > > > Lucas has been a long time Kafka contributor since Oct. 2018. He has
> been
> > > > extremely valuable for Kafka on both performance and correctness
> > > > improvements.
> > > >
> > > > The following are his performance related contributions.
> > > >
> > > > KAFKA-9820: validateMessagesAndAssignOffsetsCompressed allocates
> batch
> > > > iterator which is not used
> > > > KAFKA-9685: Solve Set concatenation perf issue in AclAuthorizer
> > > > KAFKA-9729: avoid readLock in authorizer ACL lookups
> > > > KAFKA-9039: Optimize ReplicaFetcher fetch path
> > > > KAFKA-8841: Reduce overhead of
> ReplicaManager.updateFollowerFetchState
> > > >
> > > > The following are his correctness related contributions.
> > > >
> > > > KAFKA-13194: LogCleaner may clean past highwatermark
> > > > KAFKA-10432: LeaderEpochCache is incorrectly recovered on segment
> > > recovery
> > > > for epoch 0
> > > > KAFKA-9137: Fix incorrect FetchSessionCache eviction logic
> > > >
> > > > Congratulations, Lucas!
> > > >
> > > > Thanks,
> > > >
> > > > Jun (on behalf of the Apache Kafka PMC)
> > > >
> > >
>

[jira] [Resolved] (KAFKA-14621) Don't startup in migration mode if an authorizer is enabled

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14621.
---
Fix Version/s: 3.4.0
   Resolution: Done

Close this since this is completed in kafka v3.4.0

 

> Don't startup in migration mode if an authorizer is enabled
> ---
>
> Key: KAFKA-14621
> URL: https://issues.apache.org/jira/browse/KAFKA-14621
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: David Arthur
>Assignee: David Arthur
>Priority: Blocker
> Fix For: 3.4.0
>
>
> In 3.4, we do not yet support migrating ACLs from ZK to KRaft. To avoid 
> potential confusion and security problems, we should just disallow 
> authorizers during the migration.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-906 Tools migration guidelines

[Luke Chen]

Hi Federico,

1. Sounds good.
I think you should make it clear in the KIP.

We only have one tool with this issue (kafka.tools.StateChangeLogMerger),
which is tracked in KAFKA-7735
<https://issues.apache.org/jira/browse/KAFKA-7735>. This has been broken
since 2.0 release, so it means that there is not much interest and we
should simply deprecate.
It makes me think we're going to abandon this tool since no one is
interested in it.
Could we make it clear to mention the broken tool will be fixed in an
on-going PR, but not included in the scope of this KIP?

2. For the SPI argument, currently, KIP-641 is still under discussion, not
passing the votes.
Do we have any alternative if KIP-641 is rejected?

Thank you.
Luke

On Thu, Feb 16, 2023 at 4:56 PM Federico Valeri 
wrote:

> Hi Luke, thanks for the feedback.
>
> On Thu, Feb 16, 2023 at 9:33 AM Luke Chen  wrote:
> >
> > Hi Federico,
> >
> > Thanks for the KIP.
> > It's really helpful to have a general guideline for tools migration so
> that
> > contributors and reviewers can be clear what to be expected.
> >
> > One question:
> > In the broker link tool, you said we should directly deprecate it.
> > But does that mean this tool will keep broken?
>
> No, as I already did both migration and fix in the following PR:
>
> https://github.com/apache/kafka/pull/13171
>
> > Will we fix that during the module movement?
> >
>
> If the above PR is accepted, we will have a deprecated but working
> version. The general orientation is to remove it in the next major
> release, because it was broker for a long time and nobody complained.
>
> > Thank you.
> > Luke
> >
> > On Thu, Feb 16, 2023 at 4:16 PM Federico Valeri 
> > wrote:
> >
> > > Sorry for the spam. I had some issue with my mail client.
> > >
> > > On Wed, Feb 15, 2023 at 3:42 PM Federico Valeri 
> > > wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > I opened KIP-906 to have some guidelines around tools migration:
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines
> > > >
> > > > Let me know if you have any feedback or suggestions.
> > > >
> > > > Thanks
> > > > Fede
> > >
>

Re: [DISCUSS] KIP-906 Tools migration guidelines

[Luke Chen]

Hi Federico,

Thanks for the KIP.
It's really helpful to have a general guideline for tools migration so that
contributors and reviewers can be clear what to be expected.

One question:
In the broker link tool, you said we should directly deprecate it.
But does that mean this tool will keep broken?
Will we fix that during the module movement?

Thank you.
Luke

On Thu, Feb 16, 2023 at 4:16 PM Federico Valeri 
wrote:

> Sorry for the spam. I had some issue with my mail client.
>
> On Wed, Feb 15, 2023 at 3:42 PM Federico Valeri 
> wrote:
> >
> > Hi everyone,
> >
> > I opened KIP-906 to have some guidelines around tools migration:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-906%3A+Tools+migration+guidelines
> >
> > Let me know if you have any feedback or suggestions.
> >
> > Thanks
> > Fede
>

Re: Kafka design docs

[Luke Chen]

Hi Arun,

The official doc should be good enough:
https://kafka.apache.org/documentation/#design
For detailed design doc, you can check the Kafka Improvement Proposals:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals

Thank you
Luke

On Thu, Feb 16, 2023 at 7:27 AM Arun Raju  wrote:

> Hi All,
>
> Just trying to understand if there are documents relating to high-level and
> low-level design that I can use to understand overall architecture. Please
> let me know.
>
> Arun
>

[jira] [Resolved] (KAFKA-14708) Remove kafka.examples.Consumer dependancy on ShutdownableThread

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14708.
---
Fix Version/s: 3.5.0
   Resolution: Fixed

> Remove kafka.examples.Consumer dependancy on ShutdownableThread
> ---
>
> Key: KAFKA-14708
> URL: https://issues.apache.org/jira/browse/KAFKA-14708
> Project: Kafka
>  Issue Type: Task
>Reporter: Satish Duggana
>Priority: Major
> Fix For: 3.5.0
>
>
> Remove "kafka.examples.Consumer" dependency on ShutdownableThread. "examples" 
> module should be dependent only on public APIs but not to be dependent upon 
> server common/internal components. 
>  
> This is a followup item from the discussion 
> [here|https://github.com/apache/kafka/pull/13234#discussion_r1103848236]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14695) broker will get LEADER_AND_ISR is not enabled error while ZK migrating to KRaft

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14695.
---
Resolution: Duplicate

> broker will get LEADER_AND_ISR is not enabled error while ZK migrating to 
> KRaft
> ---
>
> Key: KAFKA-14695
> URL: https://issues.apache.org/jira/browse/KAFKA-14695
> Project: Kafka
>  Issue Type: Bug
>  Components: kraft
>Affects Versions: 3.4.0
>Reporter: Luke Chen
>Assignee: Luke Chen
>Priority: Major
>
> Following the docs here:
> [https://kafka.apache.org/documentation/#kraft_zk_migration]
>  
> During the step "Migrating brokers to KRaft", after migrating, the broker 
> will get a lot of error:
>  
> {code:java}
> org.apache.kafka.common.errors.InvalidRequestException: Received request api 
> key LEADER_AND_ISR which is not enabled
> [2023-02-08 12:06:25,776] ERROR Exception while processing request from 
> 192.168.1.11:9092-192.168.1.11:57210-107 (kafka.network.Processor) {code}
>  
>  
> It blocks further migration.
> This can be workaround by explicitly setting the listener host name. That is, 
> instead of setting listeners value like this (also the default value)
> _listeners=PLAINTEXT://:9092_  
> It should be set as: _listeners=PLAINTEXT://localhost:9092_  
>  
> Will update the doc first, to unblock other users trying to migrate from ZK 
> to Kraft, then investigating why this happened.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14695) broker will get LEADER_AND_ISR is not enabled error while ZK migrating to KRaft

[Luke Chen (Jira)]

Luke Chen created KAFKA-14695:
-

 Summary: broker will get LEADER_AND_ISR is not enabled error while 
ZK migrating to KRaft
 Key: KAFKA-14695
 URL: https://issues.apache.org/jira/browse/KAFKA-14695
 Project: Kafka
  Issue Type: Bug
  Components: kraft
Affects Versions: 3.4.0
Reporter: Luke Chen
Assignee: Luke Chen


Following the docs here:

[https://kafka.apache.org/documentation/#kraft_zk_migration]

 

During the step "Migrating brokers to KRaft", after migrating, the broker will 
get a lot of error:

 
{code:java}
org.apache.kafka.common.errors.InvalidRequestException: Received request api 
key LEADER_AND_ISR which is not enabled
[2023-02-08 12:06:25,776] ERROR Exception while processing request from 
192.168.1.11:9092-192.168.1.11:57210-107 (kafka.network.Processor) {code}
 

 

It blocks further migration.

This can be workaround by explicitly setting the listener host name. That is, 
instead of setting listeners value like this (also the default value)

_listeners=PLAINTEXT://:9092_  

It should be set as: _listeners=PLAINTEXT://localhost:9092_  

 

Will update the doc first, to unblock other users trying to migrate from ZK to 
Kraft, then investigating why this happened.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Luke Chen]

Hi Mickael,
Thanks for volunteering!

Luke

On Thu, Feb 9, 2023 at 6:23 AM Chris Egerton 
wrote:

> Thanks for volunteering, Mickael!
>
> On Wed, Feb 8, 2023 at 1:12 PM José Armando García Sancio
>  wrote:
>
> > Thanks for volunteering Mickael.
> >
> > --
> > -José
> >
>

Re: Request for access

[Luke Chen]

Hi David,

Your JIRA (with id: davidmariassy) and wiki account is all set.

Thank you.
Luke

On Thu, Feb 9, 2023 at 4:46 AM David Mariassy 
wrote:

> Hi,
>
> I'd like to request access to JIRA and permission to contribute to Apache
> Kafka. My wiki ID is david.mariassy. If JIRA usernames don't support dots,
> I'm happy to take davidmariassy as my ID.
>
> Thanks,
> David
>

Re: [Possible bug] Failing to use multiple client for multiple cluster using SASL channel.

[Luke Chen]

Hi Sourav,

Your JIRA account is all set.

Thank you.
Luke

On Wed, Feb 8, 2023 at 4:24 PM Sourav Biswas
 wrote:

> Hello Chen,
>
> Please use followings for jira account.
>
> Email: sourav_biswa...@yahoo.com
>
> User name: bsourav85
>
> Display Name: "Sourav Biswas" or "Sourav"
>
>
> ~ Thanks
>
> Sourav
>
>

Re: [Possible bug] Failing to use multiple client for multiple cluster using SASL channel.

[Luke Chen]

Hi Sourav,

To create a JIRA account, in addition to email address, we also need your
preferred username, display name.

Thank you.
Luke

On Wed, Feb 8, 2023 at 2:01 PM Sourav Biswas
 wrote:

> May I get a jira account
> Email id: sourav_biswa...@yahoo.com
>
>
>
>
> On Sunday, 5 February, 2023 at 01:58:33 am IST, Sourav Biswas <
> sourav_biswa...@yahoo.com> wrote:
>
>  Hello Kafka Dev,
> Issue:Say, I need to configure multiple client (consumer/producer)
> listening and publishing to different cluster inside same application (Same
> JVM). Both cluster uses
> - sasl.mechanism = GSSAPI- security.porotocol = SASL_PLAINTEXT
>
> But, different 'sasl.kerberos.service.name'.
>
> Now, considering above configuration, client will create a KafkaChannel
> using SaslChannelBuilder, which uses a LoginManager.
> https://github.com/apache/kafka/blob/4a7fedd46a7fc1eff5411a0f4329781c9474f8e8/clients/src/main/java/org/apache/kafka/common/network/SaslChannelBuilder.java#L170
> For this case, it should create multiple LoginManager for each cluster but
> it is creating only one. Because of this Authentication is failing for all
> cluster except one.
>
> Reason:
> A static Map of login managers is maintained, with key of LoginMetadata
>STATIC_INSTANCES.put(loginMetadata, loginManager);
>
> -
> https://github.com/apache/kafka/blob/4a7fedd46a7fc1eff5411a0f4329781c9474f8e8/clients/src/main/java/org/apache/kafka/common/security/authenticator/LoginManager.java#L109
>
> -
> https://github.com/apache/kafka/blob/4a7fedd46a7fc1eff5411a0f4329781c9474f8e8/clients/src/main/java/org/apache/kafka/common/security/authenticator/LoginManager.java#L113
>
> LoginMetadata only considers following fields to maintains its uniqueness.
> final T configInfo; // "KafkaClient"; Same for all cluster
> final Class loginClass; // Same for all clusester
> final Class
> loginCallbackClass; // Same for all cluster
>
>
> Possible fix:Need to consider more fields (
> sasl.kerberos.service.name/client.id/somethin-else) to maintain more
> granular uniqueness.
>
> Note:If you feel it's a bug, then I can raise a PR if I get a jira. Please
> share your thoughts.
> ~ Sourav
>
>
>

Re: [DISCUSS] KIP-894: Use incrementalAlterConfigs API for syncing topic configurations

[Luke Chen]

Hi Tina,

Thanks for the KIP to fix the issue.

Some comments:
1. In the compatibility section, you said:
`By default, the new setting will be set to false so it does not change the
current behaviour.`

I'm confused, what is the config we need to set to `false` to avoid
breaking compatibility?
All I can see is there is one new config introduced:
use.incremental.alter.configs, and default to "requested".
Does that mean it'll change current behavior?
If so, I think we should make it clear in the compatibility section about
what will be changed after this KIP.

2. It looks like you're going to introduce a new method in the existing
interface. Could you follow the pattern in other KIP (ex: KIP-888
)
to list the javadoc and the method name together?

Thank you.
Luke


On Fri, Feb 3, 2023 at 11:46 PM Chris Egerton 
wrote:

> Hi Tina,
>
> Thanks for the KIP! I recently ran into this exact issue and it's great to
> see a fix being proposed. I have a few small comments but overall this
> looks good:
>
> 1) The current logic for syncing topic configs in MM2 is basically
> fire-and-forget; all we do is log a warning message [1] if an attempt
> fails. When "use.incremental.alter.configs" is set to "requested", we'll
> need to know whether attempts using the incremental API succeed or not, and
> then adjust our behavior accordingly. Will the plan here be to block on the
> success/failure of the first request before sending any more, or will we
> just switch over to the legacy API as soon as any request fails due to
> targeting an incompatible broker, possibly after multiple requests with the
> new API have been sent or at least queued up in the admin client?
>
> 2) We don't replicate default properties from the source cluster right now
> [2].
> Won't making ConfigPropertyFilter::shouldReplicateSourceDefault return true
> by default change that behavior? If so, what's the motivation for favoring
> defaults from the source cluster over defaults for the target cluster,
> especially given that users may already be relying on the opposite?
>
> 3) Nit: IMO the parts in the "proposed changes" section that detail changes
> to internal classes aren't really necessary since they're not relevant to
> user-facing behavior and the classes/methods described in them might change
> between now and when the PR for the KIP gets opened/reviewed/merged. I
> think the only points that need to be in the KIP are the ones beginning
> with "Extend ConfigPropertyFilter class", "Add a new configuration setting
> to MirrorMaker", and "From Kafka 4.0" (which itself can just describe the
> broker APIs that are used by MM2 in general, without referring to the
> specific name of the method in MM2 that will call them).
>
> [1] -
>
> https://github.com/apache/kafka/blob/6e2b86597d9cd7c8b2019cffb895522deb63c93a/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L429
>
> [2] -
>
> https://github.com/apache/kafka/blob/6e2b86597d9cd7c8b2019cffb895522deb63c93a/connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConnector.java#L460
>
> Thanks again for the KIP!
>
> Cheers,
>
> Chris
>
> On Wed, Jan 25, 2023 at 10:44 AM Gantigmaa Selenge 
> wrote:
>
> > Hi,
> >
> > If there are no further comments on the KIP, I will start a vote on it.
> >
> > Regards,
> >
> >
> > On Mon, Jan 16, 2023 at 11:14 AM Gantigmaa Selenge 
> > wrote:
> >
> > > Thanks everyone.
> > >
> > > I took the suggestions and updated the KIP accordingly. Please let me
> > know
> > > if there is anything else I could improve on.
> > >
> > > Regards,
> > > Tina
> > >
> > > On Sun, Jan 15, 2023 at 10:24 PM Ismael Juma 
> wrote:
> > >
> > >> Hi Tina,
> > >>
> > >> See below.
> > >>
> > >> On Wed, Jan 11, 2023 at 3:03 AM Gantigmaa Selenge <
> gsele...@redhat.com>
> > >> wrote:
> > >>
> > >> > I do like this idea, however when it's set to required, I wasn't
> sure
> > >> how
> > >> > the mirrormaker should have. It's probably not a great experience if
> > >> > mirrormaker starts crashing at some point after it's already running
> > >> due to
> > >> > an incompatible broker version.
> > >> >
> > >>
> > >> This would only happen if the user explicitly requests the strict
> > required
> > >> ("non fallback") mode. There are many reasons why one may want this:
> say
> > >> you want to be sure that your system is not susceptible to the
> > >> "alterConfigs" problems or you want to write a test that fails if
> > >> "alterConfigs' is used.
> > >>
> > >>
> > >> > If the incrementalAlterConfig request fails because the target
> cluster
> > >> is
> > >> > running an older version, then we could log a WARN message that says
> > >> > something like  "The config to use incrementalAlterConfig API for
> > >> syncing
> > >> > topic configurations has been set to true however target cluster is

Re: Re: Request for JIRA account creation

[Luke Chen]

Hi Farooq,

I've registered the user name: "fqpublic" as JIRA account for you.

Thank you.
Luke

On Sun, Feb 5, 2023 at 9:06 PM Fq Public  wrote:

> Hi Luke,
>
> Sorry for the trouble. Here are some alternative usernames (in order of
> decreasing preference):
>
> fqpublic
> fqoss
> fqsnowstorm
> fqfuchsia
>
> Thanks,
> Farooq
>
> On 2023/02/05 06:29:54 Luke Chen wrote:
> > Hi Farooq,
> >
> > Your wiki ID is all set.
> > But for the JIRA account, the user name cannot allow the special
> character:
> > ".".
> > Could you provide another user name without special characters?
> >
> > Thank you.
> > Luke
> >
> > On Sun, Feb 5, 2023 at 2:02 AM Fq Public  wrote:
> >
> > > Also, I would like permission to contribute to Apache Kafka.
> > > My wiki ID is also fq.public5
> > >
> > > Thanks,
> > > Farooq
> > >
> > > On Sat, 4 Feb 2023 at 12:23, Fq Public  wrote:
> > >
> > > > Hi, I would like to request a JIRA account in order to be able to
> file a
> > > > KIP and tickets related to this PR:
> > > > https://github.com/apache/kafka/pull/10747
> > > >
> > > > Email: fq.publ...@gmail.com
> > > > User name: fq.public5
> > > > Display name: Farooq Qaiser
> > > >
> > > > Thanks,
> > > > Farooq
> > > >
> > >
> >

Re: Request for JIRA account creation

[Luke Chen]

Hi Farooq,

Your wiki ID is all set.
But for the JIRA account, the user name cannot allow the special character:
".".
Could you provide another user name without special characters?

Thank you.
Luke

On Sun, Feb 5, 2023 at 2:02 AM Fq Public  wrote:

> Also, I would like permission to contribute to Apache Kafka.
> My wiki ID is also fq.public5
>
> Thanks,
> Farooq
>
> On Sat, 4 Feb 2023 at 12:23, Fq Public  wrote:
>
> > Hi, I would like to request a JIRA account in order to be able to file a
> > KIP and tickets related to this PR:
> > https://github.com/apache/kafka/pull/10747
> >
> > Email: fq.publ...@gmail.com
> > User name: fq.public5
> > Display name: Farooq Qaiser
> >
> > Thanks,
> > Farooq
> >
>

Re: A query on log truncation.

[Luke Chen]

Hi Vinoth,

I'm wondering what's the use case or pain point you're trying to resolve?
Like you said, the client will be notified the data is not successfully
sent or propagated and handle the error, why should we keep the un-commited
records?
Could you elaborate more on the motivation?

Thank you.
Luke

On Mon, Jan 16, 2023 at 12:33 PM Vinoth  wrote:

> I was reading through about kafka , the way leader election works , log
> truncation etc. One thing that kind of struck me was how records which were
> written to log but then were not committed (It has not propagated
> successfully through to all of the isr and and the high watermark has not
> increased and so not committed ) ,were truncated following the replication
> reconciliation logic . In case they are not committed they would not be
> available for the consumer since the reads are  only upto to the high
> watermark. the producer client will also be notified or will eventually
> know if the message has not successfully propagated and it should be
> handled thru application logic. It seems straight forward in this case.
>
> KIP-405
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage
> >
> talks about tiered storage and kafka being an important part of and an
> entry point for data infrastructure . Else where i have read that kafka
> also serves as way of replaying data to restore state / viewing data.
> KIP-320
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-320%3A+Allow+fetchers+to+detect+and+handle+log+truncation
> >
> mentions users wanting higher availability opting for unclean leader
> election.
>
> Would it be fair to assume that users might be interested in a feature or
> at least  one that can be user enabled where a write to kafka (even with a
> 0 or no acks configuration or unlcean leader election ) will remain written
> until the event where clean or delete config is acted upon?.
>
> If this is a valid use case , i have thoughts of suggesting a kip around
> picking up the data that is to be truncated at time of truncation and
> replaying it as if it came through a fresh produce request. That is a
> truncation of data will not result in the data being removed from kafka but
> rather be placed differently at a different offset.
>
> Regards,
> Vinoth
>

Re: [DISCUSS] Add "Security Implications" section to KIP template

[Luke Chen]

Hi Chris,

I like this idea.
Thanks for raising this!

One question to the template bullet:
• Does it make Kafka or any of its components more difficult to run in a
fully-secured fashion?

I don't quite understand what it means. Could you elaborate on it?

Thank you.
Luke

On Wed, Jan 11, 2023 at 11:59 PM Chris Egerton 
wrote:

> Hi all,
>
> I'd like to propose augmenting the KIP template with a "Security
> Implications" section. Similar to the recently-added "test plan" section,
> the purpose here is to draw explicit attention to the security impact of
> the changes in the KIP during the design and discussion phase. On top of
> that, it should provide a common framework for how to reason about security
> so that everyone from new contributors to seasoned committers/PMC members
> can use the same standards when evaluating the security implications of a
> proposal.
>
> Here's the draft wording I've come up with so far for the template:
>
> How does this impact the security of the project?
> • Does it make Kafka or any of its components (brokers, clients, Kafka
> Connect, Kafka Streams, Mirror Maker 2, etc.) less secure when run with
> default settings?
> • Does it give users new access to configure clients, brokers, topics, etc.
> in situations where they did not have this access before? Keep in mind that
> the ability to arbitrarily configure a Kafka client can add to the attack
> surface of a project and may be safer to disable by default.
> • Does it make Kafka or any of its components more difficult to run in a
> fully-secured fashion?
>
> Let me know your thoughts. My tentative plan is to add this (with any
> modifications after discussion) to the KIP template after at least one week
> has elapsed, there has been approval from at least a couple seasoned
> contributors, and there are no unaddressed objections.
>
> Cheers,
>
> Chris
>

[jira] [Created] (KAFKA-14604) SASL session expiration time will be overflowed when calculation

[Luke Chen (Jira)]

Luke Chen created KAFKA-14604:
-

 Summary: SASL session expiration time will be overflowed when 
calculation
 Key: KAFKA-14604
 URL: https://issues.apache.org/jira/browse/KAFKA-14604
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.3.1
Reporter: Luke Chen
Assignee: Luke Chen


When sasl server of client set a large expiration time, the timeout value might 
be overflowed, and cause the session timeout immediately.

 

[Here|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslServerAuthenticator.java#L694]'s
 the sasl server timeout's calculation

[Here|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslClientAuthenticator.java#L692]'s
 the sasl client timeout's calculation

 

something like this:
{code:java}
sessionExpirationTimeNanos = authenticationEndNanos + 1000 * 1000 * 
sessionLifetimeMs; {code}
So, if the configured or returned sessionLifetimeMs is a large number, after 
the calculation, it'll be a negative value, and cause the session timeout each 
check.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New committer: Edoardo Comar

[Luke Chen]

Congratulations, Edoardo!

Luke

On Sat, Jan 7, 2023 at 7:58 AM Mickael Maison 
wrote:

> Congratulations Edo!
>
>
> On Sat, Jan 7, 2023 at 12:05 AM Jun Rao  wrote:
> >
> > Hi, Everyone,
> >
> > The PMC of Apache Kafka is pleased to announce a new Kafka committer
> Edoardo
> > Comar.
> >
> > Edoardo has been a long time Kafka contributor since 2016. His major
> > contributions are the following.
> >
> > KIP-302: Enable Kafka clients to use all DNS resolved IP addresses
> > KIP-277: Fine Grained ACL for CreateTopics API
> > KIP-136: Add Listener name to SelectorMetrics tags
> >
> > Congratulations, Edoardo!
> >
> > Thanks,
> >
> > Jun (on behalf of the Apache Kafka PMC)
>

Re: [ANNOUNCE] New committer: Justine Olshan

[Luke Chen]

Congratulations, Justine!
Well deserved!

Luke

On Fri, Dec 30, 2022 at 9:15 AM Ron Dagostino  wrote:

> Congratulations, Justine!Well-deserved., and I’m very happy for you.
>
> Ron
>
> > On Dec 29, 2022, at 6:13 PM, Israel Ekpo  wrote:
> >
> > Congratulations Justine!
> >
> >
> >> On Thu, Dec 29, 2022 at 5:05 PM Greg Harris
> 
> >> wrote:
> >>
> >> Congratulations Justine!
> >>
> >>> On Thu, Dec 29, 2022 at 1:37 PM Bill Bejeck  wrote:
> >>>
> >>> Congratulations Justine!
> >>>
> >>>
> >>> -Bill
> >>>
>  On Thu, Dec 29, 2022 at 4:36 PM Philip Nee 
> wrote:
> >>>
>  wow congrats!
> 
>  On Thu, Dec 29, 2022 at 1:05 PM Chris Egerton <
> fearthecel...@gmail.com
> >>>
>  wrote:
> 
> > Congrats, Justine!
> >
> > On Thu, Dec 29, 2022, 15:58 David Jacot  wrote:
> >
> >> Hi all,
> >>
> >> The PMC of Apache Kafka is pleased to announce a new Kafka
> >> committer
> >> Justine
> >> Olshan.
> >>
> >> Justine has been contributing to Kafka since June 2019. She
> >>> contributed
> > 53
> >> PRs including the following KIPs.
> >>
> >> KIP-480: Sticky Partitioner
> >> KIP-516: Topic Identifiers & Topic Deletion State Improvements
> >> KIP-854: Separate configuration for producer ID expiry
> >> KIP-890: Transactions Server-Side Defense (in progress)
> >>
> >> Congratulations, Justine!
> >>
> >> Thanks,
> >>
> >> David (on behalf of the Apache Kafka PMC)
> >>
> >
> 
> >>>
> >>
>

[jira] [Created] (KAFKA-14556) log cleaner throttler start time is correct

[Luke Chen (Jira)]

Luke Chen created KAFKA-14556:
-

 Summary: log cleaner throttler start time is correct
 Key: KAFKA-14556
 URL: https://issues.apache.org/jira/browse/KAFKA-14556
 Project: Kafka
  Issue Type: Bug
Reporter: Luke Chen


Log cleaner throttler will do throttling check very 300ms. The expected 
throttling work is like this:

disk IO operation for 300ms -> check if throttling is needed (sleep to slow 
down the IO) -> continue the disk IO operation -> check ...

 

But the 300ms interval start time is started when log cleaner created 
[here|https://github.com/jolshan/kafka/blob/trunk/core/src/main/scala/kafka/utils/Throttler.scala#L49].
 So, even if there's no logs needed to be cleaned, the interval time still 
elapses. When logs appeared and ready to be cleaned, the 300ms interval will 
exceed immediately and start to verify if throttling is needed, which is 
unexpected.

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14553) RecordAccumulator hangs in infinite NOP loop

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14553.
---
Fix Version/s: 3.4.0
   3.3.2
   Resolution: Duplicate

> RecordAccumulator hangs in infinite NOP loop
> 
>
> Key: KAFKA-14553
> URL: https://issues.apache.org/jira/browse/KAFKA-14553
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 3.3.1
> Environment: - Spring Boot 3.0.1
> - Spring Cloud 2022.0.0
> Versions of dependencies are defined in boms of SB and SC:
> - micrometer-tracing-bridge-brave 1.0.0
> - zipkin-reporter-brave 2.16.3
> - zipkin-sender-kafka 2.16.3
>Reporter: Viczai Gábor
>Priority: Minor
> Fix For: 3.4.0, 3.3.2
>
>
> *Summary:*
> There is an infinite loop in RecordAccumulator, if stickyBatchSize is 
> configured to be 0 in BuiltinPartitioner.
> (Which is the default case when using KafkaSender's default Builder.)
> *Details:*
> The infinite loop is caused by this while:
> https://github.com/apache/kafka/blob/3.3.1/clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java#L293
> and this continue particularly:
> https://github.com/apache/kafka/blob/3.3.1/clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java#L316
> because the partitionChanged() call in the condition always return true if 
> batchSize is 0.
> So program flow never reaches this point:
> https://github.com/apache/kafka/blob/3.3.1/clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java#L318
> Thus no span data sent to Kafka ever.
> The problematic line in partitionChanged() is when it calls an update on the 
> BuiltInPartitioner:
> https://github.com/apache/kafka/blob/3.3.1/clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java#L242
> which in fact always updates the partition because of this condition:
> https://github.com/apache/kafka/blob/3.3.1/clients/src/main/java/org/apache/kafka/clients/producer/internals/BuiltInPartitioner.java#L218
> therefore the next confdition in RecordAccumulator will evaluate to true also:
> https://github.com/apache/kafka/blob/3.3.1/clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java#L243
> thus returning 'true' and forcing the 'continue' in the while(true) loop.
> Suggested fix:
> I think these conditions should be changed:
> https://github.com/apache/kafka/blob/3.3.1/clients/src/main/java/org/apache/kafka/clients/producer/internals/BuiltInPartitioner.java#L218
> The equal signs should be removed from the conditions:
> {code}if (producedBytes > stickyBatchSize && enableSwitch || producedBytes > 
> stickyBatchSize * 2) {{code}
> (Btw: line 213 also needs this modification.)
> *Note:*
> The problem arises because KafkaSender sets the batchSize to 0.
> https://github.com/openzipkin/zipkin-reporter-java/blob/2.16.3/kafka/src/main/java/zipkin2/reporter/kafka/KafkaSender.java#L88
> *Workaround:*
> Simply set the batch size greater than zero.
> {code:java}@Configuration
> public class SenderConfiguration {
>     @Bean
>     KafkaSender kafkaSender() {
>         Properties overrides = new Properties();
>         overrides.put(ProducerConfig.BATCH_SIZE_CONFIG, 1);
>         return KafkaSender.newBuilder()
>             .bootstrapServers("localhost:9092")
>             .topic("zipkin")
>             .overrides(overrides)
>             .build();
>     }
> }{code}
> *Using:*
> - Spring Boot 3.0.1
> - Spring Cloud 2022.0.0
> pom.xml (fragment):
> {code:xml}        
>             org.springframework.boot
>             spring-boot-autoconfigure
>         
>         
>             org.springframework.boot
>             spring-boot-starter-actuator
>         
>         
>             io.micrometer
>             micrometer-registry-prometheus
>         
>         
>             io.micrometer
>             micrometer-tracing-bridge-brave
>         
>         
>             io.zipkin.reporter2
>             zipkin-reporter-brave
>         
>         
>             io.zipkin.reporter2
>             zipkin-sender-kafka
>         {code}
> Everything is on default settings, except a KafkaSender is explicitely 
> created as illustrated above. (No autoconfiguration available for Kafka 
> sender.)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New committer: Satish Duggana

[Luke Chen]

Congratulations, Satish!

On Sat, Dec 24, 2022 at 4:12 AM Federico Valeri 
wrote:

> Hi Satish, congrats!
>
> On Fri, Dec 23, 2022, 8:46 PM Viktor Somogyi-Vass
>  wrote:
>
> > Congrats Satish!
> >
> > On Fri, Dec 23, 2022, 19:38 Mickael Maison 
> > wrote:
> >
> > > Congratulations Satish!
> > >
> > > On Fri, Dec 23, 2022 at 7:36 PM Divij Vaidya 
> > > wrote:
> > > >
> > > > Congratulations Satish! 🎉
> > > >
> > > > On Fri 23. Dec 2022 at 19:32, Josep Prat  >
> > > > wrote:
> > > >
> > > > > Congrats Satish!
> > > > >
> > > > > ———
> > > > > Josep Prat
> > > > >
> > > > > Aiven Deutschland GmbH
> > > > >
> > > > > Immanuelkirchstraße 26, 10405 Berlin
> > > > > <
> > >
> >
> https://www.google.com/maps/search/Immanuelkirchstra%C3%9Fe+26,+10405+Berlin?entry=gmail&source=g
> > > >
> > > > >
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >
> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > >
> > > > > m: +491715557497
> > > > >
> > > > > w: aiven.io
> > > > >
> > > > > e: josep.p...@aiven.io
> > > > >
> > > > > On Fri, Dec 23, 2022, 19:23 Chris Egerton  >
> > > wrote:
> > > > >
> > > > > > Congrats, Satish!
> > > > > >
> > > > > > On Fri, Dec 23, 2022, 13:19 Arun Raju 
> > wrote:
> > > > > >
> > > > > > > Congratulations 👏
> > > > > > >
> > > > > > > On Fri, Dec 23, 2022, 1:08 PM Jun Rao  >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi, Everyone,
> > > > > > > >
> > > > > > > > The PMC of Apache Kafka is pleased to announce a new Kafka
> > > committer
> > > > > > > Satish
> > > > > > > > Duggana.
> > > > > > > >
> > > > > > > > Satish has been a long time Kafka contributor since 2017. He
> is
> > > the
> > > > > > main
> > > > > > > > driver behind KIP-405 that integrates Kafka with remote
> > storage,
> > > a
> > > > > > > > significant and much anticipated feature in Kafka.
> > > > > > > >
> > > > > > > > Congratulations, Satish!
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun (on behalf of the Apache Kafka PMC)
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > --
> > > > Divij Vaidya
> > >
> >
>

Re: Kafka issue

[Luke Chen]

Hi Amoli,

> The code is working fine with 100 threads and 100 partition topic.
It's good to know you have a workable solution.

For 500 threads in 500 partitions, could you firstly make sure the topic is
created successfully?
If topic cannot be created successfully, no records will be sent even if
you have 1 thread.
So, as long as the topic can be created successfully, I think it won't be a
problem with 500 threads as long as your machine is powerful enough.

Thank you.
Luke


On Wed, Dec 21, 2022 at 11:12 PM Amoli Tandon  wrote:

> Hi Team,
>
> I am working on a POC and i am facing issues while producing messages to
> Kafka cluster.
>
> I am getting 'topic doesn't exist in the metadata after 6ms' error when
> trying to produce the message using 500 threads. Topic has a partition of
> 500.
>
> The code is working fine with 100 threads and 100 partition topic.
>
> Please suggest the resolution steps.
>
> Regards,
> Amoli.
>

Re: [DISCUSS] Apache Kafka 3.4.0 release

[Luke Chen]

Hi Ismael,

Good point! This file hasn't been changed after 2020 Oct..
So I think this is not a blocker for v3.4.0.

Thank you.
Luke

On Wed, Dec 21, 2022 at 9:49 PM Ismael Juma  wrote:

> Hi Luke,
>
> Is this a recent change? If not, that gives further credence that we don't
> use this method today.
>
> Ismael
>
> On Wed, Dec 21, 2022 at 2:22 AM Luke Chen  wrote:
>
> > Hi Sophie and devs,
> >
> > KAFKA-14540 <https://issues.apache.org/jira/browse/KAFKA-14540> is
> > reported
> > (from Michael Marshall, thanks!) that we might write corrupted data to
> the
> > output stream due to the wrong buffer position set in
> > DataOutputStreamWritable#writeByteBuffer.
> > Had a search in the project, it looks like there is no place using this
> > method directly.
> > But I'd like to make sure I didn't miss anything and make things worse
> > after we release it.
> >
> > Thank you.
> > Luke
> >
> > On Wed, Dec 21, 2022 at 4:36 PM David Jacot  >
> > wrote:
> >
> > > Hi Sophie,
> > >
> > > I have merged https://issues.apache.org/jira/browse/KAFKA-14532.
> > >
> > > Best,
> > > David
> > >
> > > On Tue, Dec 20, 2022 at 4:31 PM David Arthur
> > >  wrote:
> > > >
> > > > Hey Sophie,
> > > >
> > > > I found a KRaft blocker for 3.4
> > > > https://issues.apache.org/jira/browse/KAFKA-14531. The fix is
> > committed
> > > to
> > > > trunk and is quite small. If you agree, I'll merge the fix to the 3.4
> > > > branch.
> > > >
> > > > Thanks!
> > > > David
> > > >
> > > > On Tue, Dec 20, 2022 at 7:53 AM David Jacot
> >  > > >
> > > > wrote:
> > > >
> > > > > Hi Sophie,
> > > > >
> > > > > We just found a blocker for 3.4.0:
> > > > > https://issues.apache.org/jira/browse/KAFKA-14532. The PR is on
> the
> > > > > way.
> > > > >
> > > > > Best,
> > > > > David
> > > > >
> > > > > On Sat, Dec 17, 2022 at 1:08 AM Sophie Blee-Goldman
> > > > >  wrote:
> > > > > >
> > > > > > Thanks Jose & Kirk. I agree both those fixes should be included
> in
> > > the
> > > > > 3.4
> > > > > > release
> > > > > >
> > > > > > On Fri, Dec 16, 2022 at 12:30 PM José Armando García Sancio
> > > > > >  wrote:
> > > > > >
> > > > > > > Hi Sophie,
> > > > > > >
> > > > > > > I am interested in including a bug fix for
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-14457 in the 3.4.0
> > > > > > > release. The fix is here:
> > > https://github.com/apache/kafka/pull/12994.
> > > > > > >
> > > > > > > I think it is important to include this fix because some of the
> > > > > > > controller metrics are inaccurate without this fix. This could
> > > impact
> > > > > > > some users' ability to monitor the cluster.
> > > > > > >
> > > > > > > What do you think?
> > > > > > > --
> > > > > > > -José
> > > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > -David
> > >
> >
>

Re: [DISCUSS] Apache Kafka 3.4.0 release

[Luke Chen]

Hi Sophie and devs,

KAFKA-14540  is reported
(from Michael Marshall, thanks!) that we might write corrupted data to the
output stream due to the wrong buffer position set in
DataOutputStreamWritable#writeByteBuffer.
Had a search in the project, it looks like there is no place using this
method directly.
But I'd like to make sure I didn't miss anything and make things worse
after we release it.

Thank you.
Luke

On Wed, Dec 21, 2022 at 4:36 PM David Jacot 
wrote:

> Hi Sophie,
>
> I have merged https://issues.apache.org/jira/browse/KAFKA-14532.
>
> Best,
> David
>
> On Tue, Dec 20, 2022 at 4:31 PM David Arthur
>  wrote:
> >
> > Hey Sophie,
> >
> > I found a KRaft blocker for 3.4
> > https://issues.apache.org/jira/browse/KAFKA-14531. The fix is committed
> to
> > trunk and is quite small. If you agree, I'll merge the fix to the 3.4
> > branch.
> >
> > Thanks!
> > David
> >
> > On Tue, Dec 20, 2022 at 7:53 AM David Jacot  >
> > wrote:
> >
> > > Hi Sophie,
> > >
> > > We just found a blocker for 3.4.0:
> > > https://issues.apache.org/jira/browse/KAFKA-14532. The PR is on the
> > > way.
> > >
> > > Best,
> > > David
> > >
> > > On Sat, Dec 17, 2022 at 1:08 AM Sophie Blee-Goldman
> > >  wrote:
> > > >
> > > > Thanks Jose & Kirk. I agree both those fixes should be included in
> the
> > > 3.4
> > > > release
> > > >
> > > > On Fri, Dec 16, 2022 at 12:30 PM José Armando García Sancio
> > > >  wrote:
> > > >
> > > > > Hi Sophie,
> > > > >
> > > > > I am interested in including a bug fix for
> > > > > https://issues.apache.org/jira/browse/KAFKA-14457 in the 3.4.0
> > > > > release. The fix is here:
> https://github.com/apache/kafka/pull/12994.
> > > > >
> > > > > I think it is important to include this fix because some of the
> > > > > controller metrics are inaccurate without this fix. This could
> impact
> > > > > some users' ability to monitor the cluster.
> > > > >
> > > > > What do you think?
> > > > > --
> > > > > -José
> > > > >
> > >
> >
> >
> > --
> > -David
>

Re: [ANNOUNCE] New committer: Josep Prat

[Luke Chen]

Congratulations, Josep!

Luke

On Wed, Dec 21, 2022 at 6:26 AM Viktor Somogyi-Vass
 wrote:

> Congrats Josep!
>
> On Tue, Dec 20, 2022, 21:56 Matthias J. Sax  wrote:
>
> > Congrats!
> >
> > On 12/20/22 12:01 PM, Josep Prat wrote:
> > > Thank you all!
> > >
> > > ———
> > > Josep Prat
> > >
> > > Aiven Deutschland GmbH
> > >
> > > Immanuelkirchstraße 26, 10405 Berlin
> > >
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >
> > > m: +491715557497
> > >
> > > w: aiven.io
> > >
> > > e: josep.p...@aiven.io
> > >
> > > On Tue, Dec 20, 2022, 20:42 Bill Bejeck  wrote:
> > >
> > >> Congratulations Josep!
> > >>
> > >> -Bill
> > >>
> > >> On Tue, Dec 20, 2022 at 1:11 PM Mickael Maison <
> > mickael.mai...@gmail.com>
> > >> wrote:
> > >>
> > >>> Congratulations Josep!
> > >>>
> > >>> On Tue, Dec 20, 2022 at 6:55 PM Bruno Cadonna 
> > >> wrote:
> > 
> >  Congrats, Josep!
> > 
> >  Well deserved!
> > 
> >  Best,
> >  Bruno
> > 
> >  On 20.12.22 18:40, Kirk True wrote:
> > > Congrats Josep!
> > >
> > > On Tue, Dec 20, 2022, at 9:33 AM, Jorge Esteban Quilcate Otoya
> wrote:
> > >> Congrats Josep!!
> > >>
> > >> On Tue, 20 Dec 2022, 17:31 Greg Harris,
> > >>  > 
> > >> wrote:
> > >>
> > >>> Congratulations Josep!
> > >>>
> > >>> On Tue, Dec 20, 2022 at 9:29 AM Chris Egerton <
> > >>> fearthecel...@gmail.com>
> > >>> wrote:
> > >>>
> >  Congrats Josep! Well-earned.
> > 
> >  On Tue, Dec 20, 2022, 12:26 Jun Rao 
> > >>> wrote:
> > 
> > > Hi, Everyone,
> > >
> > > The PMC of Apache Kafka is pleased to announce a new Kafka
> > >>> committer
> >  Josep
> > >Prat.
> > >
> > > Josep has been contributing to Kafka since May 2021. He
> > >>> contributed 20
> >  PRs
> > > including the following 2 KIPs.
> > >
> > > KIP-773 Differentiate metric latency measured in ms and ns
> > > KIP-744: Migrate TaskMetadata and ThreadMetadata to an
> interface
> > >>> with
> > > internal implementation
> > >
> > > Congratulations, Josep!
> > >
> > > Thanks,
> > >
> > > Jun (on behalf of the Apache Kafka PMC)
> > >
> > 
> > >>>
> > >>
> > >
> > >>>
> > >>
> > >
> >
>

[jira] [Created] (KAFKA-14529) fix failed KafkaServerKRaftRegistrationTest

[Luke Chen (Jira)]

Luke Chen created KAFKA-14529:
-

 Summary: fix failed KafkaServerKRaftRegistrationTest
 Key: KAFKA-14529
 URL: https://issues.apache.org/jira/browse/KAFKA-14529
 Project: Kafka
  Issue Type: Test
  Components: kraft
Reporter: Luke Chen
Assignee: Luke Chen


org.opentest4j.AssertionFailedError: Did not see 3 brokers within 30 seconds at 
org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38) at 
org.junit.jupiter.api.Assertions.fail(Assertions.java:134) at 
kafka.server.KafkaServerKRaftRegistrationTest.testRegisterZkBrokerInKraft1(KafkaServerKRaftRegistrationTest.scala:78)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14498) flaky org.apache.kafka.tools.MetadataQuorumCommandTest

[Luke Chen (Jira)]

Luke Chen created KAFKA-14498:
-

 Summary: flaky org.apache.kafka.tools.MetadataQuorumCommandTest
 Key: KAFKA-14498
 URL: https://issues.apache.org/jira/browse/KAFKA-14498
 Project: Kafka
  Issue Type: Test
  Components: admin
Affects Versions: 3.3.1
Reporter: Luke Chen
Assignee: Luke Chen


Build / JDK 11 and Scala 2.13 / 
org.apache.kafka.tools.MetadataQuorumCommandTest.[3] Type=Raft-CoReside, 
Name=testDescribeQuorumReplicationSuccessful, MetadataVersion=3.4-IV0, 
Security=PLAINTEXT
Failing for the past 1 build (Since 
[#33|https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-12753/33/] )
[Took 1 min 10 
sec.|https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-12753/33/testReport/junit/org.apache.kafka.tools/MetadataQuorumCommandTest/Build___JDK_11_and_Scala_2_133__Type_Raft_CoReside__Name_testDescribeQuorumReplicationSuccessful__MetadataVersion_3_4_IV0__Security_PLAINTEXT/history]
 
 
h3. Error Message

java.util.concurrent.ExecutionException: java.lang.RuntimeException: Received a 
fatal error while waiting for the broker to catch up with the current cluster 
metadata.
h3. Stacktrace

java.util.concurrent.ExecutionException: java.lang.RuntimeException: Received a 
fatal error while waiting for the broker to catch up with the current cluster 
metadata. at 
java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) at 
java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) at 
kafka.testkit.KafkaClusterTestKit.startup(KafkaClusterTestKit.java:421) at 
kafka.test.junit.RaftClusterInvocationContext.lambda$getAdditionalExtensions$5(RaftClusterInvocationContext.java:107)
 at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeTestExecutionCallbacks$5(TestMethodTestDescriptor.java:191)
 at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeMethodsOrCallbacksUntilExceptionOccurs$6(TestMethodTestDescriptor.java:202)
 at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
 at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeMethodsOrCallbacksUntilExceptionOccurs(TestMethodTestDescriptor.java:202)
 at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeTestExecutionCallbacks(TestMethodTestDescriptor.java:190)
 at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:136)
 at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
 at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
 at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
 at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
 at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) 
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
 at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
 at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
 at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
 at 
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
 at 
org.junit.platform.engine.support.hierarchical.NodeTestTask$DefaultDynamicTestExecutor.execute(NodeTestTask.java:226)
 at 
org.junit.platform.engine.support.hierarchical.NodeTestTask$DefaultDynamicTestExecutor.execute(NodeTestTask.java:204)
 at 
org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.execute(TestTemplateTestDescriptor.java:142)
 at 
org.junit.jupiter.engine.descriptor.TestTemplateTestDescriptor.lambda$execute$2(TestTemplateTestDescriptor.java:110)
 at 
java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
 at 
java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
 at 
java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
 at 
java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
 at 
java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
 at 
java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:658)
 at 
java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:274)
 at 
java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
 at 
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) 
at 
java.base

[jira] [Created] (KAFKA-14495) Improve the RemoteIndexCacheTest

[Luke Chen (Jira)]

Luke Chen created KAFKA-14495:
-

 Summary: Improve the RemoteIndexCacheTest
 Key: KAFKA-14495
 URL: https://issues.apache.org/jira/browse/KAFKA-14495
 Project: Kafka
  Issue Type: Sub-task
Reporter: Luke Chen
Assignee: Luke Chen


https://github.com/apache/kafka/pull/11390/files#r1049392445

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14285) Delete quota node in zookeeper when configs are empty

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-14285.
---
Fix Version/s: 3.4.1
   3.5.0
   Resolution: Fixed

> Delete quota node in zookeeper when configs are empty
> -
>
> Key: KAFKA-14285
> URL: https://issues.apache.org/jira/browse/KAFKA-14285
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 1.1.0, 2.7.1, 3.3.1
>Reporter: Junyang Liu
>Assignee: Junyang Liu
>Priority: Major
> Fix For: 3.4.1, 3.5.0
>
>
> When configuring flow control quota for users and clients, Kafka will add a 
> child node to /config/users or /config/clients in zookeeper. When deleting a 
> flow control quota config, the existing node in zookeeper will not be deleted 
> even if the configs in the node are empty. So if a client repeatedly 
> configure and delete flow control quota for different users and clients, the 
> node in zookeeper will increase continuously, which may leads to severe 
> problems such as memory leak.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Kafka AdminClient DEFAULT_API_TIMEOUT_MS_CONFIG property not working as expected

[Luke Chen]

Hi Shruti,

Are you sure it creates admin client successfully after setting
default.api.timeout.ms=2000 ? It should fail at this check:
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L618-L621

Try also set request.timeout.ms=2000, please.

Thank you.
Luke

On Mon, Dec 12, 2022 at 2:54 PM Shruti Pandey 
wrote:

> Hi,
>
> In my java application for kafkaProducer i created Admin and used this to
> list topics.
>
>  Admin admin1 = Admin.create(properties).
> ListTopicsResult listTopicResult = admin1.listTopics();
>
> When producer was not able to connect to broker, thread get stucks at
> listTopic step.
>
> To avoid this i have added DEFAULT_API_TIMEOUT_MS_CONFIG property and have
> set to 2000ms. so after 2s if producer is not able to connect to broker
> timeout should happen.
> properties.put(AdminClientConfig.DEFAULT_API_TIMEOUT_MS_CONFIG,2000). But
> this does not work as expected. even DEFAULT_API_TIMEOUT_MS_CONFIG with
> value 2000ms as property timeout didn't happen and thread was struck.
>
> Is there any other way to achieve timeout or do we need to configure it in
> any other way? Also is using AdminClient instead of Admin can have any
> impact.
>
> Thanks in advance
>
> Thanks
> Shruti
>
>

Re: [DISCUSS] KIP-866 ZooKeeper to KRaft Migration

[Luke Chen]

Thanks for the reply.
Yes, I think adding a section in Rejected Alternatives to explain the
rationale why we don't support combined mode upgrade in this KIP is helpful.

Thank you.
Luke

On Wed, Nov 30, 2022 at 8:47 AM Jun Rao  wrote:

> Hi, David,
>
> Thanks for the reply. No other comments from me.
>
> Jun
>
> On Tue, Nov 29, 2022 at 2:57 PM David Arthur
>  wrote:
>
> > Jun,
> >
> > 51. You're right, I missed that in the latest update. It's fixed now.
> >
> > 54. I was thinking we could update meta.properties to v1 as the brokers
> > were being restarted in KRaft mode without the migration config set.
> > However, as you mentioned, it is still possible to downgrade even then
> (as
> > long as the controller has not exited dual-write mode). Upgrading the
> > meta.properties after seeing the final ZkMigrationRecord sounds like a
> good
> > idea to me. I've updated the KIP to include this detail under
> > "Meta.Properties" section.
> >
> > 58. Yes, the metadata migration from ZK to KRaft will not migrate the
> > contents of /brokers.
> >
> > Thanks,
> > David
> >
> > On Tue, Nov 29, 2022 at 4:50 PM Jun Rao 
> wrote:
> >
> > > Hi, David,
> > >
> > > Thanks for the reply.
> > >
> > > 51. Is this reflected in the KIP? It seems that ZkMigrationState still
> > has
> > > the None value.
> > >
> > > 54. Supporting both v0 and v1 indefinitely in a KRaft broker could be a
> > bit
> > > confusing and may complicate future upgrades. Another approach is to
> let
> > > KRaft broker write the v1 meta.properties after the KRaft controller
> > exits
> > > the dual write mode. We could extend ZkMigrationRecord to 3 states like
> > > migration, dualWrite and KRaftOnly. Once a broker sees KRaftOnly, it
> will
> > > write meta.properties in v1 format. At that point, downgrade could
> cause
> > > metadata loss and require manual work. Will that work?
> > >
> > > 58. When copying metadata from ZK to KRaft, I guess we will ignore
> broker
> > > registration since the KRaft controller has already generated a
> > > BrokerRegistrationRecord based on BrokerRegistrationRequest?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Nov 29, 2022 at 7:14 AM David Arthur
> > >  wrote:
> > >
> > > > Jun, Thanks for the comments. Igor, please see 54 below for some
> > > additional
> > > > discussion on the meta.properties
> > > >
> > > > 50.1 Yes, that field name sounds fine to me.
> > > >
> > > > 50.2 Ok, I'll add something to the KIP under the Controller section.
> To
> > > > your other question, NoOpRecords are used as part of our liveness
> check
> > > for
> > > > the quorum. It doesn't produce any metadata really, so I don't think
> it
> > > > causes any harm to let it happen before the migration.  KIP-835 has
> the
> > > > details on the NoOpRecords
> > > >
> > > > 54. Colin and I discussed the meta.properties issue last night. How
> > about
> > > > we simply let the KRaft broker accept v0 or v1 meta.properties. At
> this
> > > > point, the two versions have the same contents, but different field
> > > names.
> > > > By leaving the meta.properties intact, we can simplify the downgrade
> > > > process. If we care to, we could rewrite meta.properties once a
> broker
> > is
> > > > restarted after the migration is finalized (migration config
> disabled).
> > > >
> > > > 57. If a ZK broker can't send a BrokerRegistrationRequest because the
> > > > quorum is unavailable, it should just continue operating normally.
> > Once a
> > > > leader is available, the broker will send the registration and start
> > > > heart-beating. Unlike KRaft mode, we won't block startup on a
> > successful
> > > > BrokerRegistration response. Concretely, BrokerLifecycleManager will
> > keep
> > > > trying to contact the quorum in its own thread until the
> > > > BrokerToChannelManager gets a controller ID from KafkaRaftManager.
> This
> > > > shouldn't interfere with other ZK broker activity.
> > > >
> > > > -David
> > > >
> > > > >
> > > >
> > > > --
> > > > -David
> > > >
> > >
> >
> >
> > --
> > -David
> >
>

Re: Granting permission for Create KIP and contribute to kafka

[Luke Chen]

Hi t-mac,

I've granted your JIRA account.
But I can't find the wiki id: "ws"  in the wiki system.
Are your sure you registered with "ws" here
?

Any more info you can provide to us, like account full name?

Thank you.
Luke

On Mon, Nov 28, 2022 at 12:00 AM t-mac  wrote:

> Hi, All:
>    I'd like to request permission to contribute to kafka~
>
>    Wiki Id: ws   
>    Jira Id: yws      
>    Jira Username:  ws
>    Jira Full name:  yws
>    
>
>
>
>
>
>
>
>
>
> Thanks a lot