[jira] [Resolved] (KAFKA-15139) Optimize the performance of `Set.removeAll(List)` in `MirrorCheckpointConnector`

[hudeqi (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

hudeqi resolved KAFKA-15139.

Resolution: Fixed

> Optimize the performance of `Set.removeAll(List)` in 
> `MirrorCheckpointConnector`
> 
>
> Key: KAFKA-15139
> URL: https://issues.apache.org/jira/browse/KAFKA-15139
> Project: Kafka
>  Issue Type: Improvement
>  Components: mirrormaker
>Affects Versions: 3.5.0
>Reporter: hudeqi
>Assignee: hudeqi
>Priority: Major
>
> This is the hint of `removeAll` method in `Set`:
> _This implementation determines which is the smaller of this set and the 
> specified collection, by invoking the size method on each. If this set has 
> fewer elements, then the implementation iterates over this set, checking each 
> element returned by the iterator in turn to see if it is contained in the 
> specified collection. If it is so contained, it is removed from this set with 
> the iterator's remove method. If the specified collection has fewer elements, 
> then the implementation iterates over the specified collection, removing from 
> this set each element returned by the iterator, using this set's remove 
> method._
> That's said, assume that _M_ is the number of elements in the set and _N_ is 
> the number of elements in the List, if the type of the specified collection 
> is `List`, and {_}M<=N{_}, then the time complexity of `removeAll` is _O(MN)_ 
> (because the time complexity of searching in List is {_}O(N){_}), on the 
> contrary, if {_}N {_}O(N){_}.
> In `MirrorCheckpointConnector`, `refreshConsumerGroups` method is repeatedly 
> called in a daemon thread. There are two `removeAll` in this method. From a 
> logical point of view, when this method is called in one round, when the 
> number of groups in the source cluster simply increases or decreases, the two 
> `removeAll` execution strategies will always hit the _O(MN)_ situation 
> mentioned above. Therefore, it is better to change all the variables here to 
> Set type to avoid this "low performance".



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Permissions for contributing to Kafka

[Luke Chen]

Hi Zixuan,

Your accounts are all set.

Thanks.
Luke

On Tue, Jul 11, 2023 at 10:54 AM Zixuan Bai  wrote:

> Hi master,
>
> Could you help grant me permissions to contribute to Kafka?
>
> Wiki ID:  zixuan.bai
> Jira ID: zixuan.bai
>
> Thanks a lot!
>
> Best regards,
>

Permissions for contributing to Kafka

[Zixuan Bai]

Hi master,

Could you help grant me permissions to contribute to Kafka?

Wiki ID:  zixuan.bai
Jira ID: zixuan.bai

Thanks a lot!

Best regards,

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1986

[Apache Jenkins Server]

See 

Re: wiki id

[Willma]

Appreciated, Luke

On Tue, 11 Jul 2023 at 11:59 am, Luke Chen  wrote:

> Hi Willma,
>
> Your accounts are all set.
>
> Thank you.
> Luke
>
>
>
> On Tue, Jul 11, 2023 at 9:21 AM Willma  wrote:
>
> > Hi team, I have signed up for a wiki id under the email "
> > phuctran3...@gmail.com", I have also requested for a jira id under the
> > same
> > email, if possible can the team approve it? Thanks
> >
>

Re: wiki id

[Luke Chen]

Hi Willma,

Your accounts are all set.

Thank you.
Luke



On Tue, Jul 11, 2023 at 9:21 AM Willma  wrote:

> Hi team, I have signed up for a wiki id under the email "
> phuctran3...@gmail.com", I have also requested for a jira id under the
> same
> email, if possible can the team approve it? Thanks
>

Re: [ANNOUNCE] New committer: Greg Harris

[Luke Chen]

Congrats Greg!

Luke

On Tue, Jul 11, 2023 at 8:19 AM Matthew de Detrich
 wrote:

> Congratulations, well deserved!
>
> On Mon, Jul 10, 2023 at 5:45 PM Chris Egerton  wrote:
>
> > Hi all,
> >
> > The PMC for Apache Kafka has invited Greg Harris to become a committer,
> and
> > we are happy to announce that he has accepted!
> >
> > Greg has been contributing to Kafka since 2019. He has made over 50
> commits
> > mostly around Kafka Connect and Mirror Maker 2. His most notable
> > contributions include KIP-898: "Modernize Connect plugin discovery" and a
> > deep overhaul of the offset syncing logic in MM2 that addressed several
> > technically-difficult, long-standing, high-impact issues.
> >
> > He has also been an active participant in discussions and reviews on the
> > mailing lists and on GitHub.
> >
> > Thanks for all of your contributions, Greg. Congratulations!
> >
>
>
> --
>
> Matthew de Detrich
>
> *Aiven Deutschland GmbH*
>
> Immanuelkirchstraße 26, 10405 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> *m:* +491603708037
>
> *w:* aiven.io *e:* matthew.dedetr...@aiven.io
>

wiki id

[Willma]

Hi team, I have signed up for a wiki id under the email "
phuctran3...@gmail.com", I have also requested for a jira id under the same
email, if possible can the team approve it? Thanks

Re: [VOTE] KIP-941 Range queries to accept null lower and upper bounds

[Sophie Blee-Goldman]

Thanks for the KIP!

+1 (binding)

On Mon, Jul 10, 2023 at 4:30 PM Matthias J. Sax  wrote:

> +1 (binding)
>
> On 7/10/23 12:13 PM, Bill Bejeck wrote:
> > Hi Lucia,
> >
> > Thanks for the KIP! It will be a welcomed improvement.
> >
> > +1(binding)
> >
> > -Bill
> >
> > On Mon, Jul 10, 2023 at 2:40 PM Lucia Cerchie
> 
> > wrote:
> >
> >> Hello everyone,
> >>
> >> I'd like to call a vote on KIP-941
> >> <
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-941%3A+Range+queries+to+accept+null+lower+and+upper+bounds
> >>> .
> >> It has been under discussion since June 26, and has received edits to
> the
> >> KIP and approval by discussion participants.
> >>
> >> Best,
> >> Lucia
> >>
> >> --
> >>
> >> [image: Confluent] 
> >> Lucia Cerchie
> >> Developer Advocate
> >> Follow us: [image: Blog]
> >> <
> >>
> https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog
> >>> [image:
> >> Twitter] [image: Slack]
> >> [image: YouTube]
> >> 
> >>
> >> [image: Try Confluent Cloud for Free]
> >> <
> >>
> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic
> >>>
> >>
> >
>

Re: [ANNOUNCE] New committer: Greg Harris

[Matthew de Detrich]

Congratulations, well deserved!

On Mon, Jul 10, 2023 at 5:45 PM Chris Egerton  wrote:

> Hi all,
>
> The PMC for Apache Kafka has invited Greg Harris to become a committer, and
> we are happy to announce that he has accepted!
>
> Greg has been contributing to Kafka since 2019. He has made over 50 commits
> mostly around Kafka Connect and Mirror Maker 2. His most notable
> contributions include KIP-898: "Modernize Connect plugin discovery" and a
> deep overhaul of the offset syncing logic in MM2 that addressed several
> technically-difficult, long-standing, high-impact issues.
>
> He has also been an active participant in discussions and reviews on the
> mailing lists and on GitHub.
>
> Thanks for all of your contributions, Greg. Congratulations!
>


-- 

Matthew de Detrich

*Aiven Deutschland GmbH*

Immanuelkirchstraße 26, 10405 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

*m:* +491603708037

*w:* aiven.io *e:* matthew.dedetr...@aiven.io

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1985

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-941 Range queries to accept null lower and upper bounds

[Matthias J. Sax]

+1 (binding)

On 7/10/23 12:13 PM, Bill Bejeck wrote:

Hi Lucia,

Thanks for the KIP! It will be a welcomed improvement.

+1(binding)

-Bill

On Mon, Jul 10, 2023 at 2:40 PM Lucia Cerchie 
wrote:


Hello everyone,

I'd like to call a vote on KIP-941
<
https://cwiki.apache.org/confluence/display/KAFKA/KIP-941%3A+Range+queries+to+accept+null+lower+and+upper+bounds

.

It has been under discussion since June 26, and has received edits to the
KIP and approval by discussion participants.

Best,
Lucia

--

[image: Confluent] 
Lucia Cerchie
Developer Advocate
Follow us: [image: Blog]
<
https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog

[image:

Twitter] [image: Slack]
[image: YouTube]


[image: Try Confluent Cloud for Free]
<
https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic

Re: [DISCUSS] KIP-932: Queues for Kafka

[Matthias J. Sax]

There is another detail about EOS that is important I guess.

Messages written into topic-partitions, are only marked as 
"transactional", but when we commit (or abort), we only write an 
additional "tx marker" into the partition (the original message is not 
touched). If we deliver "pending" messages, the client would need 
additional logic to buffer pending messages, plus logic to evaluate 
tx-markers to determine if/when a pending record could be processed if 
committed or discarded if aborted. The current client has nothing like 
this built-in, because we don't need to (as explained in the original 
message, why we don't read beyond the LSO).


Or we would need to have an different way to let the client know when a 
pending message is not pending any longer, and if it was committed or 
aborted. For example, we could change the client so it would always drop 
pending messages, and it would be the broker's responsibility to 
re-deliver them after they got committed. So the client won't need to 
buffer (good), however given how the broker works, this seems to be very 
undesirable to do it this way.


Maybe there are other options? In the end, it's always going to be much 
more complex, so it's not clear if it would be worth the effort or just 
do what we do know and not read beyond the LSO and keep it simple?



-Matthias

On 7/10/23 2:43 AM, Dániel Urbán wrote:

Yes, I think it's clear now, thank you.
I agree that allowing reading behind the LSO would require more work on the
broker side (we would need 1 more state for the messages, and transition
when the LSO moves forward), but I don't see the extra complexity on the
consumer side. Based on the KIP so far, brokers will be able to return
specific batches/messages to queue consumers - consumers will need to be
able to skip messages in case another consumer of the same group has
already acquired/acked those. If we have this logic present in the protocol
and the clients, consumers could skip pending messages using the same
mechanism, and only the broker would need to know *why* exactly a specific
record/batch is skipped.

I don't think that this feature would be too important, but compared to the
complexity of the KIP, 1 more state doesn't seem too complicated to me.

Thanks,
Daniel

Matthias J. Sax  ezt írta (időpont: 2023. júl. 10., H,
7:22):


Daniel, sure.

To allow the client to filter aborted messages, the broker currently
attaches metadata that tell the client which records were aborted. But
the first message after the LSO is a messages in pending state, ie, it
was neither committed nor aborted yet, so it's not possible to filter or
deliver it. Thus, the broker cannot provide this metadata (not sure if
the client could filter without this metadata?)

The main reason why this happens broker side is to avoid that the client
needs to buffer pending messages "indefinitely" until the TX might
eventually commit or abort, and thus put a lot a memory pressure on the
client. For the "classic" case, the situation is  more severe as we
guarantee ordered delivery, and thus, the client would need to buffer
everything after the LSO. -- While it's relaxed for queuing as we might
not guarantee order (ie, instead of buffering everything, only pending
messages must be buffered), it would still imply a huge additional
burden on tracking metadata (for both the broker and the consumer), and
the wire protocol, and I am already worried about the metadata we might
need to track for queuing in general.

Does this make sense?


-Matthias



On 7/7/23 01:35, Dániel Urbán wrote:

Hi Matthias,
Can you please elaborate on this: "First, you need to understand that
aborted records are filtered client side, and thus for "read-committed"

we

can never read beyond the LSO, and the same seems to apply for queuing."
I don't understand the connection here - what does skipping aborted

records

have to do with the LSO? As you said, aborted message filtering is done

on

the client side (in consumers, yes, but not sure if it has to be the same
for queues), but being blocked on the LSO is the responsibility of the
broker, isn't it? My thought was that the broker could act differently

when

working with queues and read_committed isolation.
Thanks,
Daniel

On Thu, Jul 6, 2023 at 7:26 PM Matthias J. Sax  wrote:


Thanks for the KIP.

It seems we are in very early stage, and some very important sections in
the KIP are still marked as TODO. In particular, I am curious about the
protocol changes, how the "queuing state" will be represented and made
durable, and all the error edge case / fail-over / fencing
(broker/clients) that we need to put in place.


A few other comments/question from my side:

(1) Fetch from follower: this was already touched on, but the point is
really that the consumer does not decide about it, but the broker does.
When a consumer sends it's first fetch request it will always go to the
leader, and the broker would reply to the consumer "go and fetch from
this other 

Re: Testing FixedKeyProcessor implementation using unit tests

[Matthias J. Sax]

Not sure right now, but could be a bug.

Can you maybe share the full stack trace and the test program?

-Matthias

On 7/10/23 3:47 AM, EXT.Zlatibor.Veljkovic wrote:

Hi, I am using kafka-streams-test-utils and have problem with testing 
FixedKeyProcessor [KIP-820 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-820%3A+Extend+KStream+process+with+new+Processor+API#KIP820:ExtendKStreamprocesswithnewProcessorAPI-InfrastructureforFixedKeyRecords].

Using mock processor context to get the forwarded message doesn't work.

class org.apache.kafka.streams.processor.api.MockProcessorContext cannot be 
cast to class org.apache.kafka.streams.processor.api.FixedKeyProcessorContext

Anything I can do to get forwarded records?

Thanks,
Zed

Re: [DISCUSS] KIP-759: Unneeded repartition canceling

[Shay Lin]

Hi all,

It's been a few days so I went ahead with editing the KIP, the main change
is on the method name
https://cwiki.apache.org/confluence/display/KAFKA/KIP-759%3A+Unneeded+repartition+canceling.
I will follow up with a VOTE separately.

Best,
Shay

On Thu, Jun 29, 2023 at 4:52 PM Matthias J. Sax  wrote:

> Shay,
>
> thanks for picking up this KIP. It's a pity that the discussion stalled
> for such a long time.
>
> As expressed previously, I am happy with the name `markAsPartitioned()`
> and also believe it's ok to just document the impact and leave it to the
> user to do the right thing.
>
> If we really get a lot of users that ask about it, because they did not
> do the right thing, we could still add something (eg, a reverse-mapper
> function) in a follow-up KIP. But we don't know if it's necessary; thus,
> making a small incremental step sounds like a good approach to me.
>
> Let's see if others agree or not.
>
>
> -Matthias
>
> On 6/28/23 5:29 PM, Shay Lin wrote:
> > Hi all,
> >
> > Great discussion thread. May I take this KIP up? If it’s alright my plan
> is
> > to update the KIP with the operator `markAsPartitioned()`.
> >
> > As you have discussed and pointed out, there are implications to
> downstream
> > joins or aggregation operations. Still, the operator is intended for
> > advanced users so my two cents is it would be a valuable addition
> > nonetheless. We could add this as a caution/consideration as part of the
> > java doc.
> >
> > Let me know, thanks.
> > Shay
> >
>

[jira] [Created] (KAFKA-15175) Assess the use of nio2 asynchronous channel for KafkaConsumer

[Philip Nee (Jira)]

Philip Nee created KAFKA-15175:
--

 Summary: Assess the use of nio2 asynchronous channel for 
KafkaConsumer
 Key: KAFKA-15175
 URL: https://issues.apache.org/jira/browse/KAFKA-15175
 Project: Kafka
  Issue Type: Wish
  Components: consumer
Reporter: Philip Nee


We should assess if NIO2 is appropriate to replace the current nio library with 
more performance.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: KafkaConsumer refactor proposal

[Philip Nee]

Hey Divij,

Sorry for the late reply - I was away for some extended period of time, but
I'm back to address some of the questions here.

To your point, we will incorporate your suggestions to the 1-pager, and
well as the future KIP-945 (we decided to create a kip for this work for
better publicity).

Here are some of my feedback for your suggestions, and I will respond to
each of the bullet points above:
1. For the current design, the background thread and the main thread handle
different types of tasks, and the background thread should continue making
the system calls even if the user thread is under high load.  However,
currently, we did not implement prioritization because all calls should be
sent out by the end of each event loop.  Since the background thread is
meant to be fairly lightweight by only handling the in/out-bound of the
network requests, I believe your concern should have been addressed.
2. The background thread will poll until the heartbeat or the metadata
update timer is expired or if there are any new events coming in.  In this
case, the background would not be busy waiting for the next event loop
to happen.
3. Similar to #1, the application thread works independently of the
background thread, so there shouldn't be any concern about blocking
behavior.
4. We implemented batching for a few API calls but not all.  We will
implement this optimization after having a working client!
5. The network util should check the timer of the unsent request before
sending them out.  The request will be removed if the timer expires.
6. The network connections are reused, I believe.

7. The queues aren't used for system calls.  The system calls are issued by
the "Request Manager" at the top of the event loop, and sent out by the end
of the event loop.  As long as the background thread is running, the
heartbeat and metadata updates should continue to happen.  The
clarification: we only use the queues for communication purpose: The
application event queue sends APIs calls to the background thread, and the
background event queue allows the background thread to notify the user
about errors and events like callback invocation.
8. Noted.  KAFKA-15173  was
added per your suggestion.

9. Noted: KAFKA-15173  was
added per your suggestion.
10. Noted. We will spend time examining this item.
11. Noted.
12. Noted.

13. Similar to #7: I think this is the optimization we need to investigate
after delivering the client.  KAFKA-15175 was created for this.

Thanks!
P


On Wed, May 24, 2023 at 7:20 AM Divij Vaidya 
wrote:

> Thank you Philip for the response. I will post more comments about this in
> the upcoming days but meanwhile here are my initial thoughts. Please let me
> know what you think about them.
>
> Addition to terminology
> Let's divide the API calls from the consumer into two categories. System
> API calls and User API calls.
>
> System APIs are the ones that are required for ensuring that
> consumers remain connected to the correct broker. Examples of system API
> calls would be heartbeat, metadata etc.
> User APIs are the ones which are invoked on-demand by the user. Examples of
> user API calls would be poll, commit etc.
>
> Now, may I suggest adding the following set of requirements/principles
> which the design should uphold:
> 1. System API calls should work seamlessly even when the consumer is under
> high load from user API calls.
> 2. Threads should consume minimal CPU when not doing any work. i.e. threads
> should not be performing busy waits (infinite loops without any block)
> while idle.
> 3. It should not be possible to block the background thread by the
> application/user.
> 4. Multiple occurrences of idempotent API requests should be bulked
> together. Note that this is different from batch. In bulking, we execute
> one request and use its response as the response to other duplicates /
> similar requests.
> 5. Requests which have exceeded their timeout should not be sent to the
> broker.
> 6. Design should minimize the number of connections created to the broker.
> This means that both system and user API requests should use the same
> network connection. It also means that, in this design, we are not solving
> the problem of network layer starvation of system requests due to too many
> user requests.
>
> Based on the above requirements, I would suggest the following changes:
> 7. We should consider adding another requestQueue (similar to
> ApplicationEvent queue) which will contain only the system events. The
> background thread will first dequeue from this system queue and afterwards
> go to the user event queue (ApplicationEvent queue). A daemon thread for
> each system event will regularly add events to the system queue at a fixed
> periodic interval. As a future optimization, we will piggy-back heartbeat
> with other requests such that any request from the client to the broker
> will be 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.5 #32

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1984

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-15174) Ensure the correct thread is executing the callbacks

[Philip Nee (Jira)]

Philip Nee created KAFKA-15174:
--

 Summary: Ensure the correct thread is executing the callbacks
 Key: KAFKA-15174
 URL: https://issues.apache.org/jira/browse/KAFKA-15174
 Project: Kafka
  Issue Type: Task
  Components: consumer
Reporter: Philip Nee
Assignee: Philip Nee


We need to add assertion tests to ensure the correct thread is executing the 
offset commit callbacks and rebalance callback



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15173) ApplicationEventQueue and BackgroundEventQueue should be bounded

[Philip Nee (Jira)]

Philip Nee created KAFKA-15173:
--

 Summary: ApplicationEventQueue and BackgroundEventQueue should be 
bounded
 Key: KAFKA-15173
 URL: https://issues.apache.org/jira/browse/KAFKA-15173
 Project: Kafka
  Issue Type: Task
  Components: consumer
Reporter: Philip Nee
Assignee: Philip Nee


The async consumer uses ApplicationEventQueue and BackgroundEventQueue to 
facilitate message passing between the application thread and the background 
thread.  The current implementation is boundless, which can potentially cause 
OOM and other performance-related issues.

I think the queues need a finite bound, and we need to decide how to handle the 
situation when the bound is reached.  In particular, I would like to answer 
these questions:

 
 # What should the upper limit be for both queues: Can this be a configurable, 
memory-based bound? Or just an arbitrary number of events as the bound.
 # What should happen when the application event queue is filled up?  It seems 
like we should introduce a new exception type and notify the user that the 
consumer is full.
 # What should happen when the background event queue is filled up? This seems 
less likely to happen, but I imagine it could happen when the user stops 
polling the consumer, causing the queue to be filled.
 # Is it necessary to introduce a public configuration for the queue? I think 
initially we would select an arbitrary constant number and see the community 
feedback to make a forward plan accordingly.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-941 Range queries to accept null lower and upper bounds

[Bill Bejeck]

Hi Lucia,

Thanks for the KIP! It will be a welcomed improvement.

+1(binding)

-Bill

On Mon, Jul 10, 2023 at 2:40 PM Lucia Cerchie 
wrote:

> Hello everyone,
>
> I'd like to call a vote on KIP-941
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-941%3A+Range+queries+to+accept+null+lower+and+upper+bounds
> >.
> It has been under discussion since June 26, and has received edits to the
> KIP and approval by discussion participants.
>
> Best,
> Lucia
>
> --
>
> [image: Confluent] 
> Lucia Cerchie
> Developer Advocate
> Follow us: [image: Blog]
> <
> https://www.confluent.io/blog?utm_source=footer_medium=email_campaign=ch.email-signature_type.community_content.blog
> >[image:
> Twitter] [image: Slack]
> [image: YouTube]
> 
>
> [image: Try Confluent Cloud for Free]
> <
> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound_source=gmail_medium=organic
> >
>

Re: [ANNOUNCE] New committer: Greg Harris

[Xinli shang]

Congratulations Greg!

Xinli Shang | Tech Manager at Uber Data Kafka

On Mon, Jul 10, 2023 at 9:57 AM Satish Duggana 
wrote:

> Congratulations Greg!
>
> On Mon, 10 Jul 2023 at 22:19, Justine Olshan
>  wrote:
> >
> > Congrats Greg!
> >
> > On Mon, Jul 10, 2023 at 9:40 AM Tom Bentley 
> wrote:
> >
> > > Congratulations!
> > >
> > > On Mon, 10 Jul 2023 at 17:32,  wrote:
> > >
> > > > Congrats Greg!
> > > >
> > > > -
> > > > Gaurav
> > > >
> > > > > On 10 Jul 2023, at 17:25, Hector Geraldino (BLOOMBERG/ 919 3RD A) <
> > > > hgerald...@bloomberg.net> wrote:
> > > > >
> > > > > Congrats Greg! Well deserved
> > > > >
> > > > > From: dev@kafka.apache.org At: 07/10/23 12:18:48 UTC-4:00To:
> > > > dev@kafka.apache.org
> > > > > Subject: Re: [ANNOUNCE] New committer: Greg Harris
> > > > >
> > > > > Congratulations!
> > > > >
> > > > > On Mon, Jul 10, 2023 at 9:17 AM Randall Hauch 
> > > wrote:
> > > > >>
> > > > >> Congratulations, Greg.
> > > > >>
> > > > >> On Mon, Jul 10, 2023 at 11:13 AM Mickael Maison <
> > > > mickael.mai...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Congratulations Greg!
> > > > >>>
> > > > >>> On Mon, Jul 10, 2023 at 6:08 PM Bill Bejeck
> > >  > > > >
> > > > >>> wrote:
> > > > 
> > > >  Congrats Greg!
> > > > 
> > > >  -Bill
> > > > 
> > > >  On Mon, Jul 10, 2023 at 11:53 AM Divij Vaidya <
> > > > divijvaidy...@gmail.com>
> > > >  wrote:
> > > > 
> > > > > Congratulations Greg! I am going through a new committer
> teething
> > > > >>> process
> > > > > right now and would be happy to get you up to speed. Looking
> > > forward
> > > > to
> > > > > working with you in your new role.
> > > > >
> > > > > --
> > > > > Divij Vaidya
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Jul 10, 2023 at 5:51 PM Josep Prat
> > > >  > > > 
> > > > > wrote:
> > > > >
> > > > >> Congrats Greg!
> > > > >>
> > > > >>
> > > > >> ———
> > > > >> Josep Prat
> > > > >>
> > > > >> Aiven Deutschland GmbH
> > > > >>
> > > > >> Alexanderufer 3-7, 10117 Berlin
> > > > >>
> > > > >> Amtsgericht Charlottenburg, HRB 209739 B
> > > > >>
> > > > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > >>
> > > > >> m: +491715557497
> > > > >>
> > > > >> w: aiven.io
> > > > >>
> > > > >> e: josep.p...@aiven.io
> > > > >>
> > > > >> On Mon, Jul 10, 2023, 17:47 Matthias J. Sax  >
> > > > >>> wrote:
> > > > >>
> > > > >>> Congrats!
> > > > >>>
> > > > >>> On 7/10/23 8:45 AM, Chris Egerton wrote:
> > > >  Hi all,
> > > > 
> > > >  The PMC for Apache Kafka has invited Greg Harris to become a
> > > > > committer,
> > > > >>> and
> > > >  we are happy to announce that he has accepted!
> > > > 
> > > >  Greg has been contributing to Kafka since 2019. He has made
> over
> > > > >>> 50
> > > > >>> commits
> > > >  mostly around Kafka Connect and Mirror Maker 2. His most
> notable
> > > >  contributions include KIP-898: "Modernize Connect plugin
> > > > >>> discovery"
> > > > >> and a
> > > >  deep overhaul of the offset syncing logic in MM2 that
> addressed
> > > > > several
> > > >  technically-difficult, long-standing, high-impact issues.
> > > > 
> > > >  He has also been an active participant in discussions and
> > > > >>> reviews on
> > > > >> the
> > > >  mailing lists and on GitHub.
> > > > 
> > > >  Thanks for all of your contributions, Greg. Congratulations!
> > > > 
> > > > >>>
> > > > >>
> > > > >
> > > > >>>
> > > > >
> > > > >
> > > >
> > > >
> > >
>


-- 
Xinli Shang

[VOTE] KIP-941 Range queries to accept null lower and upper bounds

[Lucia Cerchie]

Hello everyone,

I'd like to call a vote on KIP-941
.
It has been under discussion since June 26, and has received edits to the
KIP and approval by discussion participants.

Best,
Lucia

-- 

[image: Confluent] 
Lucia Cerchie
Developer Advocate
Follow us: [image: Blog]
[image:
Twitter] [image: Slack]
[image: YouTube]


[image: Try Confluent Cloud for Free]

[jira] [Created] (KAFKA-15172) Allow exact mirroring of ACLs between clusters

[Mickael Maison (Jira)]

Mickael Maison created KAFKA-15172:
--

 Summary: Allow exact mirroring of ACLs between clusters
 Key: KAFKA-15172
 URL: https://issues.apache.org/jira/browse/KAFKA-15172
 Project: Kafka
  Issue Type: Task
  Components: mirrormaker
Reporter: Mickael Maison


When mirroring ACLs, MirrorMaker downgrades allow ALL ACLs to allow READ. The 
rationale to is prevent other clients to produce to remote topics. 

However in disaster recovery scenarios, where the target cluster is not used 
and just a "hot standby", it would be preferable to have exactly the same ACLs 
on both clusters to speed up failover.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.5 #31

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 578086 lines...]
Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StoreQueryIntegrationTest > shouldQuerySpecificStalePartitionStores() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StoreQueryIntegrationTest > shouldQueryOnlyActivePartitionStoresByDefault() 
STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StoreQueryIntegrationTest > shouldQueryOnlyActivePartitionStoresByDefault() 
PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 182 > 
StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology() PASSED
streams-2: SMOKE-TEST-CLIENT-CLOSED
streams-1: SMOKE-TEST-CLIENT-CLOSED
streams-2: SMOKE-TEST-CLIENT-CLOSED
streams-1: SMOKE-TEST-CLIENT-CLOSED
streams-0: SMOKE-TEST-CLIENT-CLOSED
streams-4: SMOKE-TEST-CLIENT-CLOSED
streams-7: SMOKE-TEST-CLIENT-CLOSED
streams-5: SMOKE-TEST-CLIENT-CLOSED
streams-6: SMOKE-TEST-CLIENT-CLOSED
streams-5: SMOKE-TEST-CLIENT-CLOSED
streams-4: SMOKE-TEST-CLIENT-CLOSED
streams-3: SMOKE-TEST-CLIENT-CLOSED
streams-0: SMOKE-TEST-CLIENT-CLOSED
streams-3: SMOKE-TEST-CLIENT-CLOSED

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

See 
https://docs.gradle.org/8.0.2/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 2h 31m 52s
230 actionable tasks: 124 executed, 106 up-to-date

See the profiling report at: 
file:///home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.5/build/reports/profile/profile-2023-07-10-14-45-51.html
A fine-grained performance profile is available: use the --scan option.
[Pipeline] junit
Recording test results
[Checks API] No suitable checks publisher found.
[Pipeline] echo
Skipping Kafka Streams archetype test for Java 11
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndPersistentStores(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndInMemoryStores(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
HighAvailabilityTaskAssignorIntegrationTest > 
shouldScaleOutWithWarmupTasksAndInMemoryStores(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
KStreamAggregationDedupIntegrationTest > shouldReduce(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
KStreamAggregationDedupIntegrationTest > shouldReduce(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
KStreamAggregationDedupIntegrationTest > shouldGroupByKey(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
KStreamAggregationDedupIntegrationTest > shouldGroupByKey(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
KStreamAggregationDedupIntegrationTest > shouldReduceWindowed(TestInfo) STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
KStreamAggregationDedupIntegrationTest > shouldReduceWindowed(TestInfo) PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
KStreamKStreamIntegrationTest > shouldOuterJoin() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
KStreamKStreamIntegrationTest > shouldOuterJoin() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicNotWrittenToDuringRestoration() STARTED

Gradle Test Run :streams:integrationTest > Gradle Test Executor 177 > 
KTableSourceTopicRestartIntegrationTest > 
shouldRestoreAndProgressWhenTopicNotWrittenToDuringRestoration() PASSED

Gradle Test Run :streams:integrationTest > Gradle Test 

RE: [DISCUSS] KIP-942: Add Power(ppc64le) support

[Vaibhav Nazare]

Hi Colin and Mickael,

Yes I agree we don't have to run on every build , may be as mentioned we can 
run the job once/twice a day at specific time.
According the test plan the tests which I mentioned are not specific to ppc64le 
, I was running tests on the trunk branch which were common across platforms on 
that specific commit and I have removed the table and mentioned the test plan 
accordingly when to run the jobs.



-Original Message-
From: Mickael Maison  
Sent: Tuesday, July 4, 2023 3:12 PM
To: dev@kafka.apache.org
Subject: [EXTERNAL] Re: [DISCUSS] KIP-942: Add Power(ppc64le) support

Hi,

Thanks for the KIP!
In the Test Plan section you mentioned a few unit and integration tests are 
failing. Are these flaky tests or did you find issues related to ppc64le? If 
they are just flaky tests, I think we can remove that table and instead 
describe how often we intend to run the tests in the CI.

Thanks,
Mickael

On Tue, Jul 4, 2023 at 12:04 AM Colin McCabe  wrote:
>
> I agree with Divij. A nightly Apache Kafka build for PowerPC would be 
> welcome. But it shouldn't run on every build, since the extra time and 
> complexity would not be worth it.
>
> By the way, are there any features or plugins we don't intend to support on 
> PPC? If there are, this KIP would be a good place to spell them out.
>
> Naively, I would think all of our Java and Scala code should work on 
> PPC without changes. However, there may be library dependencies that 
> don't exist on PPC. (We have to remember that the last desktop PowerPC 
> chip that an average user could buy shipped in 2005)
>
> best,
> Colin
>
>
> On Mon, Jun 19, 2023, at 23:12, Vaibhav Nazare wrote:
> > Thank you for response Divij.
> >
> > 1. We are going to use ASF infra provided nodes for better 
> > availability and stability as there are 3 power9 nodes managed 
> > officially by ASF infra team themselves.
> > Ref: 
> > INVALID URI REMOVED
> > rg_jira_browse_INFRA-2D24663=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=s9I
> > 3h_d72lHAurpHrTUoOkX8ByFHVUGD0XU1PTKfCiw=wok2fkRzJGACiUg-o6hyaJP11
> > WLbfqYbH78J2VuRdvseBKrVLuXW9PtUJqs9BWO4=EeWMMaaUqDLlqUjJmw3_ouI9lt
> > 4B_EUGyH8OZ_VTIRE= 
> > INVALID URI REMOVED.
> > apache.org_view_Shared-2520-2D-2520ppc64le-2520nodes_=DwIFaQ=jf_
> > iaSHvJObTbx-siA1ZOg=s9I3h_d72lHAurpHrTUoOkX8ByFHVUGD0XU1PTKfCiw=
> > wok2fkRzJGACiUg-o6hyaJP11WLbfqYbH78J2VuRdvseBKrVLuXW9PtUJqs9BWO4=O
> > R-kg7uZdehAVxU7TdCUruDpGb0yOAEa0wW2dC3P3yU=
> > previously used power node details for apache/kafka CI:
> > RAM- 16GB
> > VCPUs- 8 VCPU
> > Disk- 160GB
> > for shared VMs we need to check with ASF infra team to provide 
> > details
> >
> > 2. We can run nightly builds once or twice in a day on specific 
> > period of time instead of every commit 3. apache/camel 
> > INVALID URI REMOVED
> > rg_job_Camel_job_el_=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=s9I3h_d72lHAurpHrTUoOkX8ByFHVUGD0XU1PTKfCiw=wok2fkRzJGACiUg-o6hyaJP11WLbfqYbH78J2VuRdvseBKrVLuXW9PtUJqs9BWO4=J35xYPMn5tCJRn91aARKgyiwas_cu4RGeSsh50tV4GI=
> >   has already enabled CI for power platform they are using same H/W 
> > resources as
> > RAM- 16GB
> > VCPUs- 8 VCPU
> > Disk- 160GB
> >
> > -Original Message-
> > From: Divij Vaidya 
> > Sent: Monday, June 19, 2023 10:20 PM
> > To: dev@kafka.apache.org
> > Subject: [EXTERNAL] Re: [DISCUSS] KIP-942: Add Power(ppc64le) 
> > support
> >
> > Thank you for the KIP Vaibhav.
> >
> > 1. Builds for power architecture were intentionally disabled in the 
> > past since the infrastructure was flaky [1]. Could you please add to 
> > the KIP on what has changed since then?
> > 2. What do you think about an alternative solution where we run a 
> > nightly build for this platform instead of running the CI with every 
> > PR/commit?
> > 3. To bolster the case for this KIP, could you please add 
> > information from other Apache projects who are already running CI 
> > for this platform? Is their CI stable on Apache Infra hosts?
> >
> >
> > [1] 
> > INVALID URI REMOVED
> > he_kafka_pull_12380=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=s9I3h_d72lHA
> > urpHrTUoOkX8ByFHVUGD0XU1PTKfCiw=wok2fkRzJGACiUg-o6hyaJP11WLbfqYbH7
> > 8J2VuRdvseBKrVLuXW9PtUJqs9BWO4=Wo7q9L9UOHo8AkRRwIFG9570Y4isqaxmhIY
> > W0wwVspk=
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Mon, Jun 19, 2023 at 12:30 PM Vaibhav Nazare 
> >  wrote:
> >
> >>
> >> INVALID URI REMOVED
> >> confluence_display_KAFKA_KIP-2D942-253A-2BAdd-2BPower-2528ppc64le-2
> >> 529 
> >> -2Bsupport=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=s9I3h_d72lHAurpHrTUo
> >> OkX 
> >> 8ByFHVUGD0XU1PTKfCiw=z6ZZ_vt5XP--aKB5lpRRZxdVMA37hD_0ch7COCLdMtLh
> >> Mve 
> >> 8AJcbKfwRtBac267r=BQtj2lbWlu32mK0TP37XeZanal33QOf5HB1-33QJIqc=
> >>

[jira] [Resolved] (KAFKA-14059) Replace EasyMock and PowerMock with Mockito in WorkerSourceTaskTest

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-14059.
---
Resolution: Done

> Replace EasyMock and PowerMock with Mockito in WorkerSourceTaskTest
> ---
>
> Key: KAFKA-14059
> URL: https://issues.apache.org/jira/browse/KAFKA-14059
> Project: Kafka
>  Issue Type: Sub-task
>  Components: KafkaConnect
>Reporter: Chris Egerton
>Assignee: Hector Geraldino
>Priority: Minor
> Fix For: 3.6.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New committer: Greg Harris

[Satish Duggana]

Congratulations Greg!

On Mon, 10 Jul 2023 at 22:19, Justine Olshan
 wrote:
>
> Congrats Greg!
>
> On Mon, Jul 10, 2023 at 9:40 AM Tom Bentley  wrote:
>
> > Congratulations!
> >
> > On Mon, 10 Jul 2023 at 17:32,  wrote:
> >
> > > Congrats Greg!
> > >
> > > -
> > > Gaurav
> > >
> > > > On 10 Jul 2023, at 17:25, Hector Geraldino (BLOOMBERG/ 919 3RD A) <
> > > hgerald...@bloomberg.net> wrote:
> > > >
> > > > Congrats Greg! Well deserved
> > > >
> > > > From: dev@kafka.apache.org At: 07/10/23 12:18:48 UTC-4:00To:
> > > dev@kafka.apache.org
> > > > Subject: Re: [ANNOUNCE] New committer: Greg Harris
> > > >
> > > > Congratulations!
> > > >
> > > > On Mon, Jul 10, 2023 at 9:17 AM Randall Hauch 
> > wrote:
> > > >>
> > > >> Congratulations, Greg.
> > > >>
> > > >> On Mon, Jul 10, 2023 at 11:13 AM Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Congratulations Greg!
> > > >>>
> > > >>> On Mon, Jul 10, 2023 at 6:08 PM Bill Bejeck
> >  > > >
> > > >>> wrote:
> > > 
> > >  Congrats Greg!
> > > 
> > >  -Bill
> > > 
> > >  On Mon, Jul 10, 2023 at 11:53 AM Divij Vaidya <
> > > divijvaidy...@gmail.com>
> > >  wrote:
> > > 
> > > > Congratulations Greg! I am going through a new committer teething
> > > >>> process
> > > > right now and would be happy to get you up to speed. Looking
> > forward
> > > to
> > > > working with you in your new role.
> > > >
> > > > --
> > > > Divij Vaidya
> > > >
> > > >
> > > >
> > > > On Mon, Jul 10, 2023 at 5:51 PM Josep Prat
> > >  > > 
> > > > wrote:
> > > >
> > > >> Congrats Greg!
> > > >>
> > > >>
> > > >> ———
> > > >> Josep Prat
> > > >>
> > > >> Aiven Deutschland GmbH
> > > >>
> > > >> Alexanderufer 3-7, 10117 Berlin
> > > >>
> > > >> Amtsgericht Charlottenburg, HRB 209739 B
> > > >>
> > > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >>
> > > >> m: +491715557497
> > > >>
> > > >> w: aiven.io
> > > >>
> > > >> e: josep.p...@aiven.io
> > > >>
> > > >> On Mon, Jul 10, 2023, 17:47 Matthias J. Sax 
> > > >>> wrote:
> > > >>
> > > >>> Congrats!
> > > >>>
> > > >>> On 7/10/23 8:45 AM, Chris Egerton wrote:
> > >  Hi all,
> > > 
> > >  The PMC for Apache Kafka has invited Greg Harris to become a
> > > > committer,
> > > >>> and
> > >  we are happy to announce that he has accepted!
> > > 
> > >  Greg has been contributing to Kafka since 2019. He has made over
> > > >>> 50
> > > >>> commits
> > >  mostly around Kafka Connect and Mirror Maker 2. His most notable
> > >  contributions include KIP-898: "Modernize Connect plugin
> > > >>> discovery"
> > > >> and a
> > >  deep overhaul of the offset syncing logic in MM2 that addressed
> > > > several
> > >  technically-difficult, long-standing, high-impact issues.
> > > 
> > >  He has also been an active participant in discussions and
> > > >>> reviews on
> > > >> the
> > >  mailing lists and on GitHub.
> > > 
> > >  Thanks for all of your contributions, Greg. Congratulations!
> > > 
> > > >>>
> > > >>
> > > >
> > > >>>
> > > >
> > > >
> > >
> > >
> >

Re: [ANNOUNCE] New committer: Greg Harris

[Justine Olshan]

Congrats Greg!

On Mon, Jul 10, 2023 at 9:40 AM Tom Bentley  wrote:

> Congratulations!
>
> On Mon, 10 Jul 2023 at 17:32,  wrote:
>
> > Congrats Greg!
> >
> > -
> > Gaurav
> >
> > > On 10 Jul 2023, at 17:25, Hector Geraldino (BLOOMBERG/ 919 3RD A) <
> > hgerald...@bloomberg.net> wrote:
> > >
> > > Congrats Greg! Well deserved
> > >
> > > From: dev@kafka.apache.org At: 07/10/23 12:18:48 UTC-4:00To:
> > dev@kafka.apache.org
> > > Subject: Re: [ANNOUNCE] New committer: Greg Harris
> > >
> > > Congratulations!
> > >
> > > On Mon, Jul 10, 2023 at 9:17 AM Randall Hauch 
> wrote:
> > >>
> > >> Congratulations, Greg.
> > >>
> > >> On Mon, Jul 10, 2023 at 11:13 AM Mickael Maison <
> > mickael.mai...@gmail.com>
> > >> wrote:
> > >>
> > >>> Congratulations Greg!
> > >>>
> > >>> On Mon, Jul 10, 2023 at 6:08 PM Bill Bejeck
>  > >
> > >>> wrote:
> > 
> >  Congrats Greg!
> > 
> >  -Bill
> > 
> >  On Mon, Jul 10, 2023 at 11:53 AM Divij Vaidya <
> > divijvaidy...@gmail.com>
> >  wrote:
> > 
> > > Congratulations Greg! I am going through a new committer teething
> > >>> process
> > > right now and would be happy to get you up to speed. Looking
> forward
> > to
> > > working with you in your new role.
> > >
> > > --
> > > Divij Vaidya
> > >
> > >
> > >
> > > On Mon, Jul 10, 2023 at 5:51 PM Josep Prat
> >  > 
> > > wrote:
> > >
> > >> Congrats Greg!
> > >>
> > >>
> > >> ———
> > >> Josep Prat
> > >>
> > >> Aiven Deutschland GmbH
> > >>
> > >> Alexanderufer 3-7, 10117 Berlin
> > >>
> > >> Amtsgericht Charlottenburg, HRB 209739 B
> > >>
> > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >>
> > >> m: +491715557497
> > >>
> > >> w: aiven.io
> > >>
> > >> e: josep.p...@aiven.io
> > >>
> > >> On Mon, Jul 10, 2023, 17:47 Matthias J. Sax 
> > >>> wrote:
> > >>
> > >>> Congrats!
> > >>>
> > >>> On 7/10/23 8:45 AM, Chris Egerton wrote:
> >  Hi all,
> > 
> >  The PMC for Apache Kafka has invited Greg Harris to become a
> > > committer,
> > >>> and
> >  we are happy to announce that he has accepted!
> > 
> >  Greg has been contributing to Kafka since 2019. He has made over
> > >>> 50
> > >>> commits
> >  mostly around Kafka Connect and Mirror Maker 2. His most notable
> >  contributions include KIP-898: "Modernize Connect plugin
> > >>> discovery"
> > >> and a
> >  deep overhaul of the offset syncing logic in MM2 that addressed
> > > several
> >  technically-difficult, long-standing, high-impact issues.
> > 
> >  He has also been an active participant in discussions and
> > >>> reviews on
> > >> the
> >  mailing lists and on GitHub.
> > 
> >  Thanks for all of your contributions, Greg. Congratulations!
> > 
> > >>>
> > >>
> > >
> > >>>
> > >
> > >
> >
> >
>

Re: [ANNOUNCE] New committer: Greg Harris

[Tom Bentley]

Congratulations!

On Mon, 10 Jul 2023 at 17:32,  wrote:

> Congrats Greg!
>
> -
> Gaurav
>
> > On 10 Jul 2023, at 17:25, Hector Geraldino (BLOOMBERG/ 919 3RD A) <
> hgerald...@bloomberg.net> wrote:
> >
> > Congrats Greg! Well deserved
> >
> > From: dev@kafka.apache.org At: 07/10/23 12:18:48 UTC-4:00To:
> dev@kafka.apache.org
> > Subject: Re: [ANNOUNCE] New committer: Greg Harris
> >
> > Congratulations!
> >
> > On Mon, Jul 10, 2023 at 9:17 AM Randall Hauch  wrote:
> >>
> >> Congratulations, Greg.
> >>
> >> On Mon, Jul 10, 2023 at 11:13 AM Mickael Maison <
> mickael.mai...@gmail.com>
> >> wrote:
> >>
> >>> Congratulations Greg!
> >>>
> >>> On Mon, Jul 10, 2023 at 6:08 PM Bill Bejeck  >
> >>> wrote:
> 
>  Congrats Greg!
> 
>  -Bill
> 
>  On Mon, Jul 10, 2023 at 11:53 AM Divij Vaidya <
> divijvaidy...@gmail.com>
>  wrote:
> 
> > Congratulations Greg! I am going through a new committer teething
> >>> process
> > right now and would be happy to get you up to speed. Looking forward
> to
> > working with you in your new role.
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Mon, Jul 10, 2023 at 5:51 PM Josep Prat
>  
> > wrote:
> >
> >> Congrats Greg!
> >>
> >>
> >> ———
> >> Josep Prat
> >>
> >> Aiven Deutschland GmbH
> >>
> >> Alexanderufer 3-7, 10117 Berlin
> >>
> >> Amtsgericht Charlottenburg, HRB 209739 B
> >>
> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >>
> >> m: +491715557497
> >>
> >> w: aiven.io
> >>
> >> e: josep.p...@aiven.io
> >>
> >> On Mon, Jul 10, 2023, 17:47 Matthias J. Sax 
> >>> wrote:
> >>
> >>> Congrats!
> >>>
> >>> On 7/10/23 8:45 AM, Chris Egerton wrote:
>  Hi all,
> 
>  The PMC for Apache Kafka has invited Greg Harris to become a
> > committer,
> >>> and
>  we are happy to announce that he has accepted!
> 
>  Greg has been contributing to Kafka since 2019. He has made over
> >>> 50
> >>> commits
>  mostly around Kafka Connect and Mirror Maker 2. His most notable
>  contributions include KIP-898: "Modernize Connect plugin
> >>> discovery"
> >> and a
>  deep overhaul of the offset syncing logic in MM2 that addressed
> > several
>  technically-difficult, long-standing, high-impact issues.
> 
>  He has also been an active participant in discussions and
> >>> reviews on
> >> the
>  mailing lists and on GitHub.
> 
>  Thanks for all of your contributions, Greg. Congratulations!
> 
> >>>
> >>
> >
> >>>
> >
> >
>
>

Re: [DISCUSS] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

[Chris Egerton]

Hi Omnia,

Thanks for taking this on! I have some thoughts but the general approach
looks good.

1. Default value

One thing I'm wrestling with is what the default value of the new property
should be. I know on the Jira ticket I proposed that it should be false,
but I'm having second thoughts. Technically we'd preserve backward
compatibility with pre-KIP-690 releases by defaulting to false, but at the
same time, we'd break compatibility with post-KIP-690 releases. And if we
default to true, the opposite would be true: compatibility would be broken
with pre-KIP-690 releases, but preserved with post-KIP-690 releases.

One argument against defaulting to false (which, again, would preserve the
behavior of MM2 before we accidentally broke compatibility with KIP-690) is
that this change could possibly cause a single MM2 setup to break
twice--once when upgrading from a pre-KIP-690 release to an existing
release, and again when upgrading from that existing release to a version
that reverted (by default) to pre-KIP-690 behavior. On the other hand, if
we default to true (which would preserve the existing behavior that breaks
compatibility with pre-KIP-690 releases), then any given setup will only be
broken once.

In addition, if we default to true right now, then we don't have to worry
about changing that default in 4.0 to a more intuitive value (I hope we can
all agree that, for new clusters, it makes sense to set this property to
true and not to distinguish between internal and non-internal topics).

With that in mind, I'm now leaning more towards defaulting to true, but
would be interested in your thoughts.


2. Backport?

It's highly unlikely to backport changes for a KIP, but given the impact of
the compatibility break that we're trying to address here, and the
extremely low risk of the proposed changes, I think we should consider
backporting the proposed fix to all affected release branches (i.e., 3.1
through 3.5).


3. Extra steps

I also think we can take these additional steps to try to help prevent
users from being bitten by this change:

- Add a note to our upgrade instructions [1] for all affected versions that
instructs users on how to safely upgrade to a post-KIP-690 release, for
versions that both do and do not include the changes from this KIP
- Log a warning message on MM2 startup if the config contains an explicit
value for "replication.policy.separator" but does not contain an explicit
value for "replication.policy.internal.topic.separator.enabled"

These details don't necessarily have to be codified in the KIP, but they're
worth taking into account when considering how to design any functional
changes in order to better try to gauge how well this could go for our
users.

[1] - https://kafka.apache.org/documentation.html#upgrade


Thanks again for the KIP!

Cheers,

Chris

On Fri, Jul 7, 2023 at 10:12 AM Omnia Ibrahim 
wrote:

> Hi everyone,
> I want to start the discussion of the KIP-949. The proposal is here
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-949%3A+Add+flag+to+enable+the+usage+of+topic+separator+in+MM2+DefaultReplicationPolicy
> >
>
> Thanks for your time and feedback.
> Omnia
>

Re: [ANNOUNCE] New committer: Greg Harris

[kafka]

Congrats Greg!

-
Gaurav

> On 10 Jul 2023, at 17:25, Hector Geraldino (BLOOMBERG/ 919 3RD A) 
>  wrote:
> 
> Congrats Greg! Well deserved
> 
> From: dev@kafka.apache.org At: 07/10/23 12:18:48 UTC-4:00To:  
> dev@kafka.apache.org
> Subject: Re: [ANNOUNCE] New committer: Greg Harris
> 
> Congratulations!
> 
> On Mon, Jul 10, 2023 at 9:17 AM Randall Hauch  wrote:
>> 
>> Congratulations, Greg.
>> 
>> On Mon, Jul 10, 2023 at 11:13 AM Mickael Maison 
>> wrote:
>> 
>>> Congratulations Greg!
>>> 
>>> On Mon, Jul 10, 2023 at 6:08 PM Bill Bejeck 
>>> wrote:
 
 Congrats Greg!
 
 -Bill
 
 On Mon, Jul 10, 2023 at 11:53 AM Divij Vaidya 
 wrote:
 
> Congratulations Greg! I am going through a new committer teething
>>> process
> right now and would be happy to get you up to speed. Looking forward to
> working with you in your new role.
> 
> --
> Divij Vaidya
> 
> 
> 
> On Mon, Jul 10, 2023 at 5:51 PM Josep Prat >>> 
> wrote:
> 
>> Congrats Greg!
>> 
>> 
>> ———
>> Josep Prat
>> 
>> Aiven Deutschland GmbH
>> 
>> Alexanderufer 3-7, 10117 Berlin
>> 
>> Amtsgericht Charlottenburg, HRB 209739 B
>> 
>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> 
>> m: +491715557497
>> 
>> w: aiven.io
>> 
>> e: josep.p...@aiven.io
>> 
>> On Mon, Jul 10, 2023, 17:47 Matthias J. Sax 
>>> wrote:
>> 
>>> Congrats!
>>> 
>>> On 7/10/23 8:45 AM, Chris Egerton wrote:
 Hi all,
 
 The PMC for Apache Kafka has invited Greg Harris to become a
> committer,
>>> and
 we are happy to announce that he has accepted!
 
 Greg has been contributing to Kafka since 2019. He has made over
>>> 50
>>> commits
 mostly around Kafka Connect and Mirror Maker 2. His most notable
 contributions include KIP-898: "Modernize Connect plugin
>>> discovery"
>> and a
 deep overhaul of the offset syncing logic in MM2 that addressed
> several
 technically-difficult, long-standing, high-impact issues.
 
 He has also been an active participant in discussions and
>>> reviews on
>> the
 mailing lists and on GitHub.
 
 Thanks for all of your contributions, Greg. Congratulations!
 
>>> 
>> 
> 
>>> 
> 
> 

Re: [ANNOUNCE] New committer: Greg Harris

[Hector Geraldino (BLOOMBERG/ 919 3RD A)]

Congrats Greg! Well deserved

From: dev@kafka.apache.org At: 07/10/23 12:18:48 UTC-4:00To:  
dev@kafka.apache.org
Subject: Re: [ANNOUNCE] New committer: Greg Harris

Congratulations!

On Mon, Jul 10, 2023 at 9:17 AM Randall Hauch  wrote:
>
> Congratulations, Greg.
>
> On Mon, Jul 10, 2023 at 11:13 AM Mickael Maison 
> wrote:
>
> > Congratulations Greg!
> >
> > On Mon, Jul 10, 2023 at 6:08 PM Bill Bejeck 
> > wrote:
> > >
> > > Congrats Greg!
> > >
> > > -Bill
> > >
> > > On Mon, Jul 10, 2023 at 11:53 AM Divij Vaidya 
> > > wrote:
> > >
> > > > Congratulations Greg! I am going through a new committer teething
> > process
> > > > right now and would be happy to get you up to speed. Looking forward to
> > > > working with you in your new role.
> > > >
> > > > --
> > > > Divij Vaidya
> > > >
> > > >
> > > >
> > > > On Mon, Jul 10, 2023 at 5:51 PM Josep Prat  > >
> > > > wrote:
> > > >
> > > > > Congrats Greg!
> > > > >
> > > > >
> > > > > ———
> > > > > Josep Prat
> > > > >
> > > > > Aiven Deutschland GmbH
> > > > >
> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > >
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >
> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > >
> > > > > m: +491715557497
> > > > >
> > > > > w: aiven.io
> > > > >
> > > > > e: josep.p...@aiven.io
> > > > >
> > > > > On Mon, Jul 10, 2023, 17:47 Matthias J. Sax 
> > wrote:
> > > > >
> > > > > > Congrats!
> > > > > >
> > > > > > On 7/10/23 8:45 AM, Chris Egerton wrote:
> > > > > > > Hi all,
> > > > > > >
> > > > > > > The PMC for Apache Kafka has invited Greg Harris to become a
> > > > committer,
> > > > > > and
> > > > > > > we are happy to announce that he has accepted!
> > > > > > >
> > > > > > > Greg has been contributing to Kafka since 2019. He has made over
> > 50
> > > > > > commits
> > > > > > > mostly around Kafka Connect and Mirror Maker 2. His most notable
> > > > > > > contributions include KIP-898: "Modernize Connect plugin
> > discovery"
> > > > > and a
> > > > > > > deep overhaul of the offset syncing logic in MM2 that addressed
> > > > several
> > > > > > > technically-difficult, long-standing, high-impact issues.
> > > > > > >
> > > > > > > He has also been an active participant in discussions and
> > reviews on
> > > > > the
> > > > > > > mailing lists and on GitHub.
> > > > > > >
> > > > > > > Thanks for all of your contributions, Greg. Congratulations!
> > > > > > >
> > > > > >
> > > > >
> > > >
> >

Re: [ANNOUNCE] New committer: Greg Harris

[Guozhang Wang]

Congratulations!

On Mon, Jul 10, 2023 at 9:17 AM Randall Hauch  wrote:
>
> Congratulations, Greg.
>
> On Mon, Jul 10, 2023 at 11:13 AM Mickael Maison 
> wrote:
>
> > Congratulations Greg!
> >
> > On Mon, Jul 10, 2023 at 6:08 PM Bill Bejeck 
> > wrote:
> > >
> > > Congrats Greg!
> > >
> > > -Bill
> > >
> > > On Mon, Jul 10, 2023 at 11:53 AM Divij Vaidya 
> > > wrote:
> > >
> > > > Congratulations Greg! I am going through a new committer teething
> > process
> > > > right now and would be happy to get you up to speed. Looking forward to
> > > > working with you in your new role.
> > > >
> > > > --
> > > > Divij Vaidya
> > > >
> > > >
> > > >
> > > > On Mon, Jul 10, 2023 at 5:51 PM Josep Prat  > >
> > > > wrote:
> > > >
> > > > > Congrats Greg!
> > > > >
> > > > >
> > > > > ———
> > > > > Josep Prat
> > > > >
> > > > > Aiven Deutschland GmbH
> > > > >
> > > > > Alexanderufer 3-7, 10117 Berlin
> > > > >
> > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > >
> > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > >
> > > > > m: +491715557497
> > > > >
> > > > > w: aiven.io
> > > > >
> > > > > e: josep.p...@aiven.io
> > > > >
> > > > > On Mon, Jul 10, 2023, 17:47 Matthias J. Sax 
> > wrote:
> > > > >
> > > > > > Congrats!
> > > > > >
> > > > > > On 7/10/23 8:45 AM, Chris Egerton wrote:
> > > > > > > Hi all,
> > > > > > >
> > > > > > > The PMC for Apache Kafka has invited Greg Harris to become a
> > > > committer,
> > > > > > and
> > > > > > > we are happy to announce that he has accepted!
> > > > > > >
> > > > > > > Greg has been contributing to Kafka since 2019. He has made over
> > 50
> > > > > > commits
> > > > > > > mostly around Kafka Connect and Mirror Maker 2. His most notable
> > > > > > > contributions include KIP-898: "Modernize Connect plugin
> > discovery"
> > > > > and a
> > > > > > > deep overhaul of the offset syncing logic in MM2 that addressed
> > > > several
> > > > > > > technically-difficult, long-standing, high-impact issues.
> > > > > > >
> > > > > > > He has also been an active participant in discussions and
> > reviews on
> > > > > the
> > > > > > > mailing lists and on GitHub.
> > > > > > >
> > > > > > > Thanks for all of your contributions, Greg. Congratulations!
> > > > > > >
> > > > > >
> > > > >
> > > >
> >

Re: [ANNOUNCE] New committer: Greg Harris

[Randall Hauch]

Congratulations, Greg.

On Mon, Jul 10, 2023 at 11:13 AM Mickael Maison 
wrote:

> Congratulations Greg!
>
> On Mon, Jul 10, 2023 at 6:08 PM Bill Bejeck 
> wrote:
> >
> > Congrats Greg!
> >
> > -Bill
> >
> > On Mon, Jul 10, 2023 at 11:53 AM Divij Vaidya 
> > wrote:
> >
> > > Congratulations Greg! I am going through a new committer teething
> process
> > > right now and would be happy to get you up to speed. Looking forward to
> > > working with you in your new role.
> > >
> > > --
> > > Divij Vaidya
> > >
> > >
> > >
> > > On Mon, Jul 10, 2023 at 5:51 PM Josep Prat  >
> > > wrote:
> > >
> > > > Congrats Greg!
> > > >
> > > >
> > > > ———
> > > > Josep Prat
> > > >
> > > > Aiven Deutschland GmbH
> > > >
> > > > Alexanderufer 3-7, 10117 Berlin
> > > >
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >
> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >
> > > > m: +491715557497
> > > >
> > > > w: aiven.io
> > > >
> > > > e: josep.p...@aiven.io
> > > >
> > > > On Mon, Jul 10, 2023, 17:47 Matthias J. Sax 
> wrote:
> > > >
> > > > > Congrats!
> > > > >
> > > > > On 7/10/23 8:45 AM, Chris Egerton wrote:
> > > > > > Hi all,
> > > > > >
> > > > > > The PMC for Apache Kafka has invited Greg Harris to become a
> > > committer,
> > > > > and
> > > > > > we are happy to announce that he has accepted!
> > > > > >
> > > > > > Greg has been contributing to Kafka since 2019. He has made over
> 50
> > > > > commits
> > > > > > mostly around Kafka Connect and Mirror Maker 2. His most notable
> > > > > > contributions include KIP-898: "Modernize Connect plugin
> discovery"
> > > > and a
> > > > > > deep overhaul of the offset syncing logic in MM2 that addressed
> > > several
> > > > > > technically-difficult, long-standing, high-impact issues.
> > > > > >
> > > > > > He has also been an active participant in discussions and
> reviews on
> > > > the
> > > > > > mailing lists and on GitHub.
> > > > > >
> > > > > > Thanks for all of your contributions, Greg. Congratulations!
> > > > > >
> > > > >
> > > >
> > >
>

Re: [ANNOUNCE] New committer: Greg Harris

[Mickael Maison]

Congratulations Greg!

On Mon, Jul 10, 2023 at 6:08 PM Bill Bejeck  wrote:
>
> Congrats Greg!
>
> -Bill
>
> On Mon, Jul 10, 2023 at 11:53 AM Divij Vaidya 
> wrote:
>
> > Congratulations Greg! I am going through a new committer teething process
> > right now and would be happy to get you up to speed. Looking forward to
> > working with you in your new role.
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Mon, Jul 10, 2023 at 5:51 PM Josep Prat 
> > wrote:
> >
> > > Congrats Greg!
> > >
> > >
> > > ———
> > > Josep Prat
> > >
> > > Aiven Deutschland GmbH
> > >
> > > Alexanderufer 3-7, 10117 Berlin
> > >
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >
> > > m: +491715557497
> > >
> > > w: aiven.io
> > >
> > > e: josep.p...@aiven.io
> > >
> > > On Mon, Jul 10, 2023, 17:47 Matthias J. Sax  wrote:
> > >
> > > > Congrats!
> > > >
> > > > On 7/10/23 8:45 AM, Chris Egerton wrote:
> > > > > Hi all,
> > > > >
> > > > > The PMC for Apache Kafka has invited Greg Harris to become a
> > committer,
> > > > and
> > > > > we are happy to announce that he has accepted!
> > > > >
> > > > > Greg has been contributing to Kafka since 2019. He has made over 50
> > > > commits
> > > > > mostly around Kafka Connect and Mirror Maker 2. His most notable
> > > > > contributions include KIP-898: "Modernize Connect plugin discovery"
> > > and a
> > > > > deep overhaul of the offset syncing logic in MM2 that addressed
> > several
> > > > > technically-difficult, long-standing, high-impact issues.
> > > > >
> > > > > He has also been an active participant in discussions and reviews on
> > > the
> > > > > mailing lists and on GitHub.
> > > > >
> > > > > Thanks for all of your contributions, Greg. Congratulations!
> > > > >
> > > >
> > >
> >

Re: [ANNOUNCE] New committer: Greg Harris

[Bill Bejeck]

Congrats Greg!

-Bill

On Mon, Jul 10, 2023 at 11:53 AM Divij Vaidya 
wrote:

> Congratulations Greg! I am going through a new committer teething process
> right now and would be happy to get you up to speed. Looking forward to
> working with you in your new role.
>
> --
> Divij Vaidya
>
>
>
> On Mon, Jul 10, 2023 at 5:51 PM Josep Prat 
> wrote:
>
> > Congrats Greg!
> >
> >
> > ———
> > Josep Prat
> >
> > Aiven Deutschland GmbH
> >
> > Alexanderufer 3-7, 10117 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > m: +491715557497
> >
> > w: aiven.io
> >
> > e: josep.p...@aiven.io
> >
> > On Mon, Jul 10, 2023, 17:47 Matthias J. Sax  wrote:
> >
> > > Congrats!
> > >
> > > On 7/10/23 8:45 AM, Chris Egerton wrote:
> > > > Hi all,
> > > >
> > > > The PMC for Apache Kafka has invited Greg Harris to become a
> committer,
> > > and
> > > > we are happy to announce that he has accepted!
> > > >
> > > > Greg has been contributing to Kafka since 2019. He has made over 50
> > > commits
> > > > mostly around Kafka Connect and Mirror Maker 2. His most notable
> > > > contributions include KIP-898: "Modernize Connect plugin discovery"
> > and a
> > > > deep overhaul of the offset syncing logic in MM2 that addressed
> several
> > > > technically-difficult, long-standing, high-impact issues.
> > > >
> > > > He has also been an active participant in discussions and reviews on
> > the
> > > > mailing lists and on GitHub.
> > > >
> > > > Thanks for all of your contributions, Greg. Congratulations!
> > > >
> > >
> >
>

Re: [ANNOUNCE] New committer: Greg Harris

[Kenneth Eversole]

Congrats Greg!

On Mon, Jul 10, 2023 at 10:55 AM Viktor Somogyi-Vass
 wrote:

> Congrats Greg!
>
> On Mon, Jul 10, 2023, 17:54 Yash Mayya  wrote:
>
> > Congrats Greg!
> >
> > On Mon, Jul 10, 2023 at 9:15 PM Chris Egerton 
> wrote:
> >
> > > Hi all,
> > >
> > > The PMC for Apache Kafka has invited Greg Harris to become a committer,
> > and
> > > we are happy to announce that he has accepted!
> > >
> > > Greg has been contributing to Kafka since 2019. He has made over 50
> > commits
> > > mostly around Kafka Connect and Mirror Maker 2. His most notable
> > > contributions include KIP-898: "Modernize Connect plugin discovery"
> and a
> > > deep overhaul of the offset syncing logic in MM2 that addressed several
> > > technically-difficult, long-standing, high-impact issues.
> > >
> > > He has also been an active participant in discussions and reviews on
> the
> > > mailing lists and on GitHub.
> > >
> > > Thanks for all of your contributions, Greg. Congratulations!
> > >
> >
>

Re: [ANNOUNCE] New committer: Greg Harris

[Viktor Somogyi-Vass]

Congrats Greg!

On Mon, Jul 10, 2023, 17:54 Yash Mayya  wrote:

> Congrats Greg!
>
> On Mon, Jul 10, 2023 at 9:15 PM Chris Egerton  wrote:
>
> > Hi all,
> >
> > The PMC for Apache Kafka has invited Greg Harris to become a committer,
> and
> > we are happy to announce that he has accepted!
> >
> > Greg has been contributing to Kafka since 2019. He has made over 50
> commits
> > mostly around Kafka Connect and Mirror Maker 2. His most notable
> > contributions include KIP-898: "Modernize Connect plugin discovery" and a
> > deep overhaul of the offset syncing logic in MM2 that addressed several
> > technically-difficult, long-standing, high-impact issues.
> >
> > He has also been an active participant in discussions and reviews on the
> > mailing lists and on GitHub.
> >
> > Thanks for all of your contributions, Greg. Congratulations!
> >
>

Re: [ANNOUNCE] New committer: Greg Harris

[Yash Mayya]

Congrats Greg!

On Mon, Jul 10, 2023 at 9:15 PM Chris Egerton  wrote:

> Hi all,
>
> The PMC for Apache Kafka has invited Greg Harris to become a committer, and
> we are happy to announce that he has accepted!
>
> Greg has been contributing to Kafka since 2019. He has made over 50 commits
> mostly around Kafka Connect and Mirror Maker 2. His most notable
> contributions include KIP-898: "Modernize Connect plugin discovery" and a
> deep overhaul of the offset syncing logic in MM2 that addressed several
> technically-difficult, long-standing, high-impact issues.
>
> He has also been an active participant in discussions and reviews on the
> mailing lists and on GitHub.
>
> Thanks for all of your contributions, Greg. Congratulations!
>

Re: [ANNOUNCE] New committer: Greg Harris

[Divij Vaidya]

Congratulations Greg! I am going through a new committer teething process
right now and would be happy to get you up to speed. Looking forward to
working with you in your new role.

--
Divij Vaidya



On Mon, Jul 10, 2023 at 5:51 PM Josep Prat 
wrote:

> Congrats Greg!
>
>
> ———
> Josep Prat
>
> Aiven Deutschland GmbH
>
> Alexanderufer 3-7, 10117 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> m: +491715557497
>
> w: aiven.io
>
> e: josep.p...@aiven.io
>
> On Mon, Jul 10, 2023, 17:47 Matthias J. Sax  wrote:
>
> > Congrats!
> >
> > On 7/10/23 8:45 AM, Chris Egerton wrote:
> > > Hi all,
> > >
> > > The PMC for Apache Kafka has invited Greg Harris to become a committer,
> > and
> > > we are happy to announce that he has accepted!
> > >
> > > Greg has been contributing to Kafka since 2019. He has made over 50
> > commits
> > > mostly around Kafka Connect and Mirror Maker 2. His most notable
> > > contributions include KIP-898: "Modernize Connect plugin discovery"
> and a
> > > deep overhaul of the offset syncing logic in MM2 that addressed several
> > > technically-difficult, long-standing, high-impact issues.
> > >
> > > He has also been an active participant in discussions and reviews on
> the
> > > mailing lists and on GitHub.
> > >
> > > Thanks for all of your contributions, Greg. Congratulations!
> > >
> >
>

Re: [ANNOUNCE] New committer: Greg Harris

[Bruno Cadonna]

Congratulations Greg!

Best,
Bruno

On 10.07.23 17:47, Matthias J. Sax wrote:

Congrats!

On 7/10/23 8:45 AM, Chris Egerton wrote:

Hi all,

The PMC for Apache Kafka has invited Greg Harris to become a 
committer, and

we are happy to announce that he has accepted!

Greg has been contributing to Kafka since 2019. He has made over 50 
commits

mostly around Kafka Connect and Mirror Maker 2. His most notable
contributions include KIP-898: "Modernize Connect plugin discovery" and a
deep overhaul of the offset syncing logic in MM2 that addressed several
technically-difficult, long-standing, high-impact issues.

He has also been an active participant in discussions and reviews on the
mailing lists and on GitHub.

Thanks for all of your contributions, Greg. Congratulations!

Re: [ANNOUNCE] New committer: Greg Harris

[Josep Prat]

Congrats Greg!


———
Josep Prat

Aiven Deutschland GmbH

Alexanderufer 3-7, 10117 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Mon, Jul 10, 2023, 17:47 Matthias J. Sax  wrote:

> Congrats!
>
> On 7/10/23 8:45 AM, Chris Egerton wrote:
> > Hi all,
> >
> > The PMC for Apache Kafka has invited Greg Harris to become a committer,
> and
> > we are happy to announce that he has accepted!
> >
> > Greg has been contributing to Kafka since 2019. He has made over 50
> commits
> > mostly around Kafka Connect and Mirror Maker 2. His most notable
> > contributions include KIP-898: "Modernize Connect plugin discovery" and a
> > deep overhaul of the offset syncing logic in MM2 that addressed several
> > technically-difficult, long-standing, high-impact issues.
> >
> > He has also been an active participant in discussions and reviews on the
> > mailing lists and on GitHub.
> >
> > Thanks for all of your contributions, Greg. Congratulations!
> >
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1983

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 385734 lines...]
[INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/pom.xml to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart/3.6.0-SNAPSHOT/streams-quickstart-3.6.0-SNAPSHOT.pom
[INFO] 
[INFO] --< org.apache.kafka:streams-quickstart-java >--
[INFO] Building streams-quickstart-java 3.6.0-SNAPSHOT[2/2]
[INFO]   from java/pom.xml
[INFO] --[ maven-archetype ]---
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart-java ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart-java ---
[INFO] 
[INFO] --- resources:2.7:resources (default-resources) @ 
streams-quickstart-java ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 6 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- resources:2.7:testResources (default-testResources) @ 
streams-quickstart-java ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- archetype:2.2:jar (default-jar) @ streams-quickstart-java ---
[INFO] Building archetype jar: 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/java/target/streams-quickstart-java-3.6.0-SNAPSHOT
[INFO] 
[INFO] --- site:3.5.1:attach-descriptor (attach-descriptor) @ 
streams-quickstart-java ---
[INFO] 
[INFO] --- archetype:2.2:integration-test (default-integration-test) @ 
streams-quickstart-java ---
[WARNING]  Parameter 'skip' (user property 'archetype.test.skip') is read-only, 
must not be used in configuration
[INFO] 
[INFO] --- gpg:1.6:sign (sign-artifacts) @ streams-quickstart-java ---
[INFO] 
[INFO] --- install:2.5.2:install (default-install) @ streams-quickstart-java ---
[INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/java/target/streams-quickstart-java-3.6.0-SNAPSHOT.jar
 to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart-java/3.6.0-SNAPSHOT/streams-quickstart-java-3.6.0-SNAPSHOT.jar
[INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/java/pom.xml to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart-java/3.6.0-SNAPSHOT/streams-quickstart-java-3.6.0-SNAPSHOT.pom
[INFO] 
[INFO] --- archetype:2.2:update-local-catalog (default-update-local-catalog) @ 
streams-quickstart-java ---
[INFO] 
[INFO] Reactor Summary for Kafka Streams :: Quickstart 3.6.0-SNAPSHOT:
[INFO] 
[INFO] Kafka Streams :: Quickstart  SUCCESS [  3.688 s]
[INFO] streams-quickstart-java  SUCCESS [  1.592 s]
[INFO] 
[INFO] BUILD SUCCESS
[INFO] 
[INFO] Total time:  5.821 s
[INFO] Finished at: 2023-07-10T15:47:17Z
[INFO] 
[Pipeline] dir
Running in 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/test-streams-archetype
[Pipeline] {
[Pipeline] sh
+ echo Y
+ mvn archetype:generate -DarchetypeCatalog=local 
-DarchetypeGroupId=org.apache.kafka 
-DarchetypeArtifactId=streams-quickstart-java -DarchetypeVersion=3.6.0-SNAPSHOT 
-DgroupId=streams.examples -DartifactId=streams.examples -Dversion=0.1 
-Dpackage=myapps
[INFO] Scanning for projects...
[INFO] 
[INFO] --< org.apache.maven:standalone-pom >---
[INFO] Building Maven Stub Project (No POM) 1
[INFO] [ pom ]-
[INFO] 
[INFO] >>> archetype:3.2.1:generate (default-cli) > generate-sources @ 
standalone-pom >>>
[INFO] 
[INFO] <<< archetype:3.2.1:generate (default-cli) < generate-sources @ 
standalone-pom <<<
[INFO] 
[INFO] 
[INFO] --- archetype:3.2.1:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Interactive mode
[WARNING] Archetype not found in any catalog. Falling back to central 
repository.
[WARNING] Add a repository with id 'archetype' in your settings.xml if 
archetype's repository is elsewhere.
[INFO] Using property: groupId = streams.examples
[INFO] Using property: artifactId = streams.examples
[INFO] Using property: version = 0.1
[INFO] Using property: package = myapps
Confirm properties configuration:
groupId: streams.examples
artifactId: streams.examples
version: 0.1
package: myapps
 Y: : [INFO] 

[INFO] Using following parameters for creating project from Archetype: 
streams-quickstart-java:3.6.0-SNAPSHOT
[INFO] 

Re: [ANNOUNCE] New committer: Greg Harris

[Matthias J. Sax]

Congrats!

On 7/10/23 8:45 AM, Chris Egerton wrote:

Hi all,

The PMC for Apache Kafka has invited Greg Harris to become a committer, and
we are happy to announce that he has accepted!

Greg has been contributing to Kafka since 2019. He has made over 50 commits
mostly around Kafka Connect and Mirror Maker 2. His most notable
contributions include KIP-898: "Modernize Connect plugin discovery" and a
deep overhaul of the offset syncing logic in MM2 that addressed several
technically-difficult, long-standing, high-impact issues.

He has also been an active participant in discussions and reviews on the
mailing lists and on GitHub.

Thanks for all of your contributions, Greg. Congratulations!

Re: [DISCUSS] KIP-917: Additional custom metadata for remote log segment

[Ivan Yurchenko]

Hi!

For those who are interested, I submitted the PR with the implementation:
https://github.com/apache/kafka/pull/13984
Thanks!

Ivan


On Tue, 13 Jun 2023 at 15:05, Ivan Yurchenko 
wrote:

> Hi Satish,
>
> I understand your point and I agree with it. TBH, I didn't take into
> account the in-memory cache.
> Surely, 10 KiB is a relatively arbitrary number. Lowering it to the
> proposed 128 bytes won't probably change anything for most of the
> implementations. Others could ask for an increase.
> I've updated the KIP.
> Thanks!
>
> Ivan
>
>
> On Tue, 13 Jun 2023 at 09:47, Satish Duggana 
> wrote:
>
>> Thanks Ivan for taking the feedback/suggestions into the KIP.
>>
>> 10kb limit as default is huge for each segment metadata. This should
>> only be used for a few keys pointing to more values if needed from a
>> different store. This is intended to be used for purposes like
>> pointing to a specific cluster or a specific bucket etc. This is
>> generally required for identifiers like uuid or other formats that can
>> amount to a few 10s of bytes. So, I suggest having a default value of
>> 128 bytes instead of 10kb. If it requires metadata with a few KBs then
>> it is better to avoid keeping all of them inmemory with default topic
>> based RLMM and have them in a cache with separate storage. In that
>> case, it is better to think about why we need that large metadata with
>> each segment.
>>
>> ~Satish.
>>
>>
>> On Wed, 7 Jun 2023 at 23:38, Ivan Yurchenko 
>> wrote:
>> >
>> > Hi Satish,
>> >
>> > Thank you for your feedback.
>> >
>> > I've nothing against going from Map to byte[].
>> > Serialization should not be a problem for RSM implementations: `Struct`,
>> > `Schema` and other useful serde classes are distributed as a part of the
>> > kafka-clients library.
>> >
>> > Also a good idea to add the size limiting setting, some
>> > `remote.log.metadata.custom.metadata.max.size`. A sensible default may
>> be
>> > 10 KB, which is enough to host a struct with 10 long (almost) 1K symbol
>> > ASCII strings.
>> >
>> > If a piece of custom metadata exceeds the limit, the execution of
>> > RLMTask.copyLogSegmentsToRemote should be interrupted with an error
>> message.
>> >
>> > Does this sound good?
>> > If so, I'll update the KIP accordingly. And I think it may be time for
>> the
>> > vote after that.
>> >
>> > Best,
>> > Ivan
>> >
>> >
>> >
>> > On Sat, 3 Jun 2023 at 17:13, Satish Duggana 
>> > wrote:
>> >
>> > > Hi Ivan,
>> > > Thanks for the KIP.
>> > >
>> > > The motivation of the KIP looks reasonable to me. It requires a way
>> > > for RSM providers to add custom metadata with the existing
>> > > RemoteLogSegmentMetadata. I remember we wanted to introduce a very
>> > > similar change in the earlier proposals called
>> > > RemoteLogMetadataContext. But we dropped that as we did not feel a
>> > > strong need at that time and we wanted to revisit it if needed. But I
>> > > see there is a clear need for this kind of custom metadata to keep
>> > > with RemoteLogSegmentMetadata.
>> > >
>> > > It is better to introduce a new class for this custom metadata in
>> > > RemoteLogSegmentMetadata like below for any changes in the future.
>> > > RemoteLogSegmentMetadata will have this as an optional value.
>> > >
>> > > public class RemoteLogSegmentMetadata {
>> > > ...
>> > > public static class CustomMetadata {
>> > >  private final byte[] value;
>> > > ...
>> > > }
>> > > ...
>> > > }
>> > >
>> > > This is completely opaque and it is the RSM implementation provider's
>> > > responsibility in serializing and deserializing the bytes. We can
>> > > introduce a property to guard the size with a configurable property
>> > > with a default value to avoid any unwanted large size values.
>> > >
>> > > Thanks,
>> > > Satish.
>> > >
>> > > On Tue, 30 May 2023 at 10:59, Ivan Yurchenko <
>> ivan0yurche...@gmail.com>
>> > > wrote:
>> > > >
>> > > > Hi all,
>> > > >
>> > > > I want to bring this to a conclusion (positive or negative), so if
>> there
>> > > > are no more questions in a couple of days, I'll put the KIP to the
>> vote.
>> > > >
>> > > > Best,
>> > > > Ivan
>> > > >
>> > > >
>> > > > On Fri, 5 May 2023 at 18:42, Ivan Yurchenko <
>> ivan0yurche...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi Alexandre,
>> > > > >
>> > > > > > combining custom
>> > > > > > metadata with rlmMetadata increases coupling between Kafka and
>> the
>> > > > > > plugin.
>> > > > >
>> > > > > This is true. However, (if I understand your concern correctly,)
>> > > > > rlmMetadata in the current form may be independent from RSM
>> plugins,
>> > > but
>> > > > > data they point to are accessible only via the particular plugin
>> (the
>> > > one
>> > > > > that wrote the data or a compatible one). It seems, this coupling
>> > > already
>> > > > > exists, but it is implicit. To make my point more concrete,
>> imagine an
>> > > S3
>> > > > > RSM which maps RemoteLogSegmentMetadata objects to S3 object
>> keys. This
>> > > > > mapping logic is a 

[ANNOUNCE] New committer: Greg Harris

[Chris Egerton]

Hi all,

The PMC for Apache Kafka has invited Greg Harris to become a committer, and
we are happy to announce that he has accepted!

Greg has been contributing to Kafka since 2019. He has made over 50 commits
mostly around Kafka Connect and Mirror Maker 2. His most notable
contributions include KIP-898: "Modernize Connect plugin discovery" and a
deep overhaul of the offset syncing logic in MM2 that addressed several
technically-difficult, long-standing, high-impact issues.

He has also been an active participant in discussions and reviews on the
mailing lists and on GitHub.

Thanks for all of your contributions, Greg. Congratulations!

[GitHub] [kafka-site] divijvaidya merged pull request #532: MINOR: Add Divij PGP public keys

[via GitHub]

divijvaidya merged PR #532:
URL: https://github.com/apache/kafka-site/pull/532


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] Release plan for Apache Kafka 3.5.1 release

[Ismael Juma]

Hi David,

There is no reason to wait 3 months - we often would release 4-6 weeks
after the .0 in the past. If someone is willing to volunteer to do the
release and there are important bugs, it's actually better to release more
often.

Ismael

On Mon, Jul 3, 2023 at 5:00 AM David Jacot 
wrote:

> Hi Divij,
>
> Thanks for the release plan.
>
> I wonder if we should wait a little more as 3.5.0 was released on June
> 15th. Releasing 3.5.1 a month after seems not enough in order to have time
> to catch bugs in 3.5.0. I think that we usually release the first minor
> release ~3 months after the major one. Is there a reason to release it in
> July?
>
> As a side note, we don't have a formal code freeze for minor releases.
>
> Best,
> David
>
> On Mon, Jul 3, 2023 at 1:51 PM Divij Vaidya 
> wrote:
>
> > Hi folks
> >
> > Here's the release plan for
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+plan+3.5.1
> >
> > 3.5.1 will be a bug fix release which also addresses some of the CVEs
> such
> > as CVE-2023-34455 [1] in snappy-java. If all goes smoothly, I am
> estimating
> > a release date in the 3rd or 4th week of July. I will continue to post
> > important updates on the mailing list and you can also follow the
> progress
> > on the release plan wiki above.
> >
> > *Call for action* 
> >
> > If you think that a commit from the trunk should be backported to 3.5.1,
> > please let me know. Note that we usually backport only the critical bug
> > fixes which don't have a production work around and security fixes. Note
> > that code freeze is on 9th July and no new commits will be added to the
> 3.5
> > .1 release after that.
> >
> > *Important dates *
> >
> > 9th July - Code freeze for 3.5.1
> > 10th July - First release candidate is published for voting
> > 18th July - Expected completion of release
> >
> > --
> > Divij Vaidya
> > Release Manager for Apache Kafka 3.5.1
> >
> > [1] https://nvd.nist.gov/vuln/detail/CVE-2023-34455
> >
>

[VOTE] KIP-916: MM2 distributed mode flow log context

[Dániel Urbán]

Hello everyone,

I would like to start a vote on KIP-916: MM2 distributed mode flow log
context.
The KIP aims to improve the logging of MM2 distributed mode. It is a
relatively small change, but it has a big impact, as the current logs are
very hard to decipher. (The current logs are based on the KConnect logging
model, which assumes connector name uniqueness, which is not true in the
MM2 distributed mode.)

Thanks in advance,
Daniel

Re: [DISCUSS] Release plan for Apache Kafka 3.5.1 release

[Divij Vaidya]

Hey folks

I am postponing the release for first RC by one day to 11th July so that we
can merge in minor dependency upgrades -
https://issues.apache.org/jira/browse/KAFKA-15159

--
Divij Vaidya



On Mon, Jul 3, 2023 at 2:38 PM Divij Vaidya  wrote:

> Satish - Thank you for catching that. It is now fixed.
>
> David - Please refer to the security@kafka mailing thread with "Reg CVE
> 2023-34455" where it was proposed to have an early release for 3.5.1. The
> rationale of releasing 3.5.1 early is to have a version of Kafka released
> which does not have any known CVE, specifically
> https://issues.apache.org/jira/browse/KAFKA-15096. Separately, I am going
> to start a PR today to update the CVE list with more information on this
> CVE and the potential workaround.
>
> --
> Divij Vaidya
>
>
>
> On Mon, Jul 3, 2023 at 2:00 PM David Jacot 
> wrote:
>
>> Hi Divij,
>>
>> Thanks for the release plan.
>>
>> I wonder if we should wait a little more as 3.5.0 was released on June
>> 15th. Releasing 3.5.1 a month after seems not enough in order to have time
>> to catch bugs in 3.5.0. I think that we usually release the first minor
>> release ~3 months after the major one. Is there a reason to release it in
>> July?
>>
>> As a side note, we don't have a formal code freeze for minor releases.
>>
>> Best,
>> David
>>
>> On Mon, Jul 3, 2023 at 1:51 PM Divij Vaidya 
>> wrote:
>>
>> > Hi folks
>> >
>> > Here's the release plan for
>> > https://cwiki.apache.org/confluence/display/KAFKA/Release+plan+3.5.1
>> >
>> > 3.5.1 will be a bug fix release which also addresses some of the CVEs
>> such
>> > as CVE-2023-34455 [1] in snappy-java. If all goes smoothly, I am
>> estimating
>> > a release date in the 3rd or 4th week of July. I will continue to post
>> > important updates on the mailing list and you can also follow the
>> progress
>> > on the release plan wiki above.
>> >
>> > *Call for action* 
>> >
>> > If you think that a commit from the trunk should be backported to 3.5.1,
>> > please let me know. Note that we usually backport only the critical bug
>> > fixes which don't have a production work around and security fixes. Note
>> > that code freeze is on 9th July and no new commits will be added to the
>> 3.5
>> > .1 release after that.
>> >
>> > *Important dates *
>> >
>> > 9th July - Code freeze for 3.5.1
>> > 10th July - First release candidate is published for voting
>> > 18th July - Expected completion of release
>> >
>> > --
>> > Divij Vaidya
>> > Release Manager for Apache Kafka 3.5.1
>> >
>> > [1] https://nvd.nist.gov/vuln/detail/CVE-2023-34455
>> >
>>
>

Re: [REVIEW REQUEST] KAFKA-14591 DeleteRecordsCommand moved to tools

[Николай Ижиков]

Fixed all comments and checkstyle issues.
Please, make one more round of review.

> 10 июля 2023 г., в 16:17, Federico Valeri  написал(а):
> 
> HI Nikolay, I did the review.
> 
> Thanks again for working on this.
> 
> Br
> Fede
> 
> On Mon, Jul 10, 2023 at 12:08 PM Николай Ижиков  wrote:
>> 
>> Hello.
>> 
>> I prepared PR [1] for ticket [2].
>> DeleteRecordsCommand rewritten in java and moved to tools module.
>> Currently, commands doesn’t have any tests so PR adds some tests, also.
>> 
>> Please, review.
>> 
>> [1] https://github.com/apache/kafka/pull/13278
>> [2] https://issues.apache.org/jira/browse/KAFKA-14591
>> 

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

[Igor Soarez]

Hi Colin,

Thanks for your questions.
Please have a look at my answers below.

> In the previous email I asked, "who is responsible for assigning replicas to 
> broker directories?" Can you clarify what the answer is to that? If the 
> answer is the controller, there is no need for an "unknown" state for 
> assignments, since the controller can simply choose an assignment immediately 
> when it creates a replica.

Apologies, I thought I had made this clear in my previous email
and in the KIP. It is the Broker who is responsible for
assigning replicas to log directories.

> The broker has access to the same metadata, of course, and already knows what 
> directory a replica is supposed to be in. If that directory doesn't exist, 
> then the replica is offline. There is no need to modify replicas to have an 
> "unknown" assignment state.

You are correct, we can avoid this intermediate metadata update by
the Controller. The Controller doesn't have to reset assignments
to Uuid.ZERO, instead the Broker can just select a new directory
for replicas assigned to other UUIDs when there are no offline dirs.
I have updated the KIP with this change.

However, I still think we need reserved UUIDs.

  a) If there are any offline log directories the broker cannot
  distinguish between replicas that are assigned to offline dirs,
  vs replicas that are assigned to deconfigured directories and
  need new placement.
  Since new replicas are created with UUID.Zero, it is safe for the
  broker to select a directory for them, even if some are offline.
  This replaces the use of the `isNew` flag in ZK mode.

  b) As described in the KIP, under "Handling log directory failures",
  in a race between syncing assignments and failing log directories,
  the Broker might exceptionally need to convey to the Controller
  that some replica that was in the failed directory had an incorrect
  assignment in the metadata, and the Broker can do that with a
  AssignReplicasToDirs using a reserved UUID value.

  b) During a Migration from a ZK JBOD cluster, the Controller
  can enforce fencing of a Broker until all replica assignments
  are known (i.e. not Uuid.ZERO).

I've updated the KIP to name our use of Uuid.ZERO as
Uuid.UnknownDir, and I've introduced Uuid.OfflineDir
to deal with case b).

Let me know what you think.


Best,

--
Igor

[GitHub] [kafka-site] divijvaidya opened a new pull request, #532: MINOR: Add Divij PGP public keys

[via GitHub]

divijvaidya opened a new pull request, #532:
URL: https://github.com/apache/kafka-site/pull/532

   ## Motivation
   
   Required for 3.5.1 release
   
   ## Public keys
   
   At public server: https://keys.openpgp.org/search?q=divijv%40apache.org
   At Apache homepage: https://home.apache.org/~divijv/
   
   Note that the keys are RSA 4096 encrypted.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] KIP-917: Additional custom metadata for remote log segment

[Ivan Yurchenko]

Hi all,

For your information: while developing this, I made a couple of minor
changes to the KIP:
1. `nullableVersions` was added to the `CustomMetadata` schema.
2. `CustomMetadata` was made `null` by default.
3. The same changes as discussed for `RemoteLogSegmentMetadataRecord` were
added for `RemoteLogSegmentMetadataUpdateRecord`.
4. The documentation for `remote.log.metadata.custom.metadata.max.size` was
slightly reworded for clarity.

You can check the diff [1].

I hope this doesn't affect the consensus on this KIP.

Thanks!

Ivan

[1]
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=252513269=10=8


On Mon, 19 Jun 2023 at 13:29, Josep Prat 
wrote:

> Hi Ivan,
>
> Thanks and congrats for your first accepted KIP.
> Regarding the process, yes you are right. You should create an issue (this
> can be done pre-KIP as well), it would be good if you could link this vote
> thread in the JIRA. Then you need to update your KIP's wiki page with the
> status (approved now), in your case adding the Jira ticket link as well in
> the right section. Then you should go to the general KIP overview page and
> move your KIP from "under discussion" to "approved" and 3.6.0 as the
> version.
>
> Best,
>
> On Mon, Jun 19, 2023 at 12:20 PM Ivan Yurchenko 
> wrote:
>
> > Hi all,
> >
> > Thank you for your votes.
> >
> > The voting has been open for 7 days and we have:
> > - 4 binding +1: Divij Vaidya, Luke Chen, Josep Prat, and Satish Duggana
> > (based on today's state of https://kafka.apache.org/committers).
> > - 2 non-binding +1: Kamal Chandraprakash and Matthew Benedict de Detrich.
> > - No -1.
> >
> > AFAIU, we can declare the KIP as accepted. I will create a ticket and
> also
> > volunteer to implement it.
> >
> > Please correct me if I'm wrong, that's the first time I'm bringing a KIP
> to
> > this stage :)
> >
> > Thanks!
> >
> > Best,
> > Ivan
> >
> >
> > On Wed, 14 Jun 2023 at 18:28, Satish Duggana 
> > wrote:
> >
> > > Thanks Ivan for addressing the comments in the KIP, LGTM.
> > >
> > > +1
> > >
> > > On Tue, 13 Jun 2023 at 18:31, Luke Chen  wrote:
> > > >
> > > > Looks good. Thanks for the update.
> > > >
> > > > On Tue, Jun 13, 2023 at 8:08 PM Ivan Yurchenko <
> > ivan0yurche...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi all!
> > > > >
> > > > > Thank you for your votes.
> > > > >
> > > > > Based on the proposal of Satish in the discussion thread, I
> modified
> > > the
> > > > > KIP a little bit by lowering the default value of
> > > > > `remote.log.metadata.custom.metadata.max.size` from 10 KiB to 128
> > > bytes. I
> > > > > hope this doesn't change your vote, but feel free to raise your
> > > concerns.
> > > > > Thank you!
> > > > >
> > > > > Best,
> > > > > Ivan
> > > > >
> > > > >
> > > > > On Tue, 13 Jun 2023 at 11:09, Josep Prat
>  > >
> > > > > wrote:
> > > > >
> > > > > > Hi Ivan,
> > > > > >
> > > > > > Thank you very much for this KIP. +1 (binding) from me.
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > >
> > > > > > On Tue, Jun 13, 2023 at 10:03 AM Luke Chen 
> > > wrote:
> > > > > >
> > > > > > > +1 (binding) from me.
> > > > > > >
> > > > > > > Thanks.
> > > > > > > Luke
> > > > > > >
> > > > > > > On Tue, Jun 13, 2023 at 3:44 PM Matthew Benedict de Detrich
> > > > > > >  wrote:
> > > > > > > >
> > > > > > > > +1 (non binding). Thanks for KIP
> > > > > > > >
> > > > > > > > On Tue, Jun 13, 2023 at 3:38 AM Kamal Chandraprakash <
> > > > > > > > kamal.chandraprak...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > +1 (non-binding). Thanks for the KIP!
> > > > > > > > >
> > > > > > > > > On Mon, Jun 12, 2023, 21:23 Divij Vaidya <
> > > divijvaidy...@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I have reviewed the proposal and feel that it would be
> > > beneficial
> > > > > > to
> > > > > > > > > > implement.
> > > > > > > > > >
> > > > > > > > > > Vote +1 (non-binding)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Divij Vaidya
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Jun 12, 2023 at 4:39 PM Ivan Yurchenko <
> > > > > > > ivan0yurche...@gmail.com
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hello,
> > > > > > > > > > >
> > > > > > > > > > > Some interest in KIP-917 was expressed in the
> discussion
> > > thread
> > > > > > > [1].
> > > > > > > > > > After
> > > > > > > > > > > addressing all the comments there, I'm putting it up
> to a
> > > vote.
> > > > > > > > > > > Thanks!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Ivan
> > > > > > > > > > >
> > > > > > > > > > > [1]
> > > > > > >
> https://lists.apache.org/thread/qpccqd3jy5rzvbt5ngtzo3dg9pzp722y
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Matthew de Detrich
> > > > > > > >
> > > > > > > > *Aiven Deutschland GmbH*
> > > > > > 

Re: [REVIEW REQUEST] KAFKA-14591 DeleteRecordsCommand moved to tools

[Federico Valeri]

HI Nikolay, I did the review.

Thanks again for working on this.

Br
Fede

On Mon, Jul 10, 2023 at 12:08 PM Николай Ижиков  wrote:
>
> Hello.
>
> I prepared PR [1] for ticket [2].
> DeleteRecordsCommand rewritten in java and moved to tools module.
> Currently, commands doesn’t have any tests so PR adds some tests, also.
>
> Please, review.
>
> [1] https://github.com/apache/kafka/pull/13278
> [2] https://issues.apache.org/jira/browse/KAFKA-14591
>

[jira] [Created] (KAFKA-15171) Kafka client poll never notifies when broker is down.

[Kartik (Jira)]

Kartik created KAFKA-15171:
--

 Summary: Kafka client poll never notifies when broker is down.
 Key: KAFKA-15171
 URL: https://issues.apache.org/jira/browse/KAFKA-15171
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 3.2.0
Reporter: Kartik
 Fix For: 3.2.1


Hi All, 

We are using apache camel to connect to the Kafka endpoint which internally 
uses Kafka-client to connect to the Kafka broker.

When the broker is down, the client keeps logging warn messages _*"node -1 
disconnected, Broker may not be available"*_ and never gives up and throws an 
exception, because of this application never stops polling.

Is there any config parameter in the client that allows the poll function to 
throw RuntimeException or this is a bug?

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-882: Kafka Connect REST API timeout improvements

[Yash Mayya]

Hi all,

I just wanted to post an update on this KIP since it has been dormant for a
while now. The original issue that motivated this KIP was resolved by
tweaking some configurations for the client that the connector was using to
talk to the external system. Instead of discarding this KIP, however, I've
heavily reduced its scope to only include the following improvements:

   - Abort create / update connector requests if the config validation
   causes the request to time out (instead of the existing behavior of
   proceeding to create / update a connector even after serving a 500 response
   to the user)
   - Fix the double config validation issue in Connect's distributed mode

I was on the fence about whether or not this required a KIP but since we're
technically changing the behavior of a public endpoint in certain cases, I
decided to err on the side of caution and continue with a public discussion
and vote.

Thanks,
Yash

On Wed, Mar 1, 2023 at 6:32 PM Yash Mayya  wrote:

> Hi all,
>
> I'd like to call for a vote on the (hopefully) straightforward KIP-882
> which adds support for configuring request timeouts on Kafka Connect REST
> APIs via query parameters along with a couple of related small improvements.
>
> KIP -
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-882%3A+Kafka+Connect+REST+API+timeout+improvements
>
> Discussion thread -
> https://lists.apache.org/thread/cygy115qmwpc3nj5omnj0crws2dw8nor
>
> Thanks,
> Yash
>

[jira] [Created] (KAFKA-15170) CooperativeStickyAssignor cannot adjust assignment correctly

[li xiangyuan (Jira)]

li xiangyuan created KAFKA-15170:


 Summary: CooperativeStickyAssignor cannot adjust assignment 
correctly
 Key: KAFKA-15170
 URL: https://issues.apache.org/jira/browse/KAFKA-15170
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 3.5.0
Reporter: li xiangyuan
Assignee: li xiangyuan


AbstractStickyAssignor use ConstrainedAssignmentBuilder to build assignment 
when all consumers in group subscribe the same topic list, but it couldn't add 
all partitions move owner to another consumer to 
``partitionsWithMultiplePreviousOwners``.

 

the reason is in function assignOwnedPartitions hasn't add partitions that 
rack-mismatch with prev owner to allRevokedPartitions, then partition only in 
this list would add to partitionsWithMultiplePreviousOwners.

 

In Cooperative Rebalance, partitions have changed owner must be removed from 
final assignment or will lead to incorrect consume behavior, I have already 
raise a pr, please take a look, thx

 

https://github.com/apache/kafka/pull/13965



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-948: Allow custom prefix for internal topic names in Kafka Streams

[Bruno Cadonna]

Hi Igor,

Thanks for the KIP!


1.
Could you please elaborate why prefixing the application.id with the 
team's topic namespace is not always desirable. I would like to better 
understand the motivation of this KIP.



2.
Could you please add type and default value for the proposed config 
internal.topics.prefix?


Something like
"internal.topics.prefix"
type: string
default: "" (empty string)

Similar would be also great for the proposed argument for the Streams 
resetter tool.



3.
Could you please add a test plan to the KIP?


4.
Could you please add a JIRA ticket (if one does not already exist) and 
link it to the KIP?



Best,
Bruno



On 06.07.23 13:14, Igor Buzatovic wrote:

Hi everyone,
I would like to start a discussion on the following KIP:

KIP-948: Allow custom prefix for internal topic names in Kafka Streams
https://cwiki.apache.org/confluence/display/KAFKA/KIP-948%3A+Allow+custom+prefix+for+internal+topic+names+in+Kafka+Streams

The goal is to provide greater flexibility in naming internal topics. The
main issue is that it affects Kafka Streams application reset tool (as
described in the KIP), so I guess the pros and cons should be weighed to
decide if this proposal makes sense.

Your feedback is appreciated. Thanks!

Igor Buzatović
Porsche Digital Croatia d.o.o.

Testing FixedKeyProcessor implementation using unit tests

[EXT . Zlatibor . Veljkovic]

Hi, I am using kafka-streams-test-utils and have problem with testing 
FixedKeyProcessor [KIP-820 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-820%3A+Extend+KStream+process+with+new+Processor+API#KIP820:ExtendKStreamprocesswithnewProcessorAPI-InfrastructureforFixedKeyRecords].

Using mock processor context to get the forwarded message doesn't work.

class org.apache.kafka.streams.processor.api.MockProcessorContext cannot be 
cast to class org.apache.kafka.streams.processor.api.FixedKeyProcessorContext

Anything I can do to get forwarded records?

Thanks,
Zed

[REVIEW REQUEST] KAFKA-14591 DeleteRecordsCommand moved to tools

[Николай Ижиков]

Hello.

I prepared PR [1] for ticket [2].
DeleteRecordsCommand rewritten in java and moved to tools module.
Currently, commands doesn’t have any tests so PR adds some tests, also.

Please, review.

[1] https://github.com/apache/kafka/pull/13278
[2] https://issues.apache.org/jira/browse/KAFKA-14591

Re: [DISCUSS] KIP-932: Queues for Kafka

[Dániel Urbán]

Yes, I think it's clear now, thank you.
I agree that allowing reading behind the LSO would require more work on the
broker side (we would need 1 more state for the messages, and transition
when the LSO moves forward), but I don't see the extra complexity on the
consumer side. Based on the KIP so far, brokers will be able to return
specific batches/messages to queue consumers - consumers will need to be
able to skip messages in case another consumer of the same group has
already acquired/acked those. If we have this logic present in the protocol
and the clients, consumers could skip pending messages using the same
mechanism, and only the broker would need to know *why* exactly a specific
record/batch is skipped.

I don't think that this feature would be too important, but compared to the
complexity of the KIP, 1 more state doesn't seem too complicated to me.

Thanks,
Daniel

Matthias J. Sax  ezt írta (időpont: 2023. júl. 10., H,
7:22):

> Daniel, sure.
>
> To allow the client to filter aborted messages, the broker currently
> attaches metadata that tell the client which records were aborted. But
> the first message after the LSO is a messages in pending state, ie, it
> was neither committed nor aborted yet, so it's not possible to filter or
> deliver it. Thus, the broker cannot provide this metadata (not sure if
> the client could filter without this metadata?)
>
> The main reason why this happens broker side is to avoid that the client
> needs to buffer pending messages "indefinitely" until the TX might
> eventually commit or abort, and thus put a lot a memory pressure on the
> client. For the "classic" case, the situation is  more severe as we
> guarantee ordered delivery, and thus, the client would need to buffer
> everything after the LSO. -- While it's relaxed for queuing as we might
> not guarantee order (ie, instead of buffering everything, only pending
> messages must be buffered), it would still imply a huge additional
> burden on tracking metadata (for both the broker and the consumer), and
> the wire protocol, and I am already worried about the metadata we might
> need to track for queuing in general.
>
> Does this make sense?
>
>
> -Matthias
>
>
>
> On 7/7/23 01:35, Dániel Urbán wrote:
> > Hi Matthias,
> > Can you please elaborate on this: "First, you need to understand that
> > aborted records are filtered client side, and thus for "read-committed"
> we
> > can never read beyond the LSO, and the same seems to apply for queuing."
> > I don't understand the connection here - what does skipping aborted
> records
> > have to do with the LSO? As you said, aborted message filtering is done
> on
> > the client side (in consumers, yes, but not sure if it has to be the same
> > for queues), but being blocked on the LSO is the responsibility of the
> > broker, isn't it? My thought was that the broker could act differently
> when
> > working with queues and read_committed isolation.
> > Thanks,
> > Daniel
> >
> > On Thu, Jul 6, 2023 at 7:26 PM Matthias J. Sax  wrote:
> >
> >> Thanks for the KIP.
> >>
> >> It seems we are in very early stage, and some very important sections in
> >> the KIP are still marked as TODO. In particular, I am curious about the
> >> protocol changes, how the "queuing state" will be represented and made
> >> durable, and all the error edge case / fail-over / fencing
> >> (broker/clients) that we need to put in place.
> >>
> >>
> >> A few other comments/question from my side:
> >>
> >> (1) Fetch from follower: this was already touched on, but the point is
> >> really that the consumer does not decide about it, but the broker does.
> >> When a consumer sends it's first fetch request it will always go to the
> >> leader, and the broker would reply to the consumer "go and fetch from
> >> this other broker". -- I think it's ok to exclude fetch from follower in
> >> the first version of the KIP, but it would need a broker change such
> >> that the broker knows it's a "queue fetch" request. -- It would also be
> >> worth to explore how fetch from follow could work in the future and
> >> ensure that our initial design allows for it and is future proof.
> >>
> >>
> >> (2) Why do we not allow pattern subscription and what happens if
> >> different consumers subscribe to different topics? It's not fully
> >> explained in the KIP.
> >>
> >>
> >> (3) auto.offset.reset and SPSO/SPSE -- I don't understand why we would
> >> not allow auto.offset.reset? In the discussion, you mentioned that
> >> "first consumer would win, if two consumers have a different config" --
> >> while this is correct, it's the same for a consumer group right now.
> >> Maybe we should not try to solve a "non problem"? -- In general, my
> >> impression is that we are going to do Kafkaeque Queuing, what is fine,
> >> but it might be to our advantage to carry over as many established
> >> concepts as we can? And if not, have a very good reason not to.
> >>
> >> In the end, it find if very clumsy to only have an admin API to 

[jira] [Resolved] (KAFKA-14828) Remove R/W lock from StandardAuthorizer

[Purshotam Chauhan (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Purshotam Chauhan resolved KAFKA-14828.
---
Fix Version/s: 3.6.0
 Reviewer: Manikumar Reddy
   Resolution: Fixed

> Remove R/W lock from StandardAuthorizer
> ---
>
> Key: KAFKA-14828
> URL: https://issues.apache.org/jira/browse/KAFKA-14828
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Purshotam Chauhan
>Assignee: Purshotam Chauhan
>Priority: Major
> Fix For: 3.6.0
>
>
> Currently, StandardAuthorizer uses R/W locks to keep the data consistent 
> between reads. The intent of this Jira is to remove the R/W locks by using 
> the persistent data structures library like - 
> [pcollections|https://github.com/hrldcpr/pcollections], 
> [Paguro|https://github.com/GlenKPeterson/Paguro] and 
> [Vavr|https://github.com/vavr-io/vavr] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)