Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.7 #181

[Apache Jenkins Server]

See 

Re: [DISCUSS] Apache Kafka 3.9.0 release

[Sophie Blee-Goldman]

+1, thank you Colin

Given the July freeze deadlines, I take it we are going with the "short
3.9.0 release" option and that the existence of this release will impact
the 4.0.0 deadlines which will still follow the usual schedule -- in other
words, this is an "additional release" outside of the regular timeline.  Is
this understanding correct?

On Fri, Jun 14, 2024 at 5:57 PM Chia-Ping Tsai  wrote:

> +1 thanks Colin!
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.8 #46

[Apache Jenkins Server]

See 

Re: [VOTE] 3.7.1 RC1

[Justine Olshan]

The import fix is in. (As well as the integration tag)
I did notice a large number of failing tests on the PR build though. Not
sure if some of these were fixed in trunk and if we want to pick up those
fixes.

Justine

On Fri, Jun 14, 2024 at 6:19 PM Matthias J. Sax  wrote:

> Replied on https://issues.apache.org/jira/browse/KAFKA-16960
>
> On 6/14/24 6:05 AM, Luke Chen wrote:
> > Hi all,
> >
> > After running system tests, here are failing tests:
> >
> > 1. kafkatest.tests.core.upgrade_test failed at lz4 and snappy tests:
> > KAFKA-16962 
> > 2.  StreamsUpgradeTest failed at "fromVersion"=2.2 ~ 2.5, but passed for
> > the rest of the versions: KAFKA-16960
> > 
> > 3. TestKRaftUpgrade failed at "fromVersion"=3.3 ~ 3.5 under combined
> KRaft:
> > KAFKA-16961 
> >
> > Need help to take a look, or maybe re-run in another environment.
> >
> > Thanks.
> > Luke
> >
> > On Fri, Jun 14, 2024 at 5:03 PM Edoardo Comar 
> wrote:
> >
> >> well if you still need to build RC2, I'd cherry pick this that I
> >> missed from the other PR
> >> https://github.com/apache/kafka/pull/16326
> >>
> >> On Thu, 13 Jun 2024 at 13:20, Igor Soarez  wrote:
> >>>
> >>> Hi Edoardo,
> >>>
> >>> It is late, but not too late. I have cherry-picked your change
> >>> to the 3.7 branch and I'll build a second release candidate.
> >>>
> >>> If you could have a look at the first RC, please let me know if
> >>> you spot any issues with it that can be avoided in the next RC.
> >>>
> >>> Thanks,
> >>>
> >>> --
> >>> Igor
> >>
> >
>

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.4 #177

[Apache Jenkins Server]

See 

Re: [VOTE] 3.7.1 RC1

[Matthias J. Sax]

Replied on https://issues.apache.org/jira/browse/KAFKA-16960

On 6/14/24 6:05 AM, Luke Chen wrote:

Hi all,

After running system tests, here are failing tests:

1. kafkatest.tests.core.upgrade_test failed at lz4 and snappy tests:
KAFKA-16962 
2.  StreamsUpgradeTest failed at "fromVersion"=2.2 ~ 2.5, but passed for
the rest of the versions: KAFKA-16960

3. TestKRaftUpgrade failed at "fromVersion"=3.3 ~ 3.5 under combined KRaft:
KAFKA-16961 

Need help to take a look, or maybe re-run in another environment.

Thanks.
Luke

On Fri, Jun 14, 2024 at 5:03 PM Edoardo Comar  wrote:


well if you still need to build RC2, I'd cherry pick this that I
missed from the other PR
https://github.com/apache/kafka/pull/16326

On Thu, 13 Jun 2024 at 13:20, Igor Soarez  wrote:


Hi Edoardo,

It is late, but not too late. I have cherry-picked your change
to the 3.7 branch and I'll build a second release candidate.

If you could have a look at the first RC, please let me know if
you spot any issues with it that can be avoided in the next RC.

Thanks,

--
Igor

[jira] [Created] (KAFKA-16968) Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-16968:


 Summary: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0
 Key: KAFKA-16968
 URL: https://issues.apache.org/jira/browse/KAFKA-16968
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.8.0
Reporter: Colin McCabe
Assignee: Colin McCabe


Before we can release 3.8, we must make 3.8-IV0 a stable MetadataVersion and 
create 3.9-IV0.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.9.0 release

[Chia-Ping Tsai]

+1 thanks Colin!

[jira] [Created] (KAFKA-16967) NioEchoServer fails to register connection and causes flaky failure

[Greg Harris (Jira)]

Greg Harris created KAFKA-16967:
---

 Summary: NioEchoServer fails to register connection and causes 
flaky failure
 Key: KAFKA-16967
 URL: https://issues.apache.org/jira/browse/KAFKA-16967
 Project: Kafka
  Issue Type: Task
  Components: core
Reporter: Greg Harris


The NioEchoServer calls Selector#register for new connections. This call can 
throw exceptions, which then kill the NioEchoServer. This has been observed in 
the SslTransportLayerTest testUngracefulRemoteCloseDuringHandshake* methods.
{noformat}
Exception in thread "echoserver" java.lang.IllegalStateException: There is 
already a connection for id 127.0.0.1:40007-127.0.0.1:43710
at 
org.apache.kafka.common.network.Selector.ensureNotRegistered(Selector.java:322)
at org.apache.kafka.common.network.Selector.register(Selector.java:310)
at 
org.apache.kafka.common.network.NioEchoServer.run(NioEchoServer.java:229){noformat}
This causes the test to fail with essentially a timeout, when the connection is 
expired for becoming idle unexpectedly:
{noformat}
org.opentest4j.AssertionFailedError: Unexpected channel state EXPIRED ==> 
expected:  but was: 
at 
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
    at 
org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:214)
at 
org.apache.kafka.common.network.SslTransportLayerTest.testIOExceptionsDuringHandshake(SslTransportLayerTest.java:898)
at 
org.apache.kafka.common.network.SslTransportLayerTest.testUngracefulRemoteCloseDuringHandshakeRead(SslTransportLayerTest.java:837){noformat}
Instead, the NioEchoServer should handle exceptions from register in a similar 
fashion to the SocketServer.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.8 #45

[Apache Jenkins Server]

See 

Re: [DISCUSS] Apache Kafka 3.9.0 release

[Colin McCabe]

Thanks, Matthias. I did a quick sweep just now and moved a bunch of things that 
have already been implemented in trunk into 3.9. Interestingly, more than half 
of the list seems to be "remove deprecated thing X", which is genuinely a 4.0 
issue :)

I suppose we will be grooming the 3.8 JIRA backlog soon as well. I expect most 
of that stuff to get shifted to 3.9 unless it truly is a blocker.

best,
Colin

On Fri, Jun 14, 2024, at 15:48, Matthias J. Sax wrote:
> +1
>
> Btw: I just looked into Jira, and we have 90 tickets with "fixed 
> version" set to 4.0.0. I would assume that most of them need to be 
> updated to 3.9.0?
>
> https://issues.apache.org/jira/browse/KAFKA-16924?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%204.0.0
>
> For anybody who has context on any of these tickets, please take a look 
> and update them.
>
> I think Colin as RM should double check this list shortly before the 3.9 
> release, and it should hopefully be much shorter.
>
>
> -Matthias
>
> On 6/14/24 12:32 PM, Ismael Juma wrote:
>> +1, thanks Colin.
>> 
>> Ismael
>> 
>> On Fri, Jun 14, 2024 at 10:55 AM Colin McCabe  wrote:
>> 
>>> Hi all,
>>>
>>> Based on the discussion in the 3.8 release thread, we will need a 3.9
>>> release. I guess I should create a new email thread to discuss this.
>>>
>>> I'd like to volunteer as release manager for the Apache Kafka 3.9.0
>>> release. There's a release plan here:
>>> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.9.0
>>>
>>> best,
>>> Colin
>>>
>>

Re: [DISCUSS] Apache Kafka 3.9.0 release

[Matthias J. Sax]

+1

Btw: I just looked into Jira, and we have 90 tickets with "fixed 
version" set to 4.0.0. I would assume that most of them need to be 
updated to 3.9.0?


https://issues.apache.org/jira/browse/KAFKA-16924?jql=project%20%3D%20KAFKA%20AND%20fixVersion%20%3D%204.0.0

For anybody who has context on any of these tickets, please take a look 
and update them.


I think Colin as RM should double check this list shortly before the 3.9 
release, and it should hopefully be much shorter.



-Matthias

On 6/14/24 12:32 PM, Ismael Juma wrote:

+1, thanks Colin.

Ismael

On Fri, Jun 14, 2024 at 10:55 AM Colin McCabe  wrote:


Hi all,

Based on the discussion in the 3.8 release thread, we will need a 3.9
release. I guess I should create a new email thread to discuss this.

I'd like to volunteer as release manager for the Apache Kafka 3.9.0
release. There's a release plan here:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.9.0

best,
Colin

Re: [DISCUSS] Apache Kafka 3.8.0 release

[Colin McCabe]

+1. I think it would be good to keep 4.0 time-based. Most of the refactors we 
want to do are optional in some sense and can be descoped if time runs out. For 
example, we can drop support for JDK 8 without immediately refactoring 
everything that could benefit from the improvements in JDK9+.

best,
Colin


On Fri, Jun 14, 2024, at 15:37, Matthias J. Sax wrote:
> That's my understanding, and I would advocate strongly to keep the 4.0 
> release schedule as planed originally.
>
> The 3.9 one should really be an additional "out of schedule" release not 
> impacting any other releases.
>
>
> -Matthias
>
> On 6/14/24 1:29 PM, David Jacot wrote:
>> The plan sounds good to me. I suppose that we will follow our regular
>> cadence for 4.0 and release it four months after 3.9 (in November?). Is
>> this correct?
>> 
>> Best,
>> David
>> 
>> Le ven. 14 juin 2024 à 21:57, José Armando García Sancio
>>  a écrit :
>> 
>>> +1 on the proposed release plan for 3.8.
>>>
>>> Thanks!
>>>
>>> On Fri, Jun 14, 2024 at 3:33 PM Ismael Juma  wrote:

 +1 to the plan we converged on in this thread.

 Ismael

 On Fri, Jun 14, 2024 at 10:46 AM Josep Prat >>>
 wrote:

> Hi all,
>
> Thanks Colin, yes go ahead.
>
> As we are now past code freeze I would like to ask everyone involved
>>> in a
> KIP that is not yet complete, to verify if what landed on the 3.8
>>> branch
> needs to be reverted or if it can stay. Additionally, I'll be pinging
>>> KIPs
> and Jira reporters asking for their status as some Jiras seem to have
>>> all
> related GitHub PRs merged but their status is still Open or In
>>> Progress.
> I'll be checking all the open blockers and check if they are really a
> blocker or can be pushed.
>
>
> Regarding timeline, I'll attempt to generate the first RC on Wednesday
>>> or
> Thursday, so please revert any changes you deem necessary by then. If
>>> you
> need more time, please ping me.
>
> Best,
>
> -
> Josep Prat
> Open Source Engineering Director, Aiven
> josep.p...@aiven.io   |   +491715557497 | aiven.io
> Aiven Deutschland GmbH
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>
> On Fri, Jun 14, 2024, 19:25 Colin McCabe  wrote:
>
>> Hi all,
>>
>> We have had many delays with releases this year. We should try to get
> back
>> on schedule.
>>
>> I agree with the idea that was proposed a few times in this thread of
>> drawing a line under 3.8 now, and doing a short 3.9 release. I
>>> posted a
> 3.9
>> release plan here:
>> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.9.0
>>
>> I think we could start doing RCs for 3.8.0 as early as next week.
>>> There
>> are a few things that need to be reverted first (anything related to
>> KIP-853 or KIP-966).
>>
>> Josep, if you agree, I will update KIP-1012 to reflect that these
>>> things
>> are landing in 3.9 rather than 3.8. And we can start doing all the
>>> normal
>> release stuff. The main blocker JIRA I'm aware of is KAFKA-16946,
>>> which
> is
>> a very simple fix.
>>
>> best,
>> Colin
>>
>>
>> On Fri, Jun 14, 2024, at 03:48, Satish Duggana wrote:
>>> +1 on going with 3.8 release with the existing plan and targeting
>>> the
>>> required features in 3.9 timelines. 4.0 will be targeted in the
>>> usual
>>> cycle(4 months) after 3.9 is released.
>>>
>>>
>>> On Fri, 14 Jun 2024 at 15:19, Edoardo Comar >>>
>> wrote:

 Josep,
 past the deadline sorry but I can't see reasons not to cherry-pick
> this
 https://github.com/apache/kafka/pull/16326

 On Wed, 12 Jun 2024 at 17:14, Josep Prat
>>> >
>> wrote:
>
> Hi Edoardo,
>
> Correct, you can still cherry-pick this one.
>
> I'll send an email tomorrow morning (CEST) asking maintainers to
> stop
> cherry picking commits unless we discuss it beforehand.
>
> Best,
>
> On Wed, Jun 12, 2024 at 6:09 PM Edoardo Comar <
> edoardli...@gmail.com>
>> wrote:
>
>> Hi Josep, I understand I am still in time to cherry-pick on
>>> 3.8.0
>> https://github.com/apache/kafka/pull/16229
>>
>> right?
>> thanks
>>
>> On Wed, 12 Jun 2024 at 11:34, Ivan Yurchenko 
>> wrote:
>>>
>>> Hi,
>>>
>>> I'll try to do all the fixes and changes for KIP-899 [1]
>>> sooner
>> today,
>> but please proceed with the release if I don't manage.
>>>
>>> Ivan
>>>
>>> [1] https://github.com/apache/kafka/pull/13277
>>>
>>> On Wed, Jun 12, 2024, at 12:54, Josep Prat wrote:

Re: [DISCUSS] KIP-1056: Remove `default.` prefix for exception handler StreamsConfig

[Matthias J. Sax]

*`DEFAULT_PRODUCTION_EXCEPTION_HANDLER_CLASS_DOC` is private and does
notneed to be covered in the KIP technically -- it's ok to just leave it
inof course.*

Yea, technically it has no implications, but as we are anyways touching
code around this, could it be better to rename ?


Yes, we should still rename it. I was just pointing out, that the KIP 
does technically not need to mention this as all, as it's an internal 
change.



I think you can start a VOTE for this KIP, too.



-Matthias


On 6/14/24 1:35 PM, Muralidhar Basani wrote:

Thanks Matthias and Nick.

Regarding the prefix "uncaught." is fine with me too, but any other
opinions on this one ?

*101 follow-up:*

*Given that `DESERIALIZATION_EXCEPTION_HAN**DLER_CLASS_DOC` is public we*
*cannot just rename it, but must follow the same deprecation pattern as*
*for the config name itself. (It seems it was added as public by mistake,*
*and the new variable for the _DOC string should be private...)*

Agree, as DESERIALIZATION_EXCEPTION_HANDLER_CLASS_DOC is public, we shall
deprecate it. Updated kip.



*`DEFAULT_PRODUCTION_EXCEPTION_HANDLER_CLASS_DOC` is private and does
notneed to be covered in the KIP technically -- it's ok to just leave it
inof course.*

Yea, technically it has no implications, but as we are anyways touching
code around this, could it be better to rename ?







*102 follow-up:I believe for other config, we actually use the new config
if it's set,and only fall back to the old config if the new one is not set.
If bothare set, the new one wins and we just log a WARN that the old one
isignored in favor of the new one. -- It might be best to stick
theestablished pattern?*

Agree, we should follow the same pattern. Updated kip.

Thanks,
Murali

On Fri, Jun 14, 2024 at 3:12 AM Matthias J. Sax  wrote:


Using `uncaught.` prefix would be fine with me. (Even if I find it a
little be redundant personally?)



101 follow-up:

Given that `DESERIALIZATION_EXCEPTION_HANDLER_CLASS_DOC` is public we
cannot just rename it, but must follow the same deprecation pattern as
for the config name itself. (It seems it was added as public by mistake,
and the new variable for the _DOC string should be private...)

-> actually compare KIP-1053 (old incorrect number 1052) "Align the
naming convention for config and default variables in *Config classes"
for a related discussion.


`DEFAULT_PRODUCTION_EXCEPTION_HANDLER_CLASS_DOC` is private and does not
need to be covered in the KIP technically -- it's ok to just leave it in
of course.




102 follow-up:


But if user sets both old and new configs, a ConfigException to be

thrown.

I believe for other config, we actually use the new config if it's set,
and only fall back to the old config if the new one is not set. If both
are set, the new one wins and we just log a WARN that the old one is
ignored in favor of the new one. -- It might be best to stick the
established pattern?



-Matthias




On 6/13/24 3:14 AM, Nick Telford wrote:

Hi everyone,

On a semantic note, would it perhaps make more sense to rename them "
uncaught." instead? These handlers are essentially the "last resort"
exception handlers, because Exceptions can be caught *within* a

component,

e.g. a Deserializer can catch and handle an exception without the
configured default.deserialization.exception.handler being invoked.

I think this would better align these with JVM norms, e.g.
Thread#setUncaughtExceptionHandler, which behaves the same (albeit
configured through code).

Regards,
Nick

On Thu, 13 Jun 2024 at 10:22, Muralidhar Basani
 wrote:


Thanks Matthias and Greg.

Have updated the KIP based on Matthias's comments.


100: Config names are part of the public interface, and the KIP should
not say "none" in this section, but call out which configs are
deprecated and which ones are newly added.


Updated kip.



101: Nit. In "Propose Changes" there is the template placeholder text


Describe the new thing you want to do in appropriate detail. This may

be

fairly extensive and have large subsections of its own. Or it may be a

few

sentences. Use judgement based on the scope of the change.

Similarly in "Test Plan" section



Updated kip.



102: The "Deprecation" section should explain the behavior if both,

old

and new configs, are set.


Updated kip. I think a ConfigException has to be thrown here if both
configs are set.

Thanks,
Murali

On Thu, Jun 13, 2024 at 2:28 AM Matthias J. Sax 

wrote:



Thanks Greg, this is a valid idea.

However, there was no demand in the past to allow for error handler
callbacks in a fine grained manner, and I am frankly also not sure if

it

would make sense or would be required.

Thus, I am not concerned about this case, and believe this KIP does

make

sense.

Happy to change my mind if somebody has a different opinion. We could
re-purpose this KIP to add per-topic error handler, too.



-Matthias

On 6/12/24 1:11 PM, Greg Harris wrote:

Hi Murali,

Thanks for the KIP!

I'm not familiar with Streams so 

Re: [DISCUSS] Apache Kafka 3.8.0 release

[Matthias J. Sax]

That's my understanding, and I would advocate strongly to keep the 4.0 
release schedule as planed originally.


The 3.9 one should really be an additional "out of schedule" release not 
impacting any other releases.



-Matthias

On 6/14/24 1:29 PM, David Jacot wrote:

The plan sounds good to me. I suppose that we will follow our regular
cadence for 4.0 and release it four months after 3.9 (in November?). Is
this correct?

Best,
David

Le ven. 14 juin 2024 à 21:57, José Armando García Sancio
 a écrit :


+1 on the proposed release plan for 3.8.

Thanks!

On Fri, Jun 14, 2024 at 3:33 PM Ismael Juma  wrote:


+1 to the plan we converged on in this thread.

Ismael

On Fri, Jun 14, 2024 at 10:46 AM Josep Prat 
Hi all,

Thanks Colin, yes go ahead.

As we are now past code freeze I would like to ask everyone involved

in a

KIP that is not yet complete, to verify if what landed on the 3.8

branch

needs to be reverted or if it can stay. Additionally, I'll be pinging

KIPs

and Jira reporters asking for their status as some Jiras seem to have

all

related GitHub PRs merged but their status is still Open or In

Progress.

I'll be checking all the open blockers and check if they are really a
blocker or can be pushed.


Regarding timeline, I'll attempt to generate the first RC on Wednesday

or

Thursday, so please revert any changes you deem necessary by then. If

you

need more time, please ping me.

Best,

-
Josep Prat
Open Source Engineering Director, Aiven
josep.p...@aiven.io   |   +491715557497 | aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

On Fri, Jun 14, 2024, 19:25 Colin McCabe  wrote:


Hi all,

We have had many delays with releases this year. We should try to get

back

on schedule.

I agree with the idea that was proposed a few times in this thread of
drawing a line under 3.8 now, and doing a short 3.9 release. I

posted a

3.9

release plan here:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.9.0

I think we could start doing RCs for 3.8.0 as early as next week.

There

are a few things that need to be reverted first (anything related to
KIP-853 or KIP-966).

Josep, if you agree, I will update KIP-1012 to reflect that these

things

are landing in 3.9 rather than 3.8. And we can start doing all the

normal

release stuff. The main blocker JIRA I'm aware of is KAFKA-16946,

which

is

a very simple fix.

best,
Colin


On Fri, Jun 14, 2024, at 03:48, Satish Duggana wrote:

+1 on going with 3.8 release with the existing plan and targeting

the

required features in 3.9 timelines. 4.0 will be targeted in the

usual

cycle(4 months) after 3.9 is released.


On Fri, 14 Jun 2024 at 15:19, Edoardo Comar 


wrote:


Josep,
past the deadline sorry but I can't see reasons not to cherry-pick

this

https://github.com/apache/kafka/pull/16326

On Wed, 12 Jun 2024 at 17:14, Josep Prat



wrote:


Hi Edoardo,

Correct, you can still cherry-pick this one.

I'll send an email tomorrow morning (CEST) asking maintainers to

stop

cherry picking commits unless we discuss it beforehand.

Best,

On Wed, Jun 12, 2024 at 6:09 PM Edoardo Comar <

edoardli...@gmail.com>

wrote:



Hi Josep, I understand I am still in time to cherry-pick on

3.8.0

https://github.com/apache/kafka/pull/16229

right?
thanks

On Wed, 12 Jun 2024 at 11:34, Ivan Yurchenko 

wrote:


Hi,

I'll try to do all the fixes and changes for KIP-899 [1]

sooner

today,

but please proceed with the release if I don't manage.


Ivan

[1] https://github.com/apache/kafka/pull/13277

On Wed, Jun 12, 2024, at 12:54, Josep Prat wrote:

Hi Luke,
I think Jose, also mentioned that it won't be ready for

v3.8.0

(but he

can

confirm this). My question now would be, given that it

seems

we

would

need

a v3.9.0, do you think it's important to include
https://github.com/apache/kafka/pull/16284 in v3.8.0?

Best,

On Wed, Jun 12, 2024 at 11:40 AM Luke Chen <

show...@gmail.com


wrote:



Hi Josep

For KIP-966, I think Calvin had mentioned he won't

complete

in

v3.8.0.



https://lists.apache.org/thread/fsnr8wy5fznzfso7jgk90skgyo277fmw


For unclean leader election, all we need is this PR:
https://github.com/apache/kafka/pull/16284
For this PR, I think it needs one more week to be

completed.


Thanks.
Luke

On Wed, Jun 12, 2024 at 4:51 PM Josep Prat



wrote:


Hi all,

We are now really close to the planned code freeze

deadline

(today

EOD).

According to KIP-1012 [1] we agreed to stay in the 3.x

branch

until we

achieve feature parity regarding Zookeeper and KRaft.

The

two main

KIPs

identified that would achieve this are: KIP-853 [2]

and

KIP-966

[3].

At the moment of writing this email both KIPs are not

completed. My

question to the people driving both KIPs would be, how

much

more

time do

you think it's needed to bring these KIPs to

completion?


- If the time needed would be short, we could still

Re: [QUESTION] What to do about conflicted KIP numbers?

[Matthias J. Sax]

I don't think that there is an official guideline.

Personally, I would suggest that the corresponding KIP owners agree who 
is keeping the conflicting number, and how is changing it.


For the ones changing the number, I would propose to restart a new 
DISCUSS thread using the new number to separate the KIP threads.


Not sure if the is a better way to handle this... Just an idea on how I 
would do it.



Not sure if we can improve the wiki instruction to make the race 
condition less likely? Seems, this would happen if two people look at 
the next KIP number let's say X, but don't bump it right way to X+1 and 
publish their KIP with X a few hours/days later without verifying that X 
is still next available KIP number?



-Matthias


On 6/14/24 3:10 PM, Welch, Matt wrote:

Hi Kafka devs,

I submitted a KIP last week and encountered a KIP-process race condition where my KIP 
number was consumed by another dev without updating the wiki page containing KIPs: 
Kafka Improvement Proposals - Apache Kafka - Apache Software 
Foundation

There are now least three separate dev-list threads referencing this conflicted 
KIP number so I'm concerned that discussion around this number will now be 
permanently confusing due to the conflict and multiple concurrent unrelated 
threads referencing the same KIP number.  I've intentionally kept the KIP 
numbers out of this email to prevent yet another thread referencing them.

While I'm happy to keep going with my existing KIP number, I was wondering if I should 
"abandon" it and create a new one.
This solution seems like it could create extra confusion, however, so what is 
the best course of action here?

Thanks,
Matt

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.7 #180

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 2871 lines...]
[2024-06-14T22:12:47.275Z] [Warn] 
/home/jenkins/workspace/Kafka_kafka_3.7/streams/streams-scala/src/main/scala/org/apache/kafka/streams/scala/kstream/KStream.scala:722:4:
 Usage of named or default arguments transformed this annotation
[2024-06-14T22:12:47.275Z] constructor call into a block. The corresponding 
AnnotationInfo
[2024-06-14T22:12:47.275Z] will contain references to local values and default 
getters instead
[2024-06-14T22:12:47.275Z] of the actual argument trees
[2024-06-14T22:12:47.275Z] [Warn] 
/home/jenkins/workspace/Kafka_kafka_3.7/streams/streams-scala/src/main/scala/org/apache/kafka/streams/scala/kstream/KStream.scala:744:4:
 Usage of named or default arguments transformed this annotation
[2024-06-14T22:12:47.275Z] constructor call into a block. The corresponding 
AnnotationInfo
[2024-06-14T22:12:47.275Z] will contain references to local values and default 
getters instead
[2024-06-14T22:12:47.275Z] of the actual argument trees
[2024-06-14T22:12:47.275Z] [Warn] 
/home/jenkins/workspace/Kafka_kafka_3.7/streams/streams-scala/src/main/scala/org/apache/kafka/streams/scala/kstream/KStream.scala:766:4:
 Usage of named or default arguments transformed this annotation
[2024-06-14T22:12:47.275Z] constructor call into a block. The corresponding 
AnnotationInfo
[2024-06-14T22:12:47.275Z] will contain references to local values and default 
getters instead
[2024-06-14T22:12:47.275Z] of the actual argument trees
[2024-06-14T22:12:47.275Z] [Warn] 
/home/jenkins/workspace/Kafka_kafka_3.7/streams/streams-scala/src/main/scala/org/apache/kafka/streams/scala/kstream/KStream.scala:788:4:
 Usage of named or default arguments transformed this annotation
[2024-06-14T22:12:47.275Z] constructor call into a block. The corresponding 
AnnotationInfo
[2024-06-14T22:12:47.275Z] will contain references to local values and default 
getters instead
[2024-06-14T22:12:47.275Z] of the actual argument trees
[2024-06-14T22:12:47.275Z] [Warn] 
/home/jenkins/workspace/Kafka_kafka_3.7/streams/streams-scala/src/main/scala/org/apache/kafka/streams/scala/kstream/KStream.scala:810:4:
 Usage of named or default arguments transformed this annotation
[2024-06-14T22:12:47.275Z] constructor call into a block. The corresponding 
AnnotationInfo
[2024-06-14T22:12:47.275Z] will contain references to local values and default 
getters instead
[2024-06-14T22:12:47.275Z] of the actual argument trees
[2024-06-14T22:12:47.275Z] [Warn] 
/home/jenkins/workspace/Kafka_kafka_3.7/streams/streams-scala/src/main/scala/org/apache/kafka/streams/scala/kstream/KStream.scala:830:4:
 Usage of named or default arguments transformed this annotation
[2024-06-14T22:12:47.275Z] constructor call into a block. The corresponding 
AnnotationInfo
[2024-06-14T22:12:47.275Z] will contain references to local values and default 
getters instead
[2024-06-14T22:12:47.275Z] of the actual argument trees
[2024-06-14T22:12:47.275Z] [Warn] 
/home/jenkins/workspace/Kafka_kafka_3.7/streams/streams-scala/src/main/scala/org/apache/kafka/streams/scala/kstream/KStream.scala:852:4:
 Usage of named or default arguments transformed this annotation
[2024-06-14T22:12:47.275Z] constructor call into a block. The corresponding 
AnnotationInfo
[2024-06-14T22:12:47.275Z] will contain references to local values and default 
getters instead
[2024-06-14T22:12:47.275Z] of the actual argument trees
[2024-06-14T22:12:57.609Z] 
[2024-06-14T22:12:57.609Z] > Task :metadata:testClasses
[2024-06-14T22:12:57.609Z] > Task :metadata:spotbugsTest SKIPPED
[2024-06-14T22:13:13.248Z] 
[2024-06-14T22:13:13.248Z] > Task :core:compileScala
[2024-06-14T22:13:13.248Z] [Warn] 
/home/jenkins/workspace/Kafka_kafka_3.7/core/src/main/scala/kafka/admin/FeatureCommand.scala:20:2:
 Usage of named or default arguments transformed this annotation
[2024-06-14T22:13:13.248Z] constructor call into a block. The corresponding 
AnnotationInfo
[2024-06-14T22:13:13.248Z] will contain references to local values and default 
getters instead
[2024-06-14T22:13:13.248Z] of the actual argument trees
[2024-06-14T22:13:13.248Z] 
[2024-06-14T22:13:13.248Z] > Task :group-coordinator:checkstyleTest
[2024-06-14T22:13:17.526Z] > Task :group-coordinator:check
[2024-06-14T22:13:27.592Z] > Task :metadata:checkstyleTest
[2024-06-14T22:13:33.381Z] 
[2024-06-14T22:13:33.381Z] > Task :core:compileScala
[2024-06-14T22:13:33.381Z] [Warn] 
/home/jenkins/workspace/Kafka_kafka_3.7/core/src/main/scala/kafka/coordinator/group/MemberMetadata.scala:31:16:
 private object MemberMetadata in package group is never used
[2024-06-14T22:13:36.474Z] 
[2024-06-14T22:13:36.474Z] > Task :streams:streams-scala:compileScala
[2024-06-14T22:13:36.474Z] 14 warnings found
[2024-06-14T22:13:38.920Z] 
[2024-06-14T22:13:38.920Z] > Task :streams:streams-scala:classes
[2024-06-14T22:13:38.920Z] > Task 

[QUESTION] What to do about conflicted KIP numbers?

[Welch, Matt]

Hi Kafka devs,

I submitted a KIP last week and encountered a KIP-process race condition where 
my KIP number was consumed by another dev without updating the wiki page 
containing KIPs: Kafka Improvement Proposals - Apache Kafka - Apache Software 
Foundation

There are now least three separate dev-list threads referencing this conflicted 
KIP number so I'm concerned that discussion around this number will now be 
permanently confusing due to the conflict and multiple concurrent unrelated 
threads referencing the same KIP number.  I've intentionally kept the KIP 
numbers out of this email to prevent yet another thread referencing them.

While I'm happy to keep going with my existing KIP number, I was wondering if I 
should "abandon" it and create a new one.
This solution seems like it could create extra confusion, however, so what is 
the best course of action here?

Thanks,
Matt

Re: [DISCUSS] KIP-1056: Remove `default.` prefix for exception handler StreamsConfig

[Muralidhar Basani]

Thanks Matthias and Nick.

Regarding the prefix "uncaught." is fine with me too, but any other
opinions on this one ?

*101 follow-up:*

*Given that `DESERIALIZATION_EXCEPTION_HAN**DLER_CLASS_DOC` is public we*
*cannot just rename it, but must follow the same deprecation pattern as*
*for the config name itself. (It seems it was added as public by mistake,*
*and the new variable for the _DOC string should be private...)*

Agree, as DESERIALIZATION_EXCEPTION_HANDLER_CLASS_DOC is public, we shall
deprecate it. Updated kip.



*`DEFAULT_PRODUCTION_EXCEPTION_HANDLER_CLASS_DOC` is private and does
notneed to be covered in the KIP technically -- it's ok to just leave it
inof course.*

Yea, technically it has no implications, but as we are anyways touching
code around this, could it be better to rename ?







*102 follow-up:I believe for other config, we actually use the new config
if it's set,and only fall back to the old config if the new one is not set.
If bothare set, the new one wins and we just log a WARN that the old one
isignored in favor of the new one. -- It might be best to stick
theestablished pattern?*

Agree, we should follow the same pattern. Updated kip.

Thanks,
Murali

On Fri, Jun 14, 2024 at 3:12 AM Matthias J. Sax  wrote:

> Using `uncaught.` prefix would be fine with me. (Even if I find it a
> little be redundant personally?)
>
>
>
> 101 follow-up:
>
> Given that `DESERIALIZATION_EXCEPTION_HANDLER_CLASS_DOC` is public we
> cannot just rename it, but must follow the same deprecation pattern as
> for the config name itself. (It seems it was added as public by mistake,
> and the new variable for the _DOC string should be private...)
>
> -> actually compare KIP-1053 (old incorrect number 1052) "Align the
> naming convention for config and default variables in *Config classes"
> for a related discussion.
>
>
> `DEFAULT_PRODUCTION_EXCEPTION_HANDLER_CLASS_DOC` is private and does not
> need to be covered in the KIP technically -- it's ok to just leave it in
> of course.
>
>
>
>
> 102 follow-up:
>
> > But if user sets both old and new configs, a ConfigException to be
> thrown.
>
> I believe for other config, we actually use the new config if it's set,
> and only fall back to the old config if the new one is not set. If both
> are set, the new one wins and we just log a WARN that the old one is
> ignored in favor of the new one. -- It might be best to stick the
> established pattern?
>
>
>
> -Matthias
>
>
>
>
> On 6/13/24 3:14 AM, Nick Telford wrote:
> > Hi everyone,
> >
> > On a semantic note, would it perhaps make more sense to rename them "
> > uncaught." instead? These handlers are essentially the "last resort"
> > exception handlers, because Exceptions can be caught *within* a
> component,
> > e.g. a Deserializer can catch and handle an exception without the
> > configured default.deserialization.exception.handler being invoked.
> >
> > I think this would better align these with JVM norms, e.g.
> > Thread#setUncaughtExceptionHandler, which behaves the same (albeit
> > configured through code).
> >
> > Regards,
> > Nick
> >
> > On Thu, 13 Jun 2024 at 10:22, Muralidhar Basani
> >  wrote:
> >
> >> Thanks Matthias and Greg.
> >>
> >> Have updated the KIP based on Matthias's comments.
> >>
>  100: Config names are part of the public interface, and the KIP should
>  not say "none" in this section, but call out which configs are
>  deprecated and which ones are newly added.
> >>
> >> Updated kip.
> >>
> >>
>  101: Nit. In "Propose Changes" there is the template placeholder text
> 
> > Describe the new thing you want to do in appropriate detail. This may
> >> be
>  fairly extensive and have large subsections of its own. Or it may be a
> >> few
>  sentences. Use judgement based on the scope of the change.
> 
>  Similarly in "Test Plan" section
> 
> >>
> >> Updated kip.
> >>
> >>
>  102: The "Deprecation" section should explain the behavior if both,
> old
>  and new configs, are set.
> >>
> >> Updated kip. I think a ConfigException has to be thrown here if both
> >> configs are set.
> >>
> >> Thanks,
> >> Murali
> >>
> >> On Thu, Jun 13, 2024 at 2:28 AM Matthias J. Sax 
> wrote:
> >>
> >>> Thanks Greg, this is a valid idea.
> >>>
> >>> However, there was no demand in the past to allow for error handler
> >>> callbacks in a fine grained manner, and I am frankly also not sure if
> it
> >>> would make sense or would be required.
> >>>
> >>> Thus, I am not concerned about this case, and believe this KIP does
> make
> >>> sense.
> >>>
> >>> Happy to change my mind if somebody has a different opinion. We could
> >>> re-purpose this KIP to add per-topic error handler, too.
> >>>
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 6/12/24 1:11 PM, Greg Harris wrote:
>  Hi Murali,
> 
>  Thanks for the KIP!
> 
>  I'm not familiar with Streams so I'll pose a general question, open
> for
>  anyone to answer:
> 
>  The configs that are being 

Re: [DISCUSS] Apache Kafka 3.8.0 release

[David Jacot]

The plan sounds good to me. I suppose that we will follow our regular
cadence for 4.0 and release it four months after 3.9 (in November?). Is
this correct?

Best,
David

Le ven. 14 juin 2024 à 21:57, José Armando García Sancio
 a écrit :

> +1 on the proposed release plan for 3.8.
>
> Thanks!
>
> On Fri, Jun 14, 2024 at 3:33 PM Ismael Juma  wrote:
> >
> > +1 to the plan we converged on in this thread.
> >
> > Ismael
> >
> > On Fri, Jun 14, 2024 at 10:46 AM Josep Prat  >
> > wrote:
> >
> > > Hi all,
> > >
> > > Thanks Colin, yes go ahead.
> > >
> > > As we are now past code freeze I would like to ask everyone involved
> in a
> > > KIP that is not yet complete, to verify if what landed on the 3.8
> branch
> > > needs to be reverted or if it can stay. Additionally, I'll be pinging
> KIPs
> > > and Jira reporters asking for their status as some Jiras seem to have
> all
> > > related GitHub PRs merged but their status is still Open or In
> Progress.
> > > I'll be checking all the open blockers and check if they are really a
> > > blocker or can be pushed.
> > >
> > >
> > > Regarding timeline, I'll attempt to generate the first RC on Wednesday
> or
> > > Thursday, so please revert any changes you deem necessary by then. If
> you
> > > need more time, please ping me.
> > >
> > > Best,
> > >
> > > -
> > > Josep Prat
> > > Open Source Engineering Director, Aiven
> > > josep.p...@aiven.io   |   +491715557497 | aiven.io
> > > Aiven Deutschland GmbH
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > On Fri, Jun 14, 2024, 19:25 Colin McCabe  wrote:
> > >
> > > > Hi all,
> > > >
> > > > We have had many delays with releases this year. We should try to get
> > > back
> > > > on schedule.
> > > >
> > > > I agree with the idea that was proposed a few times in this thread of
> > > > drawing a line under 3.8 now, and doing a short 3.9 release. I
> posted a
> > > 3.9
> > > > release plan here:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.9.0
> > > >
> > > > I think we could start doing RCs for 3.8.0 as early as next week.
> There
> > > > are a few things that need to be reverted first (anything related to
> > > > KIP-853 or KIP-966).
> > > >
> > > > Josep, if you agree, I will update KIP-1012 to reflect that these
> things
> > > > are landing in 3.9 rather than 3.8. And we can start doing all the
> normal
> > > > release stuff. The main blocker JIRA I'm aware of is KAFKA-16946,
> which
> > > is
> > > > a very simple fix.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Fri, Jun 14, 2024, at 03:48, Satish Duggana wrote:
> > > > > +1 on going with 3.8 release with the existing plan and targeting
> the
> > > > > required features in 3.9 timelines. 4.0 will be targeted in the
> usual
> > > > > cycle(4 months) after 3.9 is released.
> > > > >
> > > > >
> > > > > On Fri, 14 Jun 2024 at 15:19, Edoardo Comar  >
> > > > wrote:
> > > > >>
> > > > >> Josep,
> > > > >> past the deadline sorry but I can't see reasons not to cherry-pick
> > > this
> > > > >> https://github.com/apache/kafka/pull/16326
> > > > >>
> > > > >> On Wed, 12 Jun 2024 at 17:14, Josep Prat
>  > > >
> > > > wrote:
> > > > >> >
> > > > >> > Hi Edoardo,
> > > > >> >
> > > > >> > Correct, you can still cherry-pick this one.
> > > > >> >
> > > > >> > I'll send an email tomorrow morning (CEST) asking maintainers to
> > > stop
> > > > >> > cherry picking commits unless we discuss it beforehand.
> > > > >> >
> > > > >> > Best,
> > > > >> >
> > > > >> > On Wed, Jun 12, 2024 at 6:09 PM Edoardo Comar <
> > > edoardli...@gmail.com>
> > > > wrote:
> > > > >> >
> > > > >> > > Hi Josep, I understand I am still in time to cherry-pick on
> 3.8.0
> > > > >> > > https://github.com/apache/kafka/pull/16229
> > > > >> > >
> > > > >> > > right?
> > > > >> > > thanks
> > > > >> > >
> > > > >> > > On Wed, 12 Jun 2024 at 11:34, Ivan Yurchenko 
> > > > wrote:
> > > > >> > > >
> > > > >> > > > Hi,
> > > > >> > > >
> > > > >> > > > I'll try to do all the fixes and changes for KIP-899 [1]
> sooner
> > > > today,
> > > > >> > > but please proceed with the release if I don't manage.
> > > > >> > > >
> > > > >> > > > Ivan
> > > > >> > > >
> > > > >> > > > [1] https://github.com/apache/kafka/pull/13277
> > > > >> > > >
> > > > >> > > > On Wed, Jun 12, 2024, at 12:54, Josep Prat wrote:
> > > > >> > > > > Hi Luke,
> > > > >> > > > > I think Jose, also mentioned that it won't be ready for
> v3.8.0
> > > > (but he
> > > > >> > > can
> > > > >> > > > > confirm this). My question now would be, given that it
> seems
> > > we
> > > > would
> > > > >> > > need
> > > > >> > > > > a v3.9.0, do you think it's important to include
> > > > >> > > > > https://github.com/apache/kafka/pull/16284 in v3.8.0?
> > > > >> > > > >
> > > > >> > > > > Best,
> > > > >> > > > >
> > > > >> > > > > On Wed, Jun 12, 2024 at 11:40 AM Luke Chen <
> show...@gmail.com
> > > >

[jira] [Resolved] (KAFKA-16946) Utils.getHost/getPort cannot parse SASL_PLAINTEXT://host:port

[Colin McCabe (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin McCabe resolved KAFKA-16946.
--
Resolution: Fixed

> Utils.getHost/getPort cannot parse SASL_PLAINTEXT://host:port
> -
>
> Key: KAFKA-16946
> URL: https://issues.apache.org/jira/browse/KAFKA-16946
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.8.0
>Reporter: Luke Chen
>Assignee: TengYao Chi
>Priority: Blocker
> Fix For: 3.8.0
>
>
> In KAFKA-16824, we tried to improve the regex for Utils.getHost/getPort, but 
> it failed to parse SASL_PLAINTEXT://host:port now. Need to fix it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.8.0 release

[José Armando García Sancio]

+1 on the proposed release plan for 3.8.

Thanks!

On Fri, Jun 14, 2024 at 3:33 PM Ismael Juma  wrote:
>
> +1 to the plan we converged on in this thread.
>
> Ismael
>
> On Fri, Jun 14, 2024 at 10:46 AM Josep Prat 
> wrote:
>
> > Hi all,
> >
> > Thanks Colin, yes go ahead.
> >
> > As we are now past code freeze I would like to ask everyone involved in a
> > KIP that is not yet complete, to verify if what landed on the 3.8 branch
> > needs to be reverted or if it can stay. Additionally, I'll be pinging KIPs
> > and Jira reporters asking for their status as some Jiras seem to have all
> > related GitHub PRs merged but their status is still Open or In Progress.
> > I'll be checking all the open blockers and check if they are really a
> > blocker or can be pushed.
> >
> >
> > Regarding timeline, I'll attempt to generate the first RC on Wednesday or
> > Thursday, so please revert any changes you deem necessary by then. If you
> > need more time, please ping me.
> >
> > Best,
> >
> > -
> > Josep Prat
> > Open Source Engineering Director, Aiven
> > josep.p...@aiven.io   |   +491715557497 | aiven.io
> > Aiven Deutschland GmbH
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > On Fri, Jun 14, 2024, 19:25 Colin McCabe  wrote:
> >
> > > Hi all,
> > >
> > > We have had many delays with releases this year. We should try to get
> > back
> > > on schedule.
> > >
> > > I agree with the idea that was proposed a few times in this thread of
> > > drawing a line under 3.8 now, and doing a short 3.9 release. I posted a
> > 3.9
> > > release plan here:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.9.0
> > >
> > > I think we could start doing RCs for 3.8.0 as early as next week. There
> > > are a few things that need to be reverted first (anything related to
> > > KIP-853 or KIP-966).
> > >
> > > Josep, if you agree, I will update KIP-1012 to reflect that these things
> > > are landing in 3.9 rather than 3.8. And we can start doing all the normal
> > > release stuff. The main blocker JIRA I'm aware of is KAFKA-16946, which
> > is
> > > a very simple fix.
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Fri, Jun 14, 2024, at 03:48, Satish Duggana wrote:
> > > > +1 on going with 3.8 release with the existing plan and targeting the
> > > > required features in 3.9 timelines. 4.0 will be targeted in the usual
> > > > cycle(4 months) after 3.9 is released.
> > > >
> > > >
> > > > On Fri, 14 Jun 2024 at 15:19, Edoardo Comar 
> > > wrote:
> > > >>
> > > >> Josep,
> > > >> past the deadline sorry but I can't see reasons not to cherry-pick
> > this
> > > >> https://github.com/apache/kafka/pull/16326
> > > >>
> > > >> On Wed, 12 Jun 2024 at 17:14, Josep Prat  > >
> > > wrote:
> > > >> >
> > > >> > Hi Edoardo,
> > > >> >
> > > >> > Correct, you can still cherry-pick this one.
> > > >> >
> > > >> > I'll send an email tomorrow morning (CEST) asking maintainers to
> > stop
> > > >> > cherry picking commits unless we discuss it beforehand.
> > > >> >
> > > >> > Best,
> > > >> >
> > > >> > On Wed, Jun 12, 2024 at 6:09 PM Edoardo Comar <
> > edoardli...@gmail.com>
> > > wrote:
> > > >> >
> > > >> > > Hi Josep, I understand I am still in time to cherry-pick on 3.8.0
> > > >> > > https://github.com/apache/kafka/pull/16229
> > > >> > >
> > > >> > > right?
> > > >> > > thanks
> > > >> > >
> > > >> > > On Wed, 12 Jun 2024 at 11:34, Ivan Yurchenko 
> > > wrote:
> > > >> > > >
> > > >> > > > Hi,
> > > >> > > >
> > > >> > > > I'll try to do all the fixes and changes for KIP-899 [1] sooner
> > > today,
> > > >> > > but please proceed with the release if I don't manage.
> > > >> > > >
> > > >> > > > Ivan
> > > >> > > >
> > > >> > > > [1] https://github.com/apache/kafka/pull/13277
> > > >> > > >
> > > >> > > > On Wed, Jun 12, 2024, at 12:54, Josep Prat wrote:
> > > >> > > > > Hi Luke,
> > > >> > > > > I think Jose, also mentioned that it won't be ready for v3.8.0
> > > (but he
> > > >> > > can
> > > >> > > > > confirm this). My question now would be, given that it seems
> > we
> > > would
> > > >> > > need
> > > >> > > > > a v3.9.0, do you think it's important to include
> > > >> > > > > https://github.com/apache/kafka/pull/16284 in v3.8.0?
> > > >> > > > >
> > > >> > > > > Best,
> > > >> > > > >
> > > >> > > > > On Wed, Jun 12, 2024 at 11:40 AM Luke Chen  > >
> > > wrote:
> > > >> > > > >
> > > >> > > > > > Hi Josep
> > > >> > > > > >
> > > >> > > > > > For KIP-966, I think Calvin had mentioned he won't complete
> > in
> > > >> > > v3.8.0.
> > > >> > > > > >
> > > https://lists.apache.org/thread/fsnr8wy5fznzfso7jgk90skgyo277fmw
> > > >> > > > > >
> > > >> > > > > > For unclean leader election, all we need is this PR:
> > > >> > > > > > https://github.com/apache/kafka/pull/16284
> > > >> > > > > > For this PR, I think it needs one more week to be completed.
> > > >> > > > > >
> > > >> > > > > > Thanks.
> > > >> > > > > > 

Re: [DISCUSS] Apache Kafka 3.9.0 release

[Ismael Juma]

+1, thanks Colin.

Ismael

On Fri, Jun 14, 2024 at 10:55 AM Colin McCabe  wrote:

> Hi all,
>
> Based on the discussion in the 3.8 release thread, we will need a 3.9
> release. I guess I should create a new email thread to discuss this.
>
> I'd like to volunteer as release manager for the Apache Kafka 3.9.0
> release. There's a release plan here:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.9.0
>
> best,
> Colin
>

Re: [DISCUSS] Apache Kafka 3.8.0 release

[Ismael Juma]

+1 to the plan we converged on in this thread.

Ismael

On Fri, Jun 14, 2024 at 10:46 AM Josep Prat 
wrote:

> Hi all,
>
> Thanks Colin, yes go ahead.
>
> As we are now past code freeze I would like to ask everyone involved in a
> KIP that is not yet complete, to verify if what landed on the 3.8 branch
> needs to be reverted or if it can stay. Additionally, I'll be pinging KIPs
> and Jira reporters asking for their status as some Jiras seem to have all
> related GitHub PRs merged but their status is still Open or In Progress.
> I'll be checking all the open blockers and check if they are really a
> blocker or can be pushed.
>
>
> Regarding timeline, I'll attempt to generate the first RC on Wednesday or
> Thursday, so please revert any changes you deem necessary by then. If you
> need more time, please ping me.
>
> Best,
>
> -
> Josep Prat
> Open Source Engineering Director, Aiven
> josep.p...@aiven.io   |   +491715557497 | aiven.io
> Aiven Deutschland GmbH
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>
> On Fri, Jun 14, 2024, 19:25 Colin McCabe  wrote:
>
> > Hi all,
> >
> > We have had many delays with releases this year. We should try to get
> back
> > on schedule.
> >
> > I agree with the idea that was proposed a few times in this thread of
> > drawing a line under 3.8 now, and doing a short 3.9 release. I posted a
> 3.9
> > release plan here:
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.9.0
> >
> > I think we could start doing RCs for 3.8.0 as early as next week. There
> > are a few things that need to be reverted first (anything related to
> > KIP-853 or KIP-966).
> >
> > Josep, if you agree, I will update KIP-1012 to reflect that these things
> > are landing in 3.9 rather than 3.8. And we can start doing all the normal
> > release stuff. The main blocker JIRA I'm aware of is KAFKA-16946, which
> is
> > a very simple fix.
> >
> > best,
> > Colin
> >
> >
> > On Fri, Jun 14, 2024, at 03:48, Satish Duggana wrote:
> > > +1 on going with 3.8 release with the existing plan and targeting the
> > > required features in 3.9 timelines. 4.0 will be targeted in the usual
> > > cycle(4 months) after 3.9 is released.
> > >
> > >
> > > On Fri, 14 Jun 2024 at 15:19, Edoardo Comar 
> > wrote:
> > >>
> > >> Josep,
> > >> past the deadline sorry but I can't see reasons not to cherry-pick
> this
> > >> https://github.com/apache/kafka/pull/16326
> > >>
> > >> On Wed, 12 Jun 2024 at 17:14, Josep Prat  >
> > wrote:
> > >> >
> > >> > Hi Edoardo,
> > >> >
> > >> > Correct, you can still cherry-pick this one.
> > >> >
> > >> > I'll send an email tomorrow morning (CEST) asking maintainers to
> stop
> > >> > cherry picking commits unless we discuss it beforehand.
> > >> >
> > >> > Best,
> > >> >
> > >> > On Wed, Jun 12, 2024 at 6:09 PM Edoardo Comar <
> edoardli...@gmail.com>
> > wrote:
> > >> >
> > >> > > Hi Josep, I understand I am still in time to cherry-pick on 3.8.0
> > >> > > https://github.com/apache/kafka/pull/16229
> > >> > >
> > >> > > right?
> > >> > > thanks
> > >> > >
> > >> > > On Wed, 12 Jun 2024 at 11:34, Ivan Yurchenko 
> > wrote:
> > >> > > >
> > >> > > > Hi,
> > >> > > >
> > >> > > > I'll try to do all the fixes and changes for KIP-899 [1] sooner
> > today,
> > >> > > but please proceed with the release if I don't manage.
> > >> > > >
> > >> > > > Ivan
> > >> > > >
> > >> > > > [1] https://github.com/apache/kafka/pull/13277
> > >> > > >
> > >> > > > On Wed, Jun 12, 2024, at 12:54, Josep Prat wrote:
> > >> > > > > Hi Luke,
> > >> > > > > I think Jose, also mentioned that it won't be ready for v3.8.0
> > (but he
> > >> > > can
> > >> > > > > confirm this). My question now would be, given that it seems
> we
> > would
> > >> > > need
> > >> > > > > a v3.9.0, do you think it's important to include
> > >> > > > > https://github.com/apache/kafka/pull/16284 in v3.8.0?
> > >> > > > >
> > >> > > > > Best,
> > >> > > > >
> > >> > > > > On Wed, Jun 12, 2024 at 11:40 AM Luke Chen  >
> > wrote:
> > >> > > > >
> > >> > > > > > Hi Josep
> > >> > > > > >
> > >> > > > > > For KIP-966, I think Calvin had mentioned he won't complete
> in
> > >> > > v3.8.0.
> > >> > > > > >
> > https://lists.apache.org/thread/fsnr8wy5fznzfso7jgk90skgyo277fmw
> > >> > > > > >
> > >> > > > > > For unclean leader election, all we need is this PR:
> > >> > > > > > https://github.com/apache/kafka/pull/16284
> > >> > > > > > For this PR, I think it needs one more week to be completed.
> > >> > > > > >
> > >> > > > > > Thanks.
> > >> > > > > > Luke
> > >> > > > > >
> > >> > > > > > On Wed, Jun 12, 2024 at 4:51 PM Josep Prat
> > >> > > 
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi all,
> > >> > > > > > >
> > >> > > > > > > We are now really close to the planned code freeze
> deadline
> > (today
> > >> > > EOD).
> > >> > > > > > > According to KIP-1012 [1] we agreed to stay in the 3.x
> > branch
> > >> > > until we

Re: [VOTE] KIP-1017: A health check endpoint for Kafka Connect

[Greg Harris]

Hey Chris,

Thanks for the KIP.

I think demonstrating the health of the tick thread through a longer
request is going to be more reliable than caching the health and status in
order to respond quickly to requests. It should also be a natural negative
feedback loop, so that when the herder tick thread is busy, fewer health
requests are processed, alleviating some load.

+1 (binding)

Thanks!
Greg

On Fri, Jun 14, 2024 at 8:44 AM Hector Geraldino (BLOOMBERG/ 919 3RD A) <
hgerald...@bloomberg.net> wrote:

> Thanks Chris,
>
> +1 (non-binding)
>
> From: dev@kafka.apache.org At: 06/14/24 11:43:02 UTC-4:00To:
> dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-1017: A health check endpoint for Kafka Connect
>
> Hi Chris,
>
> +1 (binding)
>
> Thanks,
> Mickael
>
> On Fri, Jun 14, 2024 at 5:01 PM Andrew Schofield
>  wrote:
> >
> > Hi Chris,
> > Thanks for the KIP.
> >
> > +1 (non-binding)
> >
> > Thanks,
> > Andrew
> >
> > > On 14 Jun 2024, at 15:48, Chris Egerton 
> wrote:
> > >
> > > Hi all,
> > >
> > > Happy Friday! I'd like to kick off a vote on KIP-1017.
> > >
> > > Design doc:
> > >
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1017%3A+Health+check+endpo
> int+for+Kafka+Connect
> > >
> > > Discussion thread:
> > > https://lists.apache.org/thread/95nto84wtogdgk3v97rxcwxr258wnjnt
> > >
> > > Cheers,
> > >
> > > Chris
> >
>
>
>

[jira] [Created] (KAFKA-16966) Allow offset commit fetch to reuse previous request if partitions are a subset

[Kirk True (Jira)]

Kirk True created KAFKA-16966:
-

 Summary: Allow offset commit fetch to reuse previous request if 
partitions are a subset
 Key: KAFKA-16966
 URL: https://issues.apache.org/jira/browse/KAFKA-16966
 Project: Kafka
  Issue Type: Improvement
  Components: clients, consumer
Affects Versions: 3.8.0
Reporter: Kirk True
Assignee: Kirk True
 Fix For: 3.9.0


In {{{}initWithCommittedOffsetsIfNeeded{}}}, the behavior of the existing 
{{LegacyKafkaConsumer}} is to allow reuse only if the partitions for the 
_current_ request equal those of the _previous_ request *exactly* 
([source|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java?rgh-link-date=2024-06-14T16%3A43%3A11Z#L927]).
 That logic is the basis for the behavior used in the 
{{{}AsyncKafkaConsumer{}}}.
 
It's unlikely to cause any problems, but it introduces a subtle difference in 
behavior between the two {{Consumer}} implementations. Also, the specific case 
that the request reuse logic solves is when the user has passed in a very low 
timeout value in a tight {{poll()}} loop, which suggests the partitions 
wouldn't be changing between those loops.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.8 #44

[Apache Jenkins Server]

See 

Re: [DISCUSS] Apache Kafka 3.9.0 release

[Josep Prat]

+1 from me. Thanks Colin!

Best,

--
Josep Prat
Open Source Engineering Director, Aiven
josep.p...@aiven.io   |   +491715557497 | aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

On Fri, Jun 14, 2024, 20:00 Alyssa Huang 
wrote:

> Thanks Colin, +1 from me.
>
> On Fri, Jun 14, 2024 at 10:55 AM Colin McCabe  wrote:
>
> > Hi all,
> >
> > Based on the discussion in the 3.8 release thread, we will need a 3.9
> > release. I guess I should create a new email thread to discuss this.
> >
> > I'd like to volunteer as release manager for the Apache Kafka 3.9.0
> > release. There's a release plan here:
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.9.0
> >
> > best,
> > Colin
> >
>

Re: [DISCUSS] Apache Kafka 3.9.0 release

[Alyssa Huang]

Thanks Colin, +1 from me.

On Fri, Jun 14, 2024 at 10:55 AM Colin McCabe  wrote:

> Hi all,
>
> Based on the discussion in the 3.8 release thread, we will need a 3.9
> release. I guess I should create a new email thread to discuss this.
>
> I'd like to volunteer as release manager for the Apache Kafka 3.9.0
> release. There's a release plan here:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.9.0
>
> best,
> Colin
>

[DISCUSS] Apache Kafka 3.9.0 release

[Colin McCabe]

Hi all,

Based on the discussion in the 3.8 release thread, we will need a 3.9 release. 
I guess I should create a new email thread to discuss this.

I'd like to volunteer as release manager for the Apache Kafka 3.9.0 release. 
There's a release plan here: 
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.9.0

best,
Colin

Re: [DISCUSS] Apache Kafka 3.8.0 release

[Josep Prat]

Hi all,

Thanks Colin, yes go ahead.

As we are now past code freeze I would like to ask everyone involved in a
KIP that is not yet complete, to verify if what landed on the 3.8 branch
needs to be reverted or if it can stay. Additionally, I'll be pinging KIPs
and Jira reporters asking for their status as some Jiras seem to have all
related GitHub PRs merged but their status is still Open or In Progress.
I'll be checking all the open blockers and check if they are really a
blocker or can be pushed.


Regarding timeline, I'll attempt to generate the first RC on Wednesday or
Thursday, so please revert any changes you deem necessary by then. If you
need more time, please ping me.

Best,

-
Josep Prat
Open Source Engineering Director, Aiven
josep.p...@aiven.io   |   +491715557497 | aiven.io
Aiven Deutschland GmbH
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

On Fri, Jun 14, 2024, 19:25 Colin McCabe  wrote:

> Hi all,
>
> We have had many delays with releases this year. We should try to get back
> on schedule.
>
> I agree with the idea that was proposed a few times in this thread of
> drawing a line under 3.8 now, and doing a short 3.9 release. I posted a 3.9
> release plan here:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.9.0
>
> I think we could start doing RCs for 3.8.0 as early as next week. There
> are a few things that need to be reverted first (anything related to
> KIP-853 or KIP-966).
>
> Josep, if you agree, I will update KIP-1012 to reflect that these things
> are landing in 3.9 rather than 3.8. And we can start doing all the normal
> release stuff. The main blocker JIRA I'm aware of is KAFKA-16946, which is
> a very simple fix.
>
> best,
> Colin
>
>
> On Fri, Jun 14, 2024, at 03:48, Satish Duggana wrote:
> > +1 on going with 3.8 release with the existing plan and targeting the
> > required features in 3.9 timelines. 4.0 will be targeted in the usual
> > cycle(4 months) after 3.9 is released.
> >
> >
> > On Fri, 14 Jun 2024 at 15:19, Edoardo Comar 
> wrote:
> >>
> >> Josep,
> >> past the deadline sorry but I can't see reasons not to cherry-pick this
> >> https://github.com/apache/kafka/pull/16326
> >>
> >> On Wed, 12 Jun 2024 at 17:14, Josep Prat 
> wrote:
> >> >
> >> > Hi Edoardo,
> >> >
> >> > Correct, you can still cherry-pick this one.
> >> >
> >> > I'll send an email tomorrow morning (CEST) asking maintainers to stop
> >> > cherry picking commits unless we discuss it beforehand.
> >> >
> >> > Best,
> >> >
> >> > On Wed, Jun 12, 2024 at 6:09 PM Edoardo Comar 
> wrote:
> >> >
> >> > > Hi Josep, I understand I am still in time to cherry-pick on 3.8.0
> >> > > https://github.com/apache/kafka/pull/16229
> >> > >
> >> > > right?
> >> > > thanks
> >> > >
> >> > > On Wed, 12 Jun 2024 at 11:34, Ivan Yurchenko 
> wrote:
> >> > > >
> >> > > > Hi,
> >> > > >
> >> > > > I'll try to do all the fixes and changes for KIP-899 [1] sooner
> today,
> >> > > but please proceed with the release if I don't manage.
> >> > > >
> >> > > > Ivan
> >> > > >
> >> > > > [1] https://github.com/apache/kafka/pull/13277
> >> > > >
> >> > > > On Wed, Jun 12, 2024, at 12:54, Josep Prat wrote:
> >> > > > > Hi Luke,
> >> > > > > I think Jose, also mentioned that it won't be ready for v3.8.0
> (but he
> >> > > can
> >> > > > > confirm this). My question now would be, given that it seems we
> would
> >> > > need
> >> > > > > a v3.9.0, do you think it's important to include
> >> > > > > https://github.com/apache/kafka/pull/16284 in v3.8.0?
> >> > > > >
> >> > > > > Best,
> >> > > > >
> >> > > > > On Wed, Jun 12, 2024 at 11:40 AM Luke Chen 
> wrote:
> >> > > > >
> >> > > > > > Hi Josep
> >> > > > > >
> >> > > > > > For KIP-966, I think Calvin had mentioned he won't complete in
> >> > > v3.8.0.
> >> > > > > >
> https://lists.apache.org/thread/fsnr8wy5fznzfso7jgk90skgyo277fmw
> >> > > > > >
> >> > > > > > For unclean leader election, all we need is this PR:
> >> > > > > > https://github.com/apache/kafka/pull/16284
> >> > > > > > For this PR, I think it needs one more week to be completed.
> >> > > > > >
> >> > > > > > Thanks.
> >> > > > > > Luke
> >> > > > > >
> >> > > > > > On Wed, Jun 12, 2024 at 4:51 PM Josep Prat
> >> > > 
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi all,
> >> > > > > > >
> >> > > > > > > We are now really close to the planned code freeze deadline
> (today
> >> > > EOD).
> >> > > > > > > According to KIP-1012 [1] we agreed to stay in the 3.x
> branch
> >> > > until we
> >> > > > > > > achieve feature parity regarding Zookeeper and KRaft. The
> two main
> >> > > KIPs
> >> > > > > > > identified that would achieve this are: KIP-853 [2] and
> KIP-966
> >> > > [3].
> >> > > > > > > At the moment of writing this email both KIPs are not
> completed. My
> >> > > > > > > question to the people driving both KIPs would be, how much
> more
> >> > > time do
> >> > > > > > > you think it's needed to 

Re: [DISCUSS] Apache Kafka 3.8.0 release

[Colin McCabe]

Hi all,

We have had many delays with releases this year. We should try to get back on 
schedule.

I agree with the idea that was proposed a few times in this thread of drawing a 
line under 3.8 now, and doing a short 3.9 release. I posted a 3.9 release plan 
here: https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.9.0

I think we could start doing RCs for 3.8.0 as early as next week. There are a 
few things that need to be reverted first (anything related to KIP-853 or 
KIP-966).

Josep, if you agree, I will update KIP-1012 to reflect that these things are 
landing in 3.9 rather than 3.8. And we can start doing all the normal release 
stuff. The main blocker JIRA I'm aware of is KAFKA-16946, which is a very 
simple fix.

best,
Colin


On Fri, Jun 14, 2024, at 03:48, Satish Duggana wrote:
> +1 on going with 3.8 release with the existing plan and targeting the
> required features in 3.9 timelines. 4.0 will be targeted in the usual
> cycle(4 months) after 3.9 is released.
>
>
> On Fri, 14 Jun 2024 at 15:19, Edoardo Comar  wrote:
>>
>> Josep,
>> past the deadline sorry but I can't see reasons not to cherry-pick this
>> https://github.com/apache/kafka/pull/16326
>>
>> On Wed, 12 Jun 2024 at 17:14, Josep Prat  wrote:
>> >
>> > Hi Edoardo,
>> >
>> > Correct, you can still cherry-pick this one.
>> >
>> > I'll send an email tomorrow morning (CEST) asking maintainers to stop
>> > cherry picking commits unless we discuss it beforehand.
>> >
>> > Best,
>> >
>> > On Wed, Jun 12, 2024 at 6:09 PM Edoardo Comar  
>> > wrote:
>> >
>> > > Hi Josep, I understand I am still in time to cherry-pick on 3.8.0
>> > > https://github.com/apache/kafka/pull/16229
>> > >
>> > > right?
>> > > thanks
>> > >
>> > > On Wed, 12 Jun 2024 at 11:34, Ivan Yurchenko  wrote:
>> > > >
>> > > > Hi,
>> > > >
>> > > > I'll try to do all the fixes and changes for KIP-899 [1] sooner today,
>> > > but please proceed with the release if I don't manage.
>> > > >
>> > > > Ivan
>> > > >
>> > > > [1] https://github.com/apache/kafka/pull/13277
>> > > >
>> > > > On Wed, Jun 12, 2024, at 12:54, Josep Prat wrote:
>> > > > > Hi Luke,
>> > > > > I think Jose, also mentioned that it won't be ready for v3.8.0 (but 
>> > > > > he
>> > > can
>> > > > > confirm this). My question now would be, given that it seems we would
>> > > need
>> > > > > a v3.9.0, do you think it's important to include
>> > > > > https://github.com/apache/kafka/pull/16284 in v3.8.0?
>> > > > >
>> > > > > Best,
>> > > > >
>> > > > > On Wed, Jun 12, 2024 at 11:40 AM Luke Chen  wrote:
>> > > > >
>> > > > > > Hi Josep
>> > > > > >
>> > > > > > For KIP-966, I think Calvin had mentioned he won't complete in
>> > > v3.8.0.
>> > > > > > https://lists.apache.org/thread/fsnr8wy5fznzfso7jgk90skgyo277fmw
>> > > > > >
>> > > > > > For unclean leader election, all we need is this PR:
>> > > > > > https://github.com/apache/kafka/pull/16284
>> > > > > > For this PR, I think it needs one more week to be completed.
>> > > > > >
>> > > > > > Thanks.
>> > > > > > Luke
>> > > > > >
>> > > > > > On Wed, Jun 12, 2024 at 4:51 PM Josep Prat
>> > > 
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi all,
>> > > > > > >
>> > > > > > > We are now really close to the planned code freeze deadline 
>> > > > > > > (today
>> > > EOD).
>> > > > > > > According to KIP-1012 [1] we agreed to stay in the 3.x branch
>> > > until we
>> > > > > > > achieve feature parity regarding Zookeeper and KRaft. The two 
>> > > > > > > main
>> > > KIPs
>> > > > > > > identified that would achieve this are: KIP-853 [2] and KIP-966
>> > > [3].
>> > > > > > > At the moment of writing this email both KIPs are not completed. 
>> > > > > > > My
>> > > > > > > question to the people driving both KIPs would be, how much more
>> > > time do
>> > > > > > > you think it's needed to bring these KIPs to completion?
>> > > > > > >
>> > > > > > > - If the time needed would be short, we could still include these
>> > > 2 KIPs
>> > > > > > in
>> > > > > > > the release.
>> > > > > > > - However, if the time needed would be on the scale of weeks, we
>> > > should
>> > > > > > > continue with the release plan for 3.8 and after start working on
>> > > the 3.9
>> > > > > > > release.
>> > > > > > >
>> > > > > > > What are your thoughts?
>> > > > > > >
>> > > > > > >
>> > > > > > > [1]:
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
>> > > > > > > [2]:
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes
>> > > > > > > [3]:
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas
>> > > > > > >
>> > > > > > > On Wed, Jun 12, 2024 at 10:40 AM Josep Prat 
>> > > wrote:
>> > > > > > >
>> > > > > > > > Hi Rajini,
>> > > > > > > > Yes, we could backport this one to the 3.8 branch. Would 

[jira] [Created] (KAFKA-16965) Add a "root cause" exception as a nested exception to TimeoutException for Producer

[Alieh Saeedi (Jira)]

Alieh Saeedi created KAFKA-16965:


 Summary: Add a "root cause" exception as a nested exception to 
TimeoutException for Producer
 Key: KAFKA-16965
 URL: https://issues.apache.org/jira/browse/KAFKA-16965
 Project: Kafka
  Issue Type: Bug
Reporter: Alieh Saeedi






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1017: A health check endpoint for Kafka Connect

[Edoardo Comar]

Chris, I was about to vote too but have one trivial question.

You designed this health check for a completely generic connect runtime,
regardless of which connectors it's running, right?

Because if I run a well-known to me set of connectors, i.e. mm2 :-) ,
then checking the connectors and tasks state is a good healthcheck already.

In such case, I think the value KIP-1017 is allow to find a problem in
the runtime rather than the connectors.
would you agree?

On Fri, 14 Jun 2024 at 15:44, Chris Egerton  wrote:
>
> Thanks Viktor! Seems as good a time as any to kick off the vote thread.
>
> On Fri, Jun 14, 2024, 10:43 Viktor Somogyi-Vass
>  wrote:
>
> > Hi Chris,
> >
> > Both points make sense to me.
> >
> > 1) I'm good with your original solution where it would return after 10
> > seconds for both 5xx endpoints. The second solution seems complicated and
> > maybe we shouldn't overcomplicate it.
> > 2) Unfortunately I can't commit to that hypothetical KIP but I'll make a
> > mental note :)
> >
> > I'm +1 on the KIP.
> >
> > Thanks,
> > Viktor
> >
> > On Tue, Jun 11, 2024 at 9:53 PM Chris Egerton 
> > wrote:
> >
> > > Hi Viktor,
> > >
> > > Thanks for your comments!
> > >
> > > 1) I'm realizing now that there's some implicit behavior in the KIP that
> > I
> > > should spell out explicitly. I was thinking that the timeout of 10
> > seconds
> > > that I mentioned in the "Public Interfaces" section would have to elapse
> > > before any 5xx responses were issued. So, if someone pinged the health
> > > endpoint while the worker was starting up, the worker would take up to 10
> > > seconds to try to complete startup before giving up and responding to the
> > > request with a 503 status. This would obviate the need for a Retry-After
> > > header because it would apply a natural rate limiting to these requests
> > > during worker startup. Does this seem reasonable? We could diverge in
> > > behavior and only apply the 10 second timeout to the 500 case (i.e.,
> > worker
> > > has completed startup but is not live for other reasons), at which point
> > a
> > > Retry-After header for the 503 case (worker still starting up) would make
> > > sense, but I can't think of any benefits to this approach. Thoughts?
> > >
> > > 2) As of KAFKA-15563 [1], we have some great infrastructure in place to
> > > identify worker actions that might be good candidates for the contents of
> > > this "worker events" topic. But I don't think conflating the retrieval of
> > > these events with the health endpoint is a good idea--IMO it should be a
> > > separate endpoint and the health endpoint should stay lightweight and
> > > simple. I'm also not sure it's necessary to expose the contents of this
> > > kind of topic via the REST API at all; we could instruct users to consume
> > > directly from the topic if they'd like to know the history of the worker.
> > > Overall it seems like a decent idea and I'd be happy to review a KIP for
> > > it, but like you mention, it seems like a pretty drastic change in scope
> > > and I don't think it needs to be included in this proposal.
> > >
> > > [1] - https://issues.apache.org/jira/browse/KAFKA-15563
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Tue, Jun 11, 2024 at 11:42 AM Viktor Somogyi-Vass
> > >  wrote:
> > >
> > > > Hi Chris,
> > > >
> > > > I also have 2 other comments:
> > > >
> > > > 1. One more thing I came across is that should we provide the
> > Retry-After
> > > > header in the response in case of 503 response? Although I'm not sure
> > how
> > > > many clients honor this, we could add it just in case some does and if
> > > you
> > > > also find it useful. (We could default it to retry.backoff.ms.)
> > > >
> > > > 2. Adding to Adrian's comments, storing timestamped worker statuses in
> > an
> > > > internal topic and then reading them from there would add valuable info
> > > > about what the worker does. For instance GET
> > > /health?startTime=45345323346
> > > > could fetch events from the given timestamp additionally to your
> > proposed
> > > > behavior. Also, if the rest server isn't available, it would serve in
> > > > itself as a log about the workers' behavior. I understand if you think
> > > it's
> > > > such a scope change that it should be an improvement KIP, but would
> > like
> > > to
> > > > gauge what you think about this.
> > > >
> > > > Regards,
> > > > Viktor
> > > >
> > > > On Tue, Jun 11, 2024 at 4:34 PM Chris Egerton  > >
> > > > wrote:
> > > >
> > > > > Hi Adrian,
> > > > >
> > > > > Thanks for your comments/questions! The responses to them are related
> > > so
> > > > > I'll try to address both at once.
> > > > >
> > > > > The most recent update I made to the KIP should help provide insight
> > > into
> > > > > what's going wrong if a non-200 response is returned. I don't plan on
> > > > > adding any structured data such as error codes or something like a
> > > > "phase"
> > > > > field with values like READING_CONFIG_TOPIC quite yet, but there is
> > > room

[jira] [Resolved] (KAFKA-16917) DescribeTopicsResult should use mutable map in order to keep compatibility

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16917.

Fix Version/s: 3.9.0
   Resolution: Fixed

> DescribeTopicsResult should use mutable map in order to keep compatibility
> --
>
> Key: KAFKA-16917
> URL: https://issues.apache.org/jira/browse/KAFKA-16917
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: TengYao Chi
>Priority: Minor
> Fix For: 3.9.0
>
>
> *new target*
> DescribeTopicsResult returned mutable map before, and it was changed to 
> immutable map exceptionally. Hence, we should revert it back to mutable map.
> *origin target*
> This jira includes following issues:
>  # KafkaAdminClient#alterClientQuotas [0] return a immutable map, but other 
> APIs return `HashMap`.
>  # KafkaAdminClient#describeConfigs [1] has duplicate copy. the returned 
> stuff from `Collectors.toMap` is a new HashMap already.
>  
> [0] 
> [https://github.com/apache/kafka/blob/d13a693ea768fa8013240dc39bc21db92bffc31f/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L4074]
> [1] 
> [https://github.com/apache/kafka/blob/d13a693ea768fa8013240dc39bc21db92bffc31f/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L2763]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.7.1 RC1

[Edoardo Comar]

thanks Justine !

On Fri, 14 Jun 2024 at 16:10, Bill Bejeck  wrote:
>
> Thanks for the quick action Justine, I just reviewed and approved.
>
> On Fri, Jun 14, 2024 at 11:08 AM Justine Olshan
>  wrote:
>
> > The PR to fix is: https://github.com/apache/kafka/pull/16342/files
> >
> > On Fri, Jun 14, 2024 at 8:02 AM Justine Olshan 
> > wrote:
> >
> > > Looks like the previous cherrypick of Edoardo's feature broke the build.
> > I
> > > have a fix coming shortly.
> > >
> > > I may also want to look at the system tests again.
> > >
> > > Justine
> > >
> > > On Fri, Jun 14, 2024 at 6:06 AM Luke Chen  wrote:
> > >
> > >> Hi all,
> > >>
> > >> After running system tests, here are failing tests:
> > >>
> > >> 1. kafkatest.tests.core.upgrade_test failed at lz4 and snappy tests:
> > >> KAFKA-16962 
> > >> 2.  StreamsUpgradeTest failed at "fromVersion"=2.2 ~ 2.5, but passed for
> > >> the rest of the versions: KAFKA-16960
> > >> 
> > >> 3. TestKRaftUpgrade failed at "fromVersion"=3.3 ~ 3.5 under combined
> > >> KRaft:
> > >> KAFKA-16961 
> > >>
> > >> Need help to take a look, or maybe re-run in another environment.
> > >>
> > >> Thanks.
> > >> Luke
> > >>
> > >> On Fri, Jun 14, 2024 at 5:03 PM Edoardo Comar 
> > >> wrote:
> > >>
> > >> > well if you still need to build RC2, I'd cherry pick this that I
> > >> > missed from the other PR
> > >> > https://github.com/apache/kafka/pull/16326
> > >> >
> > >> > On Thu, 13 Jun 2024 at 13:20, Igor Soarez  wrote:
> > >> > >
> > >> > > Hi Edoardo,
> > >> > >
> > >> > > It is late, but not too late. I have cherry-picked your change
> > >> > > to the 3.7 branch and I'll build a second release candidate.
> > >> > >
> > >> > > If you could have a look at the first RC, please let me know if
> > >> > > you spot any issues with it that can be avoided in the next RC.
> > >> > >
> > >> > > Thanks,
> > >> > >
> > >> > > --
> > >> > > Igor
> > >> >
> > >>
> > >
> >

Re: [VOTE] KIP-1017: A health check endpoint for Kafka Connect

[Hector Geraldino (BLOOMBERG/ 919 3RD A)]

Thanks Chris, 

+1 (non-binding)

From: dev@kafka.apache.org At: 06/14/24 11:43:02 UTC-4:00To:  
dev@kafka.apache.org
Subject: Re: [VOTE] KIP-1017: A health check endpoint for Kafka Connect

Hi Chris,

+1 (binding)

Thanks,
Mickael

On Fri, Jun 14, 2024 at 5:01 PM Andrew Schofield
 wrote:
>
> Hi Chris,
> Thanks for the KIP.
>
> +1 (non-binding)
>
> Thanks,
> Andrew
>
> > On 14 Jun 2024, at 15:48, Chris Egerton  wrote:
> >
> > Hi all,
> >
> > Happy Friday! I'd like to kick off a vote on KIP-1017.
> >
> > Design doc:
> > 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1017%3A+Health+check+endpo
int+for+Kafka+Connect
> >
> > Discussion thread:
> > https://lists.apache.org/thread/95nto84wtogdgk3v97rxcwxr258wnjnt
> >
> > Cheers,
> >
> > Chris
>

Re: [VOTE] KIP-1017: A health check endpoint for Kafka Connect

[Mickael Maison]

Hi Chris,

+1 (binding)

Thanks,
Mickael

On Fri, Jun 14, 2024 at 5:01 PM Andrew Schofield
 wrote:
>
> Hi Chris,
> Thanks for the KIP.
>
> +1 (non-binding)
>
> Thanks,
> Andrew
>
> > On 14 Jun 2024, at 15:48, Chris Egerton  wrote:
> >
> > Hi all,
> >
> > Happy Friday! I'd like to kick off a vote on KIP-1017.
> >
> > Design doc:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1017%3A+Health+check+endpoint+for+Kafka+Connect
> >
> > Discussion thread:
> > https://lists.apache.org/thread/95nto84wtogdgk3v97rxcwxr258wnjnt
> >
> > Cheers,
> >
> > Chris
>

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #3009

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-16964) Add and remove voter integration tests

[Jira]

José Armando García Sancio created KAFKA-16964:
--

 Summary: Add and remove voter integration tests
 Key: KAFKA-16964
 URL: https://issues.apache.org/jira/browse/KAFKA-16964
 Project: Kafka
  Issue Type: Sub-task
  Components: kraft, unit tests
Reporter: José Armando García Sancio


Add integration tests that add and remove controllers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16963) Add and remove voter system tests

[Jira]

José Armando García Sancio created KAFKA-16963:
--

 Summary: Add and remove voter system tests
 Key: KAFKA-16963
 URL: https://issues.apache.org/jira/browse/KAFKA-16963
 Project: Kafka
  Issue Type: Sub-task
  Components: kraft, system tests
Reporter: José Armando García Sancio


Add a system tests that adds and removes controllers and checks some invariant.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.7.1 RC1

[Bill Bejeck]

Thanks for the quick action Justine, I just reviewed and approved.

On Fri, Jun 14, 2024 at 11:08 AM Justine Olshan
 wrote:

> The PR to fix is: https://github.com/apache/kafka/pull/16342/files
>
> On Fri, Jun 14, 2024 at 8:02 AM Justine Olshan 
> wrote:
>
> > Looks like the previous cherrypick of Edoardo's feature broke the build.
> I
> > have a fix coming shortly.
> >
> > I may also want to look at the system tests again.
> >
> > Justine
> >
> > On Fri, Jun 14, 2024 at 6:06 AM Luke Chen  wrote:
> >
> >> Hi all,
> >>
> >> After running system tests, here are failing tests:
> >>
> >> 1. kafkatest.tests.core.upgrade_test failed at lz4 and snappy tests:
> >> KAFKA-16962 
> >> 2.  StreamsUpgradeTest failed at "fromVersion"=2.2 ~ 2.5, but passed for
> >> the rest of the versions: KAFKA-16960
> >> 
> >> 3. TestKRaftUpgrade failed at "fromVersion"=3.3 ~ 3.5 under combined
> >> KRaft:
> >> KAFKA-16961 
> >>
> >> Need help to take a look, or maybe re-run in another environment.
> >>
> >> Thanks.
> >> Luke
> >>
> >> On Fri, Jun 14, 2024 at 5:03 PM Edoardo Comar 
> >> wrote:
> >>
> >> > well if you still need to build RC2, I'd cherry pick this that I
> >> > missed from the other PR
> >> > https://github.com/apache/kafka/pull/16326
> >> >
> >> > On Thu, 13 Jun 2024 at 13:20, Igor Soarez  wrote:
> >> > >
> >> > > Hi Edoardo,
> >> > >
> >> > > It is late, but not too late. I have cherry-picked your change
> >> > > to the 3.7 branch and I'll build a second release candidate.
> >> > >
> >> > > If you could have a look at the first RC, please let me know if
> >> > > you spot any issues with it that can be avoided in the next RC.
> >> > >
> >> > > Thanks,
> >> > >
> >> > > --
> >> > > Igor
> >> >
> >>
> >
>

Re: [VOTE] 3.7.1 RC1

[Justine Olshan]

The PR to fix is: https://github.com/apache/kafka/pull/16342/files

On Fri, Jun 14, 2024 at 8:02 AM Justine Olshan  wrote:

> Looks like the previous cherrypick of Edoardo's feature broke the build. I
> have a fix coming shortly.
>
> I may also want to look at the system tests again.
>
> Justine
>
> On Fri, Jun 14, 2024 at 6:06 AM Luke Chen  wrote:
>
>> Hi all,
>>
>> After running system tests, here are failing tests:
>>
>> 1. kafkatest.tests.core.upgrade_test failed at lz4 and snappy tests:
>> KAFKA-16962 
>> 2.  StreamsUpgradeTest failed at "fromVersion"=2.2 ~ 2.5, but passed for
>> the rest of the versions: KAFKA-16960
>> 
>> 3. TestKRaftUpgrade failed at "fromVersion"=3.3 ~ 3.5 under combined
>> KRaft:
>> KAFKA-16961 
>>
>> Need help to take a look, or maybe re-run in another environment.
>>
>> Thanks.
>> Luke
>>
>> On Fri, Jun 14, 2024 at 5:03 PM Edoardo Comar 
>> wrote:
>>
>> > well if you still need to build RC2, I'd cherry pick this that I
>> > missed from the other PR
>> > https://github.com/apache/kafka/pull/16326
>> >
>> > On Thu, 13 Jun 2024 at 13:20, Igor Soarez  wrote:
>> > >
>> > > Hi Edoardo,
>> > >
>> > > It is late, but not too late. I have cherry-picked your change
>> > > to the 3.7 branch and I'll build a second release candidate.
>> > >
>> > > If you could have a look at the first RC, please let me know if
>> > > you spot any issues with it that can be avoided in the next RC.
>> > >
>> > > Thanks,
>> > >
>> > > --
>> > > Igor
>> >
>>
>

Re: [VOTE] 3.7.1 RC1

[Justine Olshan]

Looks like the previous cherrypick of Edoardo's feature broke the build. I
have a fix coming shortly.

I may also want to look at the system tests again.

Justine

On Fri, Jun 14, 2024 at 6:06 AM Luke Chen  wrote:

> Hi all,
>
> After running system tests, here are failing tests:
>
> 1. kafkatest.tests.core.upgrade_test failed at lz4 and snappy tests:
> KAFKA-16962 
> 2.  StreamsUpgradeTest failed at "fromVersion"=2.2 ~ 2.5, but passed for
> the rest of the versions: KAFKA-16960
> 
> 3. TestKRaftUpgrade failed at "fromVersion"=3.3 ~ 3.5 under combined KRaft:
> KAFKA-16961 
>
> Need help to take a look, or maybe re-run in another environment.
>
> Thanks.
> Luke
>
> On Fri, Jun 14, 2024 at 5:03 PM Edoardo Comar 
> wrote:
>
> > well if you still need to build RC2, I'd cherry pick this that I
> > missed from the other PR
> > https://github.com/apache/kafka/pull/16326
> >
> > On Thu, 13 Jun 2024 at 13:20, Igor Soarez  wrote:
> > >
> > > Hi Edoardo,
> > >
> > > It is late, but not too late. I have cherry-picked your change
> > > to the 3.7 branch and I'll build a second release candidate.
> > >
> > > If you could have a look at the first RC, please let me know if
> > > you spot any issues with it that can be avoided in the next RC.
> > >
> > > Thanks,
> > >
> > > --
> > > Igor
> >
>

Re: [VOTE] KIP-1017: A health check endpoint for Kafka Connect

[Andrew Schofield]

Hi Chris,
Thanks for the KIP.

+1 (non-binding)

Thanks,
Andrew

> On 14 Jun 2024, at 15:48, Chris Egerton  wrote:
>
> Hi all,
>
> Happy Friday! I'd like to kick off a vote on KIP-1017.
>
> Design doc:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1017%3A+Health+check+endpoint+for+Kafka+Connect
>
> Discussion thread:
> https://lists.apache.org/thread/95nto84wtogdgk3v97rxcwxr258wnjnt
>
> Cheers,
>
> Chris

[VOTE] KIP-1017: A health check endpoint for Kafka Connect

[Chris Egerton]

Hi all,

Happy Friday! I'd like to kick off a vote on KIP-1017.

Design doc:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1017%3A+Health+check+endpoint+for+Kafka+Connect

Discussion thread:
https://lists.apache.org/thread/95nto84wtogdgk3v97rxcwxr258wnjnt

Cheers,

Chris

Re: [DISCUSS] KIP-1017: A health check endpoint for Kafka Connect

[Viktor Somogyi-Vass]

Hi Chris,

Both points make sense to me.

1) I'm good with your original solution where it would return after 10
seconds for both 5xx endpoints. The second solution seems complicated and
maybe we shouldn't overcomplicate it.
2) Unfortunately I can't commit to that hypothetical KIP but I'll make a
mental note :)

I'm +1 on the KIP.

Thanks,
Viktor

On Tue, Jun 11, 2024 at 9:53 PM Chris Egerton 
wrote:

> Hi Viktor,
>
> Thanks for your comments!
>
> 1) I'm realizing now that there's some implicit behavior in the KIP that I
> should spell out explicitly. I was thinking that the timeout of 10 seconds
> that I mentioned in the "Public Interfaces" section would have to elapse
> before any 5xx responses were issued. So, if someone pinged the health
> endpoint while the worker was starting up, the worker would take up to 10
> seconds to try to complete startup before giving up and responding to the
> request with a 503 status. This would obviate the need for a Retry-After
> header because it would apply a natural rate limiting to these requests
> during worker startup. Does this seem reasonable? We could diverge in
> behavior and only apply the 10 second timeout to the 500 case (i.e., worker
> has completed startup but is not live for other reasons), at which point a
> Retry-After header for the 503 case (worker still starting up) would make
> sense, but I can't think of any benefits to this approach. Thoughts?
>
> 2) As of KAFKA-15563 [1], we have some great infrastructure in place to
> identify worker actions that might be good candidates for the contents of
> this "worker events" topic. But I don't think conflating the retrieval of
> these events with the health endpoint is a good idea--IMO it should be a
> separate endpoint and the health endpoint should stay lightweight and
> simple. I'm also not sure it's necessary to expose the contents of this
> kind of topic via the REST API at all; we could instruct users to consume
> directly from the topic if they'd like to know the history of the worker.
> Overall it seems like a decent idea and I'd be happy to review a KIP for
> it, but like you mention, it seems like a pretty drastic change in scope
> and I don't think it needs to be included in this proposal.
>
> [1] - https://issues.apache.org/jira/browse/KAFKA-15563
>
> Cheers,
>
> Chris
>
> On Tue, Jun 11, 2024 at 11:42 AM Viktor Somogyi-Vass
>  wrote:
>
> > Hi Chris,
> >
> > I also have 2 other comments:
> >
> > 1. One more thing I came across is that should we provide the Retry-After
> > header in the response in case of 503 response? Although I'm not sure how
> > many clients honor this, we could add it just in case some does and if
> you
> > also find it useful. (We could default it to retry.backoff.ms.)
> >
> > 2. Adding to Adrian's comments, storing timestamped worker statuses in an
> > internal topic and then reading them from there would add valuable info
> > about what the worker does. For instance GET
> /health?startTime=45345323346
> > could fetch events from the given timestamp additionally to your proposed
> > behavior. Also, if the rest server isn't available, it would serve in
> > itself as a log about the workers' behavior. I understand if you think
> it's
> > such a scope change that it should be an improvement KIP, but would like
> to
> > gauge what you think about this.
> >
> > Regards,
> > Viktor
> >
> > On Tue, Jun 11, 2024 at 4:34 PM Chris Egerton 
> > wrote:
> >
> > > Hi Adrian,
> > >
> > > Thanks for your comments/questions! The responses to them are related
> so
> > > I'll try to address both at once.
> > >
> > > The most recent update I made to the KIP should help provide insight
> into
> > > what's going wrong if a non-200 response is returned. I don't plan on
> > > adding any structured data such as error codes or something like a
> > "phase"
> > > field with values like READING_CONFIG_TOPIC quite yet, but there is
> room
> > > for us to add human-readable information on the causes of failure in
> the
> > > "message" field (see KAFKA-15563 [1] and its PR [2] for an example of
> > what
> > > kind of information we might provide to users). Part of the problem is
> > that
> > > while I've heard plenty of (justified!) complaints about the Kafka
> > Connect
> > > REST API becoming unavailable and the difficulties users face with
> > > debugging their workers when that happens, I still don't feel we have a
> > > strong-enough grasp on the common causes for this scenario to commit
> to a
> > > response format that could be more machine-readable, and it can be
> > > surprisingly difficult to get to a root cause in some cases.
> > >
> > > I'm anticipating that users will rely on the endpoint primarily for two
> > > things:
> > > 1) Ensuring that a worker has completed startup successfully during a
> > > rolling upgrade (if you don't get a 200 after long enough, abort the
> > > upgrade, check the error message, and start investigating)
> > > 2) Ensuring that a worker remains healthy after it 

Re: [DISCUSS] KIP-1017: A health check endpoint for Kafka Connect

[Chris Egerton]

Thanks Viktor! Seems as good a time as any to kick off the vote thread.

On Fri, Jun 14, 2024, 10:43 Viktor Somogyi-Vass
 wrote:

> Hi Chris,
>
> Both points make sense to me.
>
> 1) I'm good with your original solution where it would return after 10
> seconds for both 5xx endpoints. The second solution seems complicated and
> maybe we shouldn't overcomplicate it.
> 2) Unfortunately I can't commit to that hypothetical KIP but I'll make a
> mental note :)
>
> I'm +1 on the KIP.
>
> Thanks,
> Viktor
>
> On Tue, Jun 11, 2024 at 9:53 PM Chris Egerton 
> wrote:
>
> > Hi Viktor,
> >
> > Thanks for your comments!
> >
> > 1) I'm realizing now that there's some implicit behavior in the KIP that
> I
> > should spell out explicitly. I was thinking that the timeout of 10
> seconds
> > that I mentioned in the "Public Interfaces" section would have to elapse
> > before any 5xx responses were issued. So, if someone pinged the health
> > endpoint while the worker was starting up, the worker would take up to 10
> > seconds to try to complete startup before giving up and responding to the
> > request with a 503 status. This would obviate the need for a Retry-After
> > header because it would apply a natural rate limiting to these requests
> > during worker startup. Does this seem reasonable? We could diverge in
> > behavior and only apply the 10 second timeout to the 500 case (i.e.,
> worker
> > has completed startup but is not live for other reasons), at which point
> a
> > Retry-After header for the 503 case (worker still starting up) would make
> > sense, but I can't think of any benefits to this approach. Thoughts?
> >
> > 2) As of KAFKA-15563 [1], we have some great infrastructure in place to
> > identify worker actions that might be good candidates for the contents of
> > this "worker events" topic. But I don't think conflating the retrieval of
> > these events with the health endpoint is a good idea--IMO it should be a
> > separate endpoint and the health endpoint should stay lightweight and
> > simple. I'm also not sure it's necessary to expose the contents of this
> > kind of topic via the REST API at all; we could instruct users to consume
> > directly from the topic if they'd like to know the history of the worker.
> > Overall it seems like a decent idea and I'd be happy to review a KIP for
> > it, but like you mention, it seems like a pretty drastic change in scope
> > and I don't think it needs to be included in this proposal.
> >
> > [1] - https://issues.apache.org/jira/browse/KAFKA-15563
> >
> > Cheers,
> >
> > Chris
> >
> > On Tue, Jun 11, 2024 at 11:42 AM Viktor Somogyi-Vass
> >  wrote:
> >
> > > Hi Chris,
> > >
> > > I also have 2 other comments:
> > >
> > > 1. One more thing I came across is that should we provide the
> Retry-After
> > > header in the response in case of 503 response? Although I'm not sure
> how
> > > many clients honor this, we could add it just in case some does and if
> > you
> > > also find it useful. (We could default it to retry.backoff.ms.)
> > >
> > > 2. Adding to Adrian's comments, storing timestamped worker statuses in
> an
> > > internal topic and then reading them from there would add valuable info
> > > about what the worker does. For instance GET
> > /health?startTime=45345323346
> > > could fetch events from the given timestamp additionally to your
> proposed
> > > behavior. Also, if the rest server isn't available, it would serve in
> > > itself as a log about the workers' behavior. I understand if you think
> > it's
> > > such a scope change that it should be an improvement KIP, but would
> like
> > to
> > > gauge what you think about this.
> > >
> > > Regards,
> > > Viktor
> > >
> > > On Tue, Jun 11, 2024 at 4:34 PM Chris Egerton  >
> > > wrote:
> > >
> > > > Hi Adrian,
> > > >
> > > > Thanks for your comments/questions! The responses to them are related
> > so
> > > > I'll try to address both at once.
> > > >
> > > > The most recent update I made to the KIP should help provide insight
> > into
> > > > what's going wrong if a non-200 response is returned. I don't plan on
> > > > adding any structured data such as error codes or something like a
> > > "phase"
> > > > field with values like READING_CONFIG_TOPIC quite yet, but there is
> > room
> > > > for us to add human-readable information on the causes of failure in
> > the
> > > > "message" field (see KAFKA-15563 [1] and its PR [2] for an example of
> > > what
> > > > kind of information we might provide to users). Part of the problem
> is
> > > that
> > > > while I've heard plenty of (justified!) complaints about the Kafka
> > > Connect
> > > > REST API becoming unavailable and the difficulties users face with
> > > > debugging their workers when that happens, I still don't feel we
> have a
> > > > strong-enough grasp on the common causes for this scenario to commit
> > to a
> > > > response format that could be more machine-readable, and it can be
> > > > surprisingly difficult to get to a root cause in some cases.

Re: [VOTE] 3.7.1 RC1

[Luke Chen]

Hi all,

After running system tests, here are failing tests:

1. kafkatest.tests.core.upgrade_test failed at lz4 and snappy tests:
KAFKA-16962 
2.  StreamsUpgradeTest failed at "fromVersion"=2.2 ~ 2.5, but passed for
the rest of the versions: KAFKA-16960

3. TestKRaftUpgrade failed at "fromVersion"=3.3 ~ 3.5 under combined KRaft:
KAFKA-16961 

Need help to take a look, or maybe re-run in another environment.

Thanks.
Luke

On Fri, Jun 14, 2024 at 5:03 PM Edoardo Comar  wrote:

> well if you still need to build RC2, I'd cherry pick this that I
> missed from the other PR
> https://github.com/apache/kafka/pull/16326
>
> On Thu, 13 Jun 2024 at 13:20, Igor Soarez  wrote:
> >
> > Hi Edoardo,
> >
> > It is late, but not too late. I have cherry-picked your change
> > to the 3.7 branch and I'll build a second release candidate.
> >
> > If you could have a look at the first RC, please let me know if
> > you spot any issues with it that can be avoided in the next RC.
> >
> > Thanks,
> >
> > --
> > Igor
>

[jira] [Created] (KAFKA-16962) kafkatest.tests.core.upgrade_test system tests failed in v3.7.1 RC1

[Luke Chen (Jira)]

Luke Chen created KAFKA-16962:
-

 Summary: kafkatest.tests.core.upgrade_test system tests failed in 
v3.7.1 RC1
 Key: KAFKA-16962
 URL: https://issues.apache.org/jira/browse/KAFKA-16962
 Project: Kafka
  Issue Type: Test
Affects Versions: 3.7.1
Reporter: Luke Chen


 
{code:java}
AssertionError() Traceback (most 
recent call last): File 
"/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 
186, in _do_run data = self.run_test() File 
"/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 
246, in run_test return self.test_context.function(self.test) File 
"/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 433, in 
wrapper return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs) File 
"/opt/kafka-dev/tests/kafkatest/tests/core/upgrade_test.py", line 240, in 
test_upgrade assert self.kafka.check_protocol_errors(self) AssertionError 
AssertionError() Traceback (most recent call last): File 
"/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 
186, in _do_run data = self.run_test() File 
"/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 
246, in run_test return self.test_context.function(self.test) File 
"/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 433, in 
wrapper return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs) File 
"/opt/kafka-dev/tests/kafkatest/tests/core/upgrade_test.py", line 240, in 
test_upgrade assert self.kafka.check_protocol_errors(self) AssertionError 
...

AssertionError() Traceback (most 
recent call last): File 
"/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 
186, in _do_run data = self.run_test() File 
"/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 
246, in run_test return self.test_context.function(self.test) File 
"/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 433, in 
wrapper return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs) File 
"/opt/kafka-dev/tests/kafkatest/tests/core/upgrade_test.py", line 240, in 
test_upgrade assert self.kafka.check_protocol_errors(self) AssertionError 
AssertionError() Traceback (most recent call last): File 
"/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 
186, in _do_run data = self.run_test() File 
"/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 
246, in run_test return self.test_context.function(self.test) File 
"/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 433, in 
wrapper return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs) File 
"/opt/kafka-dev/tests/kafkatest/tests/core/upgrade_test.py", line 240, in 
test_upgrade assert self.kafka.check_protocol_errors(self) AssertionError 
...{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16961) TestKRaftUpgrade system tests fail in v3.7.1 RC1

[Luke Chen (Jira)]

Luke Chen created KAFKA-16961:
-

 Summary: TestKRaftUpgrade system tests fail in v3.7.1 RC1
 Key: KAFKA-16961
 URL: https://issues.apache.org/jira/browse/KAFKA-16961
 Project: Kafka
  Issue Type: Test
Reporter: Luke Chen


 

 
{code:java}

SESSION REPORT (ALL TESTS)
ducktape version: 0.11.4
session_id:       2024-06-14--003
run time:         86 minutes 13.705 seconds
tests run:        24
passed:           18
flaky:            0
failed:           6
ignored:          0

test_id:    
kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.1.2.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT
status:     PASS
run time:   3 minutes 44.680 seconds

test_id:    
kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.1.2.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT
status:     PASS
run time:   3 minutes 42.627 seconds

test_id:    
kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.2.3.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT
status:     PASS
run time:   3 minutes 28.205 seconds

test_id:    
kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.2.3.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT
status:     PASS
run time:   3 minutes 42.388 seconds

test_id:    
kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.3.2.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT
status:     PASS
run time:   2 minutes 57.679 seconds

test_id:    
kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.3.2.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT
status:     PASS
run time:   2 minutes 57.238 seconds

test_id:    
kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.4.1.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT
status:     PASS
run time:   2 minutes 52.545 seconds

test_id:    
kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.4.1.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT
status:     PASS
run time:   2 minutes 56.289 seconds

test_id:    
kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.5.2.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT
status:     PASS
run time:   2 minutes 54.953 seconds

test_id:    
kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=3.5.2.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT
status:     PASS
run time:   2 minutes 59.579 seconds

test_id:    
kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=dev.use_new_coordinator=False.metadata_quorum=ISOLATED_KRAFT
status:     PASS
run time:   3 minutes 21.016 seconds

test_id:    
kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade.from_kafka_version=dev.use_new_coordinator=True.metadata_quorum=ISOLATED_KRAFT
status:     PASS
run time:   2 minutes 56.175 seconds

test_id:    
kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.1.2.use_new_coordinator=False.metadata_quorum=COMBINED_KRAFT
status:     PASS
run time:   3 minutes 6.505 seconds

test_id:    
kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.1.2.use_new_coordinator=True.metadata_quorum=COMBINED_KRAFT

[jira] [Created] (KAFKA-16960) StreamsUpgradeTest system tests fail in v3.7.1 RC1

[Luke Chen (Jira)]

Luke Chen created KAFKA-16960:
-

 Summary: StreamsUpgradeTest system tests fail in v3.7.1 RC1
 Key: KAFKA-16960
 URL: https://issues.apache.org/jira/browse/KAFKA-16960
 Project: Kafka
  Issue Type: Improvement
Reporter: Luke Chen


 
{code:java}

SESSION REPORT (ALL TESTS)
ducktape version: 0.11.4
session_id:       2024-06-14--002
run time:         32 minutes 6.131 seconds
tests run:        14
passed:           10
flaky:            0
failed:           4
ignored:          0

test_id:    
kafkatest.tests.streams.streams_application_upgrade_test.StreamsUpgradeTest.test_app_upgrade.from_version=2.2.2.to_version=3.7.1-SNAPSHOT.bounce_type=full
status:     FAIL
run time:   5 minutes 2.298 seconds
    TimeoutError("Did expect to read 'processed [0-9]* records' from 
ducker@ducker05")
Traceback (most recent call last):
  File 
"/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 
186, in _do_run
    data = self.run_test()
  File 
"/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 
246, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 
433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File 
"/opt/kafka-dev/tests/kafkatest/tests/streams/streams_application_upgrade_test.py",
 line 101, in test_app_upgrade
    self.start_all_nodes_with(from_version)
  File 
"/opt/kafka-dev/tests/kafkatest/tests/streams/streams_application_upgrade_test.py",
 line 157, in start_all_nodes_with
    self.wait_for_verification(self.processor1, self.processed_msg, 
self.processor1.STDOUT_FILE)
  File 
"/opt/kafka-dev/tests/kafkatest/tests/streams/streams_application_upgrade_test.py",
 line 209, in wait_for_verification
    wait_until(lambda: self.verify_from_file(processor, message, file) >= 
num_lines,
  File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line 
58, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from 
last_exception
ducktape.errors.TimeoutError: Did expect to read 'processed [0-9]* records' 
from 
ducker@ducker05
test_id:    
kafkatest.tests.streams.streams_application_upgrade_test.StreamsUpgradeTest.test_app_upgrade.from_version=2.3.1.to_version=3.7.1-SNAPSHOT.bounce_type=full
status:     FAIL
run time:   4 minutes 37.443 seconds
    TimeoutError("Did expect to read 'SMOKE-TEST-CLIENT-STARTED' from 
ducker@ducker11")
Traceback (most recent call last):
  File 
"/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 
186, in _do_run
    data = self.run_test()
  File 
"/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 
246, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 
433, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File 
"/opt/kafka-dev/tests/kafkatest/tests/streams/streams_application_upgrade_test.py",
 line 101, in test_app_upgrade
    self.start_all_nodes_with(from_version)
  File 
"/opt/kafka-dev/tests/kafkatest/tests/streams/streams_application_upgrade_test.py",
 line 152, in start_all_nodes_with
    self.wait_for_verification(self.processor1, "SMOKE-TEST-CLIENT-STARTED", 
self.processor1.STDOUT_FILE)
  File 
"/opt/kafka-dev/tests/kafkatest/tests/streams/streams_application_upgrade_test.py",
 line 209, in wait_for_verification
    wait_until(lambda: self.verify_from_file(processor, message, file) >= 
num_lines,
  File "/usr/local/lib/python3.9/dist-packages/ducktape/utils/util.py", line 
58, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from 
last_exception
ducktape.errors.TimeoutError: Did expect to read 'SMOKE-TEST-CLIENT-STARTED' 
from 
ducker@ducker11
test_id:    
kafkatest.tests.streams.streams_application_upgrade_test.StreamsUpgradeTest.test_app_upgrade.from_version=2.4.1.to_version=3.7.1-SNAPSHOT.bounce_type=full
status:     FAIL
run time:   4 minutes 41.172 seconds
    TimeoutError("Did expect to read 'SMOKE-TEST-CLIENT-STARTED' from 
ducker@ducker04")
Traceback (most recent call last):
  File 
"/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 
186, in _do_run
    data = self.run_test()
  File 
"/usr/local/lib/python3.9/dist-packages/ducktape/tests/runner_client.py", line 
246, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.9/dist-packages/ducktape/mark/_mark.py", line 

[jira] [Resolved] (KAFKA-16752) Implement acquire functionality in SharePartition

[Apoorv Mittal (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apoorv Mittal resolved KAFKA-16752.
---
Resolution: Done

> Implement acquire functionality in SharePartition
> -
>
> Key: KAFKA-16752
> URL: https://issues.apache.org/jira/browse/KAFKA-16752
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Apoorv Mittal
>Assignee: Apoorv Mittal
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.8 #43

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 440526 lines...]
[2024-06-14T11:45:33.732Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> KafkaZkClientTest > testControllerEpochMethods() STARTED
[2024-06-14T11:45:33.732Z] 
[2024-06-14T11:45:33.732Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> KafkaZkClientTest > testControllerEpochMethods() PASSED
[2024-06-14T11:45:33.732Z] 
[2024-06-14T11:45:33.732Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> KafkaZkClientTest > testDeleteRecursive() STARTED
[2024-06-14T11:45:33.732Z] 
[2024-06-14T11:45:33.732Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> KafkaZkClientTest > testDeleteRecursive() PASSED
[2024-06-14T11:45:33.732Z] 
[2024-06-14T11:45:33.732Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> KafkaZkClientTest > testGetTopicPartitionStates() STARTED
[2024-06-14T11:45:34.920Z] 
[2024-06-14T11:45:34.920Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> KafkaZkClientTest > testGetTopicPartitionStates() PASSED
[2024-06-14T11:45:34.920Z] 
[2024-06-14T11:45:34.920Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> KafkaZkClientTest > testCreateConfigChangeNotification() STARTED
[2024-06-14T11:45:34.920Z] 
[2024-06-14T11:45:34.920Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> KafkaZkClientTest > testCreateConfigChangeNotification() PASSED
[2024-06-14T11:45:34.920Z] 
[2024-06-14T11:45:34.920Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> KafkaZkClientTest > testDelegationTokenMethods() STARTED
[2024-06-14T11:45:34.920Z] 
[2024-06-14T11:45:34.920Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> KafkaZkClientTest > testDelegationTokenMethods() PASSED
[2024-06-14T11:45:34.920Z] 
[2024-06-14T11:45:34.920Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> ReassignPartitionsZNodeTest > testDecodeInvalidJson() STARTED
[2024-06-14T11:45:34.920Z] 
[2024-06-14T11:45:34.920Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> ReassignPartitionsZNodeTest > testDecodeInvalidJson() PASSED
[2024-06-14T11:45:34.920Z] 
[2024-06-14T11:45:34.920Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> ReassignPartitionsZNodeTest > testEncode() STARTED
[2024-06-14T11:45:34.920Z] 
[2024-06-14T11:45:34.920Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> ReassignPartitionsZNodeTest > testEncode() PASSED
[2024-06-14T11:45:34.920Z] 
[2024-06-14T11:45:34.920Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> ReassignPartitionsZNodeTest > testDecodeValidJson() STARTED
[2024-06-14T11:45:34.920Z] 
[2024-06-14T11:45:34.920Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> ReassignPartitionsZNodeTest > testDecodeValidJson() PASSED
[2024-06-14T11:45:34.920Z] 
[2024-06-14T11:45:34.920Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > 
testMigrateTopicDeletions [1] Type=ZK, 
MetadataVersion=3.4-IV0,Security=PLAINTEXT STARTED
[2024-06-14T11:46:01.662Z] 
[2024-06-14T11:46:01.662Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > 
testMigrateTopicDeletions [1] Type=ZK, 
MetadataVersion=3.4-IV0,Security=PLAINTEXT SKIPPED
[2024-06-14T11:46:01.662Z] 
[2024-06-14T11:46:01.662Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > 
testMigrateTopicDeletions [2] Type=ZK, 
MetadataVersion=3.5-IV2,Security=PLAINTEXT STARTED
[2024-06-14T11:46:27.155Z] 
[2024-06-14T11:46:27.155Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > 
testMigrateTopicDeletions [2] Type=ZK, 
MetadataVersion=3.5-IV2,Security=PLAINTEXT SKIPPED
[2024-06-14T11:46:27.155Z] 
[2024-06-14T11:46:27.155Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > 
testMigrateTopicDeletions [3] Type=ZK, 
MetadataVersion=3.6-IV2,Security=PLAINTEXT STARTED
[2024-06-14T11:46:53.698Z] 
[2024-06-14T11:46:53.698Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > 
testMigrateTopicDeletions [3] Type=ZK, 
MetadataVersion=3.6-IV2,Security=PLAINTEXT SKIPPED
[2024-06-14T11:46:53.698Z] 
[2024-06-14T11:46:53.698Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > 
testMigrateTopicDeletions [4] Type=ZK, 
MetadataVersion=3.7-IV0,Security=PLAINTEXT STARTED
[2024-06-14T11:47:22.728Z] 
[2024-06-14T11:47:22.728Z] Gradle Test Run :core:test > Gradle Test Executor 51 
> ZkMigrationIntegrationTest > testMigrateTopicDeletions(ClusterInstance) > 
testMigrateTopicDeletions [4] Type=ZK, 

[jira] [Resolved] (KAFKA-16747) Implement share sessions and context

[Abhinav Dixit (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhinav Dixit resolved KAFKA-16747.
---
Resolution: Fixed

> Implement share sessions and context
> 
>
> Key: KAFKA-16747
> URL: https://issues.apache.org/jira/browse/KAFKA-16747
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Apoorv Mittal
>Assignee: Abhinav Dixit
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16959) ConfigCommand should not allow to define both `entity-default` and `entity-name`

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16959:
--

 Summary: ConfigCommand should not allow to define both 
`entity-default` and `entity-name`
 Key: KAFKA-16959
 URL: https://issues.apache.org/jira/browse/KAFKA-16959
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


When users input both `entity-default` and `entity-name`, only `entity-name` 
will get evaluated. It seems to me that is error-prone. We should throw 
exception directly.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.8.0 release

[Satish Duggana]

+1 on going with 3.8 release with the existing plan and targeting the
required features in 3.9 timelines. 4.0 will be targeted in the usual
cycle(4 months) after 3.9 is released.


On Fri, 14 Jun 2024 at 15:19, Edoardo Comar  wrote:
>
> Josep,
> past the deadline sorry but I can't see reasons not to cherry-pick this
> https://github.com/apache/kafka/pull/16326
>
> On Wed, 12 Jun 2024 at 17:14, Josep Prat  wrote:
> >
> > Hi Edoardo,
> >
> > Correct, you can still cherry-pick this one.
> >
> > I'll send an email tomorrow morning (CEST) asking maintainers to stop
> > cherry picking commits unless we discuss it beforehand.
> >
> > Best,
> >
> > On Wed, Jun 12, 2024 at 6:09 PM Edoardo Comar  wrote:
> >
> > > Hi Josep, I understand I am still in time to cherry-pick on 3.8.0
> > > https://github.com/apache/kafka/pull/16229
> > >
> > > right?
> > > thanks
> > >
> > > On Wed, 12 Jun 2024 at 11:34, Ivan Yurchenko  wrote:
> > > >
> > > > Hi,
> > > >
> > > > I'll try to do all the fixes and changes for KIP-899 [1] sooner today,
> > > but please proceed with the release if I don't manage.
> > > >
> > > > Ivan
> > > >
> > > > [1] https://github.com/apache/kafka/pull/13277
> > > >
> > > > On Wed, Jun 12, 2024, at 12:54, Josep Prat wrote:
> > > > > Hi Luke,
> > > > > I think Jose, also mentioned that it won't be ready for v3.8.0 (but he
> > > can
> > > > > confirm this). My question now would be, given that it seems we would
> > > need
> > > > > a v3.9.0, do you think it's important to include
> > > > > https://github.com/apache/kafka/pull/16284 in v3.8.0?
> > > > >
> > > > > Best,
> > > > >
> > > > > On Wed, Jun 12, 2024 at 11:40 AM Luke Chen  wrote:
> > > > >
> > > > > > Hi Josep
> > > > > >
> > > > > > For KIP-966, I think Calvin had mentioned he won't complete in
> > > v3.8.0.
> > > > > > https://lists.apache.org/thread/fsnr8wy5fznzfso7jgk90skgyo277fmw
> > > > > >
> > > > > > For unclean leader election, all we need is this PR:
> > > > > > https://github.com/apache/kafka/pull/16284
> > > > > > For this PR, I think it needs one more week to be completed.
> > > > > >
> > > > > > Thanks.
> > > > > > Luke
> > > > > >
> > > > > > On Wed, Jun 12, 2024 at 4:51 PM Josep Prat
> > > 
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > We are now really close to the planned code freeze deadline (today
> > > EOD).
> > > > > > > According to KIP-1012 [1] we agreed to stay in the 3.x branch
> > > until we
> > > > > > > achieve feature parity regarding Zookeeper and KRaft. The two main
> > > KIPs
> > > > > > > identified that would achieve this are: KIP-853 [2] and KIP-966
> > > [3].
> > > > > > > At the moment of writing this email both KIPs are not completed. 
> > > > > > > My
> > > > > > > question to the people driving both KIPs would be, how much more
> > > time do
> > > > > > > you think it's needed to bring these KIPs to completion?
> > > > > > >
> > > > > > > - If the time needed would be short, we could still include these
> > > 2 KIPs
> > > > > > in
> > > > > > > the release.
> > > > > > > - However, if the time needed would be on the scale of weeks, we
> > > should
> > > > > > > continue with the release plan for 3.8 and after start working on
> > > the 3.9
> > > > > > > release.
> > > > > > >
> > > > > > > What are your thoughts?
> > > > > > >
> > > > > > >
> > > > > > > [1]:
> > > > > > >
> > > > > > >
> > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> > > > > > > [2]:
> > > > > > >
> > > > > > >
> > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes
> > > > > > > [3]:
> > > > > > >
> > > > > > >
> > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas
> > > > > > >
> > > > > > > On Wed, Jun 12, 2024 at 10:40 AM Josep Prat 
> > > wrote:
> > > > > > >
> > > > > > > > Hi Rajini,
> > > > > > > > Yes, we could backport this one to the 3.8 branch. Would you be
> > > able to
> > > > > > > do
> > > > > > > > this once you merge this PR?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram <
> > > > > > rajinisiva...@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi Josep,
> > > > > > > >>
> > > > > > > >> The PR https://github.com/apache/kafka/pull/13277 for KIP-899
> > > looks
> > > > > > > ready
> > > > > > > >> to be merged (waiting for the PR build).The PR changes several
> > > files,
> > > > > > > but
> > > > > > > >> is relatively straightforward and not risky. Also the changes
> > > are
> > > > > > under
> > > > > > > a
> > > > > > > >> config that is not enabled by default. Since the KIP was
> > > approved
> > > > > > before
> > > > > > > >> KIP freeze, will it be ok to include in 3.8.0?
> > > > > > > >>
> > > > > > > >> Thank you,
> > > > > > > >>
> > > > > > > >> Rajini
> > > > > > > >>
> > > > > > > >>
> > > > > > > 

Re: [DISCUSS] Apache Kafka 3.8.0 release

[Edoardo Comar]

Josep,
past the deadline sorry but I can't see reasons not to cherry-pick this
https://github.com/apache/kafka/pull/16326

On Wed, 12 Jun 2024 at 17:14, Josep Prat  wrote:
>
> Hi Edoardo,
>
> Correct, you can still cherry-pick this one.
>
> I'll send an email tomorrow morning (CEST) asking maintainers to stop
> cherry picking commits unless we discuss it beforehand.
>
> Best,
>
> On Wed, Jun 12, 2024 at 6:09 PM Edoardo Comar  wrote:
>
> > Hi Josep, I understand I am still in time to cherry-pick on 3.8.0
> > https://github.com/apache/kafka/pull/16229
> >
> > right?
> > thanks
> >
> > On Wed, 12 Jun 2024 at 11:34, Ivan Yurchenko  wrote:
> > >
> > > Hi,
> > >
> > > I'll try to do all the fixes and changes for KIP-899 [1] sooner today,
> > but please proceed with the release if I don't manage.
> > >
> > > Ivan
> > >
> > > [1] https://github.com/apache/kafka/pull/13277
> > >
> > > On Wed, Jun 12, 2024, at 12:54, Josep Prat wrote:
> > > > Hi Luke,
> > > > I think Jose, also mentioned that it won't be ready for v3.8.0 (but he
> > can
> > > > confirm this). My question now would be, given that it seems we would
> > need
> > > > a v3.9.0, do you think it's important to include
> > > > https://github.com/apache/kafka/pull/16284 in v3.8.0?
> > > >
> > > > Best,
> > > >
> > > > On Wed, Jun 12, 2024 at 11:40 AM Luke Chen  wrote:
> > > >
> > > > > Hi Josep
> > > > >
> > > > > For KIP-966, I think Calvin had mentioned he won't complete in
> > v3.8.0.
> > > > > https://lists.apache.org/thread/fsnr8wy5fznzfso7jgk90skgyo277fmw
> > > > >
> > > > > For unclean leader election, all we need is this PR:
> > > > > https://github.com/apache/kafka/pull/16284
> > > > > For this PR, I think it needs one more week to be completed.
> > > > >
> > > > > Thanks.
> > > > > Luke
> > > > >
> > > > > On Wed, Jun 12, 2024 at 4:51 PM Josep Prat
> > 
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > We are now really close to the planned code freeze deadline (today
> > EOD).
> > > > > > According to KIP-1012 [1] we agreed to stay in the 3.x branch
> > until we
> > > > > > achieve feature parity regarding Zookeeper and KRaft. The two main
> > KIPs
> > > > > > identified that would achieve this are: KIP-853 [2] and KIP-966
> > [3].
> > > > > > At the moment of writing this email both KIPs are not completed. My
> > > > > > question to the people driving both KIPs would be, how much more
> > time do
> > > > > > you think it's needed to bring these KIPs to completion?
> > > > > >
> > > > > > - If the time needed would be short, we could still include these
> > 2 KIPs
> > > > > in
> > > > > > the release.
> > > > > > - However, if the time needed would be on the scale of weeks, we
> > should
> > > > > > continue with the release plan for 3.8 and after start working on
> > the 3.9
> > > > > > release.
> > > > > >
> > > > > > What are your thoughts?
> > > > > >
> > > > > >
> > > > > > [1]:
> > > > > >
> > > > > >
> > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> > > > > > [2]:
> > > > > >
> > > > > >
> > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes
> > > > > > [3]:
> > > > > >
> > > > > >
> > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas
> > > > > >
> > > > > > On Wed, Jun 12, 2024 at 10:40 AM Josep Prat 
> > wrote:
> > > > > >
> > > > > > > Hi Rajini,
> > > > > > > Yes, we could backport this one to the 3.8 branch. Would you be
> > able to
> > > > > > do
> > > > > > > this once you merge this PR?
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram <
> > > > > rajinisiva...@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Josep,
> > > > > > >>
> > > > > > >> The PR https://github.com/apache/kafka/pull/13277 for KIP-899
> > looks
> > > > > > ready
> > > > > > >> to be merged (waiting for the PR build).The PR changes several
> > files,
> > > > > > but
> > > > > > >> is relatively straightforward and not risky. Also the changes
> > are
> > > > > under
> > > > > > a
> > > > > > >> config that is not enabled by default. Since the KIP was
> > approved
> > > > > before
> > > > > > >> KIP freeze, will it be ok to include in 3.8.0?
> > > > > > >>
> > > > > > >> Thank you,
> > > > > > >>
> > > > > > >> Rajini
> > > > > > >>
> > > > > > >>
> > > > > > >> On Tue, Jun 11, 2024 at 9:35 AM Josep Prat
> > > > >  > > > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Hi all,
> > > > > > >> >
> > > > > > >> > I just want to remind everybody that the code freeze deadline
> > is
> > > > > > >> > approaching. June 12th EOD is the deadline.
> > > > > > >> >
> > > > > > >> > Please do not automatically backport any commit to the 3.8
> > branch
> > > > > > after
> > > > > > >> > June 12th EOD. Ping me if you think the commit should be
> > backported
> > > > > > >> (fixes
> > > > > > >> > 

[jira] [Created] (KAFKA-16958) add `STRICT_STUBS` to `EndToEndLatencyTest`, `OffsetCommitCallbackInvokerTest`, `ProducerPerformanceTest`, and `TopologyTest`

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16958:
--

 Summary: add `STRICT_STUBS` to `EndToEndLatencyTest`, 
`OffsetCommitCallbackInvokerTest`, `ProducerPerformanceTest`, and `TopologyTest`
 Key: KAFKA-16958
 URL: https://issues.apache.org/jira/browse/KAFKA-16958
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


as title



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16957) Enable KafkaConsumerTest#configurableObjectsShouldSeeGeneratedClientId to work with CLASSIC and CONSUMER

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16957:
--

 Summary: Enable 
KafkaConsumerTest#configurableObjectsShouldSeeGeneratedClientId to work with 
CLASSIC and CONSUMER 
 Key: KAFKA-16957
 URL: https://issues.apache.org/jira/browse/KAFKA-16957
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


The `CLIENT_IDS` is a static variable, so the latter one will see previous test 
results. We should clear it before testing.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.7.1 RC1

[Edoardo Comar]

well if you still need to build RC2, I'd cherry pick this that I
missed from the other PR
https://github.com/apache/kafka/pull/16326

On Thu, 13 Jun 2024 at 13:20, Igor Soarez  wrote:
>
> Hi Edoardo,
>
> It is late, but not too late. I have cherry-picked your change
> to the 3.7 branch and I'll build a second release candidate.
>
> If you could have a look at the first RC, please let me know if
> you spot any issues with it that can be avoided in the next RC.
>
> Thanks,
>
> --
> Igor

[jira] [Resolved] (KAFKA-16317) Add event rate in GroupCoordinatorRuntimeMetrics

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-16317.
-
Resolution: Won't Do

> Add event rate in GroupCoordinatorRuntimeMetrics
> 
>
> Key: KAFKA-16317
> URL: https://issues.apache.org/jira/browse/KAFKA-16317
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jeff Kim
>Assignee: Jeff Kim
>Priority: Major
>
> We want a sensor to record every time we process a new event in the 
> coordinator runtime.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16674) Adjust classicGroupJoinToConsumerGroup to add subscription model

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-16674.
-
Resolution: Fixed

This was done in https://github.com/apache/kafka/pull/15785.

> Adjust classicGroupJoinToConsumerGroup to add subscription model
> 
>
> Key: KAFKA-16674
> URL: https://issues.apache.org/jira/browse/KAFKA-16674
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Dongnuo Lyu
>Assignee: Dongnuo Lyu
>Priority: Major
>
> [https://github.com/apache/kafka/pull/15785] adds subscription model to the 
> group state that affects `classicGroupJoinToConsumerGroup`. We'll need to 
> make adjustment to comply with the change once #15785 is merged.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-16673) Optimize toTopicPartitions with ConsumerProtocolSubscription

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot reopened KAFKA-16673:
-

> Optimize toTopicPartitions with ConsumerProtocolSubscription
> 
>
> Key: KAFKA-16673
> URL: https://issues.apache.org/jira/browse/KAFKA-16673
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Dongnuo Lyu
>Assignee: Dongnuo Lyu
>Priority: Major
> Fix For: 3.8.0
>
>
> https://github.com/apache/kafka/pull/15798#discussion_r1582981154



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16673) Optimize toTopicPartitions with ConsumerProtocolSubscription

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-16673.
-
Fix Version/s: 3.8.0
   Resolution: Duplicate

This was done as part of https://github.com/apache/kafka/pull/15785.

> Optimize toTopicPartitions with ConsumerProtocolSubscription
> 
>
> Key: KAFKA-16673
> URL: https://issues.apache.org/jira/browse/KAFKA-16673
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Dongnuo Lyu
>Assignee: Dongnuo Lyu
>Priority: Major
> Fix For: 3.8.0
>
>
> https://github.com/apache/kafka/pull/15798#discussion_r1582981154



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-12708) Rewrite org.apache.kafka.test.Microbenchmarks by JMH

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-12708.

Fix Version/s: 3.9.0
   Resolution: Fixed

> Rewrite org.apache.kafka.test.Microbenchmarks by JMH
> 
>
> Key: KAFKA-12708
> URL: https://issues.apache.org/jira/browse/KAFKA-12708
> Project: Kafka
>  Issue Type: Task
>Reporter: Chia-Ping Tsai
>Assignee: Kuan Po Tseng
>Priority: Minor
> Fix For: 3.9.0
>
>
> The benchmark code is a bit obsolete and it would be better to rewrite it by 
> JMH



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16942) Use ConcurrentHashMap in RecordAccumulator#nodeStats

[Kuan Po Tseng (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kuan Po Tseng resolved KAFKA-16942.
---
Resolution: Won't Do

The performance in get method is not obvious between ConcurrentHashMap and 
CopyOnWriteMap. And doing this breaks the code consistency (some place use 
CopyOnWriteMap while some are not). This seems not bring too much benefit.

> Use ConcurrentHashMap in RecordAccumulator#nodeStats
> 
>
> Key: KAFKA-16942
> URL: https://issues.apache.org/jira/browse/KAFKA-16942
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Kuan Po Tseng
>Assignee: Kuan Po Tseng
>Priority: Major
>
> per discussed in 
> [https://github.com/apache/kafka/pull/16231#discussion_r1635345881]
> Through the ConcurrentMapBenchmark, we observed that in scenarios where write 
> operations (i.e., computeIfAbsent) constitute 10%, the get performance of 
> CopyOnWriteMap is lower compared to ConcurrentHashMap. However, when 
> iterating over entrySet and values, CopyOnWriteMap performs better than 
> ConcurrentHashMap.
> In RecordAccumulator#nodeStats, the computeIfAbsent method is rarely 
> triggered, and we only use the get method to read data. Therefore, switching 
> to ConcurrentHashMap would gain better performance.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

[Mario Fiore Vitale]

Hi All,

We have 4 binding +1 votes. Do you think the vote can be closed and mark
the KIP as approved?

Thanks,
Mario.

On Fri, Jun 14, 2024 at 8:44 AM Punsak Incham  wrote:

> Thanks! Chris.
> Waiting @Mario Fiore Vitale to close the voting and mark it as approved.
>
> Cheers.
>
> On 2024/06/14 06:10:15 Chris Egerton wrote:
> > We don't have to patch every SMT in the same release, we can
> > definitely move incrementally. We'll just have to note in the release
> > notes and the KIPs page (
> > https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik
> > i.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKafka%2BImprovement%2BPr
> > oposals=05%7C02%7Cpunsak%40mfec.co.th%7Ca24883ed513b45d31ca508dc8
> > c3c2de1%7C74105ed972ff4685915475f7408b6f67%7C1%7C0%7C63853943730004293
> > 1%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6
> > Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=mcPQROjQyuK1Z2vDl%2BLVki8AeB
> > FsnTIkqxaX8FR2LRA%3D=0) when the change for the KIP was
> > applied to each SMT.
> >
> > On Thu, Jun 13, 2024 at 11:03 PM Punsak Incham
> > 
> > wrote:
> >
> > > Thanks! Chris.
> > >
> > > One question : We will need to wait to fix all SMTs in the KIP and
> > > release it to user, or we can split some fixed SMTs to user early?
> > > (I noticed that the PR of Mario ready to merge and it effect only
> > > ExtractField and
> > > InsertField)
> > >
> > >
> > > On 2024/06/13 17:01:49 Chris Egerton wrote:
> > > > Hi Punsak,
> > > >
> > > > If nobody has signaled their intent to contribute that work yet
> > > > (which I believe is the case), you are welcome to take it on
> yourself!
> > > >
> > > > Cheers,
> > > >
> > > > Cheers
> > > >
> > > > On Thu, Jun 13, 2024, 12:52 Punsak Incham
> > > > 
> > > wrote:
> > > >
> > > > > My customers are using these SMTs, I think it can affect to
> > > > > their
> > > project
> > > > > (about data correctness) in the future.
> > > > >
> > > > > (You may think why I'm not developing it and send to my
> > > > > customers directly? Because they will use only SMTs that
> > > > > published by Kafka.)
> > > > >
> > > > > On 2024/06/13 16:38:45 Punsak Incham wrote:
> > > > > > Hi all,
> > > > > > I noticed that the PR of Mario effected only InsertField and
> > > > > ExtractField, so I'd like to amend others SMTs in that KIP
> > > > > because I
> > > have
> > > > > experienced to develop custom SMTs.
> > > > > > Can I join to contribute (open PR)?
> > > > > >
> > > > > > On 2024/06/13 15:59:36 Greg Harris wrote:
> > > > > > > Hey Mario,
> > > > > > >
> > > > > > > Thanks for updating the KIP.
> > > > > > > +1 (binding)
> > > > > > >
> > > > > > > Greg
> > > > > > >
> > > > > > > On Thu, Jun 13, 2024 at 1:32 AM Mario Fiore Vitale <
> > > mv...@redhat.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Greg,
> > > > > > > >
> > > > > > > > > However, the KIP still omits the MaskField and
> > > > > > > > > ValueToKey
> > > > > > > > transformations.
> > > > > > > > This looks like just a typo to me, should we update the
> > > > > > > > KIP
> > > before
> > > > > closing
> > > > > > > > the vote?
> > > > > > > >
> > > > > > > > Good catch. I have just updated the KIP.
> > > > > > > >
> > > > > > > > I think that we can close the voting and mark it as
> > > > > > > > approved,
> > > right?
> > > > > > > >
> > > > > > > > Thank you all.
> > > > > > > >
> > > > > > > > On Wed, Jun 12, 2024 at 7:21 PM Greg Harris
> > > 
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Mario,
> > > > > > > > >
> > > > > > > > > Thank you for mentioning the earlier DISCUSS thread. I
> > > > > > > > > found
> > > this
> > > > > comment
> > > > > > > > > from Chris, which was agreed upon and applied to the KIP:
> > > > > > > > >
> > > > > > > > > > Yes, I think we should just do one KIP for all the
> > > > > > > > > > SMTs. You
> > > > > don't have
> > > > > > > > > to
> > > > > > > > > > implement everything all at once or by yourself, but I
> > > > > > > > > > don't
> > > see
> > > > > why we
> > > > > > > > > > should require one or more follow-up KIPs to apply the
> > > > > > > > > > exact
> > > same
> > > > > > > > changes
> > > > > > > > > > to the SMTs we missed the first time.
> > > > > > > > >
> > > > > > > > > However, the KIP still omits the MaskField and
> > > > > > > > > ValueToKey
> > > > > > > > transformations.
> > > > > > > > > This looks like just a typo to me, should we update the
> > > > > > > > > KIP
> > > before
> > > > > > > > closing
> > > > > > > > > the vote?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Greg
> > > > > > > > >
> > > > > > > > > On Wed, Jun 12, 2024 at 6:50 AM Mario Fiore Vitale <
> > > > > mv...@redhat.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Greg,
> > > > > > > > > >
> > > > > > > > > > The same check was made by Chris Egerton during the
> > > discussion
> > > > > thread.
> > > > > > > > > >
> > > > > > > > > > The original KIP scope was just for the InsertField,
> > > > > 

[jira] [Created] (KAFKA-16956) Broker-side ability to subscribe to record delete events

[Luke Chen (Jira)]

Luke Chen created KAFKA-16956:
-

 Summary: Broker-side ability to subscribe to record delete events
 Key: KAFKA-16956
 URL: https://issues.apache.org/jira/browse/KAFKA-16956
 Project: Kafka
  Issue Type: Improvement
Reporter: Luke Chen


In some cases it would be useful for systems outside Kafka to have the ability 
to know when Kafka deletes records (tombstoning or retention).

In general the use-case is where there is a desire to link the lifecycle of a 
record in a third party system (database or filesystem etc) to the lifecycle of 
the record in Kafka.

A concrete use-case:  a system using Kafka to distribute video clips + 
metadata.  The binary content is too big to store in Kafka so the publishing 
application caches the content in cloud storage and publishes a record 
containing a S3 url to the video clip.  The desire is to have a mechanism to 
remove the clip from cloud storage at the same time the record is expunged from 
Kafka by retention or tombstoning.  Currently there is no practical way to 
achieve this.
h2. Desired solution

A pluggable broker-side mechanism that is informed as records are being 
compacted away or deleted.  The API would expose the topic from which the 
record is being deleted, the record key, record headers, timestamp and 
(possibly) record value.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

RE: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

[Punsak Incham]

Thanks! Chris.
Waiting @Mario Fiore Vitale to close the voting and mark it as approved.

Cheers.

On 2024/06/14 06:10:15 Chris Egerton wrote:
> We don't have to patch every SMT in the same release, we can 
> definitely move incrementally. We'll just have to note in the release 
> notes and the KIPs page (
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik
> i.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKafka%2BImprovement%2BPr
> oposals=05%7C02%7Cpunsak%40mfec.co.th%7Ca24883ed513b45d31ca508dc8
> c3c2de1%7C74105ed972ff4685915475f7408b6f67%7C1%7C0%7C63853943730004293
> 1%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6
> Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C=mcPQROjQyuK1Z2vDl%2BLVki8AeB
> FsnTIkqxaX8FR2LRA%3D=0) when the change for the KIP was 
> applied to each SMT.
>
> On Thu, Jun 13, 2024 at 11:03 PM Punsak Incham 
> 
> wrote:
>
> > Thanks! Chris.
> >
> > One question : We will need to wait to fix all SMTs in the KIP and 
> > release it to user, or we can split some fixed SMTs to user early? 
> > (I noticed that the PR of Mario ready to merge and it effect only 
> > ExtractField and
> > InsertField)
> >
> >
> > On 2024/06/13 17:01:49 Chris Egerton wrote:
> > > Hi Punsak,
> > >
> > > If nobody has signaled their intent to contribute that work yet 
> > > (which I believe is the case), you are welcome to take it on yourself!
> > >
> > > Cheers,
> > >
> > > Cheers
> > >
> > > On Thu, Jun 13, 2024, 12:52 Punsak Incham 
> > > 
> > wrote:
> > >
> > > > My customers are using these SMTs, I think it can affect to 
> > > > their
> > project
> > > > (about data correctness) in the future.
> > > >
> > > > (You may think why I'm not developing it and send to my 
> > > > customers directly? Because they will use only SMTs that 
> > > > published by Kafka.)
> > > >
> > > > On 2024/06/13 16:38:45 Punsak Incham wrote:
> > > > > Hi all,
> > > > > I noticed that the PR of Mario effected only InsertField and
> > > > ExtractField, so I'd like to amend others SMTs in that KIP 
> > > > because I
> > have
> > > > experienced to develop custom SMTs.
> > > > > Can I join to contribute (open PR)?
> > > > >
> > > > > On 2024/06/13 15:59:36 Greg Harris wrote:
> > > > > > Hey Mario,
> > > > > >
> > > > > > Thanks for updating the KIP.
> > > > > > +1 (binding)
> > > > > >
> > > > > > Greg
> > > > > >
> > > > > > On Thu, Jun 13, 2024 at 1:32 AM Mario Fiore Vitale <
> > mv...@redhat.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Greg,
> > > > > > >
> > > > > > > > However, the KIP still omits the MaskField and 
> > > > > > > > ValueToKey
> > > > > > > transformations.
> > > > > > > This looks like just a typo to me, should we update the 
> > > > > > > KIP
> > before
> > > > closing
> > > > > > > the vote?
> > > > > > >
> > > > > > > Good catch. I have just updated the KIP.
> > > > > > >
> > > > > > > I think that we can close the voting and mark it as 
> > > > > > > approved,
> > right?
> > > > > > >
> > > > > > > Thank you all.
> > > > > > >
> > > > > > > On Wed, Jun 12, 2024 at 7:21 PM Greg Harris
> > 
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Mario,
> > > > > > > >
> > > > > > > > Thank you for mentioning the earlier DISCUSS thread. I 
> > > > > > > > found
> > this
> > > > comment
> > > > > > > > from Chris, which was agreed upon and applied to the KIP:
> > > > > > > >
> > > > > > > > > Yes, I think we should just do one KIP for all the 
> > > > > > > > > SMTs. You
> > > > don't have
> > > > > > > > to
> > > > > > > > > implement everything all at once or by yourself, but I 
> > > > > > > > > don't
> > see
> > > > why we
> > > > > > > > > should require one or more follow-up KIPs to apply the 
> > > > > > > > > exact
> > same
> > > > > > > changes
> > > > > > > > > to the SMTs we missed the first time.
> > > > > > > >
> > > > > > > > However, the KIP still omits the MaskField and 
> > > > > > > > ValueToKey
> > > > > > > transformations.
> > > > > > > > This looks like just a typo to me, should we update the 
> > > > > > > > KIP
> > before
> > > > > > > closing
> > > > > > > > the vote?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Greg
> > > > > > > >
> > > > > > > > On Wed, Jun 12, 2024 at 6:50 AM Mario Fiore Vitale <
> > > > mv...@redhat.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Greg,
> > > > > > > > >
> > > > > > > > > The same check was made by Chris Egerton during the
> > discussion
> > > > thread.
> > > > > > > > >
> > > > > > > > > The original KIP scope was just for the InsertField,
> > > > ExtractField SMTs,
> > > > > > > > > then we decided to enlarge the scope of only the KIP 
> > > > > > > > > to also
> > > > other
> > > > > > > > > potential affected SMTs.
> > > > > > > > >
> > > > > > > > > As of now the PR scope, instead, is only for 
> > > > > > > > > InsertField and
> > > > > > > ExtractField
> > > > > > > > > SMTs.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Mario.
> > > > > > > > >
> > > > > > > > >
> > > 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #195

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 339865 lines...]
[2024-06-14T06:03:13.659Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificActivePartitionStores STARTED
[2024-06-14T06:03:25.543Z] 
[2024-06-14T06:03:25.543Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificActivePartitionStores PASSED
[2024-06-14T06:03:25.543Z] 
[2024-06-14T06:03:25.543Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryAllStalePartitionStores STARTED
[2024-06-14T06:03:28.766Z] 
[2024-06-14T06:03:28.766Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryAllStalePartitionStores PASSED
[2024-06-14T06:03:28.766Z] 
[2024-06-14T06:03:28.766Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads STARTED
[2024-06-14T06:03:34.423Z] 
[2024-06-14T06:03:34.423Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreads PASSED
[2024-06-14T06:03:34.423Z] 
[2024-06-14T06:03:34.423Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStores STARTED
[2024-06-14T06:03:39.681Z] 
[2024-06-14T06:03:39.681Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStores PASSED
[2024-06-14T06:03:39.681Z] 
[2024-06-14T06:03:39.681Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryOnlyActivePartitionStoresByDefault STARTED
[2024-06-14T06:03:44.630Z] 
[2024-06-14T06:03:44.630Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryOnlyActivePartitionStoresByDefault PASSED
[2024-06-14T06:03:44.630Z] 
[2024-06-14T06:03:44.630Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread STARTED
[2024-06-14T06:03:53.248Z] 
[2024-06-14T06:03:53.248Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQueryStoresAfterAddingAndRemovingStreamThread PASSED
[2024-06-14T06:03:53.248Z] 
[2024-06-14T06:03:53.248Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology STARTED
[2024-06-14T06:03:59.639Z] 
[2024-06-14T06:03:59.639Z] 
org.apache.kafka.streams.integration.StoreQueryIntegrationTest > 
shouldQuerySpecificStalePartitionStoresMultiStreamThreadsNamedTopology PASSED
[2024-06-14T06:03:59.639Z] 
[2024-06-14T06:03:59.639Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInnerRepartitioned[caching enabled = true] STARTED
[2024-06-14T06:04:07.115Z] 
[2024-06-14T06:04:07.115Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInnerRepartitioned[caching enabled = true] PASSED
[2024-06-14T06:04:07.115Z] 
[2024-06-14T06:04:07.115Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuterRepartitioned[caching enabled = true] STARTED
[2024-06-14T06:04:16.771Z] 
[2024-06-14T06:04:16.771Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuterRepartitioned[caching enabled = true] PASSED
[2024-06-14T06:04:16.771Z] 
[2024-06-14T06:04:16.771Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInner[caching enabled = true] STARTED
[2024-06-14T06:04:21.654Z] 
[2024-06-14T06:04:21.654Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testInner[caching enabled = true] PASSED
[2024-06-14T06:04:21.654Z] 
[2024-06-14T06:04:21.654Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuter[caching enabled = true] STARTED
[2024-06-14T06:04:28.189Z] 
[2024-06-14T06:04:28.189Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testOuter[caching enabled = true] PASSED
[2024-06-14T06:04:28.189Z] 
[2024-06-14T06:04:28.189Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeft[caching enabled = true] STARTED
[2024-06-14T06:04:35.783Z] 
[2024-06-14T06:04:35.783Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeft[caching enabled = true] PASSED
[2024-06-14T06:04:35.783Z] 
[2024-06-14T06:04:35.783Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testMultiInner[caching enabled = true] STARTED
[2024-06-14T06:04:43.496Z] 
[2024-06-14T06:04:43.496Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testMultiInner[caching enabled = true] PASSED
[2024-06-14T06:04:43.496Z] 
[2024-06-14T06:04:43.496Z] 
org.apache.kafka.streams.integration.StreamStreamJoinIntegrationTest > 
testLeftRepartitioned[caching enabled = true] STARTED
[2024-06-14T06:04:54.728Z] 
[2024-06-14T06:04:54.728Z] 

Re: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

[Chris Egerton]

We don't have to patch every SMT in the same release, we can definitely
move incrementally. We'll just have to note in the release notes and the
KIPs page (
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals)
when the change for the KIP was applied to each SMT.

On Thu, Jun 13, 2024 at 11:03 PM Punsak Incham 
wrote:

> Thanks! Chris.
>
> One question : We will need to wait to fix all SMTs in the KIP and release
> it to user, or we can split some fixed SMTs to user early? (I noticed that
> the PR of Mario ready to merge and it effect only ExtractField and
> InsertField)
>
>
> On 2024/06/13 17:01:49 Chris Egerton wrote:
> > Hi Punsak,
> >
> > If nobody has signaled their intent to contribute that work yet (which I
> > believe is the case), you are welcome to take it on yourself!
> >
> > Cheers,
> >
> > Cheers
> >
> > On Thu, Jun 13, 2024, 12:52 Punsak Incham 
> wrote:
> >
> > > My customers are using these SMTs, I think it can affect to their
> project
> > > (about data correctness) in the future.
> > >
> > > (You may think why I'm not developing it and send to my customers
> > > directly? Because they will use only SMTs that published by Kafka.)
> > >
> > > On 2024/06/13 16:38:45 Punsak Incham wrote:
> > > > Hi all,
> > > > I noticed that the PR of Mario effected only InsertField and
> > > ExtractField, so I'd like to amend others SMTs in that KIP because I
> have
> > > experienced to develop custom SMTs.
> > > > Can I join to contribute (open PR)?
> > > >
> > > > On 2024/06/13 15:59:36 Greg Harris wrote:
> > > > > Hey Mario,
> > > > >
> > > > > Thanks for updating the KIP.
> > > > > +1 (binding)
> > > > >
> > > > > Greg
> > > > >
> > > > > On Thu, Jun 13, 2024 at 1:32 AM Mario Fiore Vitale <
> mv...@redhat.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Greg,
> > > > > >
> > > > > > > However, the KIP still omits the MaskField and ValueToKey
> > > > > > transformations.
> > > > > > This looks like just a typo to me, should we update the KIP
> before
> > > closing
> > > > > > the vote?
> > > > > >
> > > > > > Good catch. I have just updated the KIP.
> > > > > >
> > > > > > I think that we can close the voting and mark it as approved,
> right?
> > > > > >
> > > > > > Thank you all.
> > > > > >
> > > > > > On Wed, Jun 12, 2024 at 7:21 PM Greg Harris
> 
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Mario,
> > > > > > >
> > > > > > > Thank you for mentioning the earlier DISCUSS thread. I found
> this
> > > comment
> > > > > > > from Chris, which was agreed upon and applied to the KIP:
> > > > > > >
> > > > > > > > Yes, I think we should just do one KIP for all the SMTs. You
> > > don't have
> > > > > > > to
> > > > > > > > implement everything all at once or by yourself, but I don't
> see
> > > why we
> > > > > > > > should require one or more follow-up KIPs to apply the exact
> same
> > > > > > changes
> > > > > > > > to the SMTs we missed the first time.
> > > > > > >
> > > > > > > However, the KIP still omits the MaskField and ValueToKey
> > > > > > transformations.
> > > > > > > This looks like just a typo to me, should we update the KIP
> before
> > > > > > closing
> > > > > > > the vote?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Greg
> > > > > > >
> > > > > > > On Wed, Jun 12, 2024 at 6:50 AM Mario Fiore Vitale <
> > > mv...@redhat.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Greg,
> > > > > > > >
> > > > > > > > The same check was made by Chris Egerton during the
> discussion
> > > thread.
> > > > > > > >
> > > > > > > > The original KIP scope was just for the InsertField,
> > > ExtractField SMTs,
> > > > > > > > then we decided to enlarge the scope of only the KIP to also
> > > other
> > > > > > > > potential affected SMTs.
> > > > > > > >
> > > > > > > > As of now the PR scope, instead, is only for InsertField and
> > > > > > ExtractField
> > > > > > > > SMTs.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Mario.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Jun 12, 2024 at 12:08 AM Greg Harris
> > > > > > >  > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Mario,
> > > > > > > > >
> > > > > > > > > Thanks for the KIP. I'm on-board with this KIP, I just
> wanted
> > > to
> > > > > > > verify a
> > > > > > > > > discrepancy I noticed.
> > > > > > > > >
> > > > > > > > > I checked all of the call-sites of Struct#get(Field) and
> > > > > > > > Struct#get(String)
> > > > > > > > > in Kafka, and noticed there are some call-sites which are
> not
> > > > > > included
> > > > > > > in
> > > > > > > > > the KIP.
> > > > > > > > > 1. The Flatten transformation seems to already have the
> > > > > > > > > "replace.null.with.default=false" behavior unconditionally.
> > > > > > > > > 2. The MaskField transformation unconditionally injects
> default
> > > > > > values
> > > > > > > > for
> > > > > > > > > top-level structs.
> > > > > > > > > 3. The ValueTokey transformation injects defaults for each
> of
> > > the
> > > >