date:20240612

Hi all,

We are now past the code freeze for the 3.8.0 release. If you think a
commit should be backported to the 3.8 branch, please ping me in the PR
(@jlprat).

Thanks!

On Wed, Jun 12, 2024 at 7:22 PM José Armando García Sancio
 wrote:

> Hi Josep,
>
> See my comment below.
>
> On Wed, Jun 12, 2024 at 1:17 PM Josep Prat 
> wrote:
> > How long do you think it will take to bring KIP-853 to completion?
>
> We are still missing a few issues/jiras that need to get implemented
> for the feature to be usable. I would say a few more weeks. May be
> early July or mid July.
>
> Thanks,
> --
> -José
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

[jira] [Resolved] (KAFKA-16901) add unit tests for ConsumerRecords#records(String)



 [ 
https://issues.apache.org/jira/browse/KAFKA-16901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16901.

Fix Version/s: 3.9.0
   Resolution: Fixed

> add unit tests for ConsumerRecords#records(String)
> --
>
> Key: KAFKA-16901
> URL: https://issues.apache.org/jira/browse/KAFKA-16901
> Project: Kafka
>  Issue Type: Test
>Reporter: Chia-Ping Tsai
>Assignee: TengYao Chi
>Priority: Minor
> Fix For: 3.9.0
>
>
> ConsumerRecords#records(String) is a public API, so it is worthy to have unit 
> test :) 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16948) Reset tier lag metrics on becoming follower

2024-06-12 Thread Kamal Chandraprakash (Jira)

Kamal Chandraprakash created KAFKA-16948:


 Summary: Reset tier lag metrics on becoming follower
 Key: KAFKA-16948
 URL: https://issues.apache.org/jira/browse/KAFKA-16948
 Project: Kafka
  Issue Type: Task
Reporter: Kamal Chandraprakash
Assignee: Kamal Chandraprakash


Tier lag metrics such as remoteCopyLagBytes and remoteCopyLagSegments are not 
cleared sometimes when the node transitions from leader to follower post a 
rolling restart.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.8 #37

See

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #3002

See

[jira] [Resolved] (KAFKA-16938) non-dynamic props gets corrupted due to circular reference between DynamicBrokerConfig and DynamicConfig



 [ 
https://issues.apache.org/jira/browse/KAFKA-16938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16938.

Resolution: Fixed

> non-dynamic props gets corrupted due to circular reference between 
> DynamicBrokerConfig and DynamicConfig
> 
>
> Key: KAFKA-16938
> URL: https://issues.apache.org/jira/browse/KAFKA-16938
> Project: Kafka
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: TengYao Chi
>Priority: Blocker
> Fix For: 3.9.0
>
>
> DynamicBrokerConfig has circular reference with DynamicConfig. The following 
> initialization order will cause incorrect non-dynamic props [0]
> 1. DynamicConfig is initializing -> brokerConfigs is created [1]
> 2. DynamicConfig is initializing -> call 
> DynamicBrokerConfig.addDynamicConfigs(brokerConfigs) [2]
> 3. DynamicBrokerConfig is initializing -> nonDynamicProps: Set[String] = 
> KafkaConfig.configNames.toSet – DynamicConfig.Broker.names.asScala [3]
> 4. DynamicConfig.Broker.names reference `brokerConfigs`, and `brokerConfigs` 
> does not have all dynamic props (step2), so nonDynamicProps get created with 
> incorrect contents.
> We should break the circular by addressing following tasks:
> 1. move `DynamicBrokerConfig.addDynamicConfigs` to `DynamicConfig.Broker`
> 2. move `DynamicBrokerConfig#nonDynamicProps` to `DynamicConfig.Broker`
> {code:scala}
> object DynamicConfig {
>   object Broker {
> private val brokerConfigs = {
>   val configs = QuotaConfigs.brokerQuotaConfigs()
>   KafkaConfig.configKeys.filter {
> case (configName, _) => AllDynamicConfigs.contains(configName)
>   }.foreach {
> case (_, config) =>
>   configs.define(config.name, config.`type`, config.defaultValue, 
> config.validator,
> config.importance, config.documentation, config.group, 
> config.orderInGroup, config.width,
> config.displayName, config.dependents, config.recommender)
>   }
>   configs
> }
> val nonDynamicProps: Set[String] = KafkaConfig.configNames.toSet -- 
> brokerConfigs.names.asScala
> {code}
> [0] 
> [https://github.com/apache/kafka/blob/638844f833b165d6f9ca52c173858d26b7254fac/core/src/main/scala/kafka/server/DynamicBrokerConfig.scala#L120]
> [1] 
> [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/DynamicConfig.scala#L35]
> [2] 
> [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/DynamicConfig.scala#L36]
> [3] 
> [https://github.com/apache/kafka/blob/638844f833b165d6f9ca52c173858d26b7254fac/core/src/main/scala/kafka/server/DynamicBrokerConfig.scala#L120]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-655: Add deduplication processor in kafka-streams

106:

For the use-case of deduplicating a "at least once written" stream,
we are sure that the duplicate record has the same key as the
original one, and will land on the same task. Here, a user would
want to specify a deduplication key different from the topic's key
in case the topic's key is not a unique identifier

For example, we have a topic with keyValue (`userId`, `transaction`)
and deduplication is done on `transaction`.`id` . Here, the application
wants to deduplicate transactions. It knows that a transaction id
maps to a single userId. Any duplicate of that record would be received
by the task which processes this userId.

This is an interesting point.

My concern is to some extend, that it seems (on the surface) to not
follow the established pattern of auto-repartitioning in the DSL. Of
course, given that the current proposal says we use an "id extractor"
and not a "key extractor" it might be ok (but it might be somewhat
subtle). Of course, JavaDocs always help to explain in detail. Would
this be enough?

Would be good to hear from others about this point. I am personally not
sure which approach I would prefer personally at this point.

The problem reminds me on
https://issues.apache.org/jira/browse/KAFKA-10844 which is not resolve
directly either. We do have KIP-759
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-759%3A+Unneeded+repartition+canceling)
which is WIP and helps with KAFKA-10844, but not sure if it would be a
viable solution for the de-duplication case?

-Matthias

On 6/11/24 2:31 PM, Ayoub Omari wrote:

Hi Sebastien & Matthias,

For 106.
My idea was to deduplicate on a per-task basis. If the user wants
to do a global deduplication over all partitions, I think it's better to
have him explicitly repartition and then call the deduplication processor.

One other thought I have when writing the KIP about global deduplication,
is that it will require to map twice the key of the stream (first map to
change the key to deduplication key, and second map to get
back the initial key). Second map may imply a second repartitioning.

However, if we do a per-task deduplication, the user may adapt to his
specific use-case.

Let me know what you think
Ayoub

Le mar. 11 juin 2024 à 20:21, Matthias J. Sax a écrit :

Thanks Sebastien,

that's a good point. Thanks for raising it. -- I like your proposal.

An alternative would be to have two overloads of `deduplicate()` one w/
and one w/o the "id extractor" parameter. This would be less explicit
though.

-Matthias

On 6/11/24 2:30 AM, Sebastien Viale wrote:

Hi,

I am really interested in this KIP.

106:
I hope I am not talking nonsense, but if you do not deduplicate based on

the key, the input stream has to be repartitioned.

Otherwise, different stream tasks may handle records that need to be

deduplicated, and thus duplicates will not be detected.

This is why I would have created two different methods, as is done for

GroupBy:

deduplicateByKey(...)
deduplicate(...)

If deduplicateByKey is used, the input stream does not need to be

repartitioned.

thanks

Sébastien

De : Matthias J. Sax
Envoyé : mardi 11 juin 2024 01:54
À : dev@kafka.apache.org
Objet : [EXT] Re: [DISCUSS] KIP-655: Add deduplication processor in

kafka-streams

Warning External sender Do not click on any links or open any

attachments unless you trust the sender and know the content is safe.

Thanks for the update Ayoub.

101: you say:

But I am not sure if we don't want to have them for this processor ?

What is your reasoning to move off the established pattern? Would be
good to understand, why `Deduplicated` class needs a different
"structure" compared to existing classes.

102: Creating iterators is very expensive. For other work, we actually
hit 100x (?) throughput degradation by creating an (for most cases
empty) iterator for every input record, and needed to find other ways to
avoid creating an iterator per record. It really kills performance.

I see the point about data expiration. We could experiment with
punctuation to expire old data, or add a second "time-ordered store"
(which we already have at hand) which acts as an index into the main
store. -- Another possibility would be to add a new version of segmented
store with a different key-layout (ie, just store the plain key)

Re: Subscribe for discussion on Kafka® development

This is self service. Please follow the instructions from the web page: 
https://kafka.apache.org/contact



-Matthias


On 6/11/24 9:38 AM, Punsak Incham wrote:

I want to subscribe for discussion on Kafka® development.

Best regards,

A black and blue logo Description automatically generated



*Punsak Incham (Mon)*

Data Platform Engineer

Data Modernization
Tel : +66(0)91-790-1302 | Email : pun...@mfec.co.th 



signature_2789072556

DISCLAIMER: This e-mail and any attachments may contain confidential or 
legally privileged information for useof MFEC Public Company Limited only.


If you are not the intended recipient, you are not authorized to copy or 
disclose all or any part of it without theprior written consent of the 
company.


Refer tohttps://www.mfec.co.th/en/privacy-policy/

Re: [VOTE] KIP-1035: StateStore managed changelog offsets


Thanks Nick.

+1 (binding)


Looking forward to get this all merged!


-Matthias

On 6/12/24 9:36 AM, Nick Telford wrote:

Hi everyone,

I'd like to call a vote on KIP-1035[1].

Regards,
Nick

1:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1035%3A+StateStore+managed+changelog+offsets

Re: [VOTE] KIP-1049: Add config log.summary.interval.ms to Kafka Streams


+1 (binding)

On 6/11/24 1:17 PM, Sophie Blee-Goldman wrote:

+1 (binding)

Thanks for the KIP!

On Tue, Jun 11, 2024 at 5:37 AM jiang dou  wrote:


HI
I would like to start a vote for KIP-1049: Add config
log.summary.interval.ms to Kafka Streams

KIP：

https://cwiki.apache.org/confluence/display/KAFKA/KIP-1049%3A+Add+config+log.summary.interval.ms+to+Kafka+Streams
Discussion thread:
https://lists.apache.org/thread/rjqslkt46y5zlg0552rloqjfm5ddzk06

Thanks

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

I believe the class name was picked on purpose, to make clear that it 
should not be used -- the problem is, that the class is in a public 
package and is by itself public (that's unfortunately require, given how 
Java works).


Of course, it's also in the JavaDocs that the class is internal and 
should not be used, but not everyone reads the JavaDocs necessarily, so 
making it part of the name makes it much more explicit, what I believe 
is a good thing.


I would consider it a fix/improvement, if we could exclude 
`InternalFixedKeyRecordFactory` from JavaDoc generation during the 
release build -- but I don't think we need a KIP for this, as I would 
rather consider it a bug-fix to exclude an internal class in the 
JavaDocs build step.



-Matthias

On 6/12/24 4:47 PM, Shashwat Pandey wrote:

Hi Matthias,

I think that strategy definitely works, abstracting away changes to
FixedKeyRecord from users,  I can put that new factory class and update the
KIP.

This might be a discussion for another KIP, but would it also make sense to
rename the
InternalFixedKeyRecordFactory to just FixedKeyRecordFactory also make sense?


Regards,
Shashwat Pandey


On Mon, Jun 10, 2024 at 5:07 PM Matthias J. Sax  wrote:


Shaswhat,

any updates on this KIP? -- I still think that recommending to use
`InternalFixedKeyRecordFactory` is not the best way to write test code.
Changing `FixedKeyRecord` constructor (as I mentioned in my last email)
might not be a good solution either.

Maybe a cleaner way would be (so sidestep this problem), to add a new
public "factory class" into the test package to generate
FixedKeyRecords, and this factory could internally use
`InternalFixedKeyRecordFactory`? It looks cleaner to me from an API POV,
and if we change anything how `FixedKeyRecord` can be created, it would
become a non-user-facing / internal change to the "factory" we provide.


-Matthias

On 5/22/24 12:02 AM, Matthias J. Sax wrote:

I was not aware of `InternalFixedKeyRecordFactory`. As the name
indicates, it's considered an internal class, so not sure if we should
recommend to use it in test...

I understand why this class is required, and why it was put into a
public package; the way Java works, enforces this. Not sure if we could
find a better solution.

Might be good to hear from others.


-Matthias

On 5/21/24 3:57 PM, Shashwat Pandey wrote:

Looking at the ticket and the sample code, I think it would be
possible to
continue using `InternalFixedKeyRecordFactory` as the avenue to create
`FixedKeyRecord`s in tests. As long as there was a
MockFixedKeyProcessorContext, I think we would be able to test
FixedKeyProcessors without a Topology.

I created a sample repo with the `MockFixedKeyProcessorContext` here is
what I think the tests would look like:


https://github.com/s7pandey/kafka-processor-tests/blob/main/src/test/java/com/example/demo/MyFixedKeyProcessorTest.java




On Mon, May 20, 2024 at 9:05 PM Matthias J. Sax 

wrote:



Had a discussion on https://issues.apache.org/jira/browse/KAFKA-15242
and it was pointed out, that we also need to do something about
`FixedKeyRecord`. It does not have a public constructor (what is
correct; it should not have one). However, this makes testing
`FixedKeyProcessor` impossible w/o extending `FixedKeyRecord` manually
what does not seem to be right (too clumsy).

It seems, we either need some helper builder method (but not clear to

me

where to add it in an elegant way) which would provide us with a
`FixedKeyRecord`, or add some sub-class to the test-utils module which
would extend `FixedKeyRecord`? -- Or maybe an even better solution? I
could not think of something else so far.


Thoughts?


On 5/3/24 9:46 AM, Matthias J. Sax wrote:

Please also update the KIP.

To get a wiki account created, please request it via a commet on this
ticket: https://issues.apache.org/jira/browse/INFRA-25451

After you have the account, please share your wiki id, and we can give
you write permission on the wiki.



-Matthias

On 5/3/24 6:30 AM, Shashwat Pandey wrote:

Hi Matthias,

Sorry this fell out of my radar for a bit.

Revisiting the topic, I think you’re right and we accept the
duplicated
nesting as an appropriate solution to not affect the larger public
API.

I can update my PR with the change.

Regards,
Shashwat Pandey


On Wed, May 1, 2024 at 11:00 PM Matthias J. Sax 

wrote:



Any updates on this KIP?

On 3/28/24 4:11 AM, Matthias J. Sax wrote:

It seems that `MockRecordMetadata` is a private class, and thus not
part
of the public API. If there are any changes required, we don't
need to
discuss on the KIP.


For `CapturedPunctuator` and `CapturedForward` it's a little bit
more
tricky. My gut feeling is, that the classes might not need to be
changed, but if we use them within `MockProcessorContext` and
`MockFixedKeyProcessorContext` it might be weird to keep the

current

nesting... The problem I see is, that it's not straightforward
how to
move the classes w/o breaking compatibility, nor if we duplic

Re: [DISCUSS] KIP-1056: Remove `default.` prefix for exception handler StreamsConfig

Thanks Greg, this is a valid idea.

However, there was no demand in the past to allow for error handler
callbacks in a fine grained manner, and I am frankly also not sure if it
would make sense or would be required.

Thus, I am not concerned about this case, and believe this KIP does make
sense.

Happy to change my mind if somebody has a different opinion. We could
re-purpose this KIP to add per-topic error handler, too.

-Matthias

On 6/12/24 1:11 PM, Greg Harris wrote:

Hi Murali,

Thanks for the KIP!

I'm not familiar with Streams so I'll pose a general question, open for
anyone to answer:

The configs that are being changed don't currently accept in-place
overwrites in the code, so the "default.*" prefix doesn't make sense. Could
there be a KIP to accept in-place overwrites in the future, such that the
"default.*" prefix would make sense?
If so, this KIP would make that other KIP harder to implement, as we would
have already recommended everyone to move off of the "default.* prefix. Or
to put it another way, this KIP closes doors rather than opening them.

Or at least that's how it looks to a Streams outsider. I'm happy to defer
to the experts in this case :)

Thanks,
Greg

On Mon, Jun 10, 2024 at 3:47 PM Matthias J. Sax wrote:

Thanks for the KIP Murali,

Overall LGTM. A few comments.

100: Config names are part of the public interface, and the KIP should
not say "none" in this section, but call out which configs are
deprecated and which ones are newly added.

101: Nit. In "Propose Changes" there is the template placeholder text

Describe the new thing you want to do in appropriate detail. This may be

fairly extensive and have large subsections of its own. Or it may be a few
sentences. Use judgement based on the scope of the change.

Similarly in "Test Plan" section

Please remove both :)

102: The "Deprecation" section should explain the behavior if both, old
and new configs, are set.

Thanks a lot!

-Matthias

On 6/9/24 9:30 PM, Muralidhar Basani wrote:

Hello all,

With this KIP
<

https://cwiki.apache.org/confluence/display/KAFKA/KIP-1056%3A+Remove+%60default.%60+prefix+for+exception+handler+StreamsConfig

,
I would like to mention that a couple of exception handler configs in
StreamsConfig are defined as default configs, despite having no

alternative

values. Hence would propose to deprecate them and introduce new configs
without the 'default.' prefix.

This KIP is briefly discussed here in the jira KAFKA-16853
too.

I would appreciate any feedback or suggestions you might have.

Thanks,
Murali

[jira] [Resolved] (KAFKA-14877) refactor InMemoryLeaderEpochCheckpoint

2024-06-12 Thread Satish Duggana (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-14877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-14877.

Resolution: Invalid

> refactor InMemoryLeaderEpochCheckpoint
> --
>
> Key: KAFKA-14877
> URL: https://issues.apache.org/jira/browse/KAFKA-14877
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Luke Chen
>Priority: Minor
> Fix For: 3.8.0
>
>
> follow up with this comment: 
> https://github.com/apache/kafka/pull/13456#discussion_r1154306477



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-16935) Automatically wait for cluster startup in embedded Connect integration tests

2024-06-12 Thread Chris Egerton (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-16935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton reopened KAFKA-16935:
---

> Automatically wait for cluster startup in embedded Connect integration tests
> 
>
> Key: KAFKA-16935
> URL: https://issues.apache.org/jira/browse/KAFKA-16935
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 3.8.0
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Major
> Fix For: 3.8.0
>
>
> It's a common idiom in our integration tests to [start an embedded Kafka and 
> Connect 
> cluster|https://github.com/apache/kafka/blob/aecaf4447561edd8da9f06e3abdf46f382dc9d89/connect/runtime/src/test/java/org/apache/kafka/connect/util/clusters/EmbeddedConnect.java#L120-L135]
>  and then immediately afterwards [wait for each worker in the Connect cluster 
> to complete 
> startup|https://github.com/apache/kafka/blob/aecaf4447561edd8da9f06e3abdf46f382dc9d89/connect/runtime/src/test/java/org/apache/kafka/connect/util/clusters/ConnectAssertions.java#L62-L92].
>  Separating these two actions into separate steps makes our tests lengthier 
> and can even lead to bugs and flakiness if the second step is accidentally 
> omitted (see [https://github.com/apache/kafka/pull/16286] for one example).
> Instead, we should default to automatically awaiting the complete startup of 
> every worker in an embedded Connect cluster when {{EmbeddedConnect::start}} 
> is invoked, and require callers to opt out if they do not want to 
> automatically wait for startup to complete when invoking that method.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16935) Automatically wait for cluster startup in embedded Connect integration tests

2024-06-12 Thread Chris Egerton (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-16935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-16935.
---
Resolution: Fixed

> Automatically wait for cluster startup in embedded Connect integration tests
> 
>
> Key: KAFKA-16935
> URL: https://issues.apache.org/jira/browse/KAFKA-16935
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 3.8.0
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Major
> Fix For: 3.8.0
>
>
> It's a common idiom in our integration tests to [start an embedded Kafka and 
> Connect 
> cluster|https://github.com/apache/kafka/blob/aecaf4447561edd8da9f06e3abdf46f382dc9d89/connect/runtime/src/test/java/org/apache/kafka/connect/util/clusters/EmbeddedConnect.java#L120-L135]
>  and then immediately afterwards [wait for each worker in the Connect cluster 
> to complete 
> startup|https://github.com/apache/kafka/blob/aecaf4447561edd8da9f06e3abdf46f382dc9d89/connect/runtime/src/test/java/org/apache/kafka/connect/util/clusters/ConnectAssertions.java#L62-L92].
>  Separating these two actions into separate steps makes our tests lengthier 
> and can even lead to bugs and flakiness if the second step is accidentally 
> omitted (see [https://github.com/apache/kafka/pull/16286] for one example).
> Instead, we should default to automatically awaiting the complete startup of 
> every worker in an embedded Connect cluster when {{EmbeddedConnect::start}} 
> is invoked, and require callers to opt out if they do not want to 
> automatically wait for startup to complete when invoking that method.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16947) Kafka Tiered Storage V2

2024-06-12 Thread Satish Duggana (Jira)

Satish Duggana created KAFKA-16947:
--

 Summary: Kafka Tiered Storage V2
 Key: KAFKA-16947
 URL: https://issues.apache.org/jira/browse/KAFKA-16947
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 3.6.0
Reporter: Satish Duggana
Assignee: Satish Duggana
 Fix For: 3.8.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-13561) Consider deprecating `StreamsBuilder#build(props)` function

2024-06-12 Thread Matthias J. Sax (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-13561.
-
Resolution: Duplicate

> Consider deprecating `StreamsBuilder#build(props)` function
> ---
>
> Key: KAFKA-13561
> URL: https://issues.apache.org/jira/browse/KAFKA-13561
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Guozhang Wang
>Priority: Major
>  Labels: needs-kip
>
> With 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-591%3A+Add+Kafka+Streams+config+to+set+default+state+store
>  being accepted that introduced the new `StreamsBuilder(TopologyConfig)` 
> constructor, we can consider deprecating the `StreamsBuilder#build(props)` 
> function now. There are still a few things we'd need to do:
> 1. Copy the `StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG` to TopologyConfig.
> 2. Make sure the overloaded `StreamsBuilder()` constructor takes in default 
> values of TopologyConfig.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16890) Failing to build aux state on broker failover

2024-06-12 Thread Satish Duggana (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Satish Duggana resolved KAFKA-16890.

Resolution: Fixed

> Failing to build aux state on broker failover
> -
>
> Key: KAFKA-16890
> URL: https://issues.apache.org/jira/browse/KAFKA-16890
> Project: Kafka
>  Issue Type: Bug
>  Components: Tiered-Storage
>Affects Versions: 3.7.0, 3.7.1
>Reporter: Francois Visconte
>Assignee: Kamal Chandraprakash
>Priority: Major
> Fix For: 3.8.0
>
>
> We have clusters where we replace machines often falling into a state where 
> we keep having "Error building remote log auxiliary state for 
> loadtest_topic-22" and the partition being under-replicated until the leader 
> is manually restarted. 
> Looking into a specific case, here is what we observed in 
> __remote_log_metadata topic:
> {code:java}
>  
> partition: 29, offset: 183593, value: 
> RemoteLogSegmentMetadata{remoteLogSegmentId=RemoteLogSegmentId{topicIdPartition=ClnIeN0MQsi_d4FAOFKaDA:loadtest_topic-22,
>  id=GZeRTXLMSNe2BQjRXkg6hQ}, startOffset=10823, endOffset=11536, 
> brokerId=10013, maxTimestampMs=1715774588597, eventTimestampMs=1715781657604, 
> segmentLeaderEpochs={125=10823, 126=10968, 128=11047, 130=11048, 131=11324, 
> 133=11442, 134=11443, 135=11445, 136=11521, 137=11533, 139=11535}, 
> segmentSizeInBytes=704895, customMetadata=Optional.empty, 
> state=COPY_SEGMENT_STARTED}
> partition: 29, offset: 183594, value: 
> RemoteLogSegmentMetadataUpdate{remoteLogSegmentId=RemoteLogSegmentId{topicIdPartition=ClnIeN0MQsi_d4FAOFKaDA:loadtest_topic-22,
>  id=GZeRTXLMSNe2BQjRXkg6hQ}, customMetadata=Optional.empty, 
> state=COPY_SEGMENT_FINISHED, eventTimestampMs=1715781658183, brokerId=10013}
> partition: 29, offset: 183669, value: 
> RemoteLogSegmentMetadata{remoteLogSegmentId=RemoteLogSegmentId{topicIdPartition=ClnIeN0MQsi_d4FAOFKaDA:loadtest_topic-22,
>  id=L1TYzx0lQkagRIF86Kp0QQ}, startOffset=10823, endOffset=11544, 
> brokerId=10008, maxTimestampMs=1715781445270, eventTimestampMs=1715782717593, 
> segmentLeaderEpochs={125=10823, 126=10968, 128=11047, 130=11048, 131=11324, 
> 133=11442, 134=11443, 135=11445, 136=11521, 137=11533, 139=11535, 140=11537, 
> 142=11543}, segmentSizeInBytes=713088, customMetadata=Optional.empty, 
> state=COPY_SEGMENT_STARTED}
> partition: 29, offset: 183670, value: 
> RemoteLogSegmentMetadataUpdate{remoteLogSegmentId=RemoteLogSegmentId{topicIdPartition=ClnIeN0MQsi_d4FAOFKaDA:loadtest_topic-22,
>  id=L1TYzx0lQkagRIF86Kp0QQ}, customMetadata=Optional.empty, 
> state=COPY_SEGMENT_FINISHED, eventTimestampMs=1715782718370, brokerId=10008}
> partition: 29, offset: 186215, value: 
> RemoteLogSegmentMetadataUpdate{remoteLogSegmentId=RemoteLogSegmentId{topicIdPartition=ClnIeN0MQsi_d4FAOFKaDA:loadtest_topic-22,
>  id=L1TYzx0lQkagRIF86Kp0QQ}, customMetadata=Optional.empty, 
> state=DELETE_SEGMENT_STARTED, eventTimestampMs=1715867874617, brokerId=10008}
> partition: 29, offset: 186216, value: 
> RemoteLogSegmentMetadataUpdate{remoteLogSegmentId=RemoteLogSegmentId{topicIdPartition=ClnIeN0MQsi_d4FAOFKaDA:loadtest_topic-22,
>  id=L1TYzx0lQkagRIF86Kp0QQ}, customMetadata=Optional.empty, 
> state=DELETE_SEGMENT_FINISHED, eventTimestampMs=1715867874725, brokerId=10008}
> partition: 29, offset: 186217, value: 
> RemoteLogSegmentMetadataUpdate{remoteLogSegmentId=RemoteLogSegmentId{topicIdPartition=ClnIeN0MQsi_d4FAOFKaDA:loadtest_topic-22,
>  id=GZeRTXLMSNe2BQjRXkg6hQ}, customMetadata=Optional.empty, 
> state=DELETE_SEGMENT_STARTED, eventTimestampMs=1715867874729, brokerId=10008}
> partition: 29, offset: 186218, value: 
> RemoteLogSegmentMetadataUpdate{remoteLogSegmentId=RemoteLogSegmentId{topicIdPartition=ClnIeN0MQsi_d4FAOFKaDA:loadtest_topic-22,
>  id=GZeRTXLMSNe2BQjRXkg6hQ}, customMetadata=Optional.empty, 
> state=DELETE_SEGMENT_FINISHED, eventTimestampMs=1715867874817, brokerId=10008}
> {code}
>  
> It seems that at the time the leader is restarted (10013), a second copy of 
> the same segment is tiered by the new leader (10008). Interestingly the 
> segment doesn't have the same end offset, which is concerning. 
> Then the follower sees the following error repeatedly until the leader is 
> restarted: 
>  
> {code:java}
> [2024-05-17 20:46:42,133] DEBUG [ReplicaFetcher replicaId=10013, 
> leaderId=10008, fetcherId=0] Handling errors in processFetchRequest for 
> partitions HashSet(loadtest_topic-22) (kafka.server.ReplicaFetcherThread)
> [2024-05-17 20:46:43,174] DEBUG [ReplicaFetcher replicaId=10013, 
> leaderId=10008, fetcherId=0] Received error OFFSET_MOVED_TO_TIERED_STORAGE, 
> at fetch offset: 11537, topic-partition: loadtest_topic-22 
> (kafka.server.ReplicaFetcherThread)
> [2024-05-17 20:46:43,175] ERROR [ReplicaFetcher replicaId=10013, 
> leaderId=10008, fetcherId=0] Error bu

[jira] [Created] (KAFKA-16946) Utils.getHost/getPort cannot parse SASL_PLAINTEXT://host:port

2024-06-12 Thread Luke Chen (Jira)

Luke Chen created KAFKA-16946:
-

 Summary: Utils.getHost/getPort cannot parse 
SASL_PLAINTEXT://host:port
 Key: KAFKA-16946
 URL: https://issues.apache.org/jira/browse/KAFKA-16946
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.8.0
Reporter: Luke Chen


In KAFKA-16824, we tried to improve the regex for Utils.getHost/getPort, but it 
failed to parse SASL_PLAINTEXT://host:port now. Need to fix it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Kafka certification

2024-06-12 Thread Matheus Pereira dos Santos

Hi!

I am a Back End developer and I am looking for official kafka
certifications, but for now I am just finding courses.
If exists any official certification for Kafka developer, I would like to
know.

Best regards!

Matheus P. S.

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

2024-06-12 Thread Shashwat Pandey

Hi Matthias,

I think that strategy definitely works, abstracting away changes to
FixedKeyRecord from users,  I can put that new factory class and update the
KIP.

This might be a discussion for another KIP, but would it also make sense to
rename the
InternalFixedKeyRecordFactory to just FixedKeyRecordFactory also make sense?


Regards,
Shashwat Pandey


On Mon, Jun 10, 2024 at 5:07 PM Matthias J. Sax  wrote:

> Shaswhat,
>
> any updates on this KIP? -- I still think that recommending to use
> `InternalFixedKeyRecordFactory` is not the best way to write test code.
> Changing `FixedKeyRecord` constructor (as I mentioned in my last email)
> might not be a good solution either.
>
> Maybe a cleaner way would be (so sidestep this problem), to add a new
> public "factory class" into the test package to generate
> FixedKeyRecords, and this factory could internally use
> `InternalFixedKeyRecordFactory`? It looks cleaner to me from an API POV,
> and if we change anything how `FixedKeyRecord` can be created, it would
> become a non-user-facing / internal change to the "factory" we provide.
>
>
> -Matthias
>
> On 5/22/24 12:02 AM, Matthias J. Sax wrote:
> > I was not aware of `InternalFixedKeyRecordFactory`. As the name
> > indicates, it's considered an internal class, so not sure if we should
> > recommend to use it in test...
> >
> > I understand why this class is required, and why it was put into a
> > public package; the way Java works, enforces this. Not sure if we could
> > find a better solution.
> >
> > Might be good to hear from others.
> >
> >
> > -Matthias
> >
> > On 5/21/24 3:57 PM, Shashwat Pandey wrote:
> >> Looking at the ticket and the sample code, I think it would be
> >> possible to
> >> continue using `InternalFixedKeyRecordFactory` as the avenue to create
> >> `FixedKeyRecord`s in tests. As long as there was a
> >> MockFixedKeyProcessorContext, I think we would be able to test
> >> FixedKeyProcessors without a Topology.
> >>
> >> I created a sample repo with the `MockFixedKeyProcessorContext` here is
> >> what I think the tests would look like:
> >>
> https://github.com/s7pandey/kafka-processor-tests/blob/main/src/test/java/com/example/demo/MyFixedKeyProcessorTest.java
> >>
> >>
> >>
> >> On Mon, May 20, 2024 at 9:05 PM Matthias J. Sax 
> wrote:
> >>
> >>> Had a discussion on https://issues.apache.org/jira/browse/KAFKA-15242
> >>> and it was pointed out, that we also need to do something about
> >>> `FixedKeyRecord`. It does not have a public constructor (what is
> >>> correct; it should not have one). However, this makes testing
> >>> `FixedKeyProcessor` impossible w/o extending `FixedKeyRecord` manually
> >>> what does not seem to be right (too clumsy).
> >>>
> >>> It seems, we either need some helper builder method (but not clear to
> me
> >>> where to add it in an elegant way) which would provide us with a
> >>> `FixedKeyRecord`, or add some sub-class to the test-utils module which
> >>> would extend `FixedKeyRecord`? -- Or maybe an even better solution? I
> >>> could not think of something else so far.
> >>>
> >>>
> >>> Thoughts?
> >>>
> >>>
> >>> On 5/3/24 9:46 AM, Matthias J. Sax wrote:
>  Please also update the KIP.
> 
>  To get a wiki account created, please request it via a commet on this
>  ticket: https://issues.apache.org/jira/browse/INFRA-25451
> 
>  After you have the account, please share your wiki id, and we can give
>  you write permission on the wiki.
> 
> 
> 
>  -Matthias
> 
>  On 5/3/24 6:30 AM, Shashwat Pandey wrote:
> > Hi Matthias,
> >
> > Sorry this fell out of my radar for a bit.
> >
> > Revisiting the topic, I think you’re right and we accept the
> > duplicated
> > nesting as an appropriate solution to not affect the larger public
> > API.
> >
> > I can update my PR with the change.
> >
> > Regards,
> > Shashwat Pandey
> >
> >
> > On Wed, May 1, 2024 at 11:00 PM Matthias J. Sax 
> >>> wrote:
> >
> >> Any updates on this KIP?
> >>
> >> On 3/28/24 4:11 AM, Matthias J. Sax wrote:
> >>> It seems that `MockRecordMetadata` is a private class, and thus not
> >>> part
> >>> of the public API. If there are any changes required, we don't
> >>> need to
> >>> discuss on the KIP.
> >>>
> >>>
> >>> For `CapturedPunctuator` and `CapturedForward` it's a little bit
> >>> more
> >>> tricky. My gut feeling is, that the classes might not need to be
> >>> changed, but if we use them within `MockProcessorContext` and
> >>> `MockFixedKeyProcessorContext` it might be weird to keep the
> current
> >>> nesting... The problem I see is, that it's not straightforward
> >>> how to
> >>> move the classes w/o breaking compatibility, nor if we duplicate
> >>> them as
> >>> standalone classes w/o a larger "splash radius". (We would need
> >>> to add
> >>> new overloads for MockProcessorContext#schedu

[jira] [Created] (KAFKA-16945) Cleanup StreamsBuilder and TopologyConfig

2024-06-12 Thread Matthias J. Sax (Jira)

Matthias J. Sax created KAFKA-16945:
---

 Summary: Cleanup StreamsBuilder and TopologyConfig
 Key: KAFKA-16945
 URL: https://issues.apache.org/jira/browse/KAFKA-16945
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Matthias J. Sax


Historically, Kafka Streams offers two ways to build a topology: either via the 
PAPI by creating a `new Topology()` explicitly, or via the `StreamsBuilder` 
which returns a topology via `build()` method.

We later added an overload `build(Properties)` to enable topology optimizations 
for the DSL layer.

Furthermore, we also added `TopologyConfig` object, which can be passed into 
`new Topology(TopologyConfig)` as well as `StreamsBuilder(TopologyConfig)`.

We should consider to unify the different approaches to simplify the rather 
complex API we have right now.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16897) Move OffsetIndexTest and OffsetMapTest to storage module



 [ 
https://issues.apache.org/jira/browse/KAFKA-16897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16897.

Fix Version/s: 3.9.0
   Resolution: Fixed

> Move OffsetIndexTest and OffsetMapTest to storage module
> 
>
> Key: KAFKA-16897
> URL: https://issues.apache.org/jira/browse/KAFKA-16897
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Chia-Ping Tsai
>Assignee: 黃竣陽
>Priority: Minor
> Fix For: 3.9.0
>
>
> as title



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.8 #36

See

Re: [PR] MINOR: Mention confirming email subscriptions to contact page [kafka-site]

2024-06-12 Thread via GitHub



gharris1727 merged PR #606:
URL: https://github.com/apache/kafka-site/pull/606


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Subscribe for discussion on Kafka(R) development

2024-06-12 Thread Greg Harris

Hi Punsak,

Please find the instructions for subscribing to this mailing list here:
https://kafka.apache.org/contact and here:
https://www.apache.org/foundation/mailinglists

Thanks,
Greg

On Wed, Jun 12, 2024 at 7:11 AM Punsak Incham 
wrote:

> I want to subscribe for discussion on Kafka® development.
>
>
>
> Best regards,
>
>
>
>[image: A black and blue logo Description automatically generated]
>
> *Punsak Incham (Mon)*
>
> Data Platform Engineer
>
> Data Modernization
> Tel : +66(0)91-790-1302 | Email : pun...@mfec.co.th
>
> [image: signature_2789072556]
>
> DISCLAIMER: This e-mail and any attachments may contain confidential or
> legally privileged information for use of MFEC Public Company Limited
> only.
>
> If you are not the intended recipient, you are not authorized to copy or
> disclose all or any part of it without the prior written consent of the
> company.
>
> Refer to https://www.mfec.co.th/en/privacy-policy/
>
>
>
>
>

[PR] MINOR: Mention confirming email subscriptions to contact page [kafka-site]

2024-06-12 Thread via GitHub



gharris1727 opened a new pull request, #606:
URL: https://github.com/apache/kafka-site/pull/606

   Some users partially subscribe to the mailing list, possibly because we 
don't mention the second step of replying to the confirmation email. Mentioning 
this succinctly in the page and linking to the ASF documentation could help 
make this less likely.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-1056: Remove `default.` prefix for exception handler StreamsConfig

2024-06-12 Thread Greg Harris

Hi Murali,

Thanks for the KIP!

I'm not familiar with Streams so I'll pose a general question, open for
anyone to answer:

The configs that are being changed don't currently accept in-place
overwrites in the code, so the "default.*" prefix doesn't make sense. Could
there be a KIP to accept in-place overwrites in the future, such that the
"default.*" prefix would make sense?
If so, this KIP would make that other KIP harder to implement, as we would
have already recommended everyone to move off of the "default.* prefix. Or
to put it another way, this KIP closes doors rather than opening them.

Or at least that's how it looks to a Streams outsider. I'm happy to defer
to the experts in this case :)

Thanks,
Greg

On Mon, Jun 10, 2024 at 3:47 PM Matthias J. Sax  wrote:

> Thanks for the KIP Murali,
>
> Overall LGTM. A few comments.
>
>
>
> 100: Config names are part of the public interface, and the KIP should
> not say "none" in this section, but call out which configs are
> deprecated and which ones are newly added.
>
>
> 101: Nit. In "Propose Changes" there is the template placeholder text
>
> > Describe the new thing you want to do in appropriate detail. This may be
> fairly extensive and have large subsections of its own. Or it may be a few
> sentences. Use judgement based on the scope of the change.
>
> Similarly in "Test Plan" section
>
> Please remove both :)
>
>
> 102: The "Deprecation" section should explain the behavior if both, old
> and new configs, are set.
>
>
> Thanks a lot!
>
>
> -Matthias
>
>
> On 6/9/24 9:30 PM, Muralidhar Basani wrote:
> > Hello all,
> >
> > With this KIP
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1056%3A+Remove+%60default.%60+prefix+for+exception+handler+StreamsConfig
> >,
> > I would like to mention that a couple of exception handler configs in
> > StreamsConfig are defined as default configs, despite having no
> alternative
> > values. Hence would propose to deprecate them and introduce new configs
> > without the 'default.' prefix.
> >
> > This KIP is briefly discussed here in the jira KAFKA-16853
> >  too.
> >
> > I would appreciate any feedback or suggestions you might have.
> >
> > Thanks,
> > Murali
> >
>

[jira] [Resolved] (KAFKA-16858) Flatten SMT throws NPE

2024-06-12 Thread Greg Harris (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-16858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Harris resolved KAFKA-16858.
-
Resolution: Fixed

> Flatten SMT throws NPE
> --
>
> Key: KAFKA-16858
> URL: https://issues.apache.org/jira/browse/KAFKA-16858
> Project: Kafka
>  Issue Type: Bug
>  Components: connect
>Affects Versions: 3.6.0
> Environment: Kafka 3.6 by way of CP 7.6.0
>Reporter: Adam Strickland
>Priority: Major
> Fix For: 3.8.0, 3.9.0
>
> Attachments: FlattenTest.java, proto.proto
>
>
> {{ConnectSchema.expectedClassesFor}} sometimes will throw an NPE as part of a 
> call to an SMT chain.  Stack trace snippet:
> {{at 
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.apply(MomentFlatten.java:84)}}
> {{at 
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.applyWithSchema(MomentFlatten.java:173)}}
> {{at 
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at 
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at 
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:280)}}
> {{at 
> com.github.momenttechnology.kafka.connect.transforms.MomentFlatten.buildWithSchema(MomentFlatten.java:274)}}
> {{at org.apache.kafka.connect.data.Struct.put(Struct.java:203)}}
> {{at org.apache.kafka.connect.data.Struct.put(Struct.java:216)}}
> {{at 
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:255)}}
> {{at 
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:213)}}
> {{at 
> org.apache.kafka.connect.data.ConnectSchema.validateValue(ConnectSchema.java:224)}}
> {{at 
> org.apache.kafka.connect.data.ConnectSchema.expectedClassesFor(ConnectSchema.java:268)}}
> (the above transform is a sub-class of 
> {{{}o.a.k.connect.transforms.Flatten{}}}; have confirmed that the error 
> occurs regardless).
> The field being transformed is an array of structs. If the call to 
> {{Schema#valueSchema()}} (o.a.k.connect.data.ConnectSchema.java:255) returns 
> {{{}null{}}}, the subsequent call to {{Schema#name()}} at 
> o.a.k.connect.data.ConnectSchema:268 throws an NPE.
> The strange thing that we have observed is that this doesn't always happen; 
> *sometimes* the struct's schema is found and sometimes it is not. We have 
> been unable to determine the root cause, but have constructed a test that 
> replicates the problem as observed (see attachment).
> In our case we have worked around the issue with the aforementioned sub-class 
> of {{{}Flatten{}}}, catching and logging the NPE on that specific use-case.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16944) Range assignor doesn't co-partition with stickiness

2024-06-12 Thread Ritika Reddy (Jira)

Ritika Reddy created KAFKA-16944:


 Summary: Range assignor doesn't co-partition with stickiness
 Key: KAFKA-16944
 URL: https://issues.apache.org/jira/browse/KAFKA-16944
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ritika Reddy
Assignee: Ritika Reddy


When stickiness is considered during range assignments, it is possible that in 
certain cases where co-partitioning is guaranteed we fail. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

RE: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

2024-06-12 Thread Punsak Incham

Hi all,
I have a few questions.

If you include all the related SMTs in the KIP, such as MaskField and 
ValueToKey,  As a user, when will I be able to use the updated SMTs? Do all the 
related SMTs in the KIP need to be fixed first before you release them for 
users like me to use?


Punsak Incham
Data Platform Engineer (Kafka and Confluent)
MFEC


-Original Message-
From: Greg Harris  
Sent: 13 มิถุนายน 2567 0:14
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-1040: Improve handling of nullable values in 
InsertField, ExtractField, and other transformations

Hi Mario,

Thank you for mentioning the earlier DISCUSS thread. I found this comment from 
Chris, which was agreed upon and applied to the KIP:

> Yes, I think we should just do one KIP for all the SMTs. You don't 
> have to implement everything all at once or by yourself, but I don't 
> see why we should require one or more follow-up KIPs to apply the 
> exact same changes to the SMTs we missed the first time.

However, the KIP still omits the MaskField and ValueToKey transformations.
This looks like just a typo to me, should we update the KIP before closing the 
vote?

Thanks,
Greg

On Wed, Jun 12, 2024 at 6:50 AM Mario Fiore Vitale 
wrote:

> Hi Greg,
>
> The same check was made by Chris Egerton during the discussion thread.
>
> The original KIP scope was just for the InsertField, ExtractField 
> SMTs, then we decided to enlarge the scope of only the KIP to also 
> other potential affected SMTs.
>
> As of now the PR scope, instead, is only for InsertField and 
> ExtractField SMTs.
>
> Thanks,
> Mario.
>
>
> On Wed, Jun 12, 2024 at 12:08 AM Greg Harris 
>  >
> wrote:
>
> > Hi Mario,
> >
> > Thanks for the KIP. I'm on-board with this KIP, I just wanted to 
> > verify a discrepancy I noticed.
> >
> > I checked all of the call-sites of Struct#get(Field) and
> Struct#get(String)
> > in Kafka, and noticed there are some call-sites which are not 
> > included in the KIP.
> > 1. The Flatten transformation seems to already have the 
> > "replace.null.with.default=false" behavior unconditionally.
> > 2. The MaskField transformation unconditionally injects default 
> > values
> for
> > top-level structs.
> > 3. The ValueTokey transformation injects defaults for each of the 
> > configured "fields"
> > 4. The Values#convertToString method injects default values, but as 
> > this isn't configurable I think we'll need to leave it as-is.
> >
> > Did you intend to add this configuration to the Flatten, MaskField, 
> > or ValueToKey transformations, or should those be addressed in a 
> > separate effort?
> > Since the "false" behavior is desirable, I think the Flatten 
> > transform could be left as-is, rather than adding a configuration 
> > that nobody
> needs.
> >
> > Thanks,
> > Greg
> >
> > On Tue, Jun 11, 2024 at 9:24 AM Yash Mayya  wrote:
> >
> > > Thanks for the KIP!
> > >
> > > +1 (binding)
> > >
> > > On Tue, Jun 11, 2024 at 7:40 PM Mario Fiore Vitale 
> > >  >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Just wanted to bump up this thread for visibility.
> > > >
> > > > Thanks!
> > > >
> > > > On Wed, May 29, 2024 at 5:31 PM Mickael Maison <
> > mickael.mai...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi Mario,
> > > > >
> > > > > +1 (binding)
> > > > > Thanks for the KIP!
> > > > >
> > > > > Mickael
> > > > >
> > > > > On Mon, May 27, 2024 at 12:06 PM Mario Fiore Vitale <
> > > mvit...@redhat.com>
> > > > > wrote:
> > > > > >
> > > > > > After 7 days I received only one vote. Should I suppose this 
> > > > > > will
> > not
> > > > be
> > > > > > approved?
> > > > > >
> > > > > > On Mon, May 20, 2024 at 4:14 PM Chris Egerton
> > >  > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks for the KIP! +1 (binding)
> > > > > > >
> > > > > > > On Mon, May 20, 2024 at 4:22 AM Mario Fiore Vitale <
> > > > mvit...@redhat.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > I'd like to call a vote on KIP-1040 which aims to 
> > > > > > > > improve
> > > handling
> > > > of
> > > > > > > > nullable values in InsertField, ExtractField, and other
> > > > > transformations
> > > > > > > >
> > > > > > > > KIP -
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik
> i.apache.org%2Fconfluence%2Fpages%2Fviewpage.action%3FpageId%3D3037946
> 77&data=05%7C02%7Cpunsak%40mfec.co.th%7C36a25ef5a026433cbcc808dc8b0323
> c8%7C74105ed972ff4685915475f7408b6f67%7C1%7C0%7C638538092832326790%7CU
> nknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1ha
> WwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=AWg1i6RQZdMsijQ0Ew0Tr3sj7bDYUsfi%
> 2B%2BVxyg%2BT2wo%3D&reserved=0
> > > > > > > >
> > > > > > > > Discussion thread -
> > > > > > > >
> > https://apc01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli
> > sts.apache.org%2Fthread%2Fggqqqjbg6ccpz8g6ztyj7oxr80q5184n&data=05%7
> > C02%7Cpunsak%40mfec.co.th%7C36a25ef5a0

[jira] [Created] (KAFKA-16943) Synchronously verify Connect worker startup failure in InternalTopicsIntegrationTest

2024-06-12 Thread Chris Egerton (Jira)

Chris Egerton created KAFKA-16943:
-

 Summary: Synchronously verify Connect worker startup failure in 
InternalTopicsIntegrationTest
 Key: KAFKA-16943
 URL: https://issues.apache.org/jira/browse/KAFKA-16943
 Project: Kafka
  Issue Type: Improvement
  Components: connect
Reporter: Chris Egerton


Created after PR discussion 
[here|https://github.com/apache/kafka/pull/16288#discussion_r1636615220].

In some of our integration tests, we want to verify that a Connect worker 
cannot start under poor conditions (such as when its internal topics do not yet 
exist and it is configured to create them with a higher replication factor than 
the number of available brokers, or when its internal topics already exist but 
they do not have the compaction cleanup policy).

This is currently not possible, and presents a possible gap in testing 
coverage, especially for the test cases 
{{testFailToCreateInternalTopicsWithMoreReplicasThanBrokers}} and 
{{{}testFailToStartWhenInternalTopicsAreNotCompacted{}}}. It'd be nice if we 
could have some way of synchronously awaiting the completion or failure of 
worker startup in our integration tests in order to guarantee that worker 
startup fails under sufficiently adverse conditions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-06-12 Thread José Armando García Sancio

Hi Josep,

See my comment below.

On Wed, Jun 12, 2024 at 1:17 PM Josep Prat  wrote:
> How long do you think it will take to bring KIP-853 to completion?

We are still missing a few issues/jiras that need to get implemented
for the feature to be usable. I would say a few more weeks. May be
early July or mid July.

Thanks,
-- 
-José

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.8 #35

See

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Colin,

How long do you think it will take to bring KIP-853 to completion?

Best,

On Wed, Jun 12, 2024 at 6:42 PM Colin McCabe  wrote:

> Hi Josep,
>
> KIP-966 is not needed for feature parity with ZK. Indeed, KIP-966 will not
> be implemented in ZK at all (and never was).
>
> There is a PR related to honoring dynamic configs for
> unclean.leader.election which we did agree to do for 3.8. It's under review
> and we hope to get it in soon.
>
> We still plan on having KIP-853 in 3.8. In fact, it will be the main
> feature of that release.
>
> best,
> Colin
>
> On Wed, Jun 12, 2024, at 01:50, Josep Prat wrote:
> > Hi all,
> >
> > We are now really close to the planned code freeze deadline (today EOD).
> > According to KIP-1012 [1] we agreed to stay in the 3.x branch until we
> > achieve feature parity regarding Zookeeper and KRaft. The two main KIPs
> > identified that would achieve this are: KIP-853 [2] and KIP-966 [3].
> > At the moment of writing this email both KIPs are not completed. My
> > question to the people driving both KIPs would be, how much more time do
> > you think it's needed to bring these KIPs to completion?
> >
> > - If the time needed would be short, we could still include these 2 KIPs
> in
> > the release.
> > - However, if the time needed would be on the scale of weeks, we should
> > continue with the release plan for 3.8 and after start working on the 3.9
> > release.
> >
> > What are your thoughts?
> >
> >
> > [1]:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> > [2]:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes
> > [3]:
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas
> >
> > On Wed, Jun 12, 2024 at 10:40 AM Josep Prat  wrote:
> >
> >> Hi Rajini,
> >> Yes, we could backport this one to the 3.8 branch. Would you be able to
> do
> >> this once you merge this PR?
> >>
> >> Thanks
> >>
> >> On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram <
> rajinisiva...@gmail.com>
> >> wrote:
> >>
> >>> Hi Josep,
> >>>
> >>> The PR https://github.com/apache/kafka/pull/13277 for KIP-899 looks
> ready
> >>> to be merged (waiting for the PR build).The PR changes several files,
> but
> >>> is relatively straightforward and not risky. Also the changes are
> under a
> >>> config that is not enabled by default. Since the KIP was approved
> before
> >>> KIP freeze, will it be ok to include in 3.8.0?
> >>>
> >>> Thank you,
> >>>
> >>> Rajini
> >>>
> >>>
> >>> On Tue, Jun 11, 2024 at 9:35 AM Josep Prat  >
> >>> wrote:
> >>>
> >>> > Hi all,
> >>> >
> >>> > I just want to remind everybody that the code freeze deadline is
> >>> > approaching. June 12th EOD is the deadline.
> >>> >
> >>> > Please do not automatically backport any commit to the 3.8 branch
> after
> >>> > June 12th EOD. Ping me if you think the commit should be backported
> >>> (fixes
> >>> > failures in the branch or critical bug fixes).
> >>> >
> >>> > Thanks all!
> >>> >
> >>> > On Sat, Jun 1, 2024 at 8:43 PM José Armando García Sancio
> >>> >  wrote:
> >>> >
> >>> > > Hi Josep,
> >>> > >
> >>> > > See my comments below.
> >>> > >
> >>> > > On Wed, May 29, 2024 at 10:52 AM Josep Prat
> >>>  >>> > >
> >>> > > wrote:
> >>> > > > So I would propose to leave the deadlines as they are and
> manually
> >>> > cherry
> >>> > > > pick the commits related to KIP-853 and KIP-966.
> >>> > >
> >>> > > Your proposal sounds good to me. I suspect that will be doing
> feature
> >>> > > development for KIP-853 past the feature freeze and code freeze
> date.
> >>> > > Maybe feature development will be finished around the end of June.
> >>> > >
> >>> > > I'll make sure to cherry pick commits for KIP-853 to the 3.8 branch
> >>> > > once we have one.
> >>> > >
> >>> > > Thanks,
> >>> > > --
> >>> > > -José
> >>> > >
> >>> >
> >>> >
> >>> > --
> >>> > [image: Aiven] 
> >>> >
> >>> > *Josep Prat*
> >>> > Open Source Engineering Director, *Aiven*
> >>> > josep.p...@aiven.io   |   +491715557497
> >>> > aiven.io    |   <
> >>> https://www.facebook.com/aivencloud
> >>> > >
> >>> >      <
> >>> > https://twitter.com/aiven_io>
> >>> > *Aiven Deutschland GmbH*
> >>> > Alexanderufer 3-7, 10117 Berlin
> >>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >>> > Amtsgericht Charlottenburg, HRB 209739 B
> >>> >
> >>>
> >>
> >>
> >> --
> >> [image: Aiven] 
> >>
> >> *Josep Prat*
> >> Open Source Engineering Director, *Aiven*
> >> josep.p...@aiven.io   |   +491715557497
> >> aiven.io    |
> >> 
> >>    <
> https://twitter.com/aiven_io>
> >> *Aiven Deutschland GmbH*
> >> Alexanderufer 3-7, 10117 Berlin
> >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >> Amtsgericht Charlottenburg, HRB 209739 B
> >>
> >
> >
>

Re: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

2024-06-12 Thread Greg Harris

Hi Mario,

Thank you for mentioning the earlier DISCUSS thread. I found this comment
from Chris, which was agreed upon and applied to the KIP:

> Yes, I think we should just do one KIP for all the SMTs. You don't have to
> implement everything all at once or by yourself, but I don't see why we
> should require one or more follow-up KIPs to apply the exact same changes
> to the SMTs we missed the first time.

However, the KIP still omits the MaskField and ValueToKey transformations.
This looks like just a typo to me, should we update the KIP before closing
the vote?

Thanks,
Greg

On Wed, Jun 12, 2024 at 6:50 AM Mario Fiore Vitale 
wrote:

> Hi Greg,
>
> The same check was made by Chris Egerton during the discussion thread.
>
> The original KIP scope was just for the InsertField, ExtractField SMTs,
> then we decided to enlarge the scope of only the KIP to also other
> potential affected SMTs.
>
> As of now the PR scope, instead, is only for InsertField and ExtractField
> SMTs.
>
> Thanks,
> Mario.
>
>
> On Wed, Jun 12, 2024 at 12:08 AM Greg Harris  >
> wrote:
>
> > Hi Mario,
> >
> > Thanks for the KIP. I'm on-board with this KIP, I just wanted to verify a
> > discrepancy I noticed.
> >
> > I checked all of the call-sites of Struct#get(Field) and
> Struct#get(String)
> > in Kafka, and noticed there are some call-sites which are not included in
> > the KIP.
> > 1. The Flatten transformation seems to already have the
> > "replace.null.with.default=false" behavior unconditionally.
> > 2. The MaskField transformation unconditionally injects default values
> for
> > top-level structs.
> > 3. The ValueTokey transformation injects defaults for each of the
> > configured "fields"
> > 4. The Values#convertToString method injects default values, but as this
> > isn't configurable I think we'll need to leave it as-is.
> >
> > Did you intend to add this configuration to the Flatten, MaskField, or
> > ValueToKey transformations, or should those be addressed in a separate
> > effort?
> > Since the "false" behavior is desirable, I think the Flatten transform
> > could be left as-is, rather than adding a configuration that nobody
> needs.
> >
> > Thanks,
> > Greg
> >
> > On Tue, Jun 11, 2024 at 9:24 AM Yash Mayya  wrote:
> >
> > > Thanks for the KIP!
> > >
> > > +1 (binding)
> > >
> > > On Tue, Jun 11, 2024 at 7:40 PM Mario Fiore Vitale  >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Just wanted to bump up this thread for visibility.
> > > >
> > > > Thanks!
> > > >
> > > > On Wed, May 29, 2024 at 5:31 PM Mickael Maison <
> > mickael.mai...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi Mario,
> > > > >
> > > > > +1 (binding)
> > > > > Thanks for the KIP!
> > > > >
> > > > > Mickael
> > > > >
> > > > > On Mon, May 27, 2024 at 12:06 PM Mario Fiore Vitale <
> > > mvit...@redhat.com>
> > > > > wrote:
> > > > > >
> > > > > > After 7 days I received only one vote. Should I suppose this will
> > not
> > > > be
> > > > > > approved?
> > > > > >
> > > > > > On Mon, May 20, 2024 at 4:14 PM Chris Egerton
> > >  > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks for the KIP! +1 (binding)
> > > > > > >
> > > > > > > On Mon, May 20, 2024 at 4:22 AM Mario Fiore Vitale <
> > > > mvit...@redhat.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > I'd like to call a vote on KIP-1040 which aims to improve
> > > handling
> > > > of
> > > > > > > > nullable values in InsertField, ExtractField, and other
> > > > > transformations
> > > > > > > >
> > > > > > > > KIP -
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=303794677
> > > > > > > >
> > > > > > > > Discussion thread -
> > > > > > > >
> > https://lists.apache.org/thread/ggqqqjbg6ccpz8g6ztyj7oxr80q5184n
> > > > > > > >
> > > > > > > > Thanks and regards,
> > > > > > > > Mario
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Mario Fiore Vitale
> > > > > >
> > > > > > Senior Software Engineer
> > > > > >
> > > > > > Red Hat 
> > > > > > 
> > > > >
> > > > >
> > > >
> > > > --
> > > >
> > > > Mario Fiore Vitale
> > > >
> > > > Senior Software Engineer
> > > >
> > > > Red Hat 
> > > > 
> > > >
> > >
> >
>
>
> --
>
> Mario Fiore Vitale
>
> Senior Software Engineer
>
> Red Hat 
> 
>

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-06-12 Thread Colin McCabe

Hi Josep,

KIP-966 is not needed for feature parity with ZK. Indeed, KIP-966 will not be 
implemented in ZK at all (and never was).

There is a PR related to honoring dynamic configs for unclean.leader.election 
which we did agree to do for 3.8. It's under review and we hope to get it in 
soon.

We still plan on having KIP-853 in 3.8. In fact, it will be the main feature of 
that release.

best,
Colin

On Wed, Jun 12, 2024, at 01:50, Josep Prat wrote:
> Hi all,
>
> We are now really close to the planned code freeze deadline (today EOD).
> According to KIP-1012 [1] we agreed to stay in the 3.x branch until we
> achieve feature parity regarding Zookeeper and KRaft. The two main KIPs
> identified that would achieve this are: KIP-853 [2] and KIP-966 [3].
> At the moment of writing this email both KIPs are not completed. My
> question to the people driving both KIPs would be, how much more time do
> you think it's needed to bring these KIPs to completion?
>
> - If the time needed would be short, we could still include these 2 KIPs in
> the release.
> - However, if the time needed would be on the scale of weeks, we should
> continue with the release plan for 3.8 and after start working on the 3.9
> release.
>
> What are your thoughts?
>
>
> [1]:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> [2]:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes
> [3]:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas
>
> On Wed, Jun 12, 2024 at 10:40 AM Josep Prat  wrote:
>
>> Hi Rajini,
>> Yes, we could backport this one to the 3.8 branch. Would you be able to do
>> this once you merge this PR?
>>
>> Thanks
>>
>> On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram 
>> wrote:
>>
>>> Hi Josep,
>>>
>>> The PR https://github.com/apache/kafka/pull/13277 for KIP-899 looks ready
>>> to be merged (waiting for the PR build).The PR changes several files, but
>>> is relatively straightforward and not risky. Also the changes are under a
>>> config that is not enabled by default. Since the KIP was approved before
>>> KIP freeze, will it be ok to include in 3.8.0?
>>>
>>> Thank you,
>>>
>>> Rajini
>>>
>>>
>>> On Tue, Jun 11, 2024 at 9:35 AM Josep Prat 
>>> wrote:
>>>
>>> > Hi all,
>>> >
>>> > I just want to remind everybody that the code freeze deadline is
>>> > approaching. June 12th EOD is the deadline.
>>> >
>>> > Please do not automatically backport any commit to the 3.8 branch after
>>> > June 12th EOD. Ping me if you think the commit should be backported
>>> (fixes
>>> > failures in the branch or critical bug fixes).
>>> >
>>> > Thanks all!
>>> >
>>> > On Sat, Jun 1, 2024 at 8:43 PM José Armando García Sancio
>>> >  wrote:
>>> >
>>> > > Hi Josep,
>>> > >
>>> > > See my comments below.
>>> > >
>>> > > On Wed, May 29, 2024 at 10:52 AM Josep Prat
>>> >> > >
>>> > > wrote:
>>> > > > So I would propose to leave the deadlines as they are and manually
>>> > cherry
>>> > > > pick the commits related to KIP-853 and KIP-966.
>>> > >
>>> > > Your proposal sounds good to me. I suspect that will be doing feature
>>> > > development for KIP-853 past the feature freeze and code freeze date.
>>> > > Maybe feature development will be finished around the end of June.
>>> > >
>>> > > I'll make sure to cherry pick commits for KIP-853 to the 3.8 branch
>>> > > once we have one.
>>> > >
>>> > > Thanks,
>>> > > --
>>> > > -José
>>> > >
>>> >
>>> >
>>> > --
>>> > [image: Aiven] 
>>> >
>>> > *Josep Prat*
>>> > Open Source Engineering Director, *Aiven*
>>> > josep.p...@aiven.io   |   +491715557497
>>> > aiven.io    |   <
>>> https://www.facebook.com/aivencloud
>>> > >
>>> >      <
>>> > https://twitter.com/aiven_io>
>>> > *Aiven Deutschland GmbH*
>>> > Alexanderufer 3-7, 10117 Berlin
>>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>>> > Amtsgericht Charlottenburg, HRB 209739 B
>>> >
>>>
>>
>>
>> --
>> [image: Aiven] 
>>
>> *Josep Prat*
>> Open Source Engineering Director, *Aiven*
>> josep.p...@aiven.io   |   +491715557497
>> aiven.io    |
>> 
>>    
>> *Aiven Deutschland GmbH*
>> Alexanderufer 3-7, 10117 Berlin
>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> Amtsgericht Charlottenburg, HRB 209739 B
>>
>
>
> -- 
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |   
>      
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B

[VOTE] KIP-1035: StateStore managed changelog offsets

2024-06-12 Thread Nick Telford

Hi everyone,

I'd like to call a vote on KIP-1035[1].

Regards,
Nick

1:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1035%3A+StateStore+managed+changelog+offsets

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-06-12 Thread Justine Olshan

I'm just going to echo what Luke said.

KIP-966 is definitely delayed to the next release and we only need the
unclean leadership election PR he shared.

Thanks,
Justine

On Wed, Jun 12, 2024 at 9:33 AM Calvin Liu 
wrote:

> Hi Josep
> Due to the limit of the review resources, KIP-966 may have to be delayed to
> the next release.
> Thanks.
>
> On Wed, Jun 12, 2024 at 2:05 AM Rajini Sivaram 
> wrote:
>
> > >
> > > Yes, we could backport this one to the 3.8 branch. Would you be able to
> > do
> > > this once you merge this PR?
> >
> >
> > Thanks Josep! There were some unit test failures in last night's PR
> build,
> > so I have commented on the PR for Ivan to fix. If he fixes it today,
> then I
> > can merge to trunk and then cherry-pick to 3.8 today.
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Wed, Jun 12, 2024 at 9:51 AM Josep Prat 
> > wrote:
> >
> > > Hi all,
> > >
> > > We are now really close to the planned code freeze deadline (today
> EOD).
> > > According to KIP-1012 [1] we agreed to stay in the 3.x branch until we
> > > achieve feature parity regarding Zookeeper and KRaft. The two main KIPs
> > > identified that would achieve this are: KIP-853 [2] and KIP-966 [3].
> > > At the moment of writing this email both KIPs are not completed. My
> > > question to the people driving both KIPs would be, how much more time
> do
> > > you think it's needed to bring these KIPs to completion?
> > >
> > > - If the time needed would be short, we could still include these 2
> KIPs
> > in
> > > the release.
> > > - However, if the time needed would be on the scale of weeks, we should
> > > continue with the release plan for 3.8 and after start working on the
> 3.9
> > > release.
> > >
> > > What are your thoughts?
> > >
> > >
> > > [1]:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> > > [2]:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes
> > > [3]:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas
> > >
> > > On Wed, Jun 12, 2024 at 10:40 AM Josep Prat 
> wrote:
> > >
> > > > Hi Rajini,
> > > > Yes, we could backport this one to the 3.8 branch. Would you be able
> to
> > > do
> > > > this once you merge this PR?
> > > >
> > > > Thanks
> > > >
> > > > On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram <
> > rajinisiva...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Hi Josep,
> > > >>
> > > >> The PR https://github.com/apache/kafka/pull/13277 for KIP-899 looks
> > > ready
> > > >> to be merged (waiting for the PR build).The PR changes several
> files,
> > > but
> > > >> is relatively straightforward and not risky. Also the changes are
> > under
> > > a
> > > >> config that is not enabled by default. Since the KIP was approved
> > before
> > > >> KIP freeze, will it be ok to include in 3.8.0?
> > > >>
> > > >> Thank you,
> > > >>
> > > >> Rajini
> > > >>
> > > >>
> > > >> On Tue, Jun 11, 2024 at 9:35 AM Josep Prat
> >  > > >
> > > >> wrote:
> > > >>
> > > >> > Hi all,
> > > >> >
> > > >> > I just want to remind everybody that the code freeze deadline is
> > > >> > approaching. June 12th EOD is the deadline.
> > > >> >
> > > >> > Please do not automatically backport any commit to the 3.8 branch
> > > after
> > > >> > June 12th EOD. Ping me if you think the commit should be
> backported
> > > >> (fixes
> > > >> > failures in the branch or critical bug fixes).
> > > >> >
> > > >> > Thanks all!
> > > >> >
> > > >> > On Sat, Jun 1, 2024 at 8:43 PM José Armando García Sancio
> > > >> >  wrote:
> > > >> >
> > > >> > > Hi Josep,
> > > >> > >
> > > >> > > See my comments below.
> > > >> > >
> > > >> > > On Wed, May 29, 2024 at 10:52 AM Josep Prat
> > > >>  > > >> > >
> > > >> > > wrote:
> > > >> > > > So I would propose to leave the deadlines as they are and
> > manually
> > > >> > cherry
> > > >> > > > pick the commits related to KIP-853 and KIP-966.
> > > >> > >
> > > >> > > Your proposal sounds good to me. I suspect that will be doing
> > > feature
> > > >> > > development for KIP-853 past the feature freeze and code freeze
> > > date.
> > > >> > > Maybe feature development will be finished around the end of
> June.
> > > >> > >
> > > >> > > I'll make sure to cherry pick commits for KIP-853 to the 3.8
> > branch
> > > >> > > once we have one.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > --
> > > >> > > -José
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > [image: Aiven] 
> > > >> >
> > > >> > *Josep Prat*
> > > >> > Open Source Engineering Director, *Aiven*
> > > >> > josep.p...@aiven.io   |   +491715557497
> > > >> > aiven.io    |   <
> > > >> https://www.facebook.com/aivencloud
> > > >> > >
> > > >> >      <
> > > >> > https://twitter.com/aiven_io>
> > > >> > *Aiven Deutschland GmbH*
> > > >> > Alexanderufer 3-7, 10117 Berlin
> > > >

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-06-12 Thread Calvin Liu

Hi Josep
Due to the limit of the review resources, KIP-966 may have to be delayed to
the next release.
Thanks.

On Wed, Jun 12, 2024 at 2:05 AM Rajini Sivaram 
wrote:

> >
> > Yes, we could backport this one to the 3.8 branch. Would you be able to
> do
> > this once you merge this PR?
>
>
> Thanks Josep! There were some unit test failures in last night's PR build,
> so I have commented on the PR for Ivan to fix. If he fixes it today, then I
> can merge to trunk and then cherry-pick to 3.8 today.
>
> Regards,
>
> Rajini
>
>
> On Wed, Jun 12, 2024 at 9:51 AM Josep Prat 
> wrote:
>
> > Hi all,
> >
> > We are now really close to the planned code freeze deadline (today EOD).
> > According to KIP-1012 [1] we agreed to stay in the 3.x branch until we
> > achieve feature parity regarding Zookeeper and KRaft. The two main KIPs
> > identified that would achieve this are: KIP-853 [2] and KIP-966 [3].
> > At the moment of writing this email both KIPs are not completed. My
> > question to the people driving both KIPs would be, how much more time do
> > you think it's needed to bring these KIPs to completion?
> >
> > - If the time needed would be short, we could still include these 2 KIPs
> in
> > the release.
> > - However, if the time needed would be on the scale of weeks, we should
> > continue with the release plan for 3.8 and after start working on the 3.9
> > release.
> >
> > What are your thoughts?
> >
> >
> > [1]:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> > [2]:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes
> > [3]:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas
> >
> > On Wed, Jun 12, 2024 at 10:40 AM Josep Prat  wrote:
> >
> > > Hi Rajini,
> > > Yes, we could backport this one to the 3.8 branch. Would you be able to
> > do
> > > this once you merge this PR?
> > >
> > > Thanks
> > >
> > > On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram <
> rajinisiva...@gmail.com
> > >
> > > wrote:
> > >
> > >> Hi Josep,
> > >>
> > >> The PR https://github.com/apache/kafka/pull/13277 for KIP-899 looks
> > ready
> > >> to be merged (waiting for the PR build).The PR changes several files,
> > but
> > >> is relatively straightforward and not risky. Also the changes are
> under
> > a
> > >> config that is not enabled by default. Since the KIP was approved
> before
> > >> KIP freeze, will it be ok to include in 3.8.0?
> > >>
> > >> Thank you,
> > >>
> > >> Rajini
> > >>
> > >>
> > >> On Tue, Jun 11, 2024 at 9:35 AM Josep Prat
>  > >
> > >> wrote:
> > >>
> > >> > Hi all,
> > >> >
> > >> > I just want to remind everybody that the code freeze deadline is
> > >> > approaching. June 12th EOD is the deadline.
> > >> >
> > >> > Please do not automatically backport any commit to the 3.8 branch
> > after
> > >> > June 12th EOD. Ping me if you think the commit should be backported
> > >> (fixes
> > >> > failures in the branch or critical bug fixes).
> > >> >
> > >> > Thanks all!
> > >> >
> > >> > On Sat, Jun 1, 2024 at 8:43 PM José Armando García Sancio
> > >> >  wrote:
> > >> >
> > >> > > Hi Josep,
> > >> > >
> > >> > > See my comments below.
> > >> > >
> > >> > > On Wed, May 29, 2024 at 10:52 AM Josep Prat
> > >>  > >> > >
> > >> > > wrote:
> > >> > > > So I would propose to leave the deadlines as they are and
> manually
> > >> > cherry
> > >> > > > pick the commits related to KIP-853 and KIP-966.
> > >> > >
> > >> > > Your proposal sounds good to me. I suspect that will be doing
> > feature
> > >> > > development for KIP-853 past the feature freeze and code freeze
> > date.
> > >> > > Maybe feature development will be finished around the end of June.
> > >> > >
> > >> > > I'll make sure to cherry pick commits for KIP-853 to the 3.8
> branch
> > >> > > once we have one.
> > >> > >
> > >> > > Thanks,
> > >> > > --
> > >> > > -José
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> > [image: Aiven] 
> > >> >
> > >> > *Josep Prat*
> > >> > Open Source Engineering Director, *Aiven*
> > >> > josep.p...@aiven.io   |   +491715557497
> > >> > aiven.io    |   <
> > >> https://www.facebook.com/aivencloud
> > >> > >
> > >> >      <
> > >> > https://twitter.com/aiven_io>
> > >> > *Aiven Deutschland GmbH*
> > >> > Alexanderufer 3-7, 10117 Berlin
> > >> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >> > Amtsgericht Charlottenburg, HRB 209739 B
> > >> >
> > >>
> > >
> > >
> > > --
> > > [image: Aiven] 
> > >
> > > *Josep Prat*
> > > Open Source Engineering Director, *Aiven*
> > > josep.p...@aiven.io   |   +491715557497
> > > aiven.io    |
> > > 
> > >    <
> > https://twitter.com/aiven_io>
> > > *Aiven Deutschland GmbH*
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäf

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Edoardo,

Correct, you can still cherry-pick this one.

I'll send an email tomorrow morning (CEST) asking maintainers to stop
cherry picking commits unless we discuss it beforehand.

Best,

On Wed, Jun 12, 2024 at 6:09 PM Edoardo Comar  wrote:

> Hi Josep, I understand I am still in time to cherry-pick on 3.8.0
> https://github.com/apache/kafka/pull/16229
>
> right?
> thanks
>
> On Wed, 12 Jun 2024 at 11:34, Ivan Yurchenko  wrote:
> >
> > Hi,
> >
> > I'll try to do all the fixes and changes for KIP-899 [1] sooner today,
> but please proceed with the release if I don't manage.
> >
> > Ivan
> >
> > [1] https://github.com/apache/kafka/pull/13277
> >
> > On Wed, Jun 12, 2024, at 12:54, Josep Prat wrote:
> > > Hi Luke,
> > > I think Jose, also mentioned that it won't be ready for v3.8.0 (but he
> can
> > > confirm this). My question now would be, given that it seems we would
> need
> > > a v3.9.0, do you think it's important to include
> > > https://github.com/apache/kafka/pull/16284 in v3.8.0?
> > >
> > > Best,
> > >
> > > On Wed, Jun 12, 2024 at 11:40 AM Luke Chen  wrote:
> > >
> > > > Hi Josep
> > > >
> > > > For KIP-966, I think Calvin had mentioned he won't complete in
> v3.8.0.
> > > > https://lists.apache.org/thread/fsnr8wy5fznzfso7jgk90skgyo277fmw
> > > >
> > > > For unclean leader election, all we need is this PR:
> > > > https://github.com/apache/kafka/pull/16284
> > > > For this PR, I think it needs one more week to be completed.
> > > >
> > > > Thanks.
> > > > Luke
> > > >
> > > > On Wed, Jun 12, 2024 at 4:51 PM Josep Prat
> 
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > We are now really close to the planned code freeze deadline (today
> EOD).
> > > > > According to KIP-1012 [1] we agreed to stay in the 3.x branch
> until we
> > > > > achieve feature parity regarding Zookeeper and KRaft. The two main
> KIPs
> > > > > identified that would achieve this are: KIP-853 [2] and KIP-966
> [3].
> > > > > At the moment of writing this email both KIPs are not completed. My
> > > > > question to the people driving both KIPs would be, how much more
> time do
> > > > > you think it's needed to bring these KIPs to completion?
> > > > >
> > > > > - If the time needed would be short, we could still include these
> 2 KIPs
> > > > in
> > > > > the release.
> > > > > - However, if the time needed would be on the scale of weeks, we
> should
> > > > > continue with the release plan for 3.8 and after start working on
> the 3.9
> > > > > release.
> > > > >
> > > > > What are your thoughts?
> > > > >
> > > > >
> > > > > [1]:
> > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> > > > > [2]:
> > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes
> > > > > [3]:
> > > > >
> > > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas
> > > > >
> > > > > On Wed, Jun 12, 2024 at 10:40 AM Josep Prat 
> wrote:
> > > > >
> > > > > > Hi Rajini,
> > > > > > Yes, we could backport this one to the 3.8 branch. Would you be
> able to
> > > > > do
> > > > > > this once you merge this PR?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram <
> > > > rajinisiva...@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hi Josep,
> > > > > >>
> > > > > >> The PR https://github.com/apache/kafka/pull/13277 for KIP-899
> looks
> > > > > ready
> > > > > >> to be merged (waiting for the PR build).The PR changes several
> files,
> > > > > but
> > > > > >> is relatively straightforward and not risky. Also the changes
> are
> > > > under
> > > > > a
> > > > > >> config that is not enabled by default. Since the KIP was
> approved
> > > > before
> > > > > >> KIP freeze, will it be ok to include in 3.8.0?
> > > > > >>
> > > > > >> Thank you,
> > > > > >>
> > > > > >> Rajini
> > > > > >>
> > > > > >>
> > > > > >> On Tue, Jun 11, 2024 at 9:35 AM Josep Prat
> > > >  > > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi all,
> > > > > >> >
> > > > > >> > I just want to remind everybody that the code freeze deadline
> is
> > > > > >> > approaching. June 12th EOD is the deadline.
> > > > > >> >
> > > > > >> > Please do not automatically backport any commit to the 3.8
> branch
> > > > > after
> > > > > >> > June 12th EOD. Ping me if you think the commit should be
> backported
> > > > > >> (fixes
> > > > > >> > failures in the branch or critical bug fixes).
> > > > > >> >
> > > > > >> > Thanks all!
> > > > > >> >
> > > > > >> > On Sat, Jun 1, 2024 at 8:43 PM José Armando García Sancio
> > > > > >> >  wrote:
> > > > > >> >
> > > > > >> > > Hi Josep,
> > > > > >> > >
> > > > > >> > > See my comments below.
> > > > > >> > >
> > > > > >> > > On Wed, May 29, 2024 at 10:52 AM Josep Prat
> > > > > >>  > > > > >> > >
> > > > > >> > > wrote:
> > > > > >> > > > So I would propose to leave the dead

Re: [VOTE] 3.7.1 RC1

2024-06-12 Thread Edoardo Comar

Hi Igor, is it too late to cherry-pick
https://github.com/apache/kafka/pull/16229 to 3.7.1 ?

Thanks, Edo


On Tue, 11 Jun 2024 at 08:29, Igor Soarez  wrote:
>
> Now attaching the container build reports, which I seem to have forgotten to 
> include before.
>
> --
> Igor

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-06-12 Thread Edoardo Comar

Hi Josep, I understand I am still in time to cherry-pick on 3.8.0
https://github.com/apache/kafka/pull/16229

right?
thanks

On Wed, 12 Jun 2024 at 11:34, Ivan Yurchenko  wrote:
>
> Hi,
>
> I'll try to do all the fixes and changes for KIP-899 [1] sooner today, but 
> please proceed with the release if I don't manage.
>
> Ivan
>
> [1] https://github.com/apache/kafka/pull/13277
>
> On Wed, Jun 12, 2024, at 12:54, Josep Prat wrote:
> > Hi Luke,
> > I think Jose, also mentioned that it won't be ready for v3.8.0 (but he can
> > confirm this). My question now would be, given that it seems we would need
> > a v3.9.0, do you think it's important to include
> > https://github.com/apache/kafka/pull/16284 in v3.8.0?
> >
> > Best,
> >
> > On Wed, Jun 12, 2024 at 11:40 AM Luke Chen  wrote:
> >
> > > Hi Josep
> > >
> > > For KIP-966, I think Calvin had mentioned he won't complete in v3.8.0.
> > > https://lists.apache.org/thread/fsnr8wy5fznzfso7jgk90skgyo277fmw
> > >
> > > For unclean leader election, all we need is this PR:
> > > https://github.com/apache/kafka/pull/16284
> > > For this PR, I think it needs one more week to be completed.
> > >
> > > Thanks.
> > > Luke
> > >
> > > On Wed, Jun 12, 2024 at 4:51 PM Josep Prat 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > We are now really close to the planned code freeze deadline (today EOD).
> > > > According to KIP-1012 [1] we agreed to stay in the 3.x branch until we
> > > > achieve feature parity regarding Zookeeper and KRaft. The two main KIPs
> > > > identified that would achieve this are: KIP-853 [2] and KIP-966 [3].
> > > > At the moment of writing this email both KIPs are not completed. My
> > > > question to the people driving both KIPs would be, how much more time do
> > > > you think it's needed to bring these KIPs to completion?
> > > >
> > > > - If the time needed would be short, we could still include these 2 KIPs
> > > in
> > > > the release.
> > > > - However, if the time needed would be on the scale of weeks, we should
> > > > continue with the release plan for 3.8 and after start working on the 
> > > > 3.9
> > > > release.
> > > >
> > > > What are your thoughts?
> > > >
> > > >
> > > > [1]:
> > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> > > > [2]:
> > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes
> > > > [3]:
> > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas
> > > >
> > > > On Wed, Jun 12, 2024 at 10:40 AM Josep Prat  wrote:
> > > >
> > > > > Hi Rajini,
> > > > > Yes, we could backport this one to the 3.8 branch. Would you be able 
> > > > > to
> > > > do
> > > > > this once you merge this PR?
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram <
> > > rajinisiva...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Hi Josep,
> > > > >>
> > > > >> The PR https://github.com/apache/kafka/pull/13277 for KIP-899 looks
> > > > ready
> > > > >> to be merged (waiting for the PR build).The PR changes several files,
> > > > but
> > > > >> is relatively straightforward and not risky. Also the changes are
> > > under
> > > > a
> > > > >> config that is not enabled by default. Since the KIP was approved
> > > before
> > > > >> KIP freeze, will it be ok to include in 3.8.0?
> > > > >>
> > > > >> Thank you,
> > > > >>
> > > > >> Rajini
> > > > >>
> > > > >>
> > > > >> On Tue, Jun 11, 2024 at 9:35 AM Josep Prat
> > >  > > > >
> > > > >> wrote:
> > > > >>
> > > > >> > Hi all,
> > > > >> >
> > > > >> > I just want to remind everybody that the code freeze deadline is
> > > > >> > approaching. June 12th EOD is the deadline.
> > > > >> >
> > > > >> > Please do not automatically backport any commit to the 3.8 branch
> > > > after
> > > > >> > June 12th EOD. Ping me if you think the commit should be backported
> > > > >> (fixes
> > > > >> > failures in the branch or critical bug fixes).
> > > > >> >
> > > > >> > Thanks all!
> > > > >> >
> > > > >> > On Sat, Jun 1, 2024 at 8:43 PM José Armando García Sancio
> > > > >> >  wrote:
> > > > >> >
> > > > >> > > Hi Josep,
> > > > >> > >
> > > > >> > > See my comments below.
> > > > >> > >
> > > > >> > > On Wed, May 29, 2024 at 10:52 AM Josep Prat
> > > > >>  > > > >> > >
> > > > >> > > wrote:
> > > > >> > > > So I would propose to leave the deadlines as they are and
> > > manually
> > > > >> > cherry
> > > > >> > > > pick the commits related to KIP-853 and KIP-966.
> > > > >> > >
> > > > >> > > Your proposal sounds good to me. I suspect that will be doing
> > > > feature
> > > > >> > > development for KIP-853 past the feature freeze and code freeze
> > > > date.
> > > > >> > > Maybe feature development will be finished around the end of 
> > > > >> > > June.
> > > > >> > >
> > > > >> > > I'll make sure to cherry pick commits for KIP-853 to the 3.8
> > > branch
> > > > >> > > once w

[jira] [Created] (KAFKA-16942) Use ConcurrentHashMap in RecordAccumulator#nodeStats

2024-06-12 Thread Kuan Po Tseng (Jira)

Kuan Po Tseng created KAFKA-16942:
-

 Summary: Use ConcurrentHashMap in RecordAccumulator#nodeStats
 Key: KAFKA-16942
 URL: https://issues.apache.org/jira/browse/KAFKA-16942
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Reporter: Kuan Po Tseng
Assignee: Kuan Po Tseng


per discussed in 
[https://github.com/apache/kafka/pull/16231#discussion_r1635345881]

Through the ConcurrentMapBenchmark, we observed that in scenarios where write 
operations (i.e., computeIfAbsent) constitute 10%, the get performance of 
CopyOnWriteMap is lower compared to ConcurrentHashMap. However, when iterating 
over entrySet and values, CopyOnWriteMap performs better than ConcurrentHashMap.

In RecordAccumulator#nodeStats, the computeIfAbsent method is rarely triggered, 
and we only use the get method to read data. Therefore, switching to 
ConcurrentHashMap would gain better performance.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16865) Admin.describeTopics behavior change after KIP-966

2024-06-12 Thread Mickael Maison (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-16865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-16865.

Resolution: Fixed

> Admin.describeTopics behavior change after KIP-966
> --
>
> Key: KAFKA-16865
> URL: https://issues.apache.org/jira/browse/KAFKA-16865
> Project: Kafka
>  Issue Type: Task
>  Components: admin, clients
>Affects Versions: 3.8.0
>Reporter: Mickael Maison
>Assignee: Gantigmaa Selenge
>Priority: Major
> Fix For: 3.9.0
>
>
> Running the following code produces different behavior between ZooKeeper and 
> KRaft:
> {code:java}
> DescribeTopicsOptions options = new 
> DescribeTopicsOptions().includeAuthorizedOperations(false);
> TopicCollection topics = 
> TopicCollection.ofTopicNames(Collections.singletonList(topic));
> DescribeTopicsResult describeTopicsResult = admin.describeTopics(topics, 
> options);
> TopicDescription topicDescription = 
> describeTopicsResult.topicNameValues().get(topic).get();
> System.out.println(topicDescription.authorizedOperations());
> {code}
> With ZooKeeper this print null, and with KRaft it prints [ALTER, READ, 
> DELETE, ALTER_CONFIGS, CREATE, DESCRIBE_CONFIGS, WRITE, DESCRIBE].
> The Admin.getTopicDescriptionFromDescribeTopicsResponseTopic does not take 
> into account the options provided to describeTopics() and always populates 
> the authorizedOperations field.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Subscribe for discussion on Kafka® development

2024-06-12 Thread Punsak Incham

I want to subscribe for discussion on Kafka® development.


Best regards,

   [A black and blue logo  Description automatically generated]
Punsak Incham (Mon)
Data Platform Engineer
Data Modernization
Tel : +66(0)91-790-1302 | Email : pun...@mfec.co.th
[signature_2789072556]
DISCLAIMER: This e-mail and any attachments may contain confidential or legally 
privileged information for use of MFEC Public Company Limited only.
If you are not the intended recipient, you are not authorized to copy or 
disclose all or any part of it without the prior written consent of the company.
Refer to https://www.mfec.co.th/en/privacy-policy/

Re: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

2024-06-12 Thread Mario Fiore Vitale

Hi Greg,

The same check was made by Chris Egerton during the discussion thread.

The original KIP scope was just for the InsertField, ExtractField SMTs,
then we decided to enlarge the scope of only the KIP to also other
potential affected SMTs.

As of now the PR scope, instead, is only for InsertField and ExtractField
SMTs.

Thanks,
Mario.


On Wed, Jun 12, 2024 at 12:08 AM Greg Harris 
wrote:

> Hi Mario,
>
> Thanks for the KIP. I'm on-board with this KIP, I just wanted to verify a
> discrepancy I noticed.
>
> I checked all of the call-sites of Struct#get(Field) and Struct#get(String)
> in Kafka, and noticed there are some call-sites which are not included in
> the KIP.
> 1. The Flatten transformation seems to already have the
> "replace.null.with.default=false" behavior unconditionally.
> 2. The MaskField transformation unconditionally injects default values for
> top-level structs.
> 3. The ValueTokey transformation injects defaults for each of the
> configured "fields"
> 4. The Values#convertToString method injects default values, but as this
> isn't configurable I think we'll need to leave it as-is.
>
> Did you intend to add this configuration to the Flatten, MaskField, or
> ValueToKey transformations, or should those be addressed in a separate
> effort?
> Since the "false" behavior is desirable, I think the Flatten transform
> could be left as-is, rather than adding a configuration that nobody needs.
>
> Thanks,
> Greg
>
> On Tue, Jun 11, 2024 at 9:24 AM Yash Mayya  wrote:
>
> > Thanks for the KIP!
> >
> > +1 (binding)
> >
> > On Tue, Jun 11, 2024 at 7:40 PM Mario Fiore Vitale 
> > wrote:
> >
> > > Hi all,
> > >
> > > Just wanted to bump up this thread for visibility.
> > >
> > > Thanks!
> > >
> > > On Wed, May 29, 2024 at 5:31 PM Mickael Maison <
> mickael.mai...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Mario,
> > > >
> > > > +1 (binding)
> > > > Thanks for the KIP!
> > > >
> > > > Mickael
> > > >
> > > > On Mon, May 27, 2024 at 12:06 PM Mario Fiore Vitale <
> > mvit...@redhat.com>
> > > > wrote:
> > > > >
> > > > > After 7 days I received only one vote. Should I suppose this will
> not
> > > be
> > > > > approved?
> > > > >
> > > > > On Mon, May 20, 2024 at 4:14 PM Chris Egerton
> >  > > >
> > > > > wrote:
> > > > >
> > > > > > Thanks for the KIP! +1 (binding)
> > > > > >
> > > > > > On Mon, May 20, 2024 at 4:22 AM Mario Fiore Vitale <
> > > mvit...@redhat.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > I'd like to call a vote on KIP-1040 which aims to improve
> > handling
> > > of
> > > > > > > nullable values in InsertField, ExtractField, and other
> > > > transformations
> > > > > > >
> > > > > > > KIP -
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=303794677
> > > > > > >
> > > > > > > Discussion thread -
> > > > > > >
> https://lists.apache.org/thread/ggqqqjbg6ccpz8g6ztyj7oxr80q5184n
> > > > > > >
> > > > > > > Thanks and regards,
> > > > > > > Mario
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Mario Fiore Vitale
> > > > >
> > > > > Senior Software Engineer
> > > > >
> > > > > Red Hat 
> > > > > 
> > > >
> > > >
> > >
> > > --
> > >
> > > Mario Fiore Vitale
> > >
> > > Senior Software Engineer
> > >
> > > Red Hat 
> > > 
> > >
> >
>


-- 

Mario Fiore Vitale

Senior Software Engineer

Red Hat

[jira] [Created] (KAFKA-16941) Flaky test - testDynamicBrokerConfigUpdateUsingKraft [1] Type=Raft-Combined, MetadataVersion=4.0-IV0,Security=PLAINTEXT – kafka.admin.ConfigCommandIntegrationTest

2024-06-12 Thread Igor Soarez (Jira)

Igor Soarez created KAFKA-16941:
---

 Summary: Flaky test - testDynamicBrokerConfigUpdateUsingKraft [1] 
Type=Raft-Combined, MetadataVersion=4.0-IV0,Security=PLAINTEXT – 
kafka.admin.ConfigCommandIntegrationTest
 Key: KAFKA-16941
 URL: https://issues.apache.org/jira/browse/KAFKA-16941
 Project: Kafka
  Issue Type: Test
Reporter: Igor Soarez


https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16077/4/tests/
{code:java}
org.opentest4j.AssertionFailedError: Condition not met within timeout 5000. 
[listener.name.internal.ssl.keystore.location] are not updated ==> expected: 
 but was: 
    at 
org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
    at 
org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
    at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
    at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
    at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:214)
    at 
org.apache.kafka.test.TestUtils.lambda$waitForCondition$3(TestUtils.java:396)
    at 
org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:444)
    at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:393)
    at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:377)
    at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:367)
    at 
kafka.admin.ConfigCommandIntegrationTest.verifyConfigDefaultValue(ConfigCommandIntegrationTest.java:519)
    at 
kafka.admin.ConfigCommandIntegrationTest.deleteAndVerifyConfig(ConfigCommandIntegrationTest.java:514)
    at 
kafka.admin.ConfigCommandIntegrationTest.testDynamicBrokerConfigUpdateUsingKraft(ConfigCommandIntegrationTest.java:237)
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-06-12 Thread Ivan Yurchenko

Hi,

I'll try to do all the fixes and changes for KIP-899 [1] sooner today, but 
please proceed with the release if I don't manage.

Ivan

[1] https://github.com/apache/kafka/pull/13277

On Wed, Jun 12, 2024, at 12:54, Josep Prat wrote:
> Hi Luke,
> I think Jose, also mentioned that it won't be ready for v3.8.0 (but he can
> confirm this). My question now would be, given that it seems we would need
> a v3.9.0, do you think it's important to include
> https://github.com/apache/kafka/pull/16284 in v3.8.0?
> 
> Best,
> 
> On Wed, Jun 12, 2024 at 11:40 AM Luke Chen  wrote:
> 
> > Hi Josep
> >
> > For KIP-966, I think Calvin had mentioned he won't complete in v3.8.0.
> > https://lists.apache.org/thread/fsnr8wy5fznzfso7jgk90skgyo277fmw
> >
> > For unclean leader election, all we need is this PR:
> > https://github.com/apache/kafka/pull/16284
> > For this PR, I think it needs one more week to be completed.
> >
> > Thanks.
> > Luke
> >
> > On Wed, Jun 12, 2024 at 4:51 PM Josep Prat 
> > wrote:
> >
> > > Hi all,
> > >
> > > We are now really close to the planned code freeze deadline (today EOD).
> > > According to KIP-1012 [1] we agreed to stay in the 3.x branch until we
> > > achieve feature parity regarding Zookeeper and KRaft. The two main KIPs
> > > identified that would achieve this are: KIP-853 [2] and KIP-966 [3].
> > > At the moment of writing this email both KIPs are not completed. My
> > > question to the people driving both KIPs would be, how much more time do
> > > you think it's needed to bring these KIPs to completion?
> > >
> > > - If the time needed would be short, we could still include these 2 KIPs
> > in
> > > the release.
> > > - However, if the time needed would be on the scale of weeks, we should
> > > continue with the release plan for 3.8 and after start working on the 3.9
> > > release.
> > >
> > > What are your thoughts?
> > >
> > >
> > > [1]:
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> > > [2]:
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes
> > > [3]:
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas
> > >
> > > On Wed, Jun 12, 2024 at 10:40 AM Josep Prat  wrote:
> > >
> > > > Hi Rajini,
> > > > Yes, we could backport this one to the 3.8 branch. Would you be able to
> > > do
> > > > this once you merge this PR?
> > > >
> > > > Thanks
> > > >
> > > > On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram <
> > rajinisiva...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Hi Josep,
> > > >>
> > > >> The PR https://github.com/apache/kafka/pull/13277 for KIP-899 looks
> > > ready
> > > >> to be merged (waiting for the PR build).The PR changes several files,
> > > but
> > > >> is relatively straightforward and not risky. Also the changes are
> > under
> > > a
> > > >> config that is not enabled by default. Since the KIP was approved
> > before
> > > >> KIP freeze, will it be ok to include in 3.8.0?
> > > >>
> > > >> Thank you,
> > > >>
> > > >> Rajini
> > > >>
> > > >>
> > > >> On Tue, Jun 11, 2024 at 9:35 AM Josep Prat
> >  > > >
> > > >> wrote:
> > > >>
> > > >> > Hi all,
> > > >> >
> > > >> > I just want to remind everybody that the code freeze deadline is
> > > >> > approaching. June 12th EOD is the deadline.
> > > >> >
> > > >> > Please do not automatically backport any commit to the 3.8 branch
> > > after
> > > >> > June 12th EOD. Ping me if you think the commit should be backported
> > > >> (fixes
> > > >> > failures in the branch or critical bug fixes).
> > > >> >
> > > >> > Thanks all!
> > > >> >
> > > >> > On Sat, Jun 1, 2024 at 8:43 PM José Armando García Sancio
> > > >> >  wrote:
> > > >> >
> > > >> > > Hi Josep,
> > > >> > >
> > > >> > > See my comments below.
> > > >> > >
> > > >> > > On Wed, May 29, 2024 at 10:52 AM Josep Prat
> > > >>  > > >> > >
> > > >> > > wrote:
> > > >> > > > So I would propose to leave the deadlines as they are and
> > manually
> > > >> > cherry
> > > >> > > > pick the commits related to KIP-853 and KIP-966.
> > > >> > >
> > > >> > > Your proposal sounds good to me. I suspect that will be doing
> > > feature
> > > >> > > development for KIP-853 past the feature freeze and code freeze
> > > date.
> > > >> > > Maybe feature development will be finished around the end of June.
> > > >> > >
> > > >> > > I'll make sure to cherry pick commits for KIP-853 to the 3.8
> > branch
> > > >> > > once we have one.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > --
> > > >> > > -José
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > [image: Aiven] 
> > > >> >
> > > >> > *Josep Prat*
> > > >> > Open Source Engineering Director, *Aiven*
> > > >> > josep.p...@aiven.io   |   +491715557497
> > > >> > aiven.io    |   <
> > > >> https://www.facebook.com/aivencloud
> > > >> > >
> > > >> >      <
> > > >>

[jira] [Created] (KAFKA-16940) Flaky test "testNoConsumeWithDescribeAclViaSubscribe(String).quorum=kraft" – kafka.api.DelegationTokenEndToEndAuthorizationWithOwnerTest

2024-06-12 Thread Igor Soarez (Jira)

Igor Soarez created KAFKA-16940:
---

 Summary: Flaky test  
"testNoConsumeWithDescribeAclViaSubscribe(String).quorum=kraft" – 
kafka.api.DelegationTokenEndToEndAuthorizationWithOwnerTest
 Key: KAFKA-16940
 URL: https://issues.apache.org/jira/browse/KAFKA-16940
 Project: Kafka
  Issue Type: Test
Reporter: Igor Soarez


In 
[https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-16077/4/tests/]

 
{code:java}
java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.SaslAuthenticationException: Authentication 
failed during authentication due to invalid credentials with SASL mechanism 
SCRAM-SHA-256
    at 
java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
    at 
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
    at 
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
    at 
kafka.api.DelegationTokenEndToEndAuthorizationTest.createDelegationTokens(DelegationTokenEndToEndAuthorizationTest.scala:164)
    at 
kafka.api.DelegationTokenEndToEndAuthorizationTest.createDelegationTokens(DelegationTokenEndToEndAuthorizationTest.scala:156)
    at 
kafka.api.DelegationTokenEndToEndAuthorizationTest.configureSecurityAfterServersStart(DelegationTokenEndToEndAuthorizationTest.scala:99)
    at 
kafka.api.DelegationTokenEndToEndAuthorizationWithOwnerTest.configureSecurityAfterServersStart(DelegationTokenEndToEndAuthorizationWithOwnerTest.scala:77)
    at 
kafka.integration.KafkaServerTestHarness.setUp(KafkaServerTestHarness.scala:129)
    at 
kafka.api.IntegrationTestHarness.doSetup(IntegrationTestHarness.scala:141)
    at kafka.api.IntegrationTestHarness.setUp(IntegrationTestHarness.scala:121)
    at 
kafka.api.EndToEndAuthorizationTest.setUp(EndToEndAuthorizationTest.scala:167)
    at 
kafka.api.DelegationTokenEndToEndAuthorizationTest.setUp(DelegationTokenEndToEndAuthorizationTest.scala:134)
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] RemoteLogSegmentMetadataRecord Schema Evolution

2024-06-12 Thread Kamal Chandraprakash

Ignore the above message. Went over the Kafka message definition README

guide and learnt about the tagged fields.

On Tue, Jun 11, 2024 at 10:32 PM Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:

> Hi,
>
> We have defined the remote log segment metadata schemas with type as
> "metadata" (controller records) in #10271
> :
>
> 1. RemoteLogSegmentMetadataRecord.json
> 
> 2. RemoteLogSegmentMetadataSnapshotRecord.json
> 
> 3. RemoteLogSegmentMetadataUpdateRecord.json
> 
> and
> 4. RemotePartitionDeleteMetadataRecord.json
> 
>
> In the future, when the schema gets evolved, then the backward
> compatibility might get broken. The broker with the old
> schema will fail to read new events and throw UnsupportedVersionException.
> This also means that we cannot rollback
> the build once some segments are uploaded to the remote storage with the
> new schema events.
>
> Let me know if my understanding is not correct.
> --
> Kamal
>
>
>
>

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Luke,
I think Jose, also mentioned that it won't be ready for v3.8.0 (but he can
confirm this). My question now would be, given that it seems we would need
a v3.9.0, do you think it's important to include
https://github.com/apache/kafka/pull/16284 in v3.8.0?

Best,

On Wed, Jun 12, 2024 at 11:40 AM Luke Chen  wrote:

> Hi Josep
>
> For KIP-966, I think Calvin had mentioned he won't complete in v3.8.0.
> https://lists.apache.org/thread/fsnr8wy5fznzfso7jgk90skgyo277fmw
>
> For unclean leader election, all we need is this PR:
> https://github.com/apache/kafka/pull/16284
> For this PR, I think it needs one more week to be completed.
>
> Thanks.
> Luke
>
> On Wed, Jun 12, 2024 at 4:51 PM Josep Prat 
> wrote:
>
> > Hi all,
> >
> > We are now really close to the planned code freeze deadline (today EOD).
> > According to KIP-1012 [1] we agreed to stay in the 3.x branch until we
> > achieve feature parity regarding Zookeeper and KRaft. The two main KIPs
> > identified that would achieve this are: KIP-853 [2] and KIP-966 [3].
> > At the moment of writing this email both KIPs are not completed. My
> > question to the people driving both KIPs would be, how much more time do
> > you think it's needed to bring these KIPs to completion?
> >
> > - If the time needed would be short, we could still include these 2 KIPs
> in
> > the release.
> > - However, if the time needed would be on the scale of weeks, we should
> > continue with the release plan for 3.8 and after start working on the 3.9
> > release.
> >
> > What are your thoughts?
> >
> >
> > [1]:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> > [2]:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes
> > [3]:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas
> >
> > On Wed, Jun 12, 2024 at 10:40 AM Josep Prat  wrote:
> >
> > > Hi Rajini,
> > > Yes, we could backport this one to the 3.8 branch. Would you be able to
> > do
> > > this once you merge this PR?
> > >
> > > Thanks
> > >
> > > On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram <
> rajinisiva...@gmail.com
> > >
> > > wrote:
> > >
> > >> Hi Josep,
> > >>
> > >> The PR https://github.com/apache/kafka/pull/13277 for KIP-899 looks
> > ready
> > >> to be merged (waiting for the PR build).The PR changes several files,
> > but
> > >> is relatively straightforward and not risky. Also the changes are
> under
> > a
> > >> config that is not enabled by default. Since the KIP was approved
> before
> > >> KIP freeze, will it be ok to include in 3.8.0?
> > >>
> > >> Thank you,
> > >>
> > >> Rajini
> > >>
> > >>
> > >> On Tue, Jun 11, 2024 at 9:35 AM Josep Prat
>  > >
> > >> wrote:
> > >>
> > >> > Hi all,
> > >> >
> > >> > I just want to remind everybody that the code freeze deadline is
> > >> > approaching. June 12th EOD is the deadline.
> > >> >
> > >> > Please do not automatically backport any commit to the 3.8 branch
> > after
> > >> > June 12th EOD. Ping me if you think the commit should be backported
> > >> (fixes
> > >> > failures in the branch or critical bug fixes).
> > >> >
> > >> > Thanks all!
> > >> >
> > >> > On Sat, Jun 1, 2024 at 8:43 PM José Armando García Sancio
> > >> >  wrote:
> > >> >
> > >> > > Hi Josep,
> > >> > >
> > >> > > See my comments below.
> > >> > >
> > >> > > On Wed, May 29, 2024 at 10:52 AM Josep Prat
> > >>  > >> > >
> > >> > > wrote:
> > >> > > > So I would propose to leave the deadlines as they are and
> manually
> > >> > cherry
> > >> > > > pick the commits related to KIP-853 and KIP-966.
> > >> > >
> > >> > > Your proposal sounds good to me. I suspect that will be doing
> > feature
> > >> > > development for KIP-853 past the feature freeze and code freeze
> > date.
> > >> > > Maybe feature development will be finished around the end of June.
> > >> > >
> > >> > > I'll make sure to cherry pick commits for KIP-853 to the 3.8
> branch
> > >> > > once we have one.
> > >> > >
> > >> > > Thanks,
> > >> > > --
> > >> > > -José
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> > [image: Aiven] 
> > >> >
> > >> > *Josep Prat*
> > >> > Open Source Engineering Director, *Aiven*
> > >> > josep.p...@aiven.io   |   +491715557497
> > >> > aiven.io    |   <
> > >> https://www.facebook.com/aivencloud
> > >> > >
> > >> >      <
> > >> > https://twitter.com/aiven_io>
> > >> > *Aiven Deutschland GmbH*
> > >> > Alexanderufer 3-7, 10117 Berlin
> > >> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >> > Amtsgericht Charlottenburg, HRB 209739 B
> > >> >
> > >>
> > >
> > >
> > > --
> > > [image: Aiven] 
> > >
> > > *Josep Prat*
> > > Open Source Engineering Director, *Aiven*
> > > josep.p...@aiven.io   |   +491715557497
> > > aiven.io    |
> > > 
> > >

[jira] [Created] (KAFKA-16939) Revisit ConfigCommandIntegrationTest

Chia-Ping Tsai created KAFKA-16939:
--

 Summary: Revisit ConfigCommandIntegrationTest
 Key: KAFKA-16939
 URL: https://issues.apache.org/jira/browse/KAFKA-16939
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


ConfigCommandIntegrationTest has some invalid operations

1. the base command should not include the broker id [0]
2. don't set the broker id when we are looking for default value [1]


[0] 
https://github.com/apache/kafka/blob/trunk/core/src/test/java/kafka/admin/ConfigCommandIntegrationTest.java#L477
[1] 
https://github.com/apache/kafka/blob/trunk/core/src/test/java/kafka/admin/ConfigCommandIntegrationTest.java#L496



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-06-12 Thread Luke Chen

Hi Josep

For KIP-966, I think Calvin had mentioned he won't complete in v3.8.0.
https://lists.apache.org/thread/fsnr8wy5fznzfso7jgk90skgyo277fmw

For unclean leader election, all we need is this PR:
https://github.com/apache/kafka/pull/16284
For this PR, I think it needs one more week to be completed.

Thanks.
Luke

On Wed, Jun 12, 2024 at 4:51 PM Josep Prat 
wrote:

> Hi all,
>
> We are now really close to the planned code freeze deadline (today EOD).
> According to KIP-1012 [1] we agreed to stay in the 3.x branch until we
> achieve feature parity regarding Zookeeper and KRaft. The two main KIPs
> identified that would achieve this are: KIP-853 [2] and KIP-966 [3].
> At the moment of writing this email both KIPs are not completed. My
> question to the people driving both KIPs would be, how much more time do
> you think it's needed to bring these KIPs to completion?
>
> - If the time needed would be short, we could still include these 2 KIPs in
> the release.
> - However, if the time needed would be on the scale of weeks, we should
> continue with the release plan for 3.8 and after start working on the 3.9
> release.
>
> What are your thoughts?
>
>
> [1]:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> [2]:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes
> [3]:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas
>
> On Wed, Jun 12, 2024 at 10:40 AM Josep Prat  wrote:
>
> > Hi Rajini,
> > Yes, we could backport this one to the 3.8 branch. Would you be able to
> do
> > this once you merge this PR?
> >
> > Thanks
> >
> > On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram  >
> > wrote:
> >
> >> Hi Josep,
> >>
> >> The PR https://github.com/apache/kafka/pull/13277 for KIP-899 looks
> ready
> >> to be merged (waiting for the PR build).The PR changes several files,
> but
> >> is relatively straightforward and not risky. Also the changes are under
> a
> >> config that is not enabled by default. Since the KIP was approved before
> >> KIP freeze, will it be ok to include in 3.8.0?
> >>
> >> Thank you,
> >>
> >> Rajini
> >>
> >>
> >> On Tue, Jun 11, 2024 at 9:35 AM Josep Prat  >
> >> wrote:
> >>
> >> > Hi all,
> >> >
> >> > I just want to remind everybody that the code freeze deadline is
> >> > approaching. June 12th EOD is the deadline.
> >> >
> >> > Please do not automatically backport any commit to the 3.8 branch
> after
> >> > June 12th EOD. Ping me if you think the commit should be backported
> >> (fixes
> >> > failures in the branch or critical bug fixes).
> >> >
> >> > Thanks all!
> >> >
> >> > On Sat, Jun 1, 2024 at 8:43 PM José Armando García Sancio
> >> >  wrote:
> >> >
> >> > > Hi Josep,
> >> > >
> >> > > See my comments below.
> >> > >
> >> > > On Wed, May 29, 2024 at 10:52 AM Josep Prat
> >>  >> > >
> >> > > wrote:
> >> > > > So I would propose to leave the deadlines as they are and manually
> >> > cherry
> >> > > > pick the commits related to KIP-853 and KIP-966.
> >> > >
> >> > > Your proposal sounds good to me. I suspect that will be doing
> feature
> >> > > development for KIP-853 past the feature freeze and code freeze
> date.
> >> > > Maybe feature development will be finished around the end of June.
> >> > >
> >> > > I'll make sure to cherry pick commits for KIP-853 to the 3.8 branch
> >> > > once we have one.
> >> > >
> >> > > Thanks,
> >> > > --
> >> > > -José
> >> > >
> >> >
> >> >
> >> > --
> >> > [image: Aiven] 
> >> >
> >> > *Josep Prat*
> >> > Open Source Engineering Director, *Aiven*
> >> > josep.p...@aiven.io   |   +491715557497
> >> > aiven.io    |   <
> >> https://www.facebook.com/aivencloud
> >> > >
> >> >      <
> >> > https://twitter.com/aiven_io>
> >> > *Aiven Deutschland GmbH*
> >> > Alexanderufer 3-7, 10117 Berlin
> >> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >> > Amtsgericht Charlottenburg, HRB 209739 B
> >> >
> >>
> >
> >
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io    |
> > 
> >    <
> https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |    >
>      <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #3000

See

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-06-12 Thread Rajini Sivaram

>
> Yes, we could backport this one to the 3.8 branch. Would you be able to do
> this once you merge this PR?


Thanks Josep! There were some unit test failures in last night's PR build,
so I have commented on the PR for Ivan to fix. If he fixes it today, then I
can merge to trunk and then cherry-pick to 3.8 today.

Regards,

Rajini


On Wed, Jun 12, 2024 at 9:51 AM Josep Prat 
wrote:

> Hi all,
>
> We are now really close to the planned code freeze deadline (today EOD).
> According to KIP-1012 [1] we agreed to stay in the 3.x branch until we
> achieve feature parity regarding Zookeeper and KRaft. The two main KIPs
> identified that would achieve this are: KIP-853 [2] and KIP-966 [3].
> At the moment of writing this email both KIPs are not completed. My
> question to the people driving both KIPs would be, how much more time do
> you think it's needed to bring these KIPs to completion?
>
> - If the time needed would be short, we could still include these 2 KIPs in
> the release.
> - However, if the time needed would be on the scale of weeks, we should
> continue with the release plan for 3.8 and after start working on the 3.9
> release.
>
> What are your thoughts?
>
>
> [1]:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> [2]:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes
> [3]:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas
>
> On Wed, Jun 12, 2024 at 10:40 AM Josep Prat  wrote:
>
> > Hi Rajini,
> > Yes, we could backport this one to the 3.8 branch. Would you be able to
> do
> > this once you merge this PR?
> >
> > Thanks
> >
> > On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram  >
> > wrote:
> >
> >> Hi Josep,
> >>
> >> The PR https://github.com/apache/kafka/pull/13277 for KIP-899 looks
> ready
> >> to be merged (waiting for the PR build).The PR changes several files,
> but
> >> is relatively straightforward and not risky. Also the changes are under
> a
> >> config that is not enabled by default. Since the KIP was approved before
> >> KIP freeze, will it be ok to include in 3.8.0?
> >>
> >> Thank you,
> >>
> >> Rajini
> >>
> >>
> >> On Tue, Jun 11, 2024 at 9:35 AM Josep Prat  >
> >> wrote:
> >>
> >> > Hi all,
> >> >
> >> > I just want to remind everybody that the code freeze deadline is
> >> > approaching. June 12th EOD is the deadline.
> >> >
> >> > Please do not automatically backport any commit to the 3.8 branch
> after
> >> > June 12th EOD. Ping me if you think the commit should be backported
> >> (fixes
> >> > failures in the branch or critical bug fixes).
> >> >
> >> > Thanks all!
> >> >
> >> > On Sat, Jun 1, 2024 at 8:43 PM José Armando García Sancio
> >> >  wrote:
> >> >
> >> > > Hi Josep,
> >> > >
> >> > > See my comments below.
> >> > >
> >> > > On Wed, May 29, 2024 at 10:52 AM Josep Prat
> >>  >> > >
> >> > > wrote:
> >> > > > So I would propose to leave the deadlines as they are and manually
> >> > cherry
> >> > > > pick the commits related to KIP-853 and KIP-966.
> >> > >
> >> > > Your proposal sounds good to me. I suspect that will be doing
> feature
> >> > > development for KIP-853 past the feature freeze and code freeze
> date.
> >> > > Maybe feature development will be finished around the end of June.
> >> > >
> >> > > I'll make sure to cherry pick commits for KIP-853 to the 3.8 branch
> >> > > once we have one.
> >> > >
> >> > > Thanks,
> >> > > --
> >> > > -José
> >> > >
> >> >
> >> >
> >> > --
> >> > [image: Aiven] 
> >> >
> >> > *Josep Prat*
> >> > Open Source Engineering Director, *Aiven*
> >> > josep.p...@aiven.io   |   +491715557497
> >> > aiven.io    |   <
> >> https://www.facebook.com/aivencloud
> >> > >
> >> >      <
> >> > https://twitter.com/aiven_io>
> >> > *Aiven Deutschland GmbH*
> >> > Alexanderufer 3-7, 10117 Berlin
> >> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >> > Amtsgericht Charlottenburg, HRB 209739 B
> >> >
> >>
> >
> >
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io    |
> > 
> >    <
> https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |    >
>      <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valto

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi all,

We are now really close to the planned code freeze deadline (today EOD).
According to KIP-1012 [1] we agreed to stay in the 3.x branch until we
achieve feature parity regarding Zookeeper and KRaft. The two main KIPs
identified that would achieve this are: KIP-853 [2] and KIP-966 [3].
At the moment of writing this email both KIPs are not completed. My
question to the people driving both KIPs would be, how much more time do
you think it's needed to bring these KIPs to completion?

- If the time needed would be short, we could still include these 2 KIPs in
the release.
- However, if the time needed would be on the scale of weeks, we should
continue with the release plan for 3.8 and after start working on the 3.9
release.

What are your thoughts?


[1]:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
[2]:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes
[3]:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas

On Wed, Jun 12, 2024 at 10:40 AM Josep Prat  wrote:

> Hi Rajini,
> Yes, we could backport this one to the 3.8 branch. Would you be able to do
> this once you merge this PR?
>
> Thanks
>
> On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram 
> wrote:
>
>> Hi Josep,
>>
>> The PR https://github.com/apache/kafka/pull/13277 for KIP-899 looks ready
>> to be merged (waiting for the PR build).The PR changes several files, but
>> is relatively straightforward and not risky. Also the changes are under a
>> config that is not enabled by default. Since the KIP was approved before
>> KIP freeze, will it be ok to include in 3.8.0?
>>
>> Thank you,
>>
>> Rajini
>>
>>
>> On Tue, Jun 11, 2024 at 9:35 AM Josep Prat 
>> wrote:
>>
>> > Hi all,
>> >
>> > I just want to remind everybody that the code freeze deadline is
>> > approaching. June 12th EOD is the deadline.
>> >
>> > Please do not automatically backport any commit to the 3.8 branch after
>> > June 12th EOD. Ping me if you think the commit should be backported
>> (fixes
>> > failures in the branch or critical bug fixes).
>> >
>> > Thanks all!
>> >
>> > On Sat, Jun 1, 2024 at 8:43 PM José Armando García Sancio
>> >  wrote:
>> >
>> > > Hi Josep,
>> > >
>> > > See my comments below.
>> > >
>> > > On Wed, May 29, 2024 at 10:52 AM Josep Prat
>> > > >
>> > > wrote:
>> > > > So I would propose to leave the deadlines as they are and manually
>> > cherry
>> > > > pick the commits related to KIP-853 and KIP-966.
>> > >
>> > > Your proposal sounds good to me. I suspect that will be doing feature
>> > > development for KIP-853 past the feature freeze and code freeze date.
>> > > Maybe feature development will be finished around the end of June.
>> > >
>> > > I'll make sure to cherry pick commits for KIP-853 to the 3.8 branch
>> > > once we have one.
>> > >
>> > > Thanks,
>> > > --
>> > > -José
>> > >
>> >
>> >
>> > --
>> > [image: Aiven] 
>> >
>> > *Josep Prat*
>> > Open Source Engineering Director, *Aiven*
>> > josep.p...@aiven.io   |   +491715557497
>> > aiven.io    |   <
>> https://www.facebook.com/aivencloud
>> > >
>> >      <
>> > https://twitter.com/aiven_io>
>> > *Aiven Deutschland GmbH*
>> > Alexanderufer 3-7, 10117 Berlin
>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> > Amtsgericht Charlottenburg, HRB 209739 B
>> >
>>
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |
> 
>    
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Re: [DISCUSS] Apache Kafka 3.8.0 release

Hi Rajini,
Yes, we could backport this one to the 3.8 branch. Would you be able to do
this once you merge this PR?

Thanks

On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram 
wrote:

> Hi Josep,
>
> The PR https://github.com/apache/kafka/pull/13277 for KIP-899 looks ready
> to be merged (waiting for the PR build).The PR changes several files, but
> is relatively straightforward and not risky. Also the changes are under a
> config that is not enabled by default. Since the KIP was approved before
> KIP freeze, will it be ok to include in 3.8.0?
>
> Thank you,
>
> Rajini
>
>
> On Tue, Jun 11, 2024 at 9:35 AM Josep Prat 
> wrote:
>
> > Hi all,
> >
> > I just want to remind everybody that the code freeze deadline is
> > approaching. June 12th EOD is the deadline.
> >
> > Please do not automatically backport any commit to the 3.8 branch after
> > June 12th EOD. Ping me if you think the commit should be backported
> (fixes
> > failures in the branch or critical bug fixes).
> >
> > Thanks all!
> >
> > On Sat, Jun 1, 2024 at 8:43 PM José Armando García Sancio
> >  wrote:
> >
> > > Hi Josep,
> > >
> > > See my comments below.
> > >
> > > On Wed, May 29, 2024 at 10:52 AM Josep Prat
>  > >
> > > wrote:
> > > > So I would propose to leave the deadlines as they are and manually
> > cherry
> > > > pick the commits related to KIP-853 and KIP-966.
> > >
> > > Your proposal sounds good to me. I suspect that will be doing feature
> > > development for KIP-853 past the feature freeze and code freeze date.
> > > Maybe feature development will be finished around the end of June.
> > >
> > > I'll make sure to cherry pick commits for KIP-853 to the 3.8 branch
> > > once we have one.
> > >
> > > Thanks,
> > > --
> > > -José
> > >
> >
> >
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io    |   <
> https://www.facebook.com/aivencloud
> > >
> >      <
> > https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

[jira] [Created] (KAFKA-16938) non-dynamic props gets corrupted due to circular reference between DynamicBrokerConfig and DynamicConfig