Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[ziming deng]

Congratulations Satish!

> On Oct 27, 2023, at 23:03, Jun Rao  wrote:
> 
> Hi, Everyone,
> 
> Satish Duggana has been a Kafka committer since 2022. He has been very
> instrumental to the community since becoming a committer. It's my pleasure
> to announce that Satish is now a member of Kafka PMC.
> 
> Congratulations Satish!
> 
> Jun
> on behalf of Apache Kafka PMC

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2336

[Apache Jenkins Server]

See 

Re: [PR] MINOR: Add note about KAFKA-15653 [kafka-site]

[via GitHub]

ijuma commented on code in PR #564:
URL: https://github.com/apache/kafka-site/pull/564#discussion_r1375131837


##
36/upgrade.html:
##
@@ -116,6 +116,12 @@ Notable changes in 3
 For more information about the early access tiered storage 
feature, please check https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage;>KIP-405
 and
 https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes;>Tiered
 Storage Early Access Release Note.
 
+Transaction partition verification (https://cwiki.apache.org/confluence/display/KAFKA/KIP-890%3A+Transactions+Server-Side+Defense;>KIP-890)
+has been added to data partitions to prevent hanging transactions. 
Workloads with compression can experience InvalidRecordExceptions and 
UnknownServerExceptions.
+For workloads with compression, 
transaction.partition.verification.enable should be set to false. 
Note that the default for 3.6 is true.

Review Comment:
   I'd just say "This feature can be disabled by setting..." (instead of "For 
workloads with compression...".



##
36/upgrade.html:
##
@@ -116,6 +116,12 @@ Notable changes in 3
 For more information about the early access tiered storage 
feature, please check https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage;>KIP-405
 and
 https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes;>Tiered
 Storage Early Access Release Note.
 
+Transaction partition verification (https://cwiki.apache.org/confluence/display/KAFKA/KIP-890%3A+Transactions+Server-Side+Defense;>KIP-890)
+has been added to data partitions to prevent hanging transactions. 
Workloads with compression can experience InvalidRecordExceptions and 
UnknownServerExceptions.
+For workloads with compression, 
transaction.partition.verification.enable should be set to false. 
Note that the default for 3.6 is true.
+The configuration can also be updated dynamically and is applied 
to the broker.
+This will be fixed in a future release. See https://issues.apache.org/jira/browse/KAFKA-15653;>KAFKA-15653 for 
more details.

Review Comment:
   Can we say it will be fixed in 3.6.1?.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[PR] MINOR: Add note about KAFKA-15653 [kafka-site]

[via GitHub]

jolshan opened a new pull request, #564:
URL: https://github.com/apache/kafka-site/pull/564

   KAFKA-15653 can be painful for folks with compression. Adding a note about 
the issue and how to mitigate it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] Road to Kafka 4.0

[Colin McCabe]

More notes on this:

As far as I can tell, KAFKA-15513 should be fixed by the recent SCRAM work. I 
hope someone can verify that this is the case. Proven and I commented on the 
JIRA.

KAFKA-15489 is a bug fix but not a regression. Hence it doesn't block any new 
(or old) releases. While we have never seen anything like this in more than a 
year of production use, I do hope we can get this fix in soon. Jose and Alyssa 
have been reviewing the PR. Thanks, Luke, for posting the fix.

The docs JIRAs, KAFKA-14941 and KAFKA-14369, certainly don't block anything, 
but we are always happy to have more documentation. One comment: as far as I 
know (am I wrong about this?) none of our sample broker configurations in 
config/ ever included non-PLAINTEXT auth. So this is a gap with ZK mode as 
well. I do think an SSL-based configuration or two would be nice to include in 
there. (SSL sample configs are probably relatively easy to do. Kerberos is the 
real hard mode.)

"KAFKA-14349: Support dynamically resizing the KRaft controller's thread pools" 
was fixed as part of the PR that implemented KAFKA-14351, but we forgot to 
close the JIRA. I have corrected this.

Also, Luke, even though I removed the blocker tag from some of these, thanks 
for taking a look at our backlog. That always helps us improve. If you find 
something that should be fixed for kraft, go ahead and add the "kraft" tag so 
we can find it more easily.

best,
Colin

On Fri, Oct 27, 2023, at 15:01, Colin McCabe wrote:
> Hi Luke,
>
> As Ismael said, the only thing on that list that is a 4.0 blocker is 
> KIP-858: JBOD support. It is the last remaining feature gap with ZK 
> mode. We previously agreed that it needed to be implemented so that 
> users of this feature could migrate to KRaft as planned.
>
> The good news is that several committers and community members are 
> actively working on KIP-858 and we expect it to land in 3.7 as planned.
>
> I will remove the 4.0-blocker label from the others since they do not block 
> 4.0.
>
> best,
> Colin
>
>
> On Wed, Oct 11, 2023, at 05:17, Luke Chen wrote:
>> Hi all,
>>
>> While Kafka 3.6.0 is released, I’d like to start the discussion for the
>> “road to Kafka 4.0”. Based on the plan in KIP-833
>> ,
>> the next release 3.7 will be the final release before moving to Kafka 
>> 4.0
>> to remove the Zookeeper from Kafka. Before making this major change, I'd
>> like to get consensus on the "must-have features/fixes for Kafka 4.0", 
>> to
>> avoid some users being surprised when upgrading to Kafka 4.0. The 
>> intent is
>> to have a clear communication about what to expect in the following 
>> months.
>> In particular we should be signaling what features and configurations 
>> are
>> not supported, or at risk (if no one is able to add support or fix known
>> bugs).
>>
>> Here is the JIRA tickets list
>>  I
>> labeled for "4.0-blocker". The criteria I labeled as “4.0-blocker” are:
>> 1. The feature is supported in Zookeeper Mode, but not supported in KRaft
>> mode, yet (ex: KIP-858: JBOD in KRaft)
>> 2. Critical bugs in KRaft, (ex: KAFKA-15489 : split brain in KRaft
>> controller quorum)
>>
>> If you disagree with my current list, welcome to have discussion in the
>> specific JIRA ticket. Or, if you think there are some tickets I missed,
>> welcome to start a discussion in the JIRA ticket and ping me or other
>> people. After we get the consensus, we can label/unlabel it afterwards.
>> Again, the goal is to have an open communication with the community about
>> what will be coming in 4.0.
>>
>> Below is the high level category of the list content:
>>
>> 1. Recovery from disk failure
>> KIP-856
>> :
>> KRaft Disk Failure Recovery
>>
>> 2. Prevote to support controllers more than 3
>> KIP-650
>> :
>> Enhance Kafkaesque Raft semantics
>>
>> 3. JBOD support
>> KIP-858
>> :
>> Handle
>> JBOD broker disk failure in KRaft
>>
>> 4. Scale up/down Controllers
>> KIP-853
>> :
>> KRaft Controller Membership Changes
>>
>> 5. Modifying dynamic configurations on the KRaft controller
>>
>> 6. Critical bugs in KRaft
>>
>> Does this make sense?
>> Any feedback is welcomed.
>>
>> Thank you.
>> Luke

Re: [DISCUSS] Road to Kafka 4.0

[Colin McCabe]

Hi Luke,

As Ismael said, the only thing on that list that is a 4.0 blocker is KIP-858: 
JBOD support. It is the last remaining feature gap with ZK mode. We previously 
agreed that it needed to be implemented so that users of this feature could 
migrate to KRaft as planned.

The good news is that several committers and community members are actively 
working on KIP-858 and we expect it to land in 3.7 as planned.

I will remove the 4.0-blocker label from the others since they do not block 4.0.

best,
Colin


On Wed, Oct 11, 2023, at 05:17, Luke Chen wrote:
> Hi all,
>
> While Kafka 3.6.0 is released, I’d like to start the discussion for the
> “road to Kafka 4.0”. Based on the plan in KIP-833
> ,
> the next release 3.7 will be the final release before moving to Kafka 
> 4.0
> to remove the Zookeeper from Kafka. Before making this major change, I'd
> like to get consensus on the "must-have features/fixes for Kafka 4.0", 
> to
> avoid some users being surprised when upgrading to Kafka 4.0. The 
> intent is
> to have a clear communication about what to expect in the following 
> months.
> In particular we should be signaling what features and configurations 
> are
> not supported, or at risk (if no one is able to add support or fix known
> bugs).
>
> Here is the JIRA tickets list
>  I
> labeled for "4.0-blocker". The criteria I labeled as “4.0-blocker” are:
> 1. The feature is supported in Zookeeper Mode, but not supported in KRaft
> mode, yet (ex: KIP-858: JBOD in KRaft)
> 2. Critical bugs in KRaft, (ex: KAFKA-15489 : split brain in KRaft
> controller quorum)
>
> If you disagree with my current list, welcome to have discussion in the
> specific JIRA ticket. Or, if you think there are some tickets I missed,
> welcome to start a discussion in the JIRA ticket and ping me or other
> people. After we get the consensus, we can label/unlabel it afterwards.
> Again, the goal is to have an open communication with the community about
> what will be coming in 4.0.
>
> Below is the high level category of the list content:
>
> 1. Recovery from disk failure
> KIP-856
> :
> KRaft Disk Failure Recovery
>
> 2. Prevote to support controllers more than 3
> KIP-650
> :
> Enhance Kafkaesque Raft semantics
>
> 3. JBOD support
> KIP-858
> :
> Handle
> JBOD broker disk failure in KRaft
>
> 4. Scale up/down Controllers
> KIP-853
> :
> KRaft Controller Membership Changes
>
> 5. Modifying dynamic configurations on the KRaft controller
>
> 6. Critical bugs in KRaft
>
> Does this make sense?
> Any feedback is welcomed.
>
> Thank you.
> Luke

[jira] [Resolved] (KAFKA-15704) ControllerRegistrationRequest must set ZkMigrationReady field if appropriate

[Colin McCabe (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin McCabe resolved KAFKA-15704.
--
Resolution: Fixed

> ControllerRegistrationRequest must set ZkMigrationReady field if appropriate
> 
>
> Key: KAFKA-15704
> URL: https://issues.apache.org/jira/browse/KAFKA-15704
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Colin McCabe
>Assignee: David Arthur
>Priority: Major
> Fix For: 3.7.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2335

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 322010 lines...]

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkAclMigrationClientTest 
> testAclsChangesInSnapshot() PASSED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkAclMigrationClientTest 
> testAclsMigrateAndDualWrite() STARTED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkAclMigrationClientTest 
> testAclsMigrateAndDualWrite() PASSED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testUpdateExistingPartitions() STARTED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testUpdateExistingPartitions() PASSED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testEmptyWrite() STARTED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testEmptyWrite() PASSED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testReadMigrateAndWriteProducerId() STARTED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testReadMigrateAndWriteProducerId() PASSED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testExistingKRaftControllerClaim() STARTED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testExistingKRaftControllerClaim() PASSED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testMigrateTopicConfigs() STARTED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testMigrateTopicConfigs() PASSED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testNonIncreasingKRaftEpoch() STARTED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testNonIncreasingKRaftEpoch() PASSED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testMigrateEmptyZk() STARTED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testMigrateEmptyZk() PASSED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testTopicAndBrokerConfigsMigrationWithSnapshots() STARTED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testTopicAndBrokerConfigsMigrationWithSnapshots() PASSED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testClaimAndReleaseExistingController() STARTED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testClaimAndReleaseExistingController() PASSED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testClaimAbsentController() STARTED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testClaimAbsentController() PASSED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testIdempotentCreateTopics() STARTED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testIdempotentCreateTopics() PASSED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testCreateNewTopic() STARTED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testCreateNewTopic() PASSED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testUpdateExistingTopicWithNewAndChangedPartitions() STARTED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZkMigrationClientTest > 
testUpdateExistingTopicWithNewAndChangedPartitions() PASSED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZooKeeperClientTest > 
testZNodeChangeHandlerForDataChange() STARTED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZooKeeperClientTest > 
testZNodeChangeHandlerForDataChange() PASSED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZooKeeperClientTest > 
testZooKeeperSessionStateMetric() STARTED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZooKeeperClientTest > 
testZooKeeperSessionStateMetric() PASSED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZooKeeperClientTest > 
testExceptionInBeforeInitializingSession() STARTED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZooKeeperClientTest > 
testExceptionInBeforeInitializingSession() PASSED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZooKeeperClientTest > 
testGetChildrenExistingZNode() STARTED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZooKeeperClientTest > 
testGetChildrenExistingZNode() PASSED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZooKeeperClientTest > 
testConnection() STARTED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZooKeeperClientTest > 
testConnection() PASSED

Gradle Test Run :core:test > Gradle Test Executor 95 > ZooKeeperClientTest > 
testZNodeChangeHandlerForCreation() STARTED

Gradle Test 

[jira] [Resolved] (KAFKA-15550) OffsetsForTimes validation for negative timestamps in new consumer

[Philip Nee (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Philip Nee resolved KAFKA-15550.

Resolution: Fixed

> OffsetsForTimes validation for negative timestamps in new consumer
> --
>
> Key: KAFKA-15550
> URL: https://issues.apache.org/jira/browse/KAFKA-15550
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, consumer
>Reporter: Lianet Magrans
>Assignee: Lianet Magrans
>Priority: Major
>  Labels: consumer-threading-refactor, kip-848, kip-848-preview
>
> OffsetsForTimes api call should fail with _IllegalArgumentException_ if 
> negative timestamps are provided as arguments. This will effectively exclude 
> earliest and latest offsets as target times, keeping the current behaviour of 
> the KafkaConsumer.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[Bill Bejeck]

Congratulations Satish!

On Fri, Oct 27, 2023 at 1:12 PM Matthias J. Sax  wrote:

> Congrats!
>
> On 10/27/23 10:01 AM, Viktor Somogyi-Vass wrote:
> > Congrats Satish!
> >
> > On Fri, Oct 27, 2023, 18:48 Ivan Yurchenko  wrote:
> >
> >> Congrats Satish!
> >>
> >> Ivan
> >>
> >> On Fri, Oct 27, 2023, at 19:02, Kamal Chandraprakash wrote:
> >>> Congratulations Satish!
> >>>
> >>> On Fri, Oct 27, 2023, 21:10 Jorge Esteban Quilcate Otoya <
> >>> quilcate.jo...@gmail.com> wrote:
> >>>
>  Congratulations Satish!!
> 
>  On Fri, 27 Oct 2023 at 18:38, Mickael Maison <
> mickael.mai...@gmail.com
> >>>
>  wrote:
> 
> > Congratulations Satish!
> >
> > On Fri, Oct 27, 2023 at 5:18 PM Lucas Brutschy
> >  wrote:
> >>
> >> Congrats!
> >>
> >> On Fri, Oct 27, 2023 at 5:06 PM Manikumar <
> >> manikumar.re...@gmail.com>
> > wrote:
> >>>
> >>> Congrats!
> >>>
> >>> On Fri, Oct 27, 2023 at 8:35 PM Jun Rao  >>>
> > wrote:
> >>>
>  Hi, Everyone,
> 
>  Satish Duggana has been a Kafka committer since 2022. He has
> >> been
> > very
>  instrumental to the community since becoming a committer. It's
> >> my
> > pleasure
>  to announce that Satish is now a member of Kafka PMC.
> 
>  Congratulations Satish!
> 
>  Jun
>  on behalf of Apache Kafka PMC
> 
> >
> 
> >>>
> >>
> >
>

[jira] [Created] (KAFKA-15705) Add integration tests for Heartbeat API and GroupLeave API

[Dongnuo Lyu (Jira)]

Dongnuo Lyu created KAFKA-15705:
---

 Summary: Add integration tests for Heartbeat API and GroupLeave API
 Key: KAFKA-15705
 URL: https://issues.apache.org/jira/browse/KAFKA-15705
 Project: Kafka
  Issue Type: Sub-task
Reporter: Dongnuo Lyu






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15704) ControllerRegistrationRequest must set ZkMigrationReady field if appropriate

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-15704:


 Summary: ControllerRegistrationRequest must set ZkMigrationReady 
field if appropriate
 Key: KAFKA-15704
 URL: https://issues.apache.org/jira/browse/KAFKA-15704
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.7.0
Reporter: Colin McCabe
 Fix For: 3.7.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[Matthias J. Sax]

Congrats!

On 10/27/23 10:01 AM, Viktor Somogyi-Vass wrote:

Congrats Satish!

On Fri, Oct 27, 2023, 18:48 Ivan Yurchenko  wrote:


Congrats Satish!

Ivan

On Fri, Oct 27, 2023, at 19:02, Kamal Chandraprakash wrote:

Congratulations Satish!

On Fri, Oct 27, 2023, 21:10 Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:


Congratulations Satish!!

On Fri, 27 Oct 2023 at 18:38, Mickael Maison 


wrote:


Congratulations Satish!

On Fri, Oct 27, 2023 at 5:18 PM Lucas Brutschy
 wrote:


Congrats!

On Fri, Oct 27, 2023 at 5:06 PM Manikumar <

manikumar.re...@gmail.com>

wrote:


Congrats!

On Fri, Oct 27, 2023 at 8:35 PM Jun Rao 


wrote:



Hi, Everyone,

Satish Duggana has been a Kafka committer since 2022. He has

been

very

instrumental to the community since becoming a committer. It's

my

pleasure

to announce that Satish is now a member of Kafka PMC.

Congratulations Satish!

Jun
on behalf of Apache Kafka PMC

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[Viktor Somogyi-Vass]

Congrats Satish!

On Fri, Oct 27, 2023, 18:48 Ivan Yurchenko  wrote:

> Congrats Satish!
>
> Ivan
>
> On Fri, Oct 27, 2023, at 19:02, Kamal Chandraprakash wrote:
> > Congratulations Satish!
> >
> > On Fri, Oct 27, 2023, 21:10 Jorge Esteban Quilcate Otoya <
> > quilcate.jo...@gmail.com> wrote:
> >
> > > Congratulations Satish!!
> > >
> > > On Fri, 27 Oct 2023 at 18:38, Mickael Maison  >
> > > wrote:
> > >
> > > > Congratulations Satish!
> > > >
> > > > On Fri, Oct 27, 2023 at 5:18 PM Lucas Brutschy
> > > >  wrote:
> > > > >
> > > > > Congrats!
> > > > >
> > > > > On Fri, Oct 27, 2023 at 5:06 PM Manikumar <
> manikumar.re...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > Congrats!
> > > > > >
> > > > > > On Fri, Oct 27, 2023 at 8:35 PM Jun Rao  >
> > > > wrote:
> > > > > >
> > > > > > > Hi, Everyone,
> > > > > > >
> > > > > > > Satish Duggana has been a Kafka committer since 2022. He has
> been
> > > > very
> > > > > > > instrumental to the community since becoming a committer. It's
> my
> > > > pleasure
> > > > > > > to announce that Satish is now a member of Kafka PMC.
> > > > > > >
> > > > > > > Congratulations Satish!
> > > > > > >
> > > > > > > Jun
> > > > > > > on behalf of Apache Kafka PMC
> > > > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[Ivan Yurchenko]

Congrats Satish!

Ivan

On Fri, Oct 27, 2023, at 19:02, Kamal Chandraprakash wrote:
> Congratulations Satish!
> 
> On Fri, Oct 27, 2023, 21:10 Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
> 
> > Congratulations Satish!!
> >
> > On Fri, 27 Oct 2023 at 18:38, Mickael Maison 
> > wrote:
> >
> > > Congratulations Satish!
> > >
> > > On Fri, Oct 27, 2023 at 5:18 PM Lucas Brutschy
> > >  wrote:
> > > >
> > > > Congrats!
> > > >
> > > > On Fri, Oct 27, 2023 at 5:06 PM Manikumar 
> > > wrote:
> > > > >
> > > > > Congrats!
> > > > >
> > > > > On Fri, Oct 27, 2023 at 8:35 PM Jun Rao 
> > > wrote:
> > > > >
> > > > > > Hi, Everyone,
> > > > > >
> > > > > > Satish Duggana has been a Kafka committer since 2022. He has been
> > > very
> > > > > > instrumental to the community since becoming a committer. It's my
> > > pleasure
> > > > > > to announce that Satish is now a member of Kafka PMC.
> > > > > >
> > > > > > Congratulations Satish!
> > > > > >
> > > > > > Jun
> > > > > > on behalf of Apache Kafka PMC
> > > > > >
> > >
> >
> 

Re: [VOTE] KIP-975: Docker Image for Apache Kafka

[Ismael Juma]

Thanks for the KIP Krishna - looking forward to this. +1 (binding).

On Thu, Oct 26, 2023 at 9:36 PM Krishna Agarwal <
krishna0608agar...@gmail.com> wrote:

> Hi,
> I'd like to call a vote on KIP-975 which aims to publish an official docker
> image for Apache Kafka.
>
> KIP -
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
>
> Discussion thread -
> https://lists.apache.org/thread/3g43hps2dmkyxgglplrlwpsf7vkywkyy
>
> Regards,
> Krishna
>

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[Kamal Chandraprakash]

Congratulations Satish!

On Fri, Oct 27, 2023, 21:10 Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:

> Congratulations Satish!!
>
> On Fri, 27 Oct 2023 at 18:38, Mickael Maison 
> wrote:
>
> > Congratulations Satish!
> >
> > On Fri, Oct 27, 2023 at 5:18 PM Lucas Brutschy
> >  wrote:
> > >
> > > Congrats!
> > >
> > > On Fri, Oct 27, 2023 at 5:06 PM Manikumar 
> > wrote:
> > > >
> > > > Congrats!
> > > >
> > > > On Fri, Oct 27, 2023 at 8:35 PM Jun Rao 
> > wrote:
> > > >
> > > > > Hi, Everyone,
> > > > >
> > > > > Satish Duggana has been a Kafka committer since 2022. He has been
> > very
> > > > > instrumental to the community since becoming a committer. It's my
> > pleasure
> > > > > to announce that Satish is now a member of Kafka PMC.
> > > > >
> > > > > Congratulations Satish!
> > > > >
> > > > > Jun
> > > > > on behalf of Apache Kafka PMC
> > > > >
> >
>

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[Jorge Esteban Quilcate Otoya]

Congratulations Satish!!

On Fri, 27 Oct 2023 at 18:38, Mickael Maison 
wrote:

> Congratulations Satish!
>
> On Fri, Oct 27, 2023 at 5:18 PM Lucas Brutschy
>  wrote:
> >
> > Congrats!
> >
> > On Fri, Oct 27, 2023 at 5:06 PM Manikumar 
> wrote:
> > >
> > > Congrats!
> > >
> > > On Fri, Oct 27, 2023 at 8:35 PM Jun Rao 
> wrote:
> > >
> > > > Hi, Everyone,
> > > >
> > > > Satish Duggana has been a Kafka committer since 2022. He has been
> very
> > > > instrumental to the community since becoming a committer. It's my
> pleasure
> > > > to announce that Satish is now a member of Kafka PMC.
> > > >
> > > > Congratulations Satish!
> > > >
> > > > Jun
> > > > on behalf of Apache Kafka PMC
> > > >
>

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[Divij Vaidya]

Congratulations Satish! And thank you for your contributions so far.

--
Divij Vaidya



On Fri, Oct 27, 2023 at 5:18 PM Lucas Brutschy
 wrote:

> Congrats!
>
> On Fri, Oct 27, 2023 at 5:06 PM Manikumar 
> wrote:
> >
> > Congrats!
> >
> > On Fri, Oct 27, 2023 at 8:35 PM Jun Rao 
> wrote:
> >
> > > Hi, Everyone,
> > >
> > > Satish Duggana has been a Kafka committer since 2022. He has been very
> > > instrumental to the community since becoming a committer. It's my
> pleasure
> > > to announce that Satish is now a member of Kafka PMC.
> > >
> > > Congratulations Satish!
> > >
> > > Jun
> > > on behalf of Apache Kafka PMC
> > >
>

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[Mickael Maison]

Congratulations Satish!

On Fri, Oct 27, 2023 at 5:18 PM Lucas Brutschy
 wrote:
>
> Congrats!
>
> On Fri, Oct 27, 2023 at 5:06 PM Manikumar  wrote:
> >
> > Congrats!
> >
> > On Fri, Oct 27, 2023 at 8:35 PM Jun Rao  wrote:
> >
> > > Hi, Everyone,
> > >
> > > Satish Duggana has been a Kafka committer since 2022. He has been very
> > > instrumental to the community since becoming a committer. It's my pleasure
> > > to announce that Satish is now a member of Kafka PMC.
> > >
> > > Congratulations Satish!
> > >
> > > Jun
> > > on behalf of Apache Kafka PMC
> > >

[jira] [Resolved] (KAFKA-15415) In Java-client, backoff should be skipped for retried producer-batch to a new leader

[Mayank Shekhar Narula (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayank Shekhar Narula resolved KAFKA-15415.
---
Resolution: Fixed

> In Java-client, backoff should be skipped for retried producer-batch to a new 
> leader
> 
>
> Key: KAFKA-15415
> URL: https://issues.apache.org/jira/browse/KAFKA-15415
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Reporter: Mayank Shekhar Narula
>Assignee: Mayank Shekhar Narula
>Priority: Major
> Fix For: 3.7.0, 3.6.1
>
>
> Right now in java-client, producer-batches backoff upto 
> retry.backoff.ms(100ms by default). This Jira proposes that backoff should be 
> skipped if client knows of a newer-leader for the partition in a sub-sequent 
> retry(typically through refresh of parition-metadata via the Metadata RPC). 
> This would help improve the latency of the produce-request around when 
> partition leadership changes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[Lucas Brutschy]

Congrats!

On Fri, Oct 27, 2023 at 5:06 PM Manikumar  wrote:
>
> Congrats!
>
> On Fri, Oct 27, 2023 at 8:35 PM Jun Rao  wrote:
>
> > Hi, Everyone,
> >
> > Satish Duggana has been a Kafka committer since 2022. He has been very
> > instrumental to the community since becoming a committer. It's my pleasure
> > to announce that Satish is now a member of Kafka PMC.
> >
> > Congratulations Satish!
> >
> > Jun
> > on behalf of Apache Kafka PMC
> >

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[Boudjelda Mohamed Said]

Congratulations!!

On Fri 27 Oct 2023 at 17:05, Jun Rao  wrote:

> Hi, Everyone,
>
> Satish Duggana has been a Kafka committer since 2022. He has been very
> instrumental to the community since becoming a committer. It's my pleasure
> to announce that Satish is now a member of Kafka PMC.
>
> Congratulations Satish!
>
> Jun
> on behalf of Apache Kafka PMC
>

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[Manikumar]

Congrats!

On Fri, Oct 27, 2023 at 8:35 PM Jun Rao  wrote:

> Hi, Everyone,
>
> Satish Duggana has been a Kafka committer since 2022. He has been very
> instrumental to the community since becoming a committer. It's my pleasure
> to announce that Satish is now a member of Kafka PMC.
>
> Congratulations Satish!
>
> Jun
> on behalf of Apache Kafka PMC
>

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[Chris Egerton]

Congrats, Satish!

On Fri, Oct 27, 2023, 11:04 Jun Rao  wrote:

> Hi, Everyone,
>
> Satish Duggana has been a Kafka committer since 2022. He has been very
> instrumental to the community since becoming a committer. It's my pleasure
> to announce that Satish is now a member of Kafka PMC.
>
> Congratulations Satish!
>
> Jun
> on behalf of Apache Kafka PMC
>

[ANNOUNCE] New Kafka PMC Member: Satish Duggana

[Jun Rao]

Hi, Everyone,

Satish Duggana has been a Kafka committer since 2022. He has been very
instrumental to the community since becoming a committer. It's my pleasure
to announce that Satish is now a member of Kafka PMC.

Congratulations Satish!

Jun
on behalf of Apache Kafka PMC

Re: [VOTE] KIP-975: Docker Image for Apache Kafka

[Stanislav Kozlovski]

Thanks for the KIP!

Great idea, well thought out and much needed. +1 (binding)

On Fri, 27 Oct 2023 at 06:36, Krishna Agarwal 
wrote:

> Hi,
> I'd like to call a vote on KIP-975 which aims to publish an official docker
> image for Apache Kafka.
>
> KIP -
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
>
> Discussion thread -
> https://lists.apache.org/thread/3g43hps2dmkyxgglplrlwpsf7vkywkyy
>
> Regards,
> Krishna
>

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

[Jorge Esteban Quilcate Otoya]

Hi Christo,

Thanks for proposing KIP, this metrics will certainly be useful to operate
Kafka Tiered Storage as it becomes production-ready.

1. Given that the scope of the KIPs has grown to cover more metrics, what
do you think about introducing latency metrics for RSM operations?
Copy and delete time metrics are quite obvious/simple on what they
represent; but fetch latency metrics would be helpful as remote fetching
clients directly. e.g. having a "time to first byte" metric could help to
know how much time is introduced by the remote tier to start serving
results to the consumer, or measuring how long it takes to return a
response to consumers.

2. Couple of requests/nits on the metrics table, could you:
- highlight the names (or have them on a separate column, as you prefer) to
make it easier to read? If you choose to have another column, maybe sort
them as "Name, Description, MBean" and adjust the width.
- group the related metrics in separate groups, e.g. Lag, Remote Delete,
Remote Log Aux State, Remote Log Size; so we can elaborate on why these set
of metrics are needed. Maybe adding some examples on usage and how
actionable they are as the ones shared in previous emails would be useful
to keep as part of the KIP.

3. On Lag metrics:
3.1 I would suggest the following renames:
- TotalRemoteRecordsLag -> RemoteCopyLagRecords
- TotalRemoteBytesLag -> RemoteCopyLagBytes
- DeleteRemoteLag -> RemoteDeleteLagRecords
3.2. I agree with Kamal that having a lag based on the number of segments
would be useful to include. Segments could give a faster proxy to
understand whether the lag is meaningful or not. e.g. if the number of
records and bytes are high, but the segment lag is only small (e.g. 1), it
may be ok; but if the number of segments is high, then it can be more
relevant to operators.
3.3. Could we consider having the same metrics for Delete Lag as there are
for Copy Lag? i.e. including RemoteDeleteLagBytes, and (potentially)
RemoteDeleteLag for segments.
3.4. The description of delete lag is unclear to me: I though it was about
the remote segments to be deleted (because of total retention) but not
deleted yet; however from description it seems that it's related to local
segments that are marked for deletion. Is this correct?

4. On Remote Delete metrics:
- Could we also include bytes-based metric as with Copy and Fetch? t would
be useful to know how many bytes are being deleted. If aggregated and
compared with copied bytes, we can get a sense of the amount of data stored
remotely, even if not exact (only considers segment size, not indexes)

5. On RemoteLogAuxState metrics: could you elaborate a bit more on the
purpose of this component and why the metrics proposed are needed?

6. On Total Remote Log Size metrics: similarly, could you elaborate on why
this metric is needed? I'm missing what makes this operation as relevant
(compared to other internals) to have some metrics attached -- maybe if you
could shared scenarios where this metrics would be useful would be helpful.

7. On the metrics naming: not sure the `Total*` prefix is really needed or
adds meaning. When I found it useful is when there are related metric that
are a subset, then the total prefix helps: e.g.
`TotalProduceRequestsPerSec` and `FailedProduceRequestsPerSec`

Cheers,
Jorge.


On Tue, 24 Oct 2023 at 12:24, Christo Lolov  wrote:

> Hello all,
>
> Now that 3.6 has been released, I would like to bring back attention to the
> following KIP for adding metrics to tiered storage targeting 3.7 -
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-963%3A+Add+more+metrics+to+Tiered+Storage
> .
>
> Let me know your thoughts about the list of metrics and their granularity!
>
> Best,
> Christo
>
> On Fri, 13 Oct 2023 at 10:14, Christo Lolov 
> wrote:
>
> > Heya Gantigmaa,
> >
> > Apologies for the (very) late reply!
> >
> > Now that 3.6 has been released and reviewers have a bit more time I will
> > be picking up this KIP again. I am more than happy to add useful new
> > metrics to the KIP, I would just ask for a couple of days to review your
> > pull request and I will come back to you.
> >
> > Best,
> > Christo
> >
> > On Mon, 25 Sept 2023 at 10:49, Gantigmaa Selenge 
> > wrote:
> >
> >> Hi Christo,
> >>
> >> Thank you for writing the KIP.
> >>
> >> I recently raised a PR to add metrics for tracking remote segment
> >> deletions
> >> (https://github.com/apache/kafka/pull/14375) but realised those metrics
> >> were not mentioned in the original KIP-405 or KIP-930. Do you think
> these
> >> would make sense to be added to this KIP and get included in the
> >> discussion?
> >>
> >> Regards,
> >> Gantigmaa
> >>
> >> On Wed, Aug 9, 2023 at 1:53 PM Christo Lolov 
> >> wrote:
> >>
> >> > Heya Kamal,
> >> >
> >> > Thank you for going through the KIP and for the question!
> >> >
> >> > I have been thinking about this and as an operator I might find it the
> >> most
> >> > useful to know all three of them actually.
> >> >
> >> > I would find 

[jira] [Created] (KAFKA-15703) Update Highwatermark while building the remote log auxillary state

[Kamal Chandraprakash (Jira)]

Kamal Chandraprakash created KAFKA-15703:


 Summary: Update Highwatermark while building the remote log 
auxillary state
 Key: KAFKA-15703
 URL: https://issues.apache.org/jira/browse/KAFKA-15703
 Project: Kafka
  Issue Type: Task
Reporter: Kamal Chandraprakash
Assignee: Kamal Chandraprakash






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-978: Allow dynamic reloading of certificates with different DN / SANs

[Mickael Maison]

Hi,

+1 (binding)
Thanks for the KIP

Mickael

On Wed, Oct 25, 2023 at 5:34 PM Federico Valeri  wrote:
>
> Hi Jakub, thanks for this KIP.
>
> +1 (non binding)
>
> Thanks
> Fede
>
> On Wed, Oct 25, 2023 at 4:45 PM Manikumar  wrote:
> >
> > Hi,
> >
> > Thanks for the KIP.
> >
> > +1 (binding)
> >
> >
> > Thanks.
> >
> > On Wed, Oct 25, 2023 at 1:37 AM Jakub Scholz  wrote:
> >
> > > Hi all,
> > >
> > > I would like to start a vote for the KIP-978: Allow dynamic reloading of
> > > certificates with different DN / SANs
> > > <
> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263429128
> > > >
> > > .
> > >
> > > Thanks & Regards
> > > Jakub
> > >

[jira] [Created] (KAFKA-15702) apiVersion request doesn't send when DNS is not ready at first

[Luke Chen (Jira)]

Luke Chen created KAFKA-15702:
-

 Summary: apiVersion request doesn't send when DNS is not ready at 
first
 Key: KAFKA-15702
 URL: https://issues.apache.org/jira/browse/KAFKA-15702
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.6.0
Reporter: Luke Chen


When ZK migrating to KRaft, we [rely on 
apiVersions|https://github.com/apache/kafka/blob/3055cd7c180cac15016169c52383ddc204ca5f16/metadata/src/main/java/org/apache/kafka/controller/QuorumFeatures.java#L140]
 to check if all controllers enabled ZkMigration flag. But if the DNS is not 
updated with the latest info (frequently happen in k8s env), then the active 
controller won't send out the apiVersion request. 

The impact is the ZK migrating to KRaft will be blocked since the active 
controller will consider the follower is not ready.

*The flow of the issue is like this:*
1. start up 3 controllers, ex: c1, c2, c3
2. The DNS doesn't update the host name entry of c3.
3. c1 becomes the leader, and send apiVersion request to c2
4. c1 is trying to connect to c3, and got unknownHost error
5. DNS is updated with c3 entry
6. c1 successfully connect to c3, but no apiVersion request sent.
7. The KRaftMigrationDriver keeps waiting for c3 ready for ZK migration


Had a look, and it looks like the channel cannot successfully finishConnect 
[here|https://github.com/apache/kafka/blob/3.6/clients/src/main/java/org/apache/kafka/common/network/Selector.java#L527],
 so the channel won't be considered as connected, and initiate a apiVersion 
request.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2334

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 321366 lines...]
Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testZNodeChangeHandlerForDataChange() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testZooKeeperSessionStateMetric() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testZooKeeperSessionStateMetric() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testExceptionInBeforeInitializingSession() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testExceptionInBeforeInitializingSession() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testGetChildrenExistingZNode() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testGetChildrenExistingZNode() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testConnection() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testConnection() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testZNodeChangeHandlerForCreation() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testZNodeChangeHandlerForCreation() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testGetAclExistingZNode() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testGetAclExistingZNode() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testSessionExpiryDuringClose() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testSessionExpiryDuringClose() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testReinitializeAfterAuthFailure() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testReinitializeAfterAuthFailure() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testSetAclNonExistentZNode() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testSetAclNonExistentZNode() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testConnectionLossRequestTermination() STARTED

Gradle Test Run :core:test > Gradle Test Executor 97 > ConsumerBounceTest > 
testConsumptionWithBrokerFailures() PASSED

5011 tests completed, 1 failed, 13 skipped

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testConnectionLossRequestTermination() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testExistsNonExistentZNode() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testExistsNonExistentZNode() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testGetDataNonExistentZNode() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testGetDataNonExistentZNode() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testConnectionTimeout() STARTED
There were failing tests. See the report at: 
file:///home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/build/reports/tests/test/index.html

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

For more on this, please refer to 
https://docs.gradle.org/8.3/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.

BUILD SUCCESSFUL in 2h 35m 34s
303 actionable tasks: 111 executed, 192 up-to-date

Publishing build scan...

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testConnectionTimeout() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testBlockOnRequestCompletionFromStateChangeHandler() STARTED
https://ge.apache.org/s/tugq25wwb4dde


See the profiling report at: 
file:///home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/build/reports/profile/profile-2023-10-27-08-56-45.html
A fine-grained performance profile is available: use the --scan option.

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testBlockOnRequestCompletionFromStateChangeHandler() PASSED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testUnresolvableConnectString() STARTED

Gradle Test Run :core:test > Gradle Test Executor 92 > ZooKeeperClientTest > 
testUnresolvableConnectString() PASSED

Gradle Test Run :core:test > Gradle Test Executor 

Re: [DISCUSS] KIP-987: Connect Static Assignments

[Sagar]

Hi Greg,

Thanks for the response.

This is detailed in the KIP two sentences earlier: "If the
> connect.protocol is set to static, each worker will send it's
> static.connectors and static.tasks to the coordinator during
> rebalances."


Yes I saw that. My point was if this is going to need a change in the
Connect Protocol format, then those changes should be outlined in the KIP.


Yes. Arbitrary here just means that the assignment is not
> influenced by the static assignment.


Okay, the usage of the word arbitrary is a bit confusing but I would leave
it to that.

Also, when I read Mickael's suggestion above about using a taint/selector
mechanism, it seems cleaner. The main reason IMO is that we are letting
connectors and tasks define where and how they want to be placed and we let
the runtime decide it(aided with a ResourceManager if needed) . With the
approach proposed in the KIP, anytime any changes are needed to the
placements, we need to modify worker configs and add/edit/remove
connectors/tasks on it. This hasn't generally been the case and in my
experience worker configs are not really modified that often. Moreover,
administrators will need to also choose where to place the connectors/tasks
based on the load on the cluster which again is an added overhead.

However you brought up a good point in the KIP

However, because tasks within a single connector can be heterogeneous, it
> is necessary to attach a different selector/affinity declaration to each
> task. But because tasks are dynamically created by the connector, either
> every task's selector must be added manually after defaulting to a wildcard
> affinity, or the Connect REST API would need a way to template multiple
> task affinities (parameterized by task-id).


as well as here above. If we consider the idea proposed by Mickael above,
and consider just the isolated config value for selector, which means that
any task being spun up for this connector would be run on a dedicated
worker. If we use this, then we might see a proliferation of workers
happening on the cluster so if we consider adding another worker level
config called max.tasks or something similar and if we couple the 2, in
effect we can achieve the same thing as the KIP propoeses (obviously I
might be over-simplifying it a bit here, but I hope the idea is clear). The
config values suggested above should cover all cases, but I still do agree
to the fact that setting the selector config value at a task level is a
challenge.

Thanks!
Sagar.


On Fri, Oct 20, 2023 at 10:43 PM Greg Harris 
wrote:

> Hey Hector,
>
> That's a cool idea for the ConnectAssignor plugin.
>
> While this proposal could be viewed as an "assignor" problem that a
> custom assignor could solve, it's really about providing additional
> context to the assignor which isn't present currently. This lack of
> context would prevent a custom assignor from solving the resource
> utilization problem adequately.
>
> Thanks!
> Greg
>
> On Fri, Oct 20, 2023 at 9:58 AM Greg Harris  wrote:
> >
> > Mickael,
> >
> > Thank you for discussing that rejected alternative, I was almost going
> > to propose it.
> >
> > > I still find the proposed mechanism limited and I'm not sure it really
> addressed the pain points I've experienced with Connect.
> >
> > I think that this KIP on its own is insufficient to solve the
> > operational difficulties of Connect, and an external management layer
> > is necessary. In this KIP i'm trying to find the minimum viable
> > abstraction to allow a management layer to make decisions about
> > placement, knowing that the abstraction may be non-ergonomic for
> > "direct users" without a management layer mediating.
> >
> > > Connectors may also change the assignment of tasks at runtime so for
> example it task-2 is really busy (because it's assigned a partition with
> high throughput), this may not be true in 10 minutes as this partition is
> now assigned to task-1
> >
> > I think this is similar to a concern (#5) that Tom raised, and a
> > limitation of the "task index" abstraction. I don't know if there is a
> > way for us to manage this sort of fine-grained dynamic utilization of
> > tasks. Once we start a task, it has some static resources assigned to
> > it (via the JVM). If you notice the resource requirements expand, it
> > will need to stop in order to move JVMs and change its resource
> > allocation, and stopping the task may cause assignments to change and
> > the workload to be distributed elsewhere.
> >
> > > I think the "hints" where to place a connector/tasks should come from
> the connector configuration as it's the engineers building a pipeline that
> knows best the requirements (in terms of isolation, resources) of their
> workload. This is basically point 3) in my initial email. The mechanism you
> propose puts this burden on the cluster administrators who may well not
> know the workloads and also have to guess/know in advance to properly
> configure workers.
> >
> > In the Strimzi 

[jira] [Resolved] (KAFKA-15659) Flaky test RestoreIntegrationTest.shouldInvokeUserDefinedGlobalStateRestoreListener

[Levani Kokhreidze (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Levani Kokhreidze resolved KAFKA-15659.
---
Resolution: Fixed

> Flaky test 
> RestoreIntegrationTest.shouldInvokeUserDefinedGlobalStateRestoreListener 
> 
>
> Key: KAFKA-15659
> URL: https://issues.apache.org/jira/browse/KAFKA-15659
> Project: Kafka
>  Issue Type: Test
>Reporter: Divij Vaidya
>Assignee: Levani Kokhreidze
>Priority: Major
>  Labels: flaky-test, streams
> Attachments: Screenshot 2023-10-20 at 13.19.20.png
>
>
> The test added in the PR [https://github.com/apache/kafka/pull/14519] 
> {{shouldInvokeUserDefinedGlobalStateRestoreListener}} has been flaky since it 
> was added. You can find the flaky build on trunk using the link 
> [https://ge.apache.org/scans/tests?search.relativeStartTime=P28D=kafka=trunk=Europe%2FBerlin=org.apache.kafka.streams.integration.RestoreIntegrationTest=shouldInvokeUserDefinedGlobalStateRestoreListener()]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker

[Krishna Agarwal]

Hi Manikumar,
Thanks for the feedback.

This image signifies 2 things:

   1. Image should be used for the local development and testing purposes
   with fast startup times. (kafka-local)
   2. To achieve (1) - we are providing a native executable for Apache
   Kafka in the docker image. (kafka-native)

While "graalvm" is the underlying tool enabling this, I'm unsure if we
should explicitly mention it in the name.
I'd love to hear your thoughts on this. Do you prefer "kafka-native"
instead of "kafka-local"?

Regards,
Krishna

On Fri, Oct 20, 2023 at 3:32 PM Manikumar  wrote:

> Hi,
>
> > For the native AK docker image, we are considering '*kafka-local*' as it
> clearly signifies that this image is intended exclusively for local
>
> I am not sure, if there is any naming pattern for graalvm based images. Can
> we include "graalvm" to the image name like "kafka-graalvm-native".
> This will clearly indicate this is graalvm based image.
>
>
> Thanks. Regards
>
>
>
>
> On Wed, Oct 18, 2023 at 9:26 PM Krishna Agarwal <
> krishna0608agar...@gmail.com> wrote:
>
> > Hi Federico,
> > Thanks for the feedback and apologies for the delay.
> >
> > I've included a section in the KIP on the release process. I would
> greatly
> > appreciate your insights after reviewing it.
> >
> > Regards,
> > Krishna
> >
> > On Fri, Sep 8, 2023 at 3:08 PM Federico Valeri 
> > wrote:
> >
> > > Hi Krishna, thanks for opening this discussion.
> > >
> > > I see you created two separate KIPs (974 and 975), but there are some
> > > common points (build system and test plan).
> > >
> > > Currently, the Docker image used for system tests is only supported in
> > > that limited scope, so the maintenance burden is minimal. Providing
> > > official Kafka images would be much more complicated. Have you
> > > considered how the image rebuild process would work in case a high
> > > severity CVE comes out for a non Kafka image dependency? In that case,
> > > there will be no Kafka release.
> > >
> > > Br
> > > Fede
> > >
> > > On Fri, Sep 8, 2023 at 9:17 AM Krishna Agarwal
> > >  wrote:
> > > >
> > > > Hi,
> > > > I want to submit a KIP to deliver an experimental Apache Kafka docker
> > > image.
> > > > The proposed docker image can launch brokers with sub-second startup
> > time
> > > > and minimal memory footprint by leveraging a GraalVM based native
> Kafka
> > > > binary.
> > > >
> > > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > > >
> > > >
> > > > Regards,
> > > > Krishna
> > >
> >
>