[jira] [Created] (KAFKA-16917) Align the returned map type of KafkaAdminClient

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16917:
--

 Summary: Align the returned map type of KafkaAdminClient
 Key: KAFKA-16917
 URL: https://issues.apache.org/jira/browse/KAFKA-16917
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


This jira includes following issues:
 # KafkaAdminClient#alterClientQuotas [0] return a immutable map, but other 
APIs return `HashMap`.
 # KafkaAdminClient#describeConfigs [1] has duplicate copy. the returned stuff 
from `Collectors.toMap` is a new HashMap already.

 

[0] 
[https://github.com/apache/kafka/blob/d13a693ea768fa8013240dc39bc21db92bffc31f/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L4074]
[1] 
[https://github.com/apache/kafka/blob/d13a693ea768fa8013240dc39bc21db92bffc31f/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L2763]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16912) Migrate ConsumerNetworkThreadTest.testPollResultTimer() to NetworkClientDelegateTest

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16912.

Fix Version/s: 3.9.0
   Resolution: Fixed

> Migrate ConsumerNetworkThreadTest.testPollResultTimer() to 
> NetworkClientDelegateTest
> 
>
> Key: KAFKA-16912
> URL: https://issues.apache.org/jira/browse/KAFKA-16912
> Project: Kafka
>  Issue Type: Test
>  Components: clients, consumer, unit tests
>Reporter: Brenden DeLuna
>Assignee: Brenden DeLuna
>Priority: Minor
>  Labels: migration, unit-test
> Fix For: 3.9.0
>
>
> ConsumerNetworkThreadTest.testPollResultTimer() only tests 
> NetworkClientDelegate, it does not test ConsumerNetworkThread. This test 
> should be moved to NetworkClientDelegateTest.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16223) Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16223.

Fix Version/s: 3.9.0
   (was: 3.8.0)
   Resolution: Fixed

> Replace EasyMock and PowerMock with Mockito for KafkaConfigBackingStoreTest
> ---
>
> Key: KAFKA-16223
> URL: https://issues.apache.org/jira/browse/KAFKA-16223
> Project: Kafka
>  Issue Type: Sub-task
>  Components: connect
>Reporter: Hector Geraldino
>Assignee: Hector Geraldino
>Priority: Minor
> Fix For: 3.9.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16909) Refactor GroupCoordinatorConfig with AbstractConfig

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16909:
--

 Summary: Refactor GroupCoordinatorConfig with AbstractConfig
 Key: KAFKA-16909
 URL: https://issues.apache.org/jira/browse/KAFKA-16909
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


This is similar to KAFKA-16884



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16908) Refactor QuorumConfig with AbstractConfig

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16908:
--

 Summary: Refactor QuorumConfig with AbstractConfig
 Key: KAFKA-16908
 URL: https://issues.apache.org/jira/browse/KAFKA-16908
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


This is similar to KAFKA-16884



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16901) add unit tests for ConsumerRecords#records(String)

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16901:
--

 Summary: add unit tests for ConsumerRecords#records(String)
 Key: KAFKA-16901
 URL: https://issues.apache.org/jira/browse/KAFKA-16901
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


ConsumerRecords#records(String) is a public API, so it is worthy to have unit 
test :) 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16898) move TimeIndexTest and TransactionIndexTest to storage module

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16898:
--

 Summary: move TimeIndexTest and TransactionIndexTest to storage 
module
 Key: KAFKA-16898
 URL: https://issues.apache.org/jira/browse/KAFKA-16898
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16897) Move OffsetIndexTest to storage module

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16897:
--

 Summary: Move OffsetIndexTest to storage module
 Key: KAFKA-16897
 URL: https://issues.apache.org/jira/browse/KAFKA-16897
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


as title



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Request permissions to Create KIP

[Chia-Ping Tsai]

> Sorry, I meant "Your accounts are now all set".

I almost filed a PR to fix the typo :)

Josep Prat  於 2024年6月5日 週三 下午9:38寫道：

> Sorry, I meant "Your accounts are now all set".
>
> On Wed, Jun 5, 2024 at 2:24 PM Josep Prat  wrote:
>
> > Hi there,
> > Thanks for your interest in Apache Kafka. Your accounts are not all set.
> >
> > Best,
> >
> > On Wed, Jun 5, 2024 at 1:52 PM Chia-Chuan Yu 
> wrote:
> >
> >> Hi team, I just got Confluence account, too. Could you grant me the
> >> permission to create KIP as well? Thank you. JIRA: chiacyu Confluence:
> >> chiacyu
> >> Best Regards, Chia-Chuan Yu
> >>
> >
> >
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io    |
> > 
> >    <
> https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |    >
>      <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>

[jira] [Resolved] (KAFKA-15305) The background thread should try to process the remaining task until the shutdown timer is expired

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-15305.

Resolution: Fixed

push to trunk and 3.8

> The background thread should try to process the remaining task until the 
> shutdown timer is expired
> --
>
> Key: KAFKA-15305
> URL: https://issues.apache.org/jira/browse/KAFKA-15305
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Reporter: Philip Nee
>    Assignee: Chia-Ping Tsai
>Priority: Major
>  Labels: consumer-threading-refactor, timeout
> Fix For: 3.8.0
>
>
> While working on https://issues.apache.org/jira/browse/KAFKA-15304
> close() API supplies a timeout parameter so that the consumer can have a 
> grace period to process things before shutting down.  The background thread 
> currently doesn't do that, when close() is initiated, it will immediately 
> close all of its dependencies.
>  
> This might not be desirable because there could be remaining tasks to be 
> processed before closing.  Maybe the correct things to do is to first stop 
> accepting API request, second, let the runOnce() continue to run before the 
> shutdown timer expires, then we can force closing all of its dependencies.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16888) Fix failed StorageToolTest.testFormatSucceedsIfAllDirectoriesAreAvailable and StorageToolTest.testFormatEmptyDirectory

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16888.

Fix Version/s: 3.9.0
   Resolution: Fixed

> Fix failed StorageToolTest.testFormatSucceedsIfAllDirectoriesAreAvailable and 
> StorageToolTest.testFormatEmptyDirectory
> --
>
> Key: KAFKA-16888
> URL: https://issues.apache.org/jira/browse/KAFKA-16888
> Project: Kafka
>  Issue Type: Bug
>    Reporter: Chia-Ping Tsai
>Assignee: Kuan Po Tseng
>Priority: Minor
> Fix For: 3.9.0
>
>
> The commit 
> (https://github.com/apache/kafka/commit/459da4795a511f6933e940fcf105a824bd9e589c#diff-4bacfdbf0e63a4d5f3deb1a0d39037a18510ac24ee5ec276fe70bc818ba4d209L505)
>  added new string to `stream`, so the test case will fail due to extra output.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16888) Fix failed StorageToolTest.testFormatSucceedsIfAllDirectoriesAreAvailable

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16888:
--

 Summary: Fix failed 
StorageToolTest.testFormatSucceedsIfAllDirectoriesAreAvailable
 Key: KAFKA-16888
 URL: https://issues.apache.org/jira/browse/KAFKA-16888
 Project: Kafka
  Issue Type: Bug
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


The commit 
(https://github.com/apache/kafka/commit/459da4795a511f6933e940fcf105a824bd9e589c#diff-4bacfdbf0e63a4d5f3deb1a0d39037a18510ac24ee5ec276fe70bc818ba4d209L505)
 added new string to `stream`, so the test case will fail due to extra output.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16885) Consider renaming RemoteLogManagerConfig#enableRemoteStorageSystem to RemoteLogManagerConfig#isRemoteStorageSystemEnabled

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16885:
--

 Summary: Consider renaming 
RemoteLogManagerConfig#enableRemoteStorageSystem to 
RemoteLogManagerConfig#isRemoteStorageSystemEnabled
 Key: KAFKA-16885
 URL: https://issues.apache.org/jira/browse/KAFKA-16885
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


see the discussion: 
https://github.com/apache/kafka/pull/16153#issuecomment-2144269279



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16877) Migrate RemoteLogSegmentLifecycleTest to use new test infra

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16877.

Resolution: Duplicate

KAFKA-16882 has a PR, so I close this as duplicate

> Migrate RemoteLogSegmentLifecycleTest to use new test infra
> ---
>
> Key: KAFKA-16877
> URL: https://issues.apache.org/jira/browse/KAFKA-16877
> Project: Kafka
>  Issue Type: Test
>    Reporter: Chia-Ping Tsai
>Assignee: Kuan Po Tseng
>Priority: Minor
>
> https://github.com/apache/kafka/blob/2c82ecd67f2f6b412f625e8efc1457e7fb7f74dd/storage/src/test/java/org/apache/kafka/server/log/remote/metadata/storage/RemoteLogSegmentLifecycleTest.java#L432



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16859) Cleanup check if tiered storage is enabled

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16859.

Fix Version/s: 3.9.0
   Resolution: Fixed

> Cleanup check if tiered storage is enabled
> --
>
> Key: KAFKA-16859
> URL: https://issues.apache.org/jira/browse/KAFKA-16859
> Project: Kafka
>  Issue Type: Task
>Reporter: Mickael Maison
>Assignee: 黃竣陽
>Priority: Minor
> Fix For: 3.9.0
>
>
> We have 2 ways to detect whether tiered storage is enabled:
> - KafkaConfig.isRemoteLogStorageSystemEnabled
> - KafkaConfig.remoteLogManagerConfig().enableRemoteStorageSystem()
> We use both in various files. We should stick with one way to do it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16878) Remove powermock from code base

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16878:
--

 Summary: Remove powermock from code base
 Key: KAFKA-16878
 URL: https://issues.apache.org/jira/browse/KAFKA-16878
 Project: Kafka
  Issue Type: Sub-task
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


This is the follow-up of KAFKA-16223. It should include following changes:

1. rename KafkaConfigBackingStoreMockitoTest back to KafkaConfigBackingStoreTest
2. remove all powermock-related code from code base



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-12572) Add import ordering checkstyle rule and configure an automatic formatter

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-12572.

Fix Version/s: 3.9.0
   Resolution: Fixed

fixed by 
https://github.com/apache/kafka/commit/342e69192f62b89b7d1ea824aeecc09c38899d72

> Add import ordering checkstyle rule and configure an automatic formatter
> 
>
> Key: KAFKA-12572
> URL: https://issues.apache.org/jira/browse/KAFKA-12572
> Project: Kafka
>  Issue Type: Sub-task
>  Components: build
>Reporter: Dongjin Lee
>Assignee: Dongjin Lee
>Priority: Minor
> Fix For: 3.9.0
>
>
> # Add import ordering checkstyle rules.
> # Configure an automatic formatter which satisfies 1.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-752: Support --bootstrap-server in ReplicaVerificationTool

[Chia-Ping Tsai]

`replica_verification_test.py` is unstable in my jenkins, and then I notice 
this thread.

Maybe kafka 4 is a good timing to remove this tool, but does it need a KIP? If 
so, I'd like to file a KIP for it.

Best,
Chia-Ping

On 2021/06/10 05:01:43 Ismael Juma wrote:
> KAFKA-12600 was a general change, not related to this tool specifically. I
> am not convinced this tool is actually useful, I haven't seen anyone using
> it in years.
> 
> Ismael
> 
> On Wed, Jun 9, 2021 at 9:51 PM Dongjin Lee  wrote:
> 
> > Hi Ismael,
> >
> > Before I submit this KIP, I reviewed some history. When KIP-499
> > <
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-499+-+Unify+connection+name+flag+for+command+line+tool
> > >
> > tried to resolve the inconsistencies between the command line tools, two
> > tools were omitted, probably by mistake.
> >
> > - KAFKA-12878: Support --bootstrap-server kafka-streams-application-reset
> > 
> > - KAFKA-12899: Support --bootstrap-server in ReplicaVerificationTool
> >  (this one)
> >
> > And it seems like this tool is still working. The last update was
> > KAFKA-12600  by you,
> > which will also be included in this 3.0.0 release. It is why I determined
> > that this tool is worth updating.
> >
> > Thanks,
> > Dongjin
> >
> > On Thu, Jun 10, 2021 at 1:26 PM Ismael Juma  wrote:
> >
> > > Hi Dongjin,
> > >
> > > Does this tool still work? I recall that there were some doubts about it
> > > and that's why it wasn't updated previously.
> > >
> > > Ismael
> > >
> > > On Sat, Jun 5, 2021 at 2:38 PM Dongjin Lee  wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to call for a vote on KIP-752: Support --bootstrap-server in
> > > > ReplicaVerificationTool:
> > > >
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-752%3A+Support+--bootstrap-server+in+ReplicaVerificationTool
> > > >
> > > > Best,
> > > > Dongjin
> > > >
> > > > --
> > > > *Dongjin Lee*
> > > >
> > > > *A hitchhiker in the mathematical world.*
> > > >
> > > >
> > > >
> > > > *github:  github.com/dongjinleekr
> > > > keybase:
> > > https://keybase.io/dongjinleekr
> > > > linkedin:
> > > kr.linkedin.com/in/dongjinleekr
> > > > speakerdeck:
> > > > speakerdeck.com/dongjin
> > > > *
> > > >
> > >
> >
> >
> > --
> > *Dongjin Lee*
> >
> > *A hitchhiker in the mathematical world.*
> >
> >
> >
> > *github:  github.com/dongjinleekr
> > keybase: https://keybase.io/dongjinleekr
> > linkedin: kr.linkedin.com/in/dongjinleekr
> > speakerdeck:
> > speakerdeck.com/dongjin
> > *
> >
> 

[jira] [Resolved] (KAFKA-16785) Migrate TopicBasedRemoteLogMetadataManagerRestartTest to new test infra

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16785.

Fix Version/s: 3.9.0
   Resolution: Fixed

> Migrate TopicBasedRemoteLogMetadataManagerRestartTest to new test infra
> ---
>
> Key: KAFKA-16785
> URL: https://issues.apache.org/jira/browse/KAFKA-16785
> Project: Kafka
>  Issue Type: Test
>    Reporter: Chia-Ping Tsai
>Assignee: Kuan Po Tseng
>Priority: Minor
>  Labels: storage_test
> Fix For: 3.9.0
>
>
> as title



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16877) Migrate RemoteLogSegmentLifecycleTest to use new test infra

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16877:
--

 Summary: Migrate RemoteLogSegmentLifecycleTest to use new test 
infra
 Key: KAFKA-16877
 URL: https://issues.apache.org/jira/browse/KAFKA-16877
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


https://github.com/apache/kafka/blob/2c82ecd67f2f6b412f625e8efc1457e7fb7f74dd/storage/src/test/java/org/apache/kafka/server/log/remote/metadata/storage/RemoteLogSegmentLifecycleTest.java#L432



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16807) DescribeLogDirsResponseData#results#topics have unexpected topics having empty partitions

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16807.

Resolution: Fixed

> DescribeLogDirsResponseData#results#topics have unexpected topics having 
> empty partitions
> -
>
> Key: KAFKA-16807
> URL: https://issues.apache.org/jira/browse/KAFKA-16807
> Project: Kafka
>  Issue Type: Bug
>    Reporter: Chia-Ping Tsai
>Assignee: 黃竣陽
>Priority: Blocker
> Fix For: 3.8.0, 3.7.1, 3.9.0
>
>
> ReplicaManager [0] could generate a response having unexpected topics which 
> have empty partitions. The root cause is it always generate the topic 
> collection even though they have no matched partitions.
> That is not a issue to Kafka clients, since we loop the "partitions" to fill 
> all future responses [1]. Hence, those unexpected topics won't be existent in 
> the final results.
> However, that could be a issue to the users who implement Kafka client based 
> on Kafka protocol [2]
> [0] 
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaManager.scala#L1252
> [1] 
> https://github.com/apache/kafka/blob/b5a013e4564ad93026b9c61431e4573a39bec766/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L3145
> [2] https://lists.apache.org/thread/lp7ktmm17pbg7nqk7p4s904lcn3mrvhy



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16574) The metrics of LogCleaner disappear after reconfiguration

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16574.

Fix Version/s: 3.9.0
   Resolution: Fixed

> The metrics of LogCleaner disappear after reconfiguration
> -
>
> Key: KAFKA-16574
> URL: https://issues.apache.org/jira/browse/KAFKA-16574
> Project: Kafka
>  Issue Type: Bug
>    Reporter: Chia-Ping Tsai
>Assignee: Chia Chuan Yu
>Priority: Minor
> Fix For: 3.9.0
>
>
> see 
> [https://github.com/apache/kafka/blob/a3dcbd4e28a35f79f75ec1bf316ef0b39c0df164/core/src/main/scala/kafka/log/LogCleaner.scala#L227]
> We don't rebuild the metrics after calling shutdown. The following test can 
> prove that.
> {code:java}
> @Test
> def testMetricsAfterReconfiguration(): Unit = {
>   val logCleaner = new LogCleaner(new CleanerConfig(true),
> logDirs = Array(TestUtils.tempDir()),
> logs = new Pool[TopicPartition, UnifiedLog](),
> logDirFailureChannel = new LogDirFailureChannel(1),
> time = time)
>   def check(): Unit =
> LogCleaner.MetricNames.foreach(name => 
> assertNotNull(KafkaYammerMetrics.defaultRegistry.allMetrics().get(logCleaner.metricsGroup
>   .metricName(name, java.util.Collections.emptyMap())), s"$name is 
> gone?"))
>   try {
> check()
> logCleaner.reconfigure(new KafkaConfig(TestUtils.createBrokerConfig(1, 
> "localhost:2181")),
>   new KafkaConfig(TestUtils.createBrokerConfig(1, "localhost:2181")))
> check()
>   } finally logCleaner.shutdown()
> } {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16652) add unit test for ClusterTemplate offering zero ClusterConfig

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16652.

Fix Version/s: 3.9.0
   Resolution: Fixed

> add unit test for ClusterTemplate offering zero ClusterConfig
> -
>
> Key: KAFKA-16652
> URL: https://issues.apache.org/jira/browse/KAFKA-16652
> Project: Kafka
>  Issue Type: Improvement
>    Reporter: Chia-Ping Tsai
>Assignee: TaiJuWu
>Priority: Minor
> Fix For: 3.9.0
>
>
> https://github.com/apache/kafka/blob/31355ef8f948f369e240ebc203f889f187116d75/core/src/test/java/kafka/test/junit/ClusterTestExtensions.java#L94
> If `ClusterTemplate`does not generate any `ClusterConfig`, we will throw 
> exception. However, we don't have UT for such scenario currently.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16639) AsyncKafkaConsumer#close does not send heartbeat to leave group

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16639.

Resolution: Fixed

> AsyncKafkaConsumer#close does not send heartbeat to leave group
> ---
>
> Key: KAFKA-16639
> URL: https://issues.apache.org/jira/browse/KAFKA-16639
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>    Reporter: Chia-Ping Tsai
>Assignee: TengYao Chi
>Priority: Critical
>  Labels: kip-848-client-support
> Fix For: 3.8.0
>
>
> This bug can be reproduced by immediately closing a consumer which is just 
> created.
> The root cause is that we skip the new heartbeat used to leave group when 
> there is a in-flight heartbeat request 
> ([https://github.com/apache/kafka/blob/5de5d967adffd864bad3ec729760a430253abf38/clients/src/main/java/org/apache/kafka/clients/consumer/internals/HeartbeatRequestManager.java#L212).]
> It seems to me the simple solution is that we create a heartbeat request when 
> meeting above situation and then send it by pollOnClose 
> ([https://github.com/apache/kafka/blob/5de5d967adffd864bad3ec729760a430253abf38/clients/src/main/java/org/apache/kafka/clients/consumer/internals/RequestManager.java#L62).]
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16629) add broker-related tests to ConfigCommandIntegrationTest

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16629.

Fix Version/s: 3.9.0
   Resolution: Fixed

> add broker-related tests to ConfigCommandIntegrationTest
> 
>
> Key: KAFKA-16629
> URL: https://issues.apache.org/jira/browse/KAFKA-16629
> Project: Kafka
>  Issue Type: Improvement
>    Reporter: Chia-Ping Tsai
>Assignee: 黃竣陽
>Priority: Major
> Fix For: 3.9.0
>
>
> [https://github.com/apache/kafka/pull/15645] will rewrite the 
> ConfigCommandIntegrationTest by java and new test infra. However, it still 
> lacks of enough tests for broker-related configs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16833) Cluster missing topicIds from equals and hashCode, PartitionInfo missing equals and hashCode

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16833.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Cluster missing topicIds from equals and hashCode, PartitionInfo missing 
> equals and hashCode
> 
>
> Key: KAFKA-16833
> URL: https://issues.apache.org/jira/browse/KAFKA-16833
> Project: Kafka
>  Issue Type: Bug
>Reporter: Alyssa Huang
>Priority: Major
> Fix For: 3.8.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16598) Mirgrate `ResetConsumerGroupOffsetTest` to new test infra

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16598.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Mirgrate `ResetConsumerGroupOffsetTest` to new test infra
> -
>
> Key: KAFKA-16598
> URL: https://issues.apache.org/jira/browse/KAFKA-16598
> Project: Kafka
>  Issue Type: Improvement
>    Reporter: Chia-Ping Tsai
>Assignee: 黃竣陽
>Priority: Major
> Fix For: 3.8.0
>
>
> as title.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16771) First log directory printed twice when formatting storage

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16771.

Fix Version/s: 3.8.0
   Resolution: Fixed

> First log directory printed twice when formatting storage
> -
>
> Key: KAFKA-16771
> URL: https://issues.apache.org/jira/browse/KAFKA-16771
> Project: Kafka
>  Issue Type: Task
>  Components: tools
>Affects Versions: 3.7.0
>Reporter: Mickael Maison
>Assignee: xuanzhang gong
>Priority: Major
> Fix For: 3.8.0
>
>
> If multiple log directories are set, when running bin/kafka-storage.sh 
> format, the first directory is printed twice. For example:
> {noformat}
> bin/kafka-storage.sh format -t $KAFKA_CLUSTER_ID -c 
> config/kraft/server.properties --release-version 3.6
> metaPropertiesEnsemble=MetaPropertiesEnsemble(metadataLogDir=Optional.empty, 
> dirs={/tmp/kraft-combined-logs: EMPTY, /tmp/kraft-combined-logs2: EMPTY})
> Formatting /tmp/kraft-combined-logs with metadata.version 3.6-IV2.
> Formatting /tmp/kraft-combined-logs with metadata.version 3.6-IV2.
> Formatting /tmp/kraft-combined-logs2 with metadata.version 3.6-IV2.
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16861) Don't convert to group to classic if the size is larger than group max size

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16861:
--

 Summary: Don't convert to group to classic if the size is larger 
than group max size
 Key: KAFKA-16861
 URL: https://issues.apache.org/jira/browse/KAFKA-16861
 Project: Kafka
  Issue Type: Bug
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


It should be one-line fix [0]

[0] 
https://github.com/apache/kafka/blob/2d9994e0de915037525f041ff9a9b9325f838938/group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java#L810



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16705) the flag "started" of RaftClusterInstance is false even though the cluster is started

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16705.

Fix Version/s: 3.8.0
   Resolution: Fixed

> the flag "started" of RaftClusterInstance is false even though the cluster is 
> started
> -
>
> Key: KAFKA-16705
> URL: https://issues.apache.org/jira/browse/KAFKA-16705
> Project: Kafka
>  Issue Type: Bug
>Reporter: Chia-Ping Tsai
>Assignee: xuanzhang gong
>Priority: Minor
> Fix For: 3.8.0
>
>
> we should set `started` to true after 
> https://github.com/apache/kafka/blob/643db430a707479c9e87eec1ad67e1d4f43c9268/core/src/test/java/kafka/test/junit/RaftClusterInvocationContext.java#L113



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16847) Revise the README for recent CI changes

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16847.

Resolution: Invalid

> Revise the README for recent CI changes 
> 
>
> Key: KAFKA-16847
> URL: https://issues.apache.org/jira/browse/KAFKA-16847
> Project: Kafka
>  Issue Type: Improvement
>    Reporter: Chia-Ping Tsai
>Assignee: Cheng-Kai, Zhang
>Priority: Minor
>
> The recent changes [0] removes the test of 11 and 17, and that is good to our 
> CI resources. However, in the root readme we still declaim "We build and test 
> Apache Kafka with Java 8, 11, 17 and 21" 
> [0] 
> https://github.com/apache/kafka/commit/adab48df6830259d33bd9705b91885c4f384f267
> [1] https://github.com/apache/kafka/blob/trunk/README.md?plain=1#L7



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16796) Introduce new org.apache.kafka.tools.api.Decoder to replace kafka.serializer.Decoder

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16796.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Introduce new org.apache.kafka.tools.api.Decoder to replace 
> kafka.serializer.Decoder
> 
>
> Key: KAFKA-16796
> URL: https://issues.apache.org/jira/browse/KAFKA-16796
> Project: Kafka
>  Issue Type: Sub-task
>    Reporter: Chia-Ping Tsai
>Assignee: PoAn Yang
>Priority: Minor
>  Labels: need-kip
> Fix For: 3.8.0
>
>
> We need a replacement in order to complete 
> https://issues.apache.org/jira/browse/KAFKA-14579 in kafak 4.0



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16805) Stop using a ClosureBackedAction to configure Spotbugs reports

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16805.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Stop using a ClosureBackedAction to configure Spotbugs reports
> --
>
> Key: KAFKA-16805
> URL: https://issues.apache.org/jira/browse/KAFKA-16805
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Greg Harris
>Assignee: 黃竣陽
>Priority: Major
>  Labels: newbie
> Fix For: 3.8.0
>
>
> The org.gradle.util.ClosureBackedAction type has been deprecated.    
> This is scheduled to be removed in Gradle 9.0.    
> [Documentation|https://docs.gradle.org/8.7/userguide/upgrading_version_7.html#org_gradle_util_reports_deprecations]
>     
> 1 usage    
> [https://github.com/apache/kafka/blob/95adb7bfbfc69b3e9f3538cc5d6f7c6a577d30ee/build.gradle#L745-L749]
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Action requested: Changes to CI for JDK 11 & 17 builds on Pull Requests

[Chia-Ping Tsai]

Please take a look at following QA. ALL PASS!!!

https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-15889/25/pipeline

I almost cried, and BIG thanks to Harris!!!

On 2024/05/28 03:20:01 Greg Harris wrote:
> Hello Apache Kafka Developers,
> 
> In order to better utilize scarce CI resources shared with other Apache
> projects, the Kafka project will no longer be running full test suites for
> the JDK 11 & 17 components of PR builds.
> 
> *Action requested: If you have an active pull request, please merge or
> rebase the latest trunk into your branch* before continuing development as
> normal. You may wait to push the resulting branch until you make another
> commit, or push the result immediately.
> 
> What to expect with this change:
> * Trunk (and release branch) builds will not be affected.
> * JDK 8 and 21 builds will not be affected.
> * Compilation will not be affected.
> * Static analysis (spotbugs, checkstyle, etc) will not be affected.
> * Overall build execution time should be similar or slightly better than
> before.
> * You can expect fewer tests to be run on your PRs (~6 instead of
> ~12).
> * Test flakiness should be similar or slightly better than before.
> 
> And as a reminder, build failures (red indicators in CloudBees) are always
> blockers for merging. Starting now, the 11 and 17 builds should always pass
> (green indicators in CloudBees) before merging, as failed tests (yellow
> indicators in CloudBees) should no longer be present.
> 
> Thanks everyone,
> Greg Harris
> 

Re: Action requested: Changes to CI for JDK 11 & 17 builds on Pull Requests

[Chia-Ping Tsai]

Dear all, 

I do love Harris's patch as no one love slow CI I believe. For another, I file 
https://issues.apache.org/jira/browse/KAFKA-16847 just now to revise our readme 
about JDK. I'd like to raise more discussion here.

> Note that compilation with Java 11/17 doesn't add any value over compiling
> with Java 21 with the appropriate --release config (which we set). So, this
> part of the build process is wasteful.

I did not see build failure that happens in 11 and 17 but not in 8 or 21, and 
also it can save more CI resources and make our CI be thinner. Hence, I'm +1 to 
drop 11 and 17 totally.

Best,
Chia-Ping


On 2024/05/28 04:40:48 Ismael Juma wrote:
> Hi Greg,
> 
> Thanks for making this change.
> 
> Note that compilation with Java 11/17 doesn't add any value over compiling
> with Java 21 with the appropriate --release config (which we set). So, this
> part of the build process is wasteful. Running the tests does add some
> value (and hence why we originally had it), but the return on investment is
> not good enough given our CI issues (and hence why the change is good).
> 
> Ismael
> 
> On Mon, May 27, 2024, 8:20 PM Greg Harris 
> wrote:
> 
> > Hello Apache Kafka Developers,
> >
> > In order to better utilize scarce CI resources shared with other Apache
> > projects, the Kafka project will no longer be running full test suites for
> > the JDK 11 & 17 components of PR builds.
> >
> > *Action requested: If you have an active pull request, please merge or
> > rebase the latest trunk into your branch* before continuing development as
> > normal. You may wait to push the resulting branch until you make another
> > commit, or push the result immediately.
> >
> > What to expect with this change:
> > * Trunk (and release branch) builds will not be affected.
> > * JDK 8 and 21 builds will not be affected.
> > * Compilation will not be affected.
> > * Static analysis (spotbugs, checkstyle, etc) will not be affected.
> > * Overall build execution time should be similar or slightly better than
> > before.
> > * You can expect fewer tests to be run on your PRs (~6 instead of
> > ~12).
> > * Test flakiness should be similar or slightly better than before.
> >
> > And as a reminder, build failures (red indicators in CloudBees) are always
> > blockers for merging. Starting now, the 11 and 17 builds should always pass
> > (green indicators in CloudBees) before merging, as failed tests (yellow
> > indicators in CloudBees) should no longer be present.
> >
> > Thanks everyone,
> > Greg Harris
> >
> 

[jira] [Created] (KAFKA-16847) Revise the README for recent CI changes

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16847:
--

 Summary: Revise the README for recent CI changes 
 Key: KAFKA-16847
 URL: https://issues.apache.org/jira/browse/KAFKA-16847
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


The recent changes [0] removes the test of 11 and 17, and that is good to our 
CI resources. However, in the root readme we still declaim "We build and test 
Apache Kafka with Java 8, 11, 17 and 21" 


[0] 
https://github.com/apache/kafka/commit/adab48df6830259d33bd9705b91885c4f384f267
[1] https://github.com/apache/kafka/blob/trunk/README.md?plain=1#L7



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16845) Migrate ReplicationQuotasTestRig to new test infra

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16845:
--

 Summary: Migrate ReplicationQuotasTestRig to new test infra
 Key: KAFKA-16845
 URL: https://issues.apache.org/jira/browse/KAFKA-16845
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


as title



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16843) Remove preAppendErrors from createPutCacheCallback

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16843:
--

 Summary: Remove preAppendErrors from createPutCacheCallback
 Key: KAFKA-16843
 URL: https://issues.apache.org/jira/browse/KAFKA-16843
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


origin discussion: 
[https://github.com/apache/kafka/pull/16072#pullrequestreview-2077368462]

The method `createPutCacheCallback` has a input argument `preAppendErrors` [0]. 
It is used to keep the "error" happens before appending. However, the 
pre-append error is handled before by calling `responseCallback` [1]. Hence, we 
can remove `preAppendErrors`.
 
[0] 
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L387
[1] 
https://github.com/apache/kafka/blob/4f55786a8a86fe228a0b10a2f28529f5128e5d6f/core/src/main/scala/kafka/coordinator/group/GroupCoordinator.scala#L927C15-L927C84



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-10234) The key/value deserializer used by ConsoleConsumer is not closed

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-10234.

Resolution: Invalid

> The key/value deserializer used by ConsoleConsumer is not closed
> 
>
> Key: KAFKA-10234
> URL: https://issues.apache.org/jira/browse/KAFKA-10234
> Project: Kafka
>  Issue Type: Bug
>    Reporter: Chia-Ping Tsai
>    Assignee: Chia-Ping Tsai
>Priority: Minor
>
> We instantiate, configure and use them but them are never closed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16828) RackAwareTaskAssignorTest failed

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16828.

Fix Version/s: 3.8.0
   Resolution: Fixed

> RackAwareTaskAssignorTest failed
> 
>
> Key: KAFKA-16828
> URL: https://issues.apache.org/jira/browse/KAFKA-16828
> Project: Kafka
>  Issue Type: Test
>Reporter: Luke Chen
>Assignee: Kuan Po Tseng
>Priority: Major
> Fix For: 3.8.0
>
>
> Found in the latest trunk build.
> It fails many tests in `RackAwareTaskAssignorTest` suite.
>  
> https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-15951/7/#showFailuresLink



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16830) Remove the scala version formatters support

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16830:
--

 Summary: Remove the scala version formatters support
 Key: KAFKA-16830
 URL: https://issues.apache.org/jira/browse/KAFKA-16830
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Kuan Po Tseng
 Fix For: 4.0.0


[https://github.com/apache/kafka/blob/trunk/tools/src/main/java/org/apache/kafka/tools/consumer/ConsoleConsumerOptions.java#L72]

 

Those deprecated formatters "strings" should be removed from 4.0.0



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16795) Fix broken compatibility in kafka.tools.NoOpMessageFormatter, kafka.tools.DefaultMessageFormatter, and kafka.tools.LoggingMessageFormatter

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16795.

Resolution: Fixed

> Fix broken compatibility in kafka.tools.NoOpMessageFormatter, 
> kafka.tools.DefaultMessageFormatter, and kafka.tools.LoggingMessageFormatter
> --
>
> Key: KAFKA-16795
> URL: https://issues.apache.org/jira/browse/KAFKA-16795
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Kuan Po Tseng
>Assignee: Kuan Po Tseng
>Priority: Major
> Fix For: 3.8.0
>
>
> [{{0bf830f}}|https://github.com/apache/kafka/commit/0bf830fc9c3915bc99b6e487e6083dabd593c5d3]
>  moved NoOpMessageFormatter, DefaultMessageFormatter and 
> LoggingMessageFormatter package from {{kafka.tools}} to 
> {{{}org.apache.kafka.tools.consumer{}}}{{{}{}}}
> These classes could be used via cmd kafka-console-consumer.sh. We should have 
> a dependency cycle before 3.8.0 comes out.
>  
> {code:java}
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 \
> --topic streams-wordcount-output \
> --from-beginning \
> --formatter kafka.tools.DefaultMessageFormatter \
> --property print.key=true \
> --property print.value=true \
> --property 
> key.deserializer=org.apache.kafka.common.serialization.StringDeserializer \
> --property 
> value.deserializer=org.apache.kafka.common.serialization.LongDeserializer{code}
> The goal in this Jira is to allow user to keep using 
> {{{}kafka.tools.NoOpMessageFormatter{}}}, 
> {{{}kafka.tools.DefaultMessageFormatter{}}}, and 
> {{{}kafka.tools.LoggingMessageFormatter{}}}, but we also display warning 
> messages to say those "strings" will be removed in 4.0.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP 1047 - Introduce new org.apache.kafka.tools.api.Decoder to replace kafka.serializer.Decoder

[Chia-Ping Tsai]

+1

Thanks for Yang to take over this!

> Frank Yang  於 2024年5月24日 凌晨12:27 寫道：
> 
> Hi all,
> 
> I would like to start a vote on KIP-1047: Introduce new
> org.apache.kafka.tools.api.Decoder to replace kafka.serializer.Decoder.
> 
> KIP: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1047+Introduce+new+org.apache.kafka.tools.api.Decoder+to+replace+kafka.serializer.Decoder
> 
> Discussion thread: 
> https://lists.apache.org/thread/n3k6vb4vddl1s5nopcyglnddtvzp4j63
> 
> Thanks and regards,
> PoAn

[jira] [Resolved] (KAFKA-16829) Consider removing delegation.token.master.key

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16829.

Resolution: Duplicate

see KAFKA-12601

> Consider removing delegation.token.master.key
> -
>
> Key: KAFKA-16829
> URL: https://issues.apache.org/jira/browse/KAFKA-16829
> Project: Kafka
>  Issue Type: Improvement
>    Reporter: Chia-Ping Tsai
>    Assignee: Chia-Ping Tsai
>Priority: Minor
>
> It was marked as deprecated since 2020 [0], and maybe we should remove it now.
> [0] 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-681:+Rename+master+key+in+delegation+token+feature



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16829) Consider removing delegation.token.master.key

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16829:
--

 Summary: Consider removing delegation.token.master.key
 Key: KAFKA-16829
 URL: https://issues.apache.org/jira/browse/KAFKA-16829
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


It was marked as deprecated since 2020 [0], and maybe we should remove it now.


[0] 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-681:+Rename+master+key+in+delegation+token+feature



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP 1047 - Introduce new org.apache.kafka.tools.api.Decoder to replace kafka.serializer.Decoder

[Chia-Ping Tsai]

Thanks for Josep's response

> We can add this to 3.8.0, but keep in mind the KIP is not voted yet (as
far
as I can see), so I would highly encourage to start the vote thread ASAP
and strat with the implementation right after.

sure. We will file a draft PR at the same time!

Josep Prat  於 2024年5月23日 週四 上午12:31寫道：

> Hi all,
>
> We can add this to 3.8.0, but keep in mind the KIP is not voted yet (as far
> as I can see), so I would highly encourage to start the vote thread ASAP
> and strat with the implementation right after.
>
> Best,
>
> -
> Josep Prat
> Open Source Engineering Director, aivenjosep.p...@aiven.io   |
> +491715557497 | aiven.io
> Aiven Deutschland GmbH
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>
> On Wed, May 22, 2024, 17:06 Chia-Ping Tsai  wrote:
>
> > > One issue I also noted is that some of the existing Decoder
> > implementations (StringDecoder for example) can accept configurations
> > but currently DumpLogSegments does not provide a way to pass any
> > configurations, it creates an empty VerifiableProperties object each
> > time it instantiates a Decoder instance. If we were to use
> > Deserializer we would also need a way to provide configurations.
> >
> > BTW, if the known bug gets fixed, we have to make new interface extend
> > `configurable`.
> >
> > Or we can just ignore the known issue as `DumpLogSegments` has no options
> > to take custom configs for `Decoder`. That allow the `Decoder` more
> simple
> >
> >
> > Chia-Ping Tsai  於 2024年5月22日 週三 下午10:58寫道：
> >
> > >
> > > Thanks for Mickael response!
> > >
> > > >I'm wondering whether we need to introduce a new Decoder interface and
> > > instead if we could reuse Deserializer. We could deprecate the
> > > key-decoder-class and value-decoder-class flags and introduce new
> > > flags like key-deserializer-class and value-deserializer-class. One
> > > benefit is that we already have many existing deserializer
> > > implementations. WDYT?
> > >
> > > I prefer to use different interface, since using the same interface
> > > (Deserializer) may obstruct us from enhancing the interface used by
> > > DumpLogSegments only in the future.
> > >
> > > > One issue I also noted is that some of the existing Decoder
> > > implementations (StringDecoder for example) can accept configurations
> > > but currently DumpLogSegments does not provide a way to pass any
> > > configurations, it creates an empty VerifiableProperties object each
> > > time it instantiates a Decoder instance. If we were to use
> > > Deserializer we would also need a way to provide configurations.
> > >
> > > yep, that is a known issue:
> > > https://issues.apache.org/jira/browse/KAFKA-12311
> > >
> > > We will file PR to fix it
> > >
> > > Mickael Maison  於 2024年5月22日 週三 下午10:51寫道：
> > >
> > >> Hi,
> > >>
> > >> Thanks for the KIP. Sorting this out in 3.8.0 would be really nice as
> > >> it would allow us to migrate this tool in 4.0.0. We're unfortunately
> > >> past the KIP deadline but maybe this is small enough to have an
> > >> exception.
> > >>
> > >> I'm wondering whether we need to introduce a new Decoder interface and
> > >> instead if we could reuse Deserializer. We could deprecate the
> > >> key-decoder-class and value-decoder-class flags and introduce new
> > >> flags like key-deserializer-class and value-deserializer-class. One
> > >> benefit is that we already have many existing deserializer
> > >> implementations. WDYT?
> > >>
> > >> One issue I also noted is that some of the existing Decoder
> > >> implementations (StringDecoder for example) can accept configurations
> > >> but currently DumpLogSegments does not provide a way to pass any
> > >> configurations, it creates an empty VerifiableProperties object each
> > >> time it instantiates a Decoder instance. If we were to use
> > >> Deserializer we would also need a way to provide configurations.
> > >>
> > >> Thanks,
> > >> Mickael
> > >>
> > >> On Wed, May 22, 2024 at 4:12 PM Chia-Ping Tsai 
> > >> wrote:
> > >> >
> > >> > Dear all,
> > >> >
> > >> > We know that  3.8.0 KIP is already frozen, but this is a small KIP
> and
> > >> we need to ship it to 3.8.0 so as to remove the deprecated scala
> > interface
> > >> from 4.0.
> > >> >
> > >> > Best,
> > >> > Chia-Ping
> > >> >
> > >> > On 2024/05/22 14:05:16 Frank Yang wrote:
> > >> > > Hi team,
> > >> > >
> > >> > > Chia-Ping Tsai and I would like to propose KIP-1047 to migrate
> > >> kafka.serializer.Decoder from core module (scala) to tools module
> > (java).
> > >> > >
> > >> > > Feedback and comments are welcome.
> > >> > >
> > >> > > KIP-1047:
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1047+Introduce+new+org.apache.kafka.tools.api.Decoder+to+replace+kafka.serializer.Decoder
> > >> > > JIRA: https://issues.apache.org/jira/browse/KAFKA-16796
> > >> > >
> > >> > > Thank you.
> > >> > > PoAn
> > >>
> > >
> >
>

Re: [DISCUSS] KIP 1047 - Introduce new org.apache.kafka.tools.api.Decoder to replace kafka.serializer.Decoder

[Chia-Ping Tsai]

> One issue I also noted is that some of the existing Decoder
implementations (StringDecoder for example) can accept configurations
but currently DumpLogSegments does not provide a way to pass any
configurations, it creates an empty VerifiableProperties object each
time it instantiates a Decoder instance. If we were to use
Deserializer we would also need a way to provide configurations.

BTW, if the known bug gets fixed, we have to make new interface extend
`configurable`.

Or we can just ignore the known issue as `DumpLogSegments` has no options
to take custom configs for `Decoder`. That allow the `Decoder` more simple


Chia-Ping Tsai  於 2024年5月22日 週三 下午10:58寫道：

>
> Thanks for Mickael response!
>
> >I'm wondering whether we need to introduce a new Decoder interface and
> instead if we could reuse Deserializer. We could deprecate the
> key-decoder-class and value-decoder-class flags and introduce new
> flags like key-deserializer-class and value-deserializer-class. One
> benefit is that we already have many existing deserializer
> implementations. WDYT?
>
> I prefer to use different interface, since using the same interface
> (Deserializer) may obstruct us from enhancing the interface used by
> DumpLogSegments only in the future.
>
> > One issue I also noted is that some of the existing Decoder
> implementations (StringDecoder for example) can accept configurations
> but currently DumpLogSegments does not provide a way to pass any
> configurations, it creates an empty VerifiableProperties object each
> time it instantiates a Decoder instance. If we were to use
> Deserializer we would also need a way to provide configurations.
>
> yep, that is a known issue:
> https://issues.apache.org/jira/browse/KAFKA-12311
>
> We will file PR to fix it
>
> Mickael Maison  於 2024年5月22日 週三 下午10:51寫道：
>
>> Hi,
>>
>> Thanks for the KIP. Sorting this out in 3.8.0 would be really nice as
>> it would allow us to migrate this tool in 4.0.0. We're unfortunately
>> past the KIP deadline but maybe this is small enough to have an
>> exception.
>>
>> I'm wondering whether we need to introduce a new Decoder interface and
>> instead if we could reuse Deserializer. We could deprecate the
>> key-decoder-class and value-decoder-class flags and introduce new
>> flags like key-deserializer-class and value-deserializer-class. One
>> benefit is that we already have many existing deserializer
>> implementations. WDYT?
>>
>> One issue I also noted is that some of the existing Decoder
>> implementations (StringDecoder for example) can accept configurations
>> but currently DumpLogSegments does not provide a way to pass any
>> configurations, it creates an empty VerifiableProperties object each
>> time it instantiates a Decoder instance. If we were to use
>> Deserializer we would also need a way to provide configurations.
>>
>> Thanks,
>> Mickael
>>
>> On Wed, May 22, 2024 at 4:12 PM Chia-Ping Tsai 
>> wrote:
>> >
>> > Dear all,
>> >
>> > We know that  3.8.0 KIP is already frozen, but this is a small KIP and
>> we need to ship it to 3.8.0 so as to remove the deprecated scala interface
>> from 4.0.
>> >
>> > Best,
>> > Chia-Ping
>> >
>> > On 2024/05/22 14:05:16 Frank Yang wrote:
>> > > Hi team,
>> > >
>> > > Chia-Ping Tsai and I would like to propose KIP-1047 to migrate
>> kafka.serializer.Decoder from core module (scala) to tools module (java).
>> > >
>> > > Feedback and comments are welcome.
>> > >
>> > > KIP-1047:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1047+Introduce+new+org.apache.kafka.tools.api.Decoder+to+replace+kafka.serializer.Decoder
>> > > JIRA: https://issues.apache.org/jira/browse/KAFKA-16796
>> > >
>> > > Thank you.
>> > > PoAn
>>
>

Re: [DISCUSS] KIP 1047 - Introduce new org.apache.kafka.tools.api.Decoder to replace kafka.serializer.Decoder

[Chia-Ping Tsai]

Thanks for Mickael response!

>I'm wondering whether we need to introduce a new Decoder interface and
instead if we could reuse Deserializer. We could deprecate the
key-decoder-class and value-decoder-class flags and introduce new
flags like key-deserializer-class and value-deserializer-class. One
benefit is that we already have many existing deserializer
implementations. WDYT?

I prefer to use different interface, since using the same interface
(Deserializer) may obstruct us from enhancing the interface used by
DumpLogSegments only in the future.

> One issue I also noted is that some of the existing Decoder
implementations (StringDecoder for example) can accept configurations
but currently DumpLogSegments does not provide a way to pass any
configurations, it creates an empty VerifiableProperties object each
time it instantiates a Decoder instance. If we were to use
Deserializer we would also need a way to provide configurations.

yep, that is a known issue:
https://issues.apache.org/jira/browse/KAFKA-12311

We will file PR to fix it

Mickael Maison  於 2024年5月22日 週三 下午10:51寫道：

> Hi,
>
> Thanks for the KIP. Sorting this out in 3.8.0 would be really nice as
> it would allow us to migrate this tool in 4.0.0. We're unfortunately
> past the KIP deadline but maybe this is small enough to have an
> exception.
>
> I'm wondering whether we need to introduce a new Decoder interface and
> instead if we could reuse Deserializer. We could deprecate the
> key-decoder-class and value-decoder-class flags and introduce new
> flags like key-deserializer-class and value-deserializer-class. One
> benefit is that we already have many existing deserializer
> implementations. WDYT?
>
> One issue I also noted is that some of the existing Decoder
> implementations (StringDecoder for example) can accept configurations
> but currently DumpLogSegments does not provide a way to pass any
> configurations, it creates an empty VerifiableProperties object each
> time it instantiates a Decoder instance. If we were to use
> Deserializer we would also need a way to provide configurations.
>
> Thanks,
> Mickael
>
> On Wed, May 22, 2024 at 4:12 PM Chia-Ping Tsai 
> wrote:
> >
> > Dear all,
> >
> > We know that  3.8.0 KIP is already frozen, but this is a small KIP and
> we need to ship it to 3.8.0 so as to remove the deprecated scala interface
> from 4.0.
> >
> > Best,
> > Chia-Ping
> >
> > On 2024/05/22 14:05:16 Frank Yang wrote:
> > > Hi team,
> > >
> > > Chia-Ping Tsai and I would like to propose KIP-1047 to migrate
> kafka.serializer.Decoder from core module (scala) to tools module (java).
> > >
> > > Feedback and comments are welcome.
> > >
> > > KIP-1047:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1047+Introduce+new+org.apache.kafka.tools.api.Decoder+to+replace+kafka.serializer.Decoder
> > > JIRA: https://issues.apache.org/jira/browse/KAFKA-16796
> > >
> > > Thank you.
> > > PoAn
>

Re: [DISCUSS] KIP 1047 - Introduce new org.apache.kafka.tools.api.Decoder to replace kafka.serializer.Decoder

[Chia-Ping Tsai]

Dear all,

We know that  3.8.0 KIP is already frozen, but this is a small KIP and we need 
to ship it to 3.8.0 so as to remove the deprecated scala interface from 4.0.

Best,
Chia-Ping

On 2024/05/22 14:05:16 Frank Yang wrote:
> Hi team,
> 
> Chia-Ping Tsai and I would like to propose KIP-1047 to migrate 
> kafka.serializer.Decoder from core module (scala) to tools module (java).
> 
> Feedback and comments are welcome.
> 
> KIP-1047: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1047+Introduce+new+org.apache.kafka.tools.api.Decoder+to+replace+kafka.serializer.Decoder
> JIRA: https://issues.apache.org/jira/browse/KAFKA-16796
> 
> Thank you.
> PoAn

[jira] [Created] (KAFKA-16813) Add global timeout for "@Test" and "@TestTemplate"

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16813:
--

 Summary: Add global timeout for "@Test" and "@TestTemplate"
 Key: KAFKA-16813
 URL: https://issues.apache.org/jira/browse/KAFKA-16813
 Project: Kafka
  Issue Type: Improvement
    Reporter: Chia-Ping Tsai
    Assignee: Chia-Ping Tsai


in code base `@Test` is used by unit test and `@TestTemplate` is used by 
integration test. The later includes `ParameterizedTest`, `ClusterTest`, 
`ClusterTests`, and `ClusterTemplate`. Hence, we can add two different timeout 
for `@Test` and `@TestTemplate`. For example:

junit.jupiter.execution.timeout.default = 30s
junit.jupiter.execution.timeout.testtemplate.method.default = 120s

The accurate timeout value may need more discussion, but we can try it in small 
junit5 module first. For example: tools module and storage module.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16812) The tools-related tests are slow

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16812:
--

 Summary: The tools-related tests are slow
 Key: KAFKA-16812
 URL: https://issues.apache.org/jira/browse/KAFKA-16812
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai


see 
https://ci-builds.apache.org/job/Kafka/job/kafka/job/trunk/2923/testReport/org.apache.kafka.tools/

Maybe we run too many cluster types (5), and we can remove some unrelated types 
for those tests.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16784) Migrate TopicBasedRemoteLogMetadataManagerMultipleSubscriptionsTest to new test infra

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16784.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Migrate TopicBasedRemoteLogMetadataManagerMultipleSubscriptionsTest to new 
> test infra
> -
>
> Key: KAFKA-16784
> URL: https://issues.apache.org/jira/browse/KAFKA-16784
> Project: Kafka
>  Issue Type: Test
>    Reporter: Chia-Ping Tsai
>Assignee: PoAn Yang
>Priority: Minor
>  Labels: storage_test
> Fix For: 3.8.0
>
>
> as title



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16654) Refactor kafka.test.annotation.Type and ClusterTestExtensions

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16654.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Refactor kafka.test.annotation.Type and ClusterTestExtensions
> -
>
> Key: KAFKA-16654
> URL: https://issues.apache.org/jira/browse/KAFKA-16654
> Project: Kafka
>  Issue Type: Improvement
>    Reporter: Chia-Ping Tsai
>Assignee: TaiJuWu
>Priority: Minor
> Fix For: 3.8.0
>
>
> It seems to me the refactor could include following tasks.
> 1. change `invocationContexts`, method invoked by `ClusterTemplate`, and 
> generate-related methods in `ClusterTestExtensions` to return a 
> java.util.Collection instead of accepting a `java.util.function.Consumer`. 
> That can brings two benefit. 1) more simple in production: we don't need to 
> create a List and then pass it to be a function to collect stuff. 2)  more 
> easy to write unit test.
> 2. separate `provideTestTemplateInvocationContexts` to multi methods to 
> handle each annotation. That can help us to write tests, and make core more 
> readable.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1031: Control offset translation in MirrorSourceConnector

[Chia-Ping Tsai]

> Which SourceRecord are you referring to here?

I have added comments to your PR (
https://github.com/apache/kafka/pull/15999#pullrequestreview-2066823538)

in short, `sourcePartition` and `sourceOffset` are unused if
emit.offset-syncs.enabled=false

BTW, I'm +1 to this KIP, and I noticed my previous comments are related to
code. Hence, please feel free to open votes. We can have discussion about
the code later.



Omnia Ibrahim  於 2024年5月21日 週二 下午9:10寫道：

> Hi Chia-Ping
> >  It seems we can disable the sync of idle consumers by setting
> `sync.group.offsets.interval.seconds` to -1, so the fail-fast should
> include sync.group.offsets.interval.seconds too. For another, maybe we
> should do fail-fast for MirrorCheckpointConnector even though we don't have
> this KIP
> I don’t think we need to fail fast with
> `sync.group.offsets.interval.seconds` to -1, as `MirrorCheckpointConnector`
> runs two functionality based on offset-syncs topic that can run separately
> 1. Write group offset to checkpoints internal topic can be disabled with
> `emit.checkpoints.interval.seconds` -1
> 2. Schedule syncing the group offset to __consumer_offsets later can be
> disabled with  `sync.group.offsets.interval.seconds` to -1
>
> So technically `MirrorCheckpointConnector` can run if only one of these
> intervals is set to -1 however, if we want to fail fast we should check
> both `sync.group.offsets.interval.seconds`  and
> `emit.checkpoints.interval.seconds` not set to -1 as this would be useless.
>
>
> > 2) Should we do similar fail-fast for MirrorSourceConnector if user set
> custom producer configs with emit.offset-syncs.enabled=false? I assume the
> producer which sending records to offset-syncs topic won't be created if
> emit.offset-syncs.enabled=false
> This is a good point I’ll update MirrorSourceConnector’s validate method
> to address this. I think we should also address
> `offset-syncs.topic.location` and `offset-syncs.topic.replication.factor`
> as well as custom consumer, and admin client configs.
>
>
> > 3) Should we simplify the SourceRecord if
> emit.offset-syncs.enabled=false? Maybe that can get a bit performance
> improvement.
> Which SourceRecord are you referring to here?
>
> Omnia
> > On 20 May 2024, at 16:58, Chia-Ping Tsai  wrote:
> >
> > Nice KIP. some minor comments/questions are listed below.
> >
> > 1) It seems we can disable the sync of idle consumers by setting
> `sync.group.offsets.interval.seconds` to -1, so the fail-fast should
> include sync.group.offsets.interval.seconds too. For another, maybe we
> should do fail-fast for MirrorCheckpointConnector even though we don't have
> this KIP
> >
> > 2) Should we do similar fail-fast for MirrorSourceConnector if user set
> custom producer configs with emit.offset-syncs.enabled=false? I assume the
> producer which sending records to offset-syncs topic won't be created if
> emit.offset-syncs.enabled=false
> >
> > 3) Should we simplify the SourceRecord if
> emit.offset-syncs.enabled=false? Maybe that can get a bit performance
> improvement.
> >
> > Best,
> > Chia-Ping
> >
> > On 2024/04/08 10:03:50 Omnia Ibrahim wrote:
> >> Hi Chris,
> >> Validation method is a good call. I updated the KIP to state that the
> checkpoint connector will fail if the configs aren’t correct. And updated
> the description of the new config to explain the impact of it on checkpoint
> connector as well.
> >>
> >> If there is no any other feedback from anyone I would like to start the
> voting thread in few days.
> >> Thanks
> >> Omnia
> >>
> >>> On 8 Apr 2024, at 06:31, Chris Egerton 
> wrote:
> >>>
> >>> Hi Omnia,
> >>>
> >>> Ah, good catch. I think failing to start the checkpoint connector if
> offset
> >>> syncs are disabled is fine. We'd probably want to do that via the
> >>> Connector::validate [1] method in order to be able to catch invalid
> configs
> >>> during preflight validation, but it's not necessary to get that
> specific in
> >>> the KIP (especially since we may add other checks as well).
> >>>
> >>> [1] -
> >>>
> https://kafka.apache.org/37/javadoc/org/apache/kafka/connect/connector/Connector.html#validate(java.util.Map)
> >>>
> >>> Cheers,
> >>>
> >>> Chris
> >>>
> >>> On Thu, Apr 4, 2024 at 8:07 PM Omnia Ibrahim 
> >>> wrote:
> >>>
> >>>> Thanks Chris for the feedback
> >>>>> 1. It'd be nice to mention that increasing the max offset lag to
> INT_MAX
> >>>

Re: DescribeLogDirs in Kafka 3.3.1 returns all topics instead of one provided in the request. Bug or "bad user error"?

[Chia-Ping Tsai]

Dear all,

I file https://issues.apache.org/jira/browse/KAFKA-16807 to fix it.

Thanks to Maxim for this nice finding. Also, thanks to Gaurav for the quick
response/dig-in

Cheers,
Chia-Ping

Gaurav Narula  於 2024年5月21日 週二 下午2:56寫道：

> Hello Maxim,
>
> Thanks for sharing this.
>
> I had a look and it seems like the behaviour on the wire changed with
> KAFKA-9435. I believe this change [0] in ReplicaManager causes all topics
> in
> online log dirs to be a part of the response inadvertently. We do however
> set
> partition information only for the queried topic. I'd suggest creating a
> JIRA
> for this.
>
> As for an option to specify ALL_PARTITIONS, I reckon that would require a
> KIP
> since it would be a change to the public interface.
>
> [0]:
>
> https://github.com/apache/kafka/pull/7972/files#diff-78812e247ffeae6f8c49b1b22506434701b1e1bafe7f92ef8f8708059e292bf0R674
>
> Regards,
> Gaurav
>
> On Mon, May 20, 2024 at 10:35 PM Maxim Senin 
> wrote:
>
> > Hello.
> >
> > I’m having a problem with Kafka protocol API.
> >
> > Requests:
> > DescribeLogDirs Request (Version: 0) => [topics]
> >   topics => topic [partitions]
> > topic => STRING
> > partitions => INT32
> >
> > My request contains `[{topic: “blah”, partitions:
> > [0,1,2,3,4,5,6,7,8,9]}]`, but the result
> >
> > Responses:
> > DescribeLogDirs Response (Version: 0) => throttle_time_ms [results]
> >   throttle_time_ms => INT32
> >   results => error_code log_dir [topics]
> > error_code => INT16
> > log_dir => STRING
> > topics => name [partitions]
> >   name => STRING
> >   partitions => partition_index partition_size offset_lag
> is_future_key
> > partition_index => INT32
> > partition_size => INT64
> > offset_lag => INT64
> > is_future_key => BOOLEAN
> >
> >
> >
> >  contains entries for *all* topics. My workaround had been to filter the
> > returned list by topic name to find the one I was requesting the data
> for,
> > but I don’t understand why it’s not limiting the results to just the
> topic
> > I requested in the first place.
> >
> > Also, I think there should be an option to just specify ALL_PARTITIONS
> > because that would save me from having to retrieve topic metadata from
> the
> > broker to count the number of partitions. Kafka server would probably
> have
> > means to do that more efficiently.
> >
> > Is this a bug or am I doing something wrong?
> >
> > Thanks,
> > Maxim
> >
> > 
> >
> > COGILITY SOFTWARE CORPORATION LEGAL DISCLAIMER: The information in this
> > email is confidential and is intended solely for the addressee. Access to
> > this email by anyone else is unauthorized. If you are not the intended
> > recipient, any disclosure, copying, distribution or any action taken or
> > omitted to be taken in reliance on it, is prohibited and may be unlawful.
> >
>

[jira] [Created] (KAFKA-16807) DescribeLogDirsResponseData#results#topics have unexpected topics having empty partitions

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16807:
--

 Summary: DescribeLogDirsResponseData#results#topics have 
unexpected topics having empty partitions
 Key: KAFKA-16807
 URL: https://issues.apache.org/jira/browse/KAFKA-16807
 Project: Kafka
  Issue Type: Bug
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


ReplicaManager [0] could generate a response having unexpected topics which 
have empty partitions. The root cause is it always generate the topic 
collection even though they have no matched partitions.

That is not a issue to Kafka clients, since we loop the "partitions" to fill 
all future responses [1]. Hence, those unexpected topics won't be existent in 
the final results.

However, that could be a issue to the users who implement Kafka client based on 
Kafka protocol [2]


[0] 
https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/ReplicaManager.scala#L1252
[1] 
https://github.com/apache/kafka/blob/b5a013e4564ad93026b9c61431e4573a39bec766/clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java#L3145
[2] https://lists.apache.org/thread/lp7ktmm17pbg7nqk7p4s904lcn3mrvhy



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16794) Can't open videos in streams documentation

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16794.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Can't open videos in streams documentation
> --
>
> Key: KAFKA-16794
> URL: https://issues.apache.org/jira/browse/KAFKA-16794
> Project: Kafka
>  Issue Type: Bug
>  Components: docs, streams
>Reporter: Kuan Po Tseng
>Assignee: 黃竣陽
>Priority: Minor
> Fix For: 3.8.0
>
> Attachments: IMG_4445.png, image.png
>
>
> Can't open videos in page [https://kafka.apache.org/documentation/streams/]
> Open console in chrome browser and it shows error message:
> {{Refused to frame 'https://www.youtube.com/' because it violates the 
> following Content Security Policy directive: "frame-src 'self'".}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-1031: Control offset translation in MirrorSourceConnector

[Chia-Ping Tsai]

Nice KIP. some minor comments/questions are listed below.

1) It seems we can disable the sync of idle consumers by setting 
`sync.group.offsets.interval.seconds` to -1, so the fail-fast should include 
sync.group.offsets.interval.seconds too. For another, maybe we should do 
fail-fast for MirrorCheckpointConnector even though we don't have this KIP

2) Should we do similar fail-fast for MirrorSourceConnector if user set custom 
producer configs with emit.offset-syncs.enabled=false? I assume the producer 
which sending records to offset-syncs topic won't be created if 
emit.offset-syncs.enabled=false

3) Should we simplify the SourceRecord if emit.offset-syncs.enabled=false? 
Maybe that can get a bit performance improvement.

Best,
Chia-Ping

On 2024/04/08 10:03:50 Omnia Ibrahim wrote:
> Hi Chris, 
> Validation method is a good call. I updated the KIP to state that the 
> checkpoint connector will fail if the configs aren’t correct. And updated the 
> description of the new config to explain the impact of it on checkpoint 
> connector as well. 
> 
> If there is no any other feedback from anyone I would like to start the 
> voting thread in few days. 
> Thanks 
> Omnia
> 
> > On 8 Apr 2024, at 06:31, Chris Egerton  wrote:
> > 
> > Hi Omnia,
> > 
> > Ah, good catch. I think failing to start the checkpoint connector if offset
> > syncs are disabled is fine. We'd probably want to do that via the
> > Connector::validate [1] method in order to be able to catch invalid configs
> > during preflight validation, but it's not necessary to get that specific in
> > the KIP (especially since we may add other checks as well).
> > 
> > [1] -
> > https://kafka.apache.org/37/javadoc/org/apache/kafka/connect/connector/Connector.html#validate(java.util.Map)
> > 
> > Cheers,
> > 
> > Chris
> > 
> > On Thu, Apr 4, 2024 at 8:07 PM Omnia Ibrahim 
> > wrote:
> > 
> >> Thanks Chris for the feedback
> >>> 1. It'd be nice to mention that increasing the max offset lag to INT_MAX
> >>> could work as a partial workaround for users on existing versions (though
> >>> of course this wouldn't prevent creation of the syncs topic).
> >> I updated the KIP
> >> 
> >>> 2. Will it be illegal to disable offset syncs if other features that rely
> >>> on offset syncs are explicitly enabled in the connector config? If
> >> they're
> >>> not explicitly enabled then it should probably be fine to silently
> >> disable
> >>> them, but I'd be interested in your thoughts.
> >> The rest of the features that relays on this is controlled by
> >> emit.checkpoints.enabled (enabled by default) and
> >> sync.group.offsets.enabled (disabled by default) which are part of
> >> MirrorCheckpointConnector config not MirrorSourceConnector, I was thinking
> >> that MirrorCheckpointConnector should fail to start if
> >> emit.offset-syncs.enabled is disabled while emit.checkpoints.enabled and/or
> >> sync.group.offsets.enabled are enabled as no point of creating this
> >> connector if the main part is disabled. WDYT?
> >> 
> >> Thanks
> >> Omnia
> >> 
> >>> On 3 Apr 2024, at 12:45, Chris Egerton  wrote:
> >>> 
> >>> Hi Omnia,
> >>> 
> >>> Thanks for the KIP! Two small things come to mind:
> >>> 
> >>> 1. It'd be nice to mention that increasing the max offset lag to INT_MAX
> >>> could work as a partial workaround for users on existing versions (though
> >>> of course this wouldn't prevent creation of the syncs topic).
> >>> 
> >>> 2. Will it be illegal to disable offset syncs if other features that rely
> >>> on offset syncs are explicitly enabled in the connector config? If
> >> they're
> >>> not explicitly enabled then it should probably be fine to silently
> >> disable
> >>> them, but I'd be interested in your thoughts.
> >>> 
> >>> Cheers,
> >>> 
> >>> Chris
> >>> 
> >>> On Wed, Apr 3, 2024, 20:41 Luke Chen  wrote:
> >>> 
>  Hi Omnia,
>  
>  Thanks for the KIP!
>  It LGTM!
>  But I'm not an expert of MM2, it would be good to see if there is any
> >> other
>  comment from MM2 experts.
>  
>  Thanks.
>  Luke
>  
>  On Thu, Mar 14, 2024 at 6:08 PM Omnia Ibrahim 
>  wrote:
>  
> > Hi everyone, I would like to start a discussion thread for KIP-1031:
> > Control offset translation in MirrorSourceConnector
> > 
> > 
>  
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1031%3A+Control+offset+translation+in+MirrorSourceConnector
> > 
> > Thanks
> > Omnia
> > 
>  
> >> 
> >> 
> 
> 

[jira] [Resolved] (KAFKA-16797) A bit cleanup of FeatureControlManager

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16797.

Fix Version/s: 3.8.0
   Resolution: Fixed

> A bit cleanup of FeatureControlManager
> --
>
> Key: KAFKA-16797
> URL: https://issues.apache.org/jira/browse/KAFKA-16797
> Project: Kafka
>  Issue Type: Improvement
>    Reporter: Chia-Ping Tsai
>Assignee: 黃竣陽
>Priority: Trivial
> Fix For: 3.8.0
>
>
> [https://github.com/apache/kafka/blob/trunk/metadata/src/main/java/org/apache/kafka/controller/FeatureControlManager.java#L62]
>  
> that can be replaced by `Collections.emptyIterator()`



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16797) A bit cleanup of FeatureControlManager

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16797:
--

 Summary: A bit cleanup of FeatureControlManager
 Key: KAFKA-16797
 URL: https://issues.apache.org/jira/browse/KAFKA-16797
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai


[https://github.com/apache/kafka/blob/trunk/metadata/src/main/java/org/apache/kafka/controller/FeatureControlManager.java#L62]

 

that can be replaced by `Collections.emptyIterator()`



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16796) Introduce new org.apache.kafka.tools.api.MessageParser to replace kafka.tools.DumpLogSegments.MessageParser

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16796:
--

 Summary: Introduce new org.apache.kafka.tools.api.MessageParser to 
replace kafka.tools.DumpLogSegments.MessageParser
 Key: KAFKA-16796
 URL: https://issues.apache.org/jira/browse/KAFKA-16796
 Project: Kafka
  Issue Type: Sub-task
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


We need a replacement in order to complete 
https://issues.apache.org/jira/browse/KAFKA-14579 in kafak 4.0



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15723) KRaft support in ListOffsetsRequestTest

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-15723.

Fix Version/s: 3.8.0
   Resolution: Fixed

> KRaft support in ListOffsetsRequestTest
> ---
>
> Key: KAFKA-15723
> URL: https://issues.apache.org/jira/browse/KAFKA-15723
> Project: Kafka
>  Issue Type: Task
>  Components: core
>Reporter: Sameer Tejani
>Assignee: Mickael Maison
>Priority: Minor
>  Labels: kraft, kraft-test, newbie
> Fix For: 3.8.0
>
>
> The following tests in ListOffsetsRequestTest in 
> core/src/test/scala/unit/kafka/server/ListOffsetsRequestTest.scala need to be 
> updated to support KRaft
> 37 : def testListOffsetsErrorCodes(): Unit = {
> 84 : def testListOffsetsMaxTimeStampOldestVersion(): Unit = {
> 112 : def testCurrentEpochValidation(): Unit = {
> 173 : def testResponseIncludesLeaderEpoch(): Unit = {
> 210 : def testResponseDefaultOffsetAndLeaderEpochForAllVersions(): Unit = {
> Scanned 261 lines. Found 0 KRaft tests out of 5 tests



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16544) DescribeTopicsResult#allTopicIds and DescribeTopicsResult#allTopicNames should return null instead of throwing NPE

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16544.

Fix Version/s: 3.8
   Resolution: Fixed

> DescribeTopicsResult#allTopicIds and DescribeTopicsResult#allTopicNames 
> should return null instead of throwing NPE
> --
>
> Key: KAFKA-16544
> URL: https://issues.apache.org/jira/browse/KAFKA-16544
> Project: Kafka
>  Issue Type: Improvement
>    Reporter: Chia-Ping Tsai
>Assignee: Kuan Po Tseng
>Priority: Major
> Fix For: 3.8
>
>
> {code:java}
>  * @return A future map from topic names to descriptions which can be 
> used to check
>  * the status of individual description if the describe topic 
> request used
>  * topic names, otherwise return null, this request succeeds only 
> if all the
>  * topic descriptions succeed
> {code}
> According the docs, it should return null if we try to get the result 
> unmatched to the request. For example, we call `allTopicNames` in passing 
> `TopicIdCollection`. However, the current implementation will throw NPE 
> directly



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16791) Add thread detection to ClusterTestExtensions

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16791:
--

 Summary: Add thread detection to ClusterTestExtensions
 Key: KAFKA-16791
 URL: https://issues.apache.org/jira/browse/KAFKA-16791
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


`ClusterTestExtensions` should implement `BeforeAllCallback` and 
`AfterAllCallback` by `TestUtils.verifyNoUnexpectedThreads`



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16763) Upgrade to scala 2.12.19 and scala 2.13.14

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16763.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Upgrade to scala 2.12.19 and scala 2.13.14
> --
>
> Key: KAFKA-16763
> URL: https://issues.apache.org/jira/browse/KAFKA-16763
> Project: Kafka
>  Issue Type: Improvement
>    Reporter: Chia-Ping Tsai
>Assignee: 黃竣陽
>Priority: Minor
> Fix For: 3.8.0
>
>
> scala 2.12.19 (https://github.com/scala/scala/releases/tag/v2.12.19)
>  
> scala 2.13.14 (https://github.com/scala/scala/releases/tag/v2.13.14)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16785) Migrate TopicBasedRemoteLogMetadataManagerRestartTest to new test infra

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16785:
--

 Summary: Migrate TopicBasedRemoteLogMetadataManagerRestartTest to 
new test infra
 Key: KAFKA-16785
 URL: https://issues.apache.org/jira/browse/KAFKA-16785
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


as title



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16784) Migrate TopicBasedRemoteLogMetadataManagerMultipleSubscriptionsTest to new test infra

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16784:
--

 Summary: Migrate 
TopicBasedRemoteLogMetadataManagerMultipleSubscriptionsTest to new test infra
 Key: KAFKA-16784
 URL: https://issues.apache.org/jira/browse/KAFKA-16784
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


as title



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16783) Migrate RemoteLogMetadataManagerTest to new test infra

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16783:
--

 Summary: Migrate RemoteLogMetadataManagerTest to new test infra
 Key: KAFKA-16783
 URL: https://issues.apache.org/jira/browse/KAFKA-16783
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


as title

`TopicBasedRemoteLogMetadataManagerWrapperWithHarness` could be replaced by 
`RemoteLogMetadataManagerTestUtils#builder`



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16668) Enable to set tags by `ClusterTest`

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16668.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Enable to set tags by `ClusterTest` 
> 
>
> Key: KAFKA-16668
> URL: https://issues.apache.org/jira/browse/KAFKA-16668
> Project: Kafka
>  Issue Type: Improvement
>    Reporter: Chia-Ping Tsai
>Assignee: Johnny Hsu
>Priority: Minor
> Fix For: 3.8.0
>
>
> Currently, the display name can be customized by only `name` 
> (https://github.com/apache/kafka/blob/trunk/core/src/test/java/kafka/test/annotation/ClusterTest.java#L42).
>  However, the "key" is hard-code to "name=xxx". Also, it is impossible to set 
> more "tags" for display name. 
> https://github.com/apache/kafka/pull/15766 is a example that we want to add 
> "xxx=bbb" to display name.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16539) Can't update specific broker configs in pre-migration mode

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16539.

Resolution: Fixed

> Can't update specific broker configs in pre-migration mode
> --
>
> Key: KAFKA-16539
> URL: https://issues.apache.org/jira/browse/KAFKA-16539
> Project: Kafka
>  Issue Type: Bug
>  Components: config, kraft
>Affects Versions: 3.6.0, 3.7.0, 3.6.1, 3.6.2
>Reporter: David Arthur
>Assignee: David Arthur
>Priority: Major
> Fix For: 3.8.0, 3.7.1
>
>
> In migration mode, ZK brokers will have a forwarding manager configured. This 
> is used to forward requests to the KRaft controller once we get to that part 
> of the migration. However, prior to KRaft taking over as the controller 
> (known as pre-migration mode), the ZK brokers are still attempting to forward 
> IncrementalAlterConfigs to the controller.
> This works fine for cluster level configs (e.g., "-entity-type broker 
> --entity-default"), but this fails for specific broker configs (e.g., 
> "-entity-type broker --entity-id 1").
> This affects BROKER and BROKER_LOGGER config types.
> To workaround this bug, you can either disable migrations on the brokers 
> (assuming no migration has taken place), or proceed with the migration and 
> get to the point where KRaft is the controller.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-950: Tiered Storage Disablement

[Chia-Ping Tsai]

+1 but I prefer to ship it to KRaft only. 

I do concern that community have enough time to accept more feature in 3.8 :(

Best,
Chia-Ping

On 2024/05/14 15:20:50 Christo Lolov wrote:
> Heya!
> 
> I would like to start a vote on KIP-950: Tiered Storage Disablement in
> order to catch the last Kafka release targeting Zookeeper -
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement
> 
> Best,
> Christo
> 

Re: [DISCUSS] Apache Kafka 3.7.1 release

[Chia-Ping Tsai]

+1

Our production will be upgraded to your 3.7.1 regardless of how many paper
works I have to handle :)

Lucas Brutschy  於 2024年5月16日 週四 下午3:25寫道：

> Hi Igor,
>
> thanks a lot!
> +1
>
> Lucas
>
> On Thu, May 16, 2024 at 5:21 AM Luke Chen  wrote:
> >
> > Hi Igor,
> >
> > Thanks for volunteering!
> > +1
> >
> > Luke
> >
> > On Wed, May 15, 2024 at 11:15 PM Mickael Maison <
> mickael.mai...@gmail.com>
> > wrote:
> >
> > > Hi Igor,
> > >
> > > Thanks for volunteering, +1
> > >
> > > Mickael
> > >
> > > On Thu, Apr 25, 2024 at 11:09 AM Igor Soarez  wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > I'd like to volunteer to be the release manager for a 3.7.1 release.
> > > >
> > > > Please keep in mind, this would be my first release, so I might have
> > > some questions,
> > > > and it might also take me a bit longer to work through the release
> > > process.
> > > > So I'm thinking a good target would be toward the end of May.
> > > >
> > > > Please let me know your thoughts and if there are any objections or
> > > concerns.
> > > >
> > > > Thanks,
> > > >
> > > > --
> > > > Igor
> > >
>

[jira] [Created] (KAFKA-16775) Fix flaky PlaintextAdminIntegrationTest#testCreateExistingTopicsThrowTopicExistsException

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16775:
--

 Summary: Fix flaky 
PlaintextAdminIntegrationTest#testCreateExistingTopicsThrowTopicExistsException
 Key: KAFKA-16775
 URL: https://issues.apache.org/jira/browse/KAFKA-16775
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai


org.opentest4j.AssertionFailedError: timed out waiting for topics
 at app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
 at app//org.junit.jupiter.api.Assertions.fail(Assertions.java:138)
 at 
app//kafka.api.BaseAdminIntegrationTest.waitForTopics(BaseAdminIntegrationTest.scala:236)
 at 
app//kafka.api.PlaintextAdminIntegrationTest.testCreateExistingTopicsThrowTopicExistsException(PlaintextAdminIntegrationTest.scala:140)
 at 
java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
 at 
java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
 at 
java.base@17.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base@17.0.7/java.lang.reflect.Method.invoke(Method.java:568)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16774) fix flaky StreamThreadTest#shouldCloseAllTaskProducersOnCloseIfEosEnabled

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16774:
--

 Summary: fix flaky 
StreamThreadTest#shouldCloseAllTaskProducersOnCloseIfEosEnabled
 Key: KAFKA-16774
 URL: https://issues.apache.org/jira/browse/KAFKA-16774
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai


java.util.ConcurrentModificationException
 at 
java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1720)
 at 
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
 at 
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
 at 
java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
 at 
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
 at 
java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
 at 
org.apache.kafka.streams.processor.internals.TaskManager.allTasks(TaskManager.java:1686)
 at 
org.apache.kafka.streams.processor.internals.TaskManager.releaseLockedUnassignedTaskDirectories(TaskManager.java:1364)
 at 
org.apache.kafka.streams.processor.internals.TaskManager.handleRebalanceComplete(TaskManager.java:208)
 at 
org.apache.kafka.streams.processor.internals.StreamsRebalanceListener.onPartitionsAssigned(StreamsRebalanceListener.java:79)
 at 
org.apache.kafka.streams.processor.internals.StreamThreadTest.shouldCloseAllTaskProducersOnCloseIfEosEnabled(StreamThreadTest.java:1408)
 at 
java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16773) Fix flaky QuorumControllerTest#testDelayedConfigurationOperations

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16773:
--

 Summary: Fix flaky 
QuorumControllerTest#testDelayedConfigurationOperations
 Key: KAFKA-16773
 URL: https://issues.apache.org/jira/browse/KAFKA-16773
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai


org.apache.kafka.server.fault.FaultHandlerException: fatalFaultHandler: 
exception while renouncing leadership: Attempt to resign from epoch 1 which is 
larger than the current epoch 0
 at org.apache.kafka.metalog.LocalLogManager.resign(LocalLogManager.java:788)
 at 
org.apache.kafka.controller.QuorumController.renounce(QuorumController.java:1267)
 at 
org.apache.kafka.controller.QuorumController.handleEventException(QuorumController.java:546)
 at 
org.apache.kafka.controller.QuorumController.access$800(QuorumController.java:179)
 at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.complete(QuorumController.java:878)
 at 
org.apache.kafka.controller.QuorumController$ControllerWriteEvent.handleException(QuorumController.java:868)
 at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.completeWithException(KafkaEventQueue.java:149)
 at 
org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:138)
 at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:211)
 at 
org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:182)
 at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.IllegalArgumentException: Attempt to resign from epoch 1 
which is larger than the current epoch 0



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16671) Revisit SessionedProtocolIntegrationTest.ensureInternalEndpointIsSecured

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16671.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Revisit SessionedProtocolIntegrationTest.ensureInternalEndpointIsSecured
> 
>
> Key: KAFKA-16671
> URL: https://issues.apache.org/jira/browse/KAFKA-16671
> Project: Kafka
>  Issue Type: Test
>    Reporter: Chia-Ping Tsai
>Assignee: PoAn Yang
>Priority: Minor
> Fix For: 3.8.0
>
>
> loop 1000times on my local, and all pass. Let's enable the test to see what 
> happens in our CI



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16763) Upgrade to scala 2.12.19 and scala 2.13.14

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16763:
--

 Summary: Upgrade to scala 2.12.19 and scala 2.13.14
 Key: KAFKA-16763
 URL: https://issues.apache.org/jira/browse/KAFKA-16763
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


scala 2.12.19 (https://github.com/scala/scala/releases/tag/v2.12.19)

 

scala 2.13.14 (https://github.com/scala/scala/releases/tag/v2.13.14)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16761) ZkClusterInstance#start does not create ClusterConfigurableIntegrationHarness

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16761:
--

 Summary: ZkClusterInstance#start does not create 
ClusterConfigurableIntegrationHarness
 Key: KAFKA-16761
 URL: https://issues.apache.org/jira/browse/KAFKA-16761
 Project: Kafka
  Issue Type: Bug
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


[https://github.com/apache/kafka/blob/440f5f6c09720bb9414524781342bbf35973c281/core/src/test/java/kafka/test/junit/ZkClusterInvocationContext.java#L103]

`ClusterConfigurableIntegrationHarness` is created only in "beforeEach" phase, 
and that makes `ZkClusterInstance#start` does not work as it could cause NPE.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16677) Replace ClusterType#ALL and ClusterType#DEFAULT by Array

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16677.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Replace ClusterType#ALL and ClusterType#DEFAULT by Array
> 
>
> Key: KAFKA-16677
> URL: https://issues.apache.org/jira/browse/KAFKA-16677
> Project: Kafka
>  Issue Type: Improvement
>    Reporter: Chia-Ping Tsai
>Assignee: PoAn Yang
>Priority: Minor
> Fix For: 3.8.0
>
>
> Both ClusterType#ALL and ClusterType#DEFAULT are a kind of "tag" instead of 
> true "type". It seems to me they can be removed by using Array. For example:
> ClusterType#ALL -> {Type.ZK, Type.KRAFT, Type.CO_KRAFT}
> ClusterType#DEFAULT -> {}
> There are two benefits
> 1. That is more readable for "ALL type". 
> 2. We don't throw the awkward "exception" when seeing "DEFAULT".



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16705) the flag "started" of RaftClusterInstance is false even though the cluster is started

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16705:
--

 Summary: the flag "started" of RaftClusterInstance is false even 
though the cluster is started
 Key: KAFKA-16705
 URL: https://issues.apache.org/jira/browse/KAFKA-16705
 Project: Kafka
  Issue Type: Bug
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


we should set `started` to true after 
https://github.com/apache/kafka/blob/643db430a707479c9e87eec1ad67e1d4f43c9268/core/src/test/java/kafka/test/junit/RaftClusterInvocationContext.java#L113



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16679) Merge unit test down to the class of integration test

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16679.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Merge unit test down to the class of integration test
> -
>
> Key: KAFKA-16679
> URL: https://issues.apache.org/jira/browse/KAFKA-16679
> Project: Kafka
>  Issue Type: Test
>    Reporter: Chia-Ping Tsai
>Assignee: Cheng-Kai, Zhang
>Priority: Minor
> Fix For: 3.8.0
>
>
> Normally, we don't put multi test classes into single file. Those test 
> classes can be extracted into a new class file. Or we can merge them into 
> single class by using "@Test" annotation. That can make those test cases run 
> without embedded cluster.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16704) Fix flaky kafka.log.remote.RemoteIndexCacheTest#testIndexFileAlreadyExistOnDiskButNotInCache

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16704:
--

 Summary: Fix flaky  
kafka.log.remote.RemoteIndexCacheTest#testIndexFileAlreadyExistOnDiskButNotInCache
 Key: KAFKA-16704
 URL: https://issues.apache.org/jira/browse/KAFKA-16704
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai


java.io.UncheckedIOException: java.nio.file.NoSuchFileException: 
/tmp/kafka-RemoteIndexCacheTest3690189103734187552/R68MBnutRfmqJY66XXFoOA:foo-0/remote-log-index-cache/2147584984_Ma8JCqucS7mqKIHfSSDeow.txnindex.deleted
 at 
java.base/java.nio.file.FileTreeIterator.fetchNextIfNeeded(FileTreeIterator.java:87)
 at java.base/java.nio.file.FileTreeIterator.hasNext(FileTreeIterator.java:103)
 at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132)
 at 
java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
 at 
java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
 at 
java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
 at 
java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
 at 
java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
 at 
java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
 at 
java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
 at 
kafka.log.remote.RemoteIndexCacheTest.renameRemoteCacheIndexFileFromDisk$1(RemoteIndexCacheTest.scala:832)
 at 
kafka.log.remote.RemoteIndexCacheTest.testIndexFileAlreadyExistOnDiskButNotInCache(RemoteIndexCacheTest.scala:851)
 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
 at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16698) Fix flaky kafka.network.DynamicConnectionQuotaTest#testDynamicIpConnectionRateQuota

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16698:
--

 Summary: Fix flaky 
kafka.network.DynamicConnectionQuotaTest#testDynamicIpConnectionRateQuota
 Key: KAFKA-16698
 URL: https://issues.apache.org/jira/browse/KAFKA-16698
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai


{code:java}
org.opentest4j.AssertionFailedError: Timed out waiting for connection rate 
update to propagate  at 
app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)   at 
app//org.junit.jupiter.api.Assertions.fail(Assertions.java:138)  at 
app//kafka.network.DynamicConnectionQuotaTest.updateIpConnectionRate(DynamicConnectionQuotaTest.scala:279)
   at 
app//kafka.network.DynamicConnectionQuotaTest.testDynamicIpConnectionRateQuota(DynamicConnectionQuotaTest.scala:255)
 at 
java.base@21.0.2/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
 at java.base@21.0.2/java.lang.reflect.Method.invoke(Method.java:580)at 
app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
  at 
app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
   at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
 at 
app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
at 
app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
  at 
app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:94)
   at 
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
at 
app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16697) Fix flaky kafka.api.SaslGssapiSslEndToEndAuthorizationTest#testProduceConsumeWithWildcardAcls

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16697:
--

 Summary: Fix flaky 
kafka.api.SaslGssapiSslEndToEndAuthorizationTest#testProduceConsumeWithWildcardAcls
 Key: KAFKA-16697
 URL: https://issues.apache.org/jira/browse/KAFKA-16697
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai


{code:java}
org.opentest4j.AssertionFailedError: Should have been zero expired connections 
killed: 1(total=0.0) ==> expected: <0> but was: <1>  at 
app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
   at 
app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
   at 
app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)  at 
app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:166)  at 
app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:664)  at 
app//kafka.api.EndToEndAuthorizationTest.$anonfun$confirmReauthenticationMetrics$1(EndToEndAuthorizationTest.scala:202)
  at 
app//kafka.api.EndToEndAuthorizationTest.$anonfun$confirmReauthenticationMetrics$1$adapted(EndToEndAuthorizationTest.scala:200)
  at app//scala.collection.IterableOnceOps.foreach(IterableOnce.scala:576)  
  at app//scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:574) 
  at app//scala.collection.AbstractIterable.foreach(Iterable.scala:933)   
at 
app//kafka.api.EndToEndAuthorizationTest.confirmReauthenticationMetrics(EndToEndAuthorizationTest.scala:200)
 at 
app//kafka.api.EndToEndAuthorizationTest.testProduceConsumeWithWildcardAcls(EndToEndAuthorizationTest.scala:236)
 at 
java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)at 
java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at 
java.base@17.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.base@17.0.7/java.lang.reflect.Method.invoke(Method.java:568)
at 
app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
  at 
app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
   at 
app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChai
 {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16660) reduce the check interval to speedup DelegationTokenRequestsTest

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16660.

Fix Version/s: 3.8.0
   Resolution: Fixed

> reduce the check interval to speedup DelegationTokenRequestsTest
> 
>
> Key: KAFKA-16660
> URL: https://issues.apache.org/jira/browse/KAFKA-16660
> Project: Kafka
>  Issue Type: Improvement
>    Reporter: Chia-Ping Tsai
>Assignee: Kuan Po Tseng
>Priority: Minor
> Fix For: 3.8.0
>
>
> the check interval is 1 minute 
> (https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/server/DelegationTokenRequestsTest.scala#L49),
>  and `DelegationTokenRequestsTest` waits 2 minutes before running the check 
> (https://github.com/apache/kafka/blob/trunk/core/src/test/scala/unit/kafka/server/DelegationTokenRequestsTest.scala#L159)
>  ...
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-12947) Replace EasyMock and PowerMock with Mockito for StreamsMetricsImplTest ...

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-12947.

Resolution: Duplicate

this is fixed by https://github.com/apache/kafka/pull/14623

> Replace EasyMock and PowerMock with Mockito for StreamsMetricsImplTest ...
> --
>
> Key: KAFKA-12947
> URL: https://issues.apache.org/jira/browse/KAFKA-12947
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: YI-CHEN WANG
>Assignee: Dalibor Plavcic
>Priority: Major
>
> For Kafka-7438



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16484) Support to define per broker/controller property by ClusterConfigProperty

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16484.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Support to define per broker/controller property by ClusterConfigProperty
> -
>
> Key: KAFKA-16484
> URL: https://issues.apache.org/jira/browse/KAFKA-16484
> Project: Kafka
>  Issue Type: Test
>    Reporter: Chia-Ping Tsai
>Assignee: Kuan Po Tseng
>Priority: Major
> Fix For: 3.8.0
>
>
> the property set to `ClusterConfigProperty` gets applied to all brokers, and 
> hence we can't have individual props for each broker to test racks.
>  
> It seems to me we can add new field "id" to `ClusterConfigProperty` to 
> declare the property should be applied to specific broker (or controller). 
> the default value is -1 and it should be applied to all nodes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16640) Replace TestUtils#resource by scala.util.Using

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16640.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Replace TestUtils#resource by scala.util.Using
> --
>
> Key: KAFKA-16640
> URL: https://issues.apache.org/jira/browse/KAFKA-16640
> Project: Kafka
>  Issue Type: Improvement
>    Reporter: Chia-Ping Tsai
>Assignee: TengYao Chi
>Priority: Minor
> Fix For: 3.8.0
>
>
> `scala.util.Using` is in both scala 2.13 and scala-collection-compat so we 
> don't need to have custom try-resource function.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16678) Remove unimplementedquorum from EndToEndAuthorizationTest

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16678.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Remove unimplementedquorum from EndToEndAuthorizationTest
> -
>
> Key: KAFKA-16678
> URL: https://issues.apache.org/jira/browse/KAFKA-16678
> Project: Kafka
>  Issue Type: Test
>    Reporter: Chia-Ping Tsai
>Assignee: TengYao Chi
>Priority: Minor
> Fix For: 3.8.0
>
>
> `unimplementedquorum`[0] is used to skip test cases if they don't support to 
> run by kraft. However, KAFKA-15219 , KAFKA-14765 and KAFKA-14776 make related 
> tests support to run by kraft.
> In short, it is time to remove the unused variable :)
> [0] 
> [https://github.com/apache/kafka/blob/d76352e2151178521dc447e3406dabb8fcd4c57c/core/src/test/scala/integration/kafka/api/EndToEndAuthorizationTest.scala#L146]
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16689) Move LogValidatorTest to storage module

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16689:
--

 Summary: Move LogValidatorTest to storage module
 Key: KAFKA-16689
 URL: https://issues.apache.org/jira/browse/KAFKA-16689
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


`LogValidator` is moved to storage module already but its unit test is still in 
core module. That is a bit weird. We ought to rewrite it by java and then move 
it to storage module.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16684) FetchResponse#responseData could return incorrect data

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16684:
--

 Summary: FetchResponse#responseData could return incorrect data
 Key: KAFKA-16684
 URL: https://issues.apache.org/jira/browse/KAFKA-16684
 Project: Kafka
  Issue Type: Bug
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


[https://github.com/apache/kafka/commit/2b8aff58b575c199ee8372e5689420c9d77357a5]
 make it accept input to return "partial" data. The content of output is based 
on the input but we cache the output ... It will return same output even though 
we pass different input. That is a potential bug.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16683) Extract security-related helpers from scala.TestUtils to java class

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16683:
--

 Summary: Extract security-related helpers from scala.TestUtils to 
java class
 Key: KAFKA-16683
 URL: https://issues.apache.org/jira/browse/KAFKA-16683
 Project: Kafka
  Issue Type: Sub-task
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


We can merge them into `JaasTestUtils and then rename `JaasTestUtils` to 
`SecurityTestUtils.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16682) Rewrite JassTestUtils by Java

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16682:
--

 Summary: Rewrite JassTestUtils by Java
 Key: KAFKA-16682
 URL: https://issues.apache.org/jira/browse/KAFKA-16682
 Project: Kafka
  Issue Type: Sub-task
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


as title

one more thing is that we should change the package name from kafka.utils to 
kafka.security



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16681) Rewrite MiniKDC by Java

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16681:
--

 Summary: Rewrite MiniKDC by Java
 Key: KAFKA-16681
 URL: https://issues.apache.org/jira/browse/KAFKA-16681
 Project: Kafka
  Issue Type: Sub-task
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


Noted:
 # we need to move it from scala folder to java folder
 # don't change the package name since system tests requires it



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16680) Make ClusterTestExtensions support SASL

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16680:
--

 Summary: Make ClusterTestExtensions support SASL
 Key: KAFKA-16680
 URL: https://issues.apache.org/jira/browse/KAFKA-16680
 Project: Kafka
  Issue Type: New Feature
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


This is a umbrella issue.

In order to migrate more tests to new test infra, we ought to make it support 
SASL at least.

*phase1: reuse/rewrite existent SASL utils by Java*
 # MiniKdc
 # JaasTestUtils
 # Move security-related helpers from scala.TestUtils to java.TestUtils
 # extract/rewrite non-zk code from SaslSetup to new java class

*phase2: make `ClusterTest#securityProtocol` works. It does not work for kraft 
mode :(*
 # add client-related helper to generate consumer/producer/admin class with 
security configs
 # configure kraft server with security settings
 # migrate tests of tools to use new test infra with security

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16679) Merge `DeleteRecordsCommandUnitTest` into `DeleteRecordsCommandTest`, `FeatureCommandUnitTest` into

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16679:
--

 Summary: Merge `DeleteRecordsCommandUnitTest` into 
`DeleteRecordsCommandTest`, `FeatureCommandUnitTest` into 
 Key: KAFKA-16679
 URL: https://issues.apache.org/jira/browse/KAFKA-16679
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


Normally, we don't put multi test classes into single file. Those test classes 
can be extracted into a new class file. Or we can merge them into single class 
by using "@Test" annotation. That can make those test cases run without 
embedded cluster.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16678) Remove unimplementedquorum from EndToEndAuthorizationTest

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16678:
--

 Summary: Remove unimplementedquorum from EndToEndAuthorizationTest
 Key: KAFKA-16678
 URL: https://issues.apache.org/jira/browse/KAFKA-16678
 Project: Kafka
  Issue Type: Test
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


`unimplementedquorum`[0] is used to skip test cases if they don't support to 
run by kraft. However, KAFKA-15219 , KAFKA-14765 and KAFKA-14776 make related 
tests support to run by kraft.

In short, it is time to remove the unused variable :)

[0] 
[https://github.com/apache/kafka/blob/d76352e2151178521dc447e3406dabb8fcd4c57c/core/src/test/scala/integration/kafka/api/EndToEndAuthorizationTest.scala#L146]

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16593) Rewrite DeleteConsumerGroupsTest by ClusterTestExtensions

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16593.

Fix Version/s: 3.8.0
   Resolution: Fixed

> Rewrite DeleteConsumerGroupsTest by ClusterTestExtensions 
> --
>
> Key: KAFKA-16593
> URL: https://issues.apache.org/jira/browse/KAFKA-16593
> Project: Kafka
>  Issue Type: Improvement
>    Reporter: Chia-Ping Tsai
>Assignee: TengYao Chi
>Priority: Minor
> Fix For: 3.8.0
>
>
> as title. the test is in tools module and it does not need to depend on test 
> code of core module directly



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16677) Replace ClusterType#ALL and ClusterType#DEFAULT by Array

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-16677:
--

 Summary: Replace ClusterType#ALL and ClusterType#DEFAULT by Array
 Key: KAFKA-16677
 URL: https://issues.apache.org/jira/browse/KAFKA-16677
 Project: Kafka
  Issue Type: Improvement
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


Both ClusterType#ALL and ClusterType#DEFAULT are a kind of "tag" instead of 
true "type". It seems to me they can be removed by using Array. For example:

ClusterType#ALL -> {Type.ZK, Type.KRAFT, Type.CO_KRAFT}

ClusterType#DEFAULT -> {}

There are two benefits

1. That is more readable for "ALL type". 
2. We don't throw the awkward "exception" when seeing "DEFAULT".



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16470) kafka-dump-log --offsets-decoder should support new records

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16470.

Fix Version/s: 3.8.0
   Resolution: Fixed

> kafka-dump-log --offsets-decoder should support new records
> ---
>
> Key: KAFKA-16470
> URL: https://issues.apache.org/jira/browse/KAFKA-16470
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: David Jacot
>Assignee: David Jacot
>Priority: Major
> Fix For: 3.8.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)