[jira] [Assigned] (KAFKA-14507) Add ConsumerGroupPrepareAssignment API

2023-11-17 Thread Zihao Lin (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zihao Lin reassigned KAFKA-14507: - Assignee: Zihao Lin > Add ConsumerGroupPrepareAssignment API >

Re: [PR] KAFKA-15174: Ensure CommitAsync propagate the exception to the user [kafka]

2023-11-17 Thread via GitHub
lucasbru commented on PR #14680: URL: https://github.com/apache/kafka/pull/14680#issuecomment-1815926638 Scala compilation in 2.12 is broken on master, which is fixed in https://github.com/apache/kafka/pull/14786, otherwise this is looking good -- This is an automated message from the Apa

Re: [PR] HOTFIX: Fix compilation error in ReplicaManagerConcurrencyTest for Scala 2.12 [kafka]

2023-11-17 Thread via GitHub
dajac commented on PR #14786: URL: https://github.com/apache/kafka/pull/14786#issuecomment-1815931886 Thanks @lucasbru! The compilation step has succeeded for all build and I have also confirmed that the test passes locally with Scala 2.12 and 2.13. Therefore, I will go ahead and merge it t

Re: [PR] HOTFIX: Fix compilation error in ReplicaManagerConcurrencyTest for Scala 2.12 [kafka]

2023-11-17 Thread via GitHub
dajac merged PR #14786: URL: https://github.com/apache/kafka/pull/14786 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.or

Re: [PR] KAFKA-15833: Restrict Consumer API to be used from one thread [kafka]

2023-11-17 Thread via GitHub
cadonna commented on PR #14779: URL: https://github.com/apache/kafka/pull/14779#issuecomment-1815989094 I agree with @philipnee that testing the locking behavior would be great. However, I also agree with @lucasbru that testing the locking behavior is quite hard. Given the tight deadline to

Re: [PR] KAFKA-15555: Ensure wakeups are handled correctly in poll() [kafka]

2023-11-17 Thread via GitHub
cadonna commented on PR #14746: URL: https://github.com/apache/kafka/pull/14746#issuecomment-1816005251 > I wonder if we could just use a BlockingQueue for the fetchBuffer because fetchBuffer.poll(time) blocks until non-empty or timeout. As far as I see from the javadocs of a `Blockin

Re: [PR] KAFKA-15555: Ensure wakeups are handled correctly in poll() [kafka]

2023-11-17 Thread via GitHub
cadonna commented on PR #14746: URL: https://github.com/apache/kafka/pull/14746#issuecomment-1816006382 > Do you think it is worth adding a ticket about using the BackgroundEventQueue for the fetches? Let me know if you have a concrete idea about how to implement this. Let me think a

Re: [PR] KAFKA-15327: ensure the commit manager commit on close [kafka]

2023-11-17 Thread via GitHub
philipnee commented on PR #14710: URL: https://github.com/apache/kafka/pull/14710#issuecomment-1816006917 Hi @lucasbru @cadonna - If you get a chance, could you take a look at the PR? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please lo

[jira] [Commented] (KAFKA-15836) KafkaConsumer subscribes to multiple topics does not respect max.poll.records

2023-11-17 Thread Mickael Maison (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787118#comment-17787118 ] Mickael Maison commented on KAFKA-15836: This is marked as a blocker for 3.6.1.

Re: [PR] KAFKA-15799 Handle full metadata updates on ZK brokers [kafka]

2023-11-17 Thread via GitHub
mimaison commented on PR #14719: URL: https://github.com/apache/kafka/pull/14719#issuecomment-1816083065 Can we now close https://issues.apache.org/jira/browse/KAFKA-15799? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[PR] KAFKA-15241: Compute tiered copied offset by keeping the respective epochs in scope [kafka]

2023-11-17 Thread via GitHub
kamalcph opened a new pull request, #14787: URL: https://github.com/apache/kafka/pull/14787 `findHighestRemoteOffset` does not take into account the leader-epoch end offset. This can cause log divergence between the local and remote log segments when there is unclean leader election.

Re: [PR] KAFKA-15241: Compute tiered copied offset by keeping the respective epochs in scope [kafka]

2023-11-17 Thread via GitHub
kamalcph commented on code in PR #14787: URL: https://github.com/apache/kafka/pull/14787#discussion_r1397045499 ## core/src/main/java/kafka/log/remote/RemoteLogManager.java: ## @@ -625,9 +626,10 @@ private void maybeUpdateCopiedOffset(UnifiedLog log) throws RemoteStorageExcepti

Re: [PR] KAFKA-15241: Compute tiered copied offset by keeping the respective epochs in scope [kafka]

2023-11-17 Thread via GitHub
clolov commented on PR #14787: URL: https://github.com/apache/kafka/pull/14787#issuecomment-1816112742 I will aim to provide a review today! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

[jira] [Assigned] (KAFKA-14877) refactor InMemoryLeaderEpochCheckpoint

2023-11-17 Thread Kamal Chandraprakash (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash reassigned KAFKA-14877: Assignee: (was: Kamal Chandraprakash) > refactor InMemoryLeaderEpochC

[jira] [Assigned] (KAFKA-15836) KafkaConsumer subscribes to multiple topics does not respect max.poll.records

2023-11-17 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield reassigned KAFKA-15836: Assignee: Andrew Schofield (was: Kirk True) > KafkaConsumer subscribes to multip

Re: [PR] KAFKA-15826: Close consumer when sink task is cancelled [kafka]

2023-11-17 Thread via GitHub
yashmayya commented on code in PR #14762: URL: https://github.com/apache/kafka/pull/14762#discussion_r1397042412 ## connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java: ## @@ -163,6 +163,12 @@ public void initialize(TaskConfig taskConfig) {

[jira] [Commented] (KAFKA-15836) KafkaConsumer subscribes to multiple topics does not respect max.poll.records

2023-11-17 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787128#comment-17787128 ] Andrew Schofield commented on KAFKA-15836: -- Yes, I would say so. I think I'll h

[PR] MINOR: Expose earliest local timestamp via the GetOffsetShell [kafka]

2023-11-17 Thread via GitHub
clolov opened a new pull request, #14788: URL: https://github.com/apache/kafka/pull/14788 With the introduction of tiered storage the ListOffsets API can now return the earliest local timestamp. The purpose of this pull request is to enable the Kafka tools to get this information in

Re: [PR] KAFKA-15555: Ensure wakeups are handled correctly in poll() [kafka]

2023-11-17 Thread via GitHub
lucasbru commented on code in PR #14746: URL: https://github.com/apache/kafka/pull/14746#discussion_r1397061812 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchBuffer.java: ## @@ -52,6 +53,8 @@ public class FetchBuffer implements AutoCloseable { pr

[jira] [Updated] (KAFKA-15851) broker under replicated due to error java.nio.BufferOverflowException

2023-11-17 Thread wangliucheng (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangliucheng updated KAFKA-15851: - Attachment: (was: server.log) > broker under replicated due to error java.nio.BufferOverflow

Re: [PR] KAFKA-15833: Restrict Consumer API to be used from one thread [kafka]

2023-11-17 Thread via GitHub
lucasbru commented on code in PR #14779: URL: https://github.com/apache/kafka/pull/14779#discussion_r1397125030 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java: ## @@ -1068,12 +1168,17 @@ public void onComplete(Map offsets, Exception

Re: [PR] KAFKA-15833: Restrict Consumer API to be used from one thread [kafka]

2023-11-17 Thread via GitHub
lucasbru commented on code in PR #14779: URL: https://github.com/apache/kafka/pull/14779#discussion_r1397125331 ## clients/src/test/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumerTest.java: ## @@ -112,6 +112,12 @@ public void testInvalidGroupId() { a

[jira] [Assigned] (KAFKA-15815) JsonRestServer leaks sockets via HttpURLConnection when keep-alive enabled

2023-11-17 Thread Ashwin Pankaj (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Pankaj reassigned KAFKA-15815: - Assignee: Ashwin Pankaj > JsonRestServer leaks sockets via HttpURLConnection when keep-

Re: [PR] KAFKA-15833: Restrict Consumer API to be used from one thread [kafka]

2023-11-17 Thread via GitHub
lucasbru commented on code in PR #14779: URL: https://github.com/apache/kafka/pull/14779#discussion_r1397139312 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java: ## @@ -1102,46 +1207,94 @@ public void subscribe(Pattern pattern, Consum

Re: [PR] KAFKA-14133: Migrate ProcessorStateManagerTest and StreamThreadTest to Mockito [kafka]

2023-11-17 Thread via GitHub
cadonna commented on code in PR #13932: URL: https://github.com/apache/kafka/pull/13932#discussion_r1396975782 ## streams/src/test/java/org/apache/kafka/streams/processor/internals/ProcessorStateManagerTest.java: ## @@ -125,11 +122,11 @@ public class ProcessorStateManagerTest {

Re: [PR] KAFKA-15555: Ensure wakeups are handled correctly in poll() [kafka]

2023-11-17 Thread via GitHub
cadonna commented on code in PR #14746: URL: https://github.com/apache/kafka/pull/14746#discussion_r1397154783 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchBuffer.java: ## @@ -52,6 +53,8 @@ public class FetchBuffer implements AutoCloseable { pri

Re: [PR] KAFKA-15555: Ensure wakeups are handled correctly in poll() [kafka]

2023-11-17 Thread via GitHub
cadonna commented on code in PR #14746: URL: https://github.com/apache/kafka/pull/14746#discussion_r1397156239 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchBuffer.java: ## @@ -185,6 +188,16 @@ void awaitNotEmpty(Timer timer) { } } +

Re: [PR] KAFKA-15849: Fix ListGroups API when runtime partition size is zero [kafka]

2023-11-17 Thread via GitHub
dajac merged PR #14785: URL: https://github.com/apache/kafka/pull/14785 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.or

[jira] [Resolved] (KAFKA-15849) Fix ListGroups API when runtime partition size is zero

2023-11-17 Thread David Jacot (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-15849. - Fix Version/s: 3.7.0 Reviewer: David Jacot Resolution: Fixed > Fix ListGroups AP

[jira] [Commented] (KAFKA-14507) Add ConsumerGroupPrepareAssignment API

2023-11-17 Thread Zihao Lin (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787193#comment-17787193 ] Zihao Lin commented on KAFKA-14507: --- Hi team, If this ticket is open I will pick it up

[jira] [Updated] (KAFKA-15799) ZK brokers incorrectly handle KRaft metadata snapshots

2023-11-17 Thread David Arthur (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur updated KAFKA-15799: - Fix Version/s: 3.7.0 > ZK brokers incorrectly handle KRaft metadata snapshots >

Re: [PR] KAFKA-15038: Add metadatacache into RemoteLogManager, and refactor all relevant codes [kafka]

2023-11-17 Thread via GitHub
Owen-CH-Leung commented on PR #14136: URL: https://github.com/apache/kafka/pull/14136#issuecomment-1816419916 @kamalcph Thanks. Just rebased. Btw when I rebased, I notice that there's a failing test `kafka.server.AssignmentsManagerTest.testAssignmentAggregation` at trunk. May

Re: [PR] KAFKA-15799 Handle full metadata updates on ZK brokers [kafka]

2023-11-17 Thread via GitHub
mumrah commented on PR #14719: URL: https://github.com/apache/kafka/pull/14719#issuecomment-1816420476 @mimaison yup, I closed it just now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] KAFKA-15201: Allow git push to fail gracefully [kafka]

2023-11-17 Thread via GitHub
Owen-CH-Leung commented on PR #14645: URL: https://github.com/apache/kafka/pull/14645#issuecomment-1816422275 @divijvaidya shall we merge this PR ? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[jira] [Commented] (KAFKA-15793) Flaky test ZkMigrationIntegrationTest.testMigrateTopicDeletions

2023-11-17 Thread Mickael Maison (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787216#comment-17787216 ] Mickael Maison commented on KAFKA-15793: Since KAFKA-15799 is now done, marking

[jira] [Resolved] (KAFKA-15793) Flaky test ZkMigrationIntegrationTest.testMigrateTopicDeletions

2023-11-17 Thread Mickael Maison (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison resolved KAFKA-15793. Resolution: Fixed > Flaky test ZkMigrationIntegrationTest.testMigrateTopicDeletions >

[jira] [Commented] (KAFKA-15836) KafkaConsumer subscribes to multiple topics does not respect max.poll.records

2023-11-17 Thread Mickael Maison (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787217#comment-17787217 ] Mickael Maison commented on KAFKA-15836: Awesome, thanks [~schofielaj]  > Kafka

Re: [PR] KAFKA-15327: ensure the commit manager commit on close [kafka]

2023-11-17 Thread via GitHub
lucasbru commented on code in PR #14710: URL: https://github.com/apache/kafka/pull/14710#discussion_r1397321134 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java: ## @@ -180,6 +180,18 @@ public void maybeAutoCommit(final Map offsets

[PR] KAFKA-15836: KafkaConsumer subscribes to multiple topics does not respect max.poll.records [kafka]

2023-11-17 Thread via GitHub
AndrewJSchofield opened a new pull request, #14789: URL: https://github.com/apache/kafka/pull/14789 When returning fetched records, the code was not properly honouring `max.poll.records`. When it had fetched records for multiple topic-partitions, it was intended to accumulate records up

Re: [PR] KAFKA-15362: Resolve offline replicas in metadata cache [kafka]

2023-11-17 Thread via GitHub
soarez commented on PR #14737: URL: https://github.com/apache/kafka/pull/14737#issuecomment-1816570498 Thanks for the review @cmccabe. I think I have incorporated all your suggestions. PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log

[jira] [Commented] (KAFKA-15836) KafkaConsumer subscribes to multiple topics does not respect max.poll.records

2023-11-17 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787243#comment-17787243 ] Andrew Schofield commented on KAFKA-15836: -- I would say that it's not applicabl

Re: [PR] KAFKA-15836: KafkaConsumer subscribes to multiple topics does not respect max.poll.records [kafka]

2023-11-17 Thread via GitHub
kirktrue commented on code in PR #14789: URL: https://github.com/apache/kafka/pull/14789#discussion_r1397472048 ## clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java: ## @@ -1124,6 +1126,62 @@ public void testFetchMaxPollRecords() { asser

Re: [PR] KAFKA-15836: KafkaConsumer subscribes to multiple topics does not respect max.poll.records [kafka]

2023-11-17 Thread via GitHub
kirktrue commented on PR #14789: URL: https://github.com/apache/kafka/pull/14789#issuecomment-1816615350 🤦‍♂️ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [PR] KAFKA-15802: Validate remote segment state before fetching index [kafka]

2023-11-17 Thread via GitHub
mimaison commented on PR #14759: URL: https://github.com/apache/kafka/pull/14759#issuecomment-1816625357 Is the work complete? If so can we close [KAFKA-15802](https://issues.apache.org/jira/browse/KAFKA-15802)? -- This is an automated message from the Apache Git Service. To respond to th

Re: [PR] KAFKA-15836: KafkaConsumer subscribes to multiple topics does not respect max.poll.records [kafka]

2023-11-17 Thread via GitHub
AndrewJSchofield commented on code in PR #14789: URL: https://github.com/apache/kafka/pull/14789#discussion_r1397505461 ## clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java: ## @@ -1124,6 +1126,62 @@ public void testFetchMaxPollRecords() {

Re: [PR] KAFKA-15362: Resolve offline replicas in metadata cache [kafka]

2023-11-17 Thread via GitHub
pprovenzano commented on code in PR #14737: URL: https://github.com/apache/kafka/pull/14737#discussion_r1397547750 ## core/src/main/scala/kafka/server/metadata/KRaftMetadataCache.scala: ## @@ -143,21 +143,30 @@ class KRaftMetadataCache(val brokerId: Int) extends MetadataCache w

Re: [PR] KAFKA-15357: Aggregate and propagate assignments [kafka]

2023-11-17 Thread via GitHub
dajac commented on PR #14369: URL: https://github.com/apache/kafka/pull/14369#issuecomment-1816721578 I've also seen `Build / JDK 8 and Scala 2.12 / testAssignmentAggregation() – kafka.server.AssignmentsManagerTest` failing consistently in a few builds ([example](https://ci-builds.apache.or

Re: [PR] KAFKA-15362: Resolve offline replicas in metadata cache [kafka]

2023-11-17 Thread via GitHub
soarez commented on code in PR #14737: URL: https://github.com/apache/kafka/pull/14737#discussion_r1397572189 ## core/src/main/scala/kafka/server/metadata/KRaftMetadataCache.scala: ## @@ -143,21 +143,30 @@ class KRaftMetadataCache(val brokerId: Int) extends MetadataCache with L

Re: [PR] KAFKA-15484: General Rack Aware Assignor [kafka]

2023-11-17 Thread via GitHub
dajac commented on code in PR #14481: URL: https://github.com/apache/kafka/pull/14481#discussion_r1397596576 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GeneralUniformAssignmentBuilder.java: ## @@ -14,17 +14,891 @@ * See the License for the s

Re: [PR] KAFKA-15484: General Rack Aware Assignor [kafka]

2023-11-17 Thread via GitHub
dajac commented on code in PR #14481: URL: https://github.com/apache/kafka/pull/14481#discussion_r1397600750 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GeneralUniformAssignmentBuilder.java: ## @@ -14,17 +14,891 @@ * See the License for the s

Re: [PR] KAFKA-15578: Migrating other system tests to use the group coordinator [kafka]

2023-11-17 Thread via GitHub
dajac commented on code in PR #14582: URL: https://github.com/apache/kafka/pull/14582#discussion_r1397610751 ## tests/kafkatest/tests/core/transactions_test.py: ## @@ -246,8 +246,9 @@ def setup_topics(self): @matrix(failure_mode=["hard_bounce", "clean_bounce"],

Re: [PR] KAFKA-14516: [1/N] Static Member leave, join, re-join request using ConsumerGroupHeartbeats [kafka]

2023-11-17 Thread via GitHub
dajac commented on code in PR #14432: URL: https://github.com/apache/kafka/pull/14432#discussion_r1397287511 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java: ## @@ -829,21 +902,50 @@ private CoordinatorResult consumerGr

[PR] KAFKA-15363: Broker log directory failure changes [kafka]

2023-11-17 Thread via GitHub
OmniaGM opened a new pull request, #14790: URL: https://github.com/apache/kafka/pull/14790 - Extend the ReplicaAlterLogDirsThread to send an AssignReplicasToDirsRequestData RPC before promoting the future replica in JBOD mode with KRAFT. - Extend AssignmentsManager and DirectoryEventHan

[PR] MINOR: Do not check whether updating tasks exist in the waiting loop [kafka]

2023-11-17 Thread via GitHub
cadonna opened a new pull request, #14791: URL: https://github.com/apache/kafka/pull/14791 The state updater waits on a condition variable if no tasks exist that need to be updated. The condition variable is wrapped by a loop to account for spurious wake-ups. The check whether updating task

[jira] [Assigned] (KAFKA-15365) Broker-side replica management changes

2023-11-17 Thread Omnia Ibrahim (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omnia Ibrahim reassigned KAFKA-15365: - Assignee: Omnia Ibrahim > Broker-side replica management changes >

[PR] MINOR: Log the ZK dual-write time [kafka]

2023-11-17 Thread via GitHub
mumrah opened a new pull request, #14792: URL: https://github.com/apache/kafka/pull/14792 We already capture the ZK dual write timings as metrics. This patch includes those timings in the existing log messages. -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] KAFKA-15484: General Rack Aware Assignor [kafka]

2023-11-17 Thread via GitHub
rreddy-22 commented on code in PR #14481: URL: https://github.com/apache/kafka/pull/14481#discussion_r1397683148 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GeneralUniformAssignmentBuilder.java: ## @@ -14,17 +14,891 @@ * See the License for t

Re: [PR] KAFKA-15826: Close consumer when sink task is cancelled [kafka]

2023-11-17 Thread via GitHub
gharris1727 commented on PR #14762: URL: https://github.com/apache/kafka/pull/14762#issuecomment-1816905171 > What was the conclusion here? Is triggering wakeup on a separate thread (through WorkerSinkTask::stop) before also eventually closing the consumer on a separate thread (through Work

[PR] [LI-HOTFIX] Enable register watch when getting children for federated topic namespaces [kafka]

2023-11-17 Thread via GitHub
kehuum opened a new pull request, #14793: URL: https://github.com/apache/kafka/pull/14793 *More detailed description of your change, if necessary. The PR title and PR message become the squashed commit message, so use a separate comment to ping reviewers.* This pr sets registerWat

Re: [PR] [LI-HOTFIX] Enable register watch when getting children for federated topic namespaces [kafka]

2023-11-17 Thread via GitHub
kehuum closed pull request #14793: [LI-HOTFIX] Enable register watch when getting children for federated topic namespaces URL: https://github.com/apache/kafka/pull/14793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] KAFKA-15836: KafkaConsumer subscribes to multiple topics does not respect max.poll.records [kafka]

2023-11-17 Thread via GitHub
AndrewJSchofield commented on PR #14789: URL: https://github.com/apache/kafka/pull/14789#issuecomment-1816929338 Test failures reviewed and unrelated to the changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] KAFKA-15778 & KAFKA-15779: Implement metrics manager (KIP-714) [kafka]

2023-11-17 Thread via GitHub
junrao commented on code in PR #14699: URL: https://github.com/apache/kafka/pull/14699#discussion_r1397714746 ## clients/src/main/java/org/apache/kafka/common/requests/PushTelemetryRequest.java: ## @@ -60,17 +62,47 @@ public PushTelemetryRequest(PushTelemetryRequestData data, s

Re: [PR] KAFKA-15417 flip joinSpuriousLookBackTimeMs and emit non-joined items [kafka]

2023-11-17 Thread via GitHub
lihaosky commented on code in PR #14426: URL: https://github.com/apache/kafka/pull/14426#discussion_r1397715022 ## streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamKStreamJoin.java: ## @@ -104,20 +103,16 @@ public void init(final ProcessorContext context)

Re: [PR] KAFKA-15778 & KAFKA-15779: Implement metrics manager (KIP-714) [kafka]

2023-11-17 Thread via GitHub
apoorvmittal10 commented on code in PR #14699: URL: https://github.com/apache/kafka/pull/14699#discussion_r1397795152 ## core/src/main/java/kafka/metrics/ClientMetricsReceiverPlugin.java: ## @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or m

Re: [PR] KAFKA-15778 & KAFKA-15779: Implement metrics manager (KIP-714) [kafka]

2023-11-17 Thread via GitHub
junrao commented on code in PR #14699: URL: https://github.com/apache/kafka/pull/14699#discussion_r1397714746 ## clients/src/main/java/org/apache/kafka/common/requests/PushTelemetryRequest.java: ## @@ -60,17 +62,47 @@ public PushTelemetryRequest(PushTelemetryRequestData data, s

Re: [PR] KAFKA-15778 & KAFKA-15779: Implement metrics manager (KIP-714) [kafka]

2023-11-17 Thread via GitHub
junrao commented on code in PR #14699: URL: https://github.com/apache/kafka/pull/14699#discussion_r1397714746 ## clients/src/main/java/org/apache/kafka/common/requests/PushTelemetryRequest.java: ## @@ -60,17 +62,47 @@ public PushTelemetryRequest(PushTelemetryRequestData data, s

Re: [PR] KAFKA-15778 & KAFKA-15779: Implement metrics manager (KIP-714) [kafka]

2023-11-17 Thread via GitHub
junrao commented on code in PR #14699: URL: https://github.com/apache/kafka/pull/14699#discussion_r1397714746 ## clients/src/main/java/org/apache/kafka/common/requests/PushTelemetryRequest.java: ## @@ -60,17 +62,47 @@ public PushTelemetryRequest(PushTelemetryRequestData data, s

Re: [PR] KAFKA-15629: Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery [kafka]

2023-11-17 Thread via GitHub
hanyuzheng7 commented on code in PR #14570: URL: https://github.com/apache/kafka/pull/14570#discussion_r1397839267 ## streams/src/main/java/org/apache/kafka/streams/query/TimestampedRangeQuery.java: ## @@ -0,0 +1,130 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

[PR] MINOR: Fix unstable sorting in AssignmentsManagerTest [kafka]

2023-11-17 Thread via GitHub
soarez opened a new pull request, #14794: URL: https://github.com/apache/kafka/pull/14794 Building AssignReplicasToDirsRequestData relies on iteration over Map entries, which can result in different sorting order. The order does not matter to the semantics of the request, but it can cause i

Re: [PR] MINOR: Fix unstable sorting in AssignmentsManagerTest [kafka]

2023-11-17 Thread via GitHub
soarez commented on PR #14794: URL: https://github.com/apache/kafka/pull/14794#issuecomment-1817065152 @dajac please take a look -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] KAFKA-15778 & KAFKA-15779: Implement metrics manager (KIP-714) [kafka]

2023-11-17 Thread via GitHub
apoorvmittal10 commented on code in PR #14699: URL: https://github.com/apache/kafka/pull/14699#discussion_r1397871398 ## clients/src/main/java/org/apache/kafka/common/requests/PushTelemetryRequest.java: ## @@ -60,17 +62,47 @@ public PushTelemetryRequest(PushTelemetryRequestData

Re: [PR] KAFKA-15778 & KAFKA-15779: Implement metrics manager (KIP-714) [kafka]

2023-11-17 Thread via GitHub
apoorvmittal10 commented on code in PR #14699: URL: https://github.com/apache/kafka/pull/14699#discussion_r1397873419 ## core/src/main/java/kafka/metrics/ClientMetricsReceiverPlugin.java: ## @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or m

Re: [PR] KAFKA-15778 & KAFKA-15779: Implement metrics manager (KIP-714) [kafka]

2023-11-17 Thread via GitHub
apoorvmittal10 commented on code in PR #14699: URL: https://github.com/apache/kafka/pull/14699#discussion_r1397875576 ## core/src/test/java/kafka/metrics/ClientMetricsInstanceTest.java: ## @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or mor

Re: [PR] KAFKA-15778 & KAFKA-15779: Implement metrics manager (KIP-714) [kafka]

2023-11-17 Thread via GitHub
apoorvmittal10 commented on PR #14699: URL: https://github.com/apache/kafka/pull/14699#issuecomment-1817091571 Thanks @junrao for leaving the comments, I have tried to address them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] KAFKA-15618: Kafka metrics collector and supporting classes (KIP-714) [kafka]

2023-11-17 Thread via GitHub
xvrl commented on code in PR #14620: URL: https://github.com/apache/kafka/pull/14620#discussion_r1397895468 ## clients/src/main/java/org/apache/kafka/common/telemetry/internals/KafkaMetricsCollector.java: ## @@ -0,0 +1,334 @@ +/* + * Licensed to the Apache Software Foundation (A

Re: [PR] KAFKA-15618: Kafka metrics collector and supporting classes (KIP-714) [kafka]

2023-11-17 Thread via GitHub
xvrl commented on code in PR #14620: URL: https://github.com/apache/kafka/pull/14620#discussion_r1397895468 ## clients/src/main/java/org/apache/kafka/common/telemetry/internals/KafkaMetricsCollector.java: ## @@ -0,0 +1,334 @@ +/* + * Licensed to the Apache Software Foundation (A

Re: [PR] KAFKA-15778 & KAFKA-15779: Implement metrics manager (KIP-714) [kafka]

2023-11-17 Thread via GitHub
junrao commented on code in PR #14699: URL: https://github.com/apache/kafka/pull/14699#discussion_r1397751986 ## core/src/main/java/kafka/server/ClientMetricsManager.java: ## @@ -30,17 +69,376 @@ public class ClientMetricsManager implements Closeable { private static fina

Re: [PR] KAFKA-15618: Kafka metrics collector and supporting classes (KIP-714) [kafka]

2023-11-17 Thread via GitHub
xvrl commented on code in PR #14620: URL: https://github.com/apache/kafka/pull/14620#discussion_r1397923196 ## clients/src/main/java/org/apache/kafka/common/telemetry/internals/KafkaMetricsCollector.java: ## @@ -0,0 +1,334 @@ +/* + * Licensed to the Apache Software Foundation (A

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1397982732 ## streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreFactory.java: ## @@ -66,6 +66,10 @@ default void configure(final StreamsConfig config) {

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1397987652 ## streams/src/main/java/org/apache/kafka/streams/kstream/internals/StreamJoinedStoreFactory.java: ## @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1397992729 ## streams/src/main/java/org/apache/kafka/streams/state/BuiltInDslStoreSuppliers.java: ## @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1398002400 ## streams/src/main/java/org/apache/kafka/streams/state/BuiltInDslStoreSuppliers.java: ## @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1398002738 ## streams/src/main/java/org/apache/kafka/streams/state/DslSessionParams.java: ## @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1398003330 ## streams/src/main/java/org/apache/kafka/streams/state/DslSessionParams.java: ## @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1398003882 ## streams/src/main/java/org/apache/kafka/streams/state/DslStoreSuppliers.java: ## @@ -0,0 +1,41 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1398004582 ## streams/src/main/java/org/apache/kafka/streams/state/DslWindowParams.java: ## @@ -0,0 +1,129 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1398005349 ## streams/src/main/java/org/apache/kafka/streams/state/DslWindowParams.java: ## @@ -0,0 +1,129 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1398006947 ## streams/src/test/java/org/apache/kafka/streams/StreamsConfigTest.java: ## @@ -1038,13 +1038,15 @@ public void shouldThrowConfigExceptionWhenOptimizationConfigN

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1398012157 ## streams/src/main/java/org/apache/kafka/streams/kstream/internals/OuterStreamJoinStoreFactory.java: ## @@ -0,0 +1,188 @@ +/* + * Licensed to the Apache Software

[PR] MINOR: Change test logging capture to per-test, reducing jenkins truncation [kafka]

2023-11-17 Thread via GitHub
gharris1727 opened a new pull request, #14795: URL: https://github.com/apache/kafka/pull/14795 Jenkins truncates stdout/stderr from tests which exceed 100,000 bytes. This truncation is computed once per-suite, meaning that each suite gets a 100kb budget for logs, and suites that log too muc

Re: [PR] KAFKA-15836: KafkaConsumer subscribes to multiple topics does not respect max.poll.records [kafka]

2023-11-17 Thread via GitHub
jolshan commented on code in PR #14789: URL: https://github.com/apache/kafka/pull/14789#discussion_r1398026217 ## clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java: ## @@ -1124,6 +1126,62 @@ public void testFetchMaxPollRecords() { assert

Re: [PR] KAFKA-15836: KafkaConsumer subscribes to multiple topics does not respect max.poll.records [kafka]

2023-11-17 Thread via GitHub
jolshan commented on code in PR #14789: URL: https://github.com/apache/kafka/pull/14789#discussion_r1398026217 ## clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java: ## @@ -1124,6 +1126,62 @@ public void testFetchMaxPollRecords() { assert

Re: [PR] KAFKA-15836: KafkaConsumer subscribes to multiple topics does not respect max.poll.records [kafka]

2023-11-17 Thread via GitHub
jolshan commented on code in PR #14789: URL: https://github.com/apache/kafka/pull/14789#discussion_r1398026217 ## clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java: ## @@ -1124,6 +1126,62 @@ public void testFetchMaxPollRecords() { assert

Re: [PR] KAFKA-15629: Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery [kafka]

2023-11-17 Thread via GitHub
hanyuzheng7 commented on PR #14570: URL: https://github.com/apache/kafka/pull/14570#issuecomment-1817349035 @mjsax I have already updated the javadoc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] KAFKA-15174: Ensure CommitAsync propagate the exception to the user [kafka]

2023-11-17 Thread via GitHub
philipnee commented on PR #14680: URL: https://github.com/apache/kafka/pull/14680#issuecomment-1817393641 Thanks @lucasbru -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] MINOR: Expose earliest local timestamp via the GetOffsetShell [kafka]

2023-11-17 Thread via GitHub
divijvaidya commented on PR #14788: URL: https://github.com/apache/kafka/pull/14788#issuecomment-1817397316 This is a public facing change. Is this part of a KIP? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL