[GitHub] [kafka] cmccabe opened a new pull request, #14010: KAFKA-15183: Add more controller, loader, snapshot emitter metrics

cmccabe opened a new pull request, #14010: URL: https://github.com/apache/kafka/pull/14010 Implement some of the metrics from KIP-938: Add more metrics for measuring KRaft performance. Add these metrics to QuorumControllerMetrics:

[GitHub] [kafka] vamossagar12 commented on pull request #11433: KAFKA-13295: Avoiding Transation timeouts arising due to long restora…

vamossagar12 commented on PR #11433: URL: https://github.com/apache/kafka/pull/11433#issuecomment-1633559424 @mjsax thanks for notifying and that's ok :D Even I am not too up to date with the latest developments on Streams so in a way I dont think I was able to do full justification to

[GitHub] [kafka] yashmayya commented on a diff in pull request #14003: KAFKA-15182: Normalize source connector offsets before invoking SourceConnector::alterOffsets

yashmayya commented on code in PR #14003: URL: https://github.com/apache/kafka/pull/14003#discussion_r1261985349 ## connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java: ## @@ -1608,6 +1612,32 @@ void modifySourceConnectorOffsets(String connName,

[GitHub] [kafka] yashmayya commented on a diff in pull request #13945: KAFKA-15121: Implement the alterOffsets method in the FileStreamSourceConnector and the FileStreamSinkConnector

yashmayya commented on code in PR #13945: URL: https://github.com/apache/kafka/pull/13945#discussion_r1261948519 ## connect/file/src/main/java/org/apache/kafka/connect/file/FileStreamSourceConnector.java: ## @@ -101,4 +105,50 @@ public ExactlyOnceSupport exactlyOnceSupport(Map

[GitHub] [kafka] yashmayya commented on a diff in pull request #13945: KAFKA-15121: Implement the alterOffsets method in the FileStreamSourceConnector and the FileStreamSinkConnector

yashmayya commented on code in PR #13945: URL: https://github.com/apache/kafka/pull/13945#discussion_r1261948519 ## connect/file/src/main/java/org/apache/kafka/connect/file/FileStreamSourceConnector.java: ## @@ -101,4 +105,50 @@ public ExactlyOnceSupport exactlyOnceSupport(Map

[jira] [Commented] (KAFKA-14912) Introduce a configuration for remote index cache size, preferably a dynamic config.

[ https://issues.apache.org/jira/browse/KAFKA-14912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742624#comment-17742624 ] hudeqi commented on KAFKA-14912: Hi [~divijvaidya] [~manyanda] I am also interested, if you are busy

[GitHub] [kafka] mjsax commented on a diff in pull request #13996: KAFKA-15022: [2/N] introduce graph to compute min cost

mjsax commented on code in PR #13996: URL: https://github.com/apache/kafka/pull/13996#discussion_r1261835084 ## streams/src/main/java/org/apache/kafka/streams/processor/internals/assignment/Graph.java: ## @@ -0,0 +1,363 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [kafka] hni61223 opened a new pull request, #14009: MINOR: Add dual write offset metric

hni61223 opened a new pull request, #14009: URL: https://github.com/apache/kafka/pull/14009 - Add dual write offset metric for ZK migration - Tested in jconsole *More detailed description of your change, if necessary. The PR title and PR message become the squashed commit

[jira] [Commented] (KAFKA-15169) Add tests for RemoteIndexCache

[ https://issues.apache.org/jira/browse/KAFKA-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742602#comment-17742602 ] Lan Ding commented on KAFKA-15169: -- Sure, I will add tests for RemoteIndexCache in days. > Add tests

[jira] [Assigned] (KAFKA-15169) Add tests for RemoteIndexCache

[ https://issues.apache.org/jira/browse/KAFKA-15169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lan Ding reassigned KAFKA-15169: Assignee: Lan Ding > Add tests for RemoteIndexCache > -- > >

[GitHub] [kafka] junrao commented on a diff in pull request #13947: KAFKA-15130: Delete remote segments when delete a topic

junrao commented on code in PR #13947: URL: https://github.com/apache/kafka/pull/13947#discussion_r1261801130 ## core/src/main/scala/kafka/server/ReplicaManager.scala: ## @@ -500,11 +504,13 @@ class ReplicaManager(val config: KafkaConfig, // Delete log and

[GitHub] [kafka] mjsax closed pull request #11433: KAFKA-13295: Avoiding Transation timeouts arising due to long restora…

mjsax closed pull request #11433: KAFKA-13295: Avoiding Transation timeouts arising due to long restora… URL: https://github.com/apache/kafka/pull/11433 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[jira] [Resolved] (KAFKA-13295) Long restoration times for new tasks can lead to transaction timeouts

[ https://issues.apache.org/jira/browse/KAFKA-13295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-13295. - Resolution: Fixed With the new restore-thread, this issue should be resolved

[jira] [Updated] (KAFKA-15184) New consumer internals refactoring and clean up

[ https://issues.apache.org/jira/browse/KAFKA-15184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kirk True updated KAFKA-15184: -- Description: Minor refactoring of the new consumer internals including introduction of the

[GitHub] [kafka] hni61223 closed pull request #14007: MINOR: Add metrics for ZK migration

hni61223 closed pull request #14007: MINOR: Add metrics for ZK migration URL: https://github.com/apache/kafka/pull/14007 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[jira] [Created] (KAFKA-15184) New consumer internals refactoring and clean up

Kirk True created KAFKA-15184: - Summary: New consumer internals refactoring and clean up Key: KAFKA-15184 URL: https://issues.apache.org/jira/browse/KAFKA-15184 Project: Kafka Issue Type: Task

[GitHub] [kafka] philipnee commented on a diff in pull request #13797: KAFKA-14950: implement assign() and assignment()

philipnee commented on code in PR #13797: URL: https://github.com/apache/kafka/pull/13797#discussion_r1261798307 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/PrototypeAsyncConsumer.java: ## @@ -522,7 +525,35 @@ public void subscribe(Collection topics,

[GitHub] [kafka] junrao commented on pull request #1864: KAFKA-4177: Remove ThrottledReplicationRateLimit from Server Config

junrao commented on PR #1864: URL: https://github.com/apache/kafka/pull/1864#issuecomment-1633289594 @nkostoulas : Since this requires a config change, it will need a KIP. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [kafka] junrao commented on a diff in pull request #13797: KAFKA-14950: implement assign() and assignment()

junrao commented on code in PR #13797: URL: https://github.com/apache/kafka/pull/13797#discussion_r1261763698 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/PrototypeAsyncConsumer.java: ## @@ -522,7 +525,35 @@ public void subscribe(Collection topics,

[GitHub] [kafka] jolshan commented on a diff in pull request #13991: KAFKA-14462; [23/23] Wire GroupCoordinatorService in BrokerServer

jolshan commented on code in PR #13991: URL: https://github.com/apache/kafka/pull/13991#discussion_r1261758965 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/util/SystemTimerReaper.java: ## @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software

[GitHub] [kafka] hni61223 opened a new pull request, #14007: KMETA-988: Add metrics for ZK migration

hni61223 opened a new pull request, #14007: URL: https://github.com/apache/kafka/pull/14007 - Add dual write offset metric for ZK migration - Tested in jconsole *More detailed description of your change, if necessary. The PR title and PR message become the squashed commit

[GitHub] [kafka] ruslankrivoshein commented on pull request #13562: KAFKA-14581: Moving GetOffsetShell to tools

ruslankrivoshein commented on PR #13562: URL: https://github.com/apache/kafka/pull/13562#issuecomment-1633242202 @fvaleri, I worked a bit after your comments and still have a few questions. First, tell me, please, how you prepare your environment to run `$ bin/kafka-get-offsets.sh`. Is

[GitHub] [kafka] mumrah closed pull request #14006: MINOR Add metrics for ZK migration

mumrah closed pull request #14006: MINOR Add metrics for ZK migration URL: https://github.com/apache/kafka/pull/14006 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[GitHub] [kafka] jeffkbkim commented on a diff in pull request #13991: KAFKA-14462; [23/23] Wire GroupCoordinatorService in BrokerServer

jeffkbkim commented on code in PR #13991: URL: https://github.com/apache/kafka/pull/13991#discussion_r1261730522 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/util/SystemTimerReaper.java: ## @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software

[jira] [Comment Edited] (KAFKA-14912) Introduce a configuration for remote index cache size, preferably a dynamic config.

[ https://issues.apache.org/jira/browse/KAFKA-14912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742554#comment-17742554 ] Manyanda Chitimbo edited comment on KAFKA-14912 at 7/12/23 8:35 PM:

[jira] [Assigned] (KAFKA-14912) Introduce a configuration for remote index cache size, preferably a dynamic config.

[ https://issues.apache.org/jira/browse/KAFKA-14912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manyanda Chitimbo reassigned KAFKA-14912: - Assignee: Manyanda Chitimbo (was: Kamal Chandraprakash) > Introduce a

[jira] [Commented] (KAFKA-14912) Introduce a configuration for remote index cache size, preferably a dynamic config.

[ https://issues.apache.org/jira/browse/KAFKA-14912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742554#comment-17742554 ] Manyanda Chitimbo commented on KAFKA-14912: --- Hi [~divijvaidya] Yes, I'll be interested in

[GitHub] [kafka] jeffkbkim commented on a diff in pull request #13870: KAFKA-14500; [5/N] Implement JoinGroup protocol in new GroupCoordinator

jeffkbkim commented on code in PR #13870: URL: https://github.com/apache/kafka/pull/13870#discussion_r1259040832 ## group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java: ## @@ -228,14 +261,21 @@ public List build(TopicsImage

[GitHub] [kafka] hni61223 opened a new pull request, #14006: KMETA-988: Add metrics for ZK migration

hni61223 opened a new pull request, #14006: URL: https://github.com/apache/kafka/pull/14006 - Add dual write offset metric for ZK migration - Tested in jconsole *More detailed description of your change, if necessary. The PR title and PR message become the squashed commit

[jira] [Updated] (KAFKA-14912) Introduce a configuration for remote index cache size, preferably a dynamic config.

[ https://issues.apache.org/jira/browse/KAFKA-14912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Divij Vaidya updated KAFKA-14912: - Description: Context: We need to make the 1024 value here [1] as dynamically configurable [1]

[jira] [Commented] (KAFKA-14912) Introduce a configuration for remote index cache size, preferably a dynamic config.

[ https://issues.apache.org/jira/browse/KAFKA-14912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742550#comment-17742550 ] Divij Vaidya commented on KAFKA-14912: -- [~hudeqi] [~manyanda] [~ivanyu] are you folks interested in

[GitHub] [kafka] gharris1727 commented on pull request #13971: KAFKA-15150: Add ServiceLoaderScanner implementation

gharris1727 commented on PR #13971: URL: https://github.com/apache/kafka/pull/13971#issuecomment-1633140141 @C0urante Could you review this? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [kafka] gharris1727 commented on pull request #13313: KAFKA-14760: Move ThroughputThrottler from tools to clients, remove tools dependency from connect-runtime

gharris1727 commented on PR #13313: URL: https://github.com/apache/kafka/pull/13313#issuecomment-1633138815 Hey @ijuma Do you have the system test results for this branch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

[GitHub] [kafka] C0urante commented on a diff in pull request #13945: KAFKA-15121: Implement the alterOffsets method in the FileStreamSourceConnector and the FileStreamSinkConnector

C0urante commented on code in PR #13945: URL: https://github.com/apache/kafka/pull/13945#discussion_r1261646923 ## connect/file/src/main/java/org/apache/kafka/connect/file/FileStreamSourceConnector.java: ## @@ -101,4 +105,50 @@ public ExactlyOnceSupport exactlyOnceSupport(Map

[GitHub] [kafka] C0urante commented on a diff in pull request #13945: KAFKA-15121: Implement the alterOffsets method in the FileStreamSourceConnector and the FileStreamSinkConnector

C0urante commented on code in PR #13945: URL: https://github.com/apache/kafka/pull/13945#discussion_r1261646923 ## connect/file/src/main/java/org/apache/kafka/connect/file/FileStreamSourceConnector.java: ## @@ -101,4 +105,50 @@ public ExactlyOnceSupport exactlyOnceSupport(Map

[GitHub] [kafka] C0urante commented on pull request #14005: KAFKA-15177: Implement KIP-875 SourceConnector::alterOffset API in MirrorMaker 2 connectors

C0urante commented on PR #14005: URL: https://github.com/apache/kafka/pull/14005#issuecomment-1633085269 @yashmayya do you have time to take a look at this? Feel like it may be useful to you with the work we were discussing on https://github.com/apache/kafka/pull/13945 about how we want to

[GitHub] [kafka] gharris1727 merged pull request #13992: MINOR:Avoid slow Set.removeAll(List) in MirrorSourceConnector

gharris1727 merged PR #13992: URL: https://github.com/apache/kafka/pull/13992 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[GitHub] [kafka] C0urante commented on a diff in pull request #14005: KAFKA-15177: Implement KIP-875 SourceConnector::alterOffset API in MirrorMaker 2 connectors

C0urante commented on code in PR #14005: URL: https://github.com/apache/kafka/pull/14005#discussion_r1261630589 ## connect/runtime/src/test/java/org/apache/kafka/connect/util/clusters/EmbeddedKafkaCluster.java: ## @@ -597,7 +596,9 @@ private Set listPartitions(

[GitHub] [kafka] gharris1727 commented on pull request #13992: MINOR:Avoid slow Set.removeAll(List) in MirrorSourceConnector

gharris1727 commented on PR #13992: URL: https://github.com/apache/kafka/pull/13992#issuecomment-1633079621 The flaky CI failures look unrelated and the mirror unit tests pass locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [kafka] C0urante opened a new pull request, #14005: KAFKA-15177: Implement KIP-875 SourceConnector::alterOffset API in MirrorMaker 2 connectors

C0urante opened a new pull request, #14005: URL: https://github.com/apache/kafka/pull/14005 [Jira](https://issues.apache.org/jira/browse/KAFKA-15177), [KIP-875](https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect) ### Summary

[GitHub] [kafka] kamalcph opened a new pull request, #14004: KAFKA-15168: Handle overlapping remote log segments in RemoteLogMetadata cache

kamalcph opened a new pull request, #14004: URL: https://github.com/apache/kafka/pull/14004 *More detailed description of your change, if necessary. The PR title and PR message become the squashed commit message, so use a separate comment to ping reviewers.* *Summary of

[GitHub] [kafka] kamalcph commented on a diff in pull request #13837: KAFKA-9564: Local Tiered Storage implementation for Remote Storage Manager

kamalcph commented on code in PR #13837: URL: https://github.com/apache/kafka/pull/13837#discussion_r1261602111 ## storage/src/test/java/org/apache/kafka/server/log/remote/storage/LocalTieredStorage.java: ## @@ -0,0 +1,564 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [kafka] kamalcph commented on a diff in pull request #13837: KAFKA-9564: Local Tiered Storage implementation for Remote Storage Manager

kamalcph commented on code in PR #13837: URL: https://github.com/apache/kafka/pull/13837#discussion_r1261600532 ## storage/src/test/java/org/apache/kafka/server/log/remote/storage/LocalTieredStorageHistory.java: ## @@ -0,0 +1,115 @@ +/* + * Licensed to the Apache Software

[GitHub] [kafka] kamalcph commented on a diff in pull request #13837: KAFKA-9564: Local Tiered Storage implementation for Remote Storage Manager

kamalcph commented on code in PR #13837: URL: https://github.com/apache/kafka/pull/13837#discussion_r1261592637 ## storage/src/test/java/org/apache/kafka/server/log/remote/storage/LocalTieredStorage.java: ## @@ -0,0 +1,578 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [kafka] kamalcph commented on a diff in pull request #13837: KAFKA-9564: Local Tiered Storage implementation for Remote Storage Manager

kamalcph commented on code in PR #13837: URL: https://github.com/apache/kafka/pull/13837#discussion_r1261590988 ## storage/src/test/java/org/apache/kafka/server/log/remote/storage/LocalTieredStorage.java: ## @@ -0,0 +1,578 @@ +/* + * Licensed to the Apache Software Foundation

[jira] [Updated] (KAFKA-14993) Improve TransactionIndex instance handling while copying to and fetching from RSM.

[ https://issues.apache.org/jira/browse/KAFKA-14993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash updated KAFKA-14993: - Description: RSM should throw a ResourceNotFoundException if it does not have

[GitHub] [kafka] kamalcph commented on a diff in pull request #13837: KAFKA-9564: Local Tiered Storage implementation for Remote Storage Manager

kamalcph commented on code in PR #13837: URL: https://github.com/apache/kafka/pull/13837#discussion_r1261588861 ## storage/src/test/java/org/apache/kafka/server/log/remote/storage/LocalTieredStorage.java: ## @@ -0,0 +1,578 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [kafka] kamalcph commented on a diff in pull request #13837: KAFKA-9564: Local Tiered Storage implementation for Remote Storage Manager

kamalcph commented on code in PR #13837: URL: https://github.com/apache/kafka/pull/13837#discussion_r1261587596 ## storage/src/test/java/org/apache/kafka/server/log/remote/storage/LocalTieredStorage.java: ## @@ -0,0 +1,578 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [kafka] jolshan commented on a diff in pull request #13991: KAFKA-14462; [23/23] Wire GroupCoordinatorService in BrokerServer

jolshan commented on code in PR #13991: URL: https://github.com/apache/kafka/pull/13991#discussion_r1261578373 ## group-coordinator/src/test/java/org/apache/kafka/coordinator/group/util/SystemTimerReaperTest.java: ## @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software

[GitHub] [kafka] kamalcph commented on a diff in pull request #13837: KAFKA-9564: Local Tiered Storage implementation for Remote Storage Manager

kamalcph commented on code in PR #13837: URL: https://github.com/apache/kafka/pull/13837#discussion_r1261578158 ## storage/src/test/java/org/apache/kafka/server/log/remote/storage/LocalTieredStorage.java: ## @@ -0,0 +1,578 @@ +/* + * Licensed to the Apache Software Foundation

[jira] [Created] (KAFKA-15183) Add more controller, loader, snapshot emitter metrics

Colin McCabe created KAFKA-15183: Summary: Add more controller, loader, snapshot emitter metrics Key: KAFKA-15183 URL: https://issues.apache.org/jira/browse/KAFKA-15183 Project: Kafka Issue

[GitHub] [kafka] kamalcph commented on a diff in pull request #13837: KAFKA-9564: Local Tiered Storage implementation for Remote Storage Manager

kamalcph commented on code in PR #13837: URL: https://github.com/apache/kafka/pull/13837#discussion_r1261567166 ## storage/src/test/java/org/apache/kafka/server/log/remote/storage/LocalTieredStorage.java: ## @@ -0,0 +1,578 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [kafka] jolshan commented on a diff in pull request #13991: KAFKA-14462; [23/23] Wire GroupCoordinatorService in BrokerServer

jolshan commented on code in PR #13991: URL: https://github.com/apache/kafka/pull/13991#discussion_r1261559993 ## group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java: ## @@ -2360,6 +2360,19 @@ public void testGroupIdsByTopics() {

[GitHub] [kafka] kamalcph commented on a diff in pull request #13837: KAFKA-9564: Local Tiered Storage implementation for Remote Storage Manager

kamalcph commented on code in PR #13837: URL: https://github.com/apache/kafka/pull/13837#discussion_r1261560074 ## storage/src/test/java/org/apache/kafka/server/log/remote/storage/LocalTieredStorage.java: ## @@ -0,0 +1,564 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] [kafka] jolshan commented on a diff in pull request #13991: KAFKA-14462; [23/23] Wire GroupCoordinatorService in BrokerServer

jolshan commented on code in PR #13991: URL: https://github.com/apache/kafka/pull/13991#discussion_r1261559993 ## group-coordinator/src/test/java/org/apache/kafka/coordinator/group/GroupMetadataManagerTest.java: ## @@ -2360,6 +2360,19 @@ public void testGroupIdsByTopics() {

[GitHub] [kafka] jolshan commented on a diff in pull request #13991: KAFKA-14462; [23/23] Wire GroupCoordinatorService in BrokerServer

jolshan commented on code in PR #13991: URL: https://github.com/apache/kafka/pull/13991#discussion_r1261554932 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/util/SystemTimerReaper.java: ## @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software

[GitHub] [kafka] jolshan commented on a diff in pull request #13991: KAFKA-14462; [23/23] Wire GroupCoordinatorService in BrokerServer

jolshan commented on code in PR #13991: URL: https://github.com/apache/kafka/pull/13991#discussion_r1261554932 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/util/SystemTimerReaper.java: ## @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software

[jira] [Commented] (KAFKA-14912) Introduce a configuration for remote index cache size, preferably a dynamic config.

[ https://issues.apache.org/jira/browse/KAFKA-14912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742508#comment-17742508 ] Kamal Chandraprakash commented on KAFKA-14912: -- I'm not working on this one currently. We

[GitHub] [kafka] divijvaidya commented on pull request #13997: KAFKA-15180: Generalize integration tests to change use of KafkaConsumer to Consumer

divijvaidya commented on PR #13997: URL: https://github.com/apache/kafka/pull/13997#issuecomment-1632968446 Unrelated test failures: ``` [Build / JDK 11 and Scala 2.13 /

[GitHub] [kafka] jolshan commented on a diff in pull request #13991: KAFKA-14462; [23/23] Wire GroupCoordinatorService in BrokerServer

jolshan commented on code in PR #13991: URL: https://github.com/apache/kafka/pull/13991#discussion_r1261521364 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/util/SystemTimerReaper.java: ## @@ -0,0 +1,81 @@ +/* + * Licensed to the Apache Software

[GitHub] [kafka] yashmayya commented on a diff in pull request #13945: KAFKA-15121: Implement the alterOffsets method in the FileStreamSourceConnector and the FileStreamSinkConnector

yashmayya commented on code in PR #13945: URL: https://github.com/apache/kafka/pull/13945#discussion_r1261515978 ## connect/file/src/main/java/org/apache/kafka/connect/file/FileStreamSourceConnector.java: ## @@ -101,4 +105,50 @@ public ExactlyOnceSupport exactlyOnceSupport(Map

[GitHub] [kafka] yashmayya opened a new pull request, #14003: KAFKA-15182: Normalize source connector offsets before invoking SourceConnector::alterOffsets

yashmayya opened a new pull request, #14003: URL: https://github.com/apache/kafka/pull/14003 - From https://issues.apache.org/jira/browse/KAFKA-15182: >See discussion [here](https://github.com/apache/kafka/pull/13945#discussion_r1260946148) > > TLDR: When users attempt to

[jira] [Created] (KAFKA-15182) Normalize offsets before invoking SourceConnector::alterOffsets

Yash Mayya created KAFKA-15182: -- Summary: Normalize offsets before invoking SourceConnector::alterOffsets Key: KAFKA-15182 URL: https://issues.apache.org/jira/browse/KAFKA-15182 Project: Kafka

[GitHub] [kafka] gharris1727 commented on pull request #13853: KAFKA-15088: Fixing Incorrect Reference Usage in Connector State Changes

gharris1727 commented on PR #13853: URL: https://github.com/apache/kafka/pull/13853#issuecomment-1632939211 Hey @daehokimm Thanks for the PR! I think that there is some semantic difference between `AbstractStatus.State` and `ConnectorStatus.State`, and certainly between

[GitHub] [kafka] philipnee commented on pull request #13956: MINOR: Remove thread leak from ConsumerBounceTest

philipnee commented on PR #13956: URL: https://github.com/apache/kafka/pull/13956#issuecomment-1632936434 Hey @divijvaidya - I actually tried to fix this probably a year ago, and my guess was some of the consumer didn't close cleanly and caused leak. -- This is an automated message from

[GitHub] [kafka] jolshan commented on a diff in pull request #13991: KAFKA-14462; [23/23] Wire GroupCoordinatorService in BrokerServer

jolshan commented on code in PR #13991: URL: https://github.com/apache/kafka/pull/13991#discussion_r1261496901 ## core/src/test/scala/unit/kafka/server/ConsumerGroupHeartbeatRequestTest.scala: ## @@ -54,18 +58,92 @@ class ConsumerGroupHeartbeatRequestTest(cluster:

[GitHub] [kafka] jolshan commented on a diff in pull request #13991: KAFKA-14462; [23/23] Wire GroupCoordinatorService in BrokerServer

jolshan commented on code in PR #13991: URL: https://github.com/apache/kafka/pull/13991#discussion_r1261492024 ## core/src/test/scala/unit/kafka/server/ConsumerGroupHeartbeatRequestTest.scala: ## @@ -54,18 +58,92 @@ class ConsumerGroupHeartbeatRequestTest(cluster:

[GitHub] [kafka] jolshan commented on a diff in pull request #13991: KAFKA-14462; [23/23] Wire GroupCoordinatorService in BrokerServer

jolshan commented on code in PR #13991: URL: https://github.com/apache/kafka/pull/13991#discussion_r1261492024 ## core/src/test/scala/unit/kafka/server/ConsumerGroupHeartbeatRequestTest.scala: ## @@ -54,18 +58,92 @@ class ConsumerGroupHeartbeatRequestTest(cluster:

[GitHub] [kafka] jolshan commented on a diff in pull request #13991: KAFKA-14462; [23/23] Wire GroupCoordinatorService in BrokerServer

jolshan commented on code in PR #13991: URL: https://github.com/apache/kafka/pull/13991#discussion_r1261475273 ## core/src/main/scala/kafka/server/KafkaConfig.scala: ## @@ -175,7 +176,7 @@ object Defaults { val ConsumerGroupMinHeartbeatIntervalMs = 5000 val

[GitHub] [kafka] vamossagar12 commented on pull request #13158: KAFKA-14647: Moving TopicFilter to server-common/utils

vamossagar12 commented on PR #13158: URL: https://github.com/apache/kafka/pull/13158#issuecomment-1632883931 > I took a look at the changes and they look fine. If I understand correctly we're moving these classes to server-common while we have to keep MirrorMaker. Then in Kafka 4.0 (and

[GitHub] [kafka] jolshan commented on a diff in pull request #13787: KAFKA-14884: Include check transaction is still ongoing right before append (take 2)

jolshan commented on code in PR #13787: URL: https://github.com/apache/kafka/pull/13787#discussion_r1261444204 ## clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java: ## @@ -3057,6 +3057,56 @@ public void

[GitHub] [kafka] jolshan commented on a diff in pull request #13787: KAFKA-14884: Include check transaction is still ongoing right before append (take 2)

jolshan commented on code in PR #13787: URL: https://github.com/apache/kafka/pull/13787#discussion_r1261443466 ## clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java: ## @@ -3057,6 +3057,56 @@ public void

[GitHub] [kafka] C0urante commented on a diff in pull request #13945: KAFKA-15121: Implement the alterOffsets method in the FileStreamSourceConnector and the FileStreamSinkConnector

C0urante commented on code in PR #13945: URL: https://github.com/apache/kafka/pull/13945#discussion_r1261413255 ## connect/file/src/main/java/org/apache/kafka/connect/file/FileStreamSourceConnector.java: ## @@ -101,4 +105,50 @@ public ExactlyOnceSupport exactlyOnceSupport(Map

[GitHub] [kafka] nkostoulas opened a new pull request, #14002: Move throttling rate to KafkaConfig

nkostoulas opened a new pull request, #14002: URL: https://github.com/apache/kafka/pull/14002 *More detailed description of your change, if necessary. The PR title and PR message become the squashed commit message, so use a separate comment to ping reviewers.* *Summary of

[jira] [Updated] (KAFKA-14938) Flaky test org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest#testConnectorBoundary

[ https://issues.apache.org/jira/browse/KAFKA-14938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-14938: -- Fix Version/s: 3.5.2 > Flaky test >

[jira] [Updated] (KAFKA-14938) Flaky test org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest#testConnectorBoundary

[ https://issues.apache.org/jira/browse/KAFKA-14938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-14938: -- Fix Version/s: 3.4.2 > Flaky test >

[GitHub] [kafka] jolshan commented on a diff in pull request #13787: KAFKA-14884: Include check transaction is still ongoing right before append (take 2)

jolshan commented on code in PR #13787: URL: https://github.com/apache/kafka/pull/13787#discussion_r1261385072 ## clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java: ## @@ -3057,6 +3057,56 @@ public void

[GitHub] [kafka] jolshan commented on a diff in pull request #13787: KAFKA-14884: Include check transaction is still ongoing right before append (take 2)

jolshan commented on code in PR #13787: URL: https://github.com/apache/kafka/pull/13787#discussion_r1261385072 ## clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java: ## @@ -3057,6 +3057,56 @@ public void

[GitHub] [kafka] viktorsomogyi commented on a diff in pull request #13975: KAFKA-15161: Fix InvalidReplicationFactorException at connect startup

viktorsomogyi commented on code in PR #13975: URL: https://github.com/apache/kafka/pull/13975#discussion_r1261378645 ## core/src/main/scala/kafka/server/metadata/KRaftMetadataCache.scala: ## @@ -174,14 +174,25 @@ class KRaftMetadataCache(val brokerId: Int) extends

[GitHub] [kafka] viktorsomogyi commented on a diff in pull request #13975: KAFKA-15161: Fix InvalidReplicationFactorException at connect startup

viktorsomogyi commented on code in PR #13975: URL: https://github.com/apache/kafka/pull/13975#discussion_r1261378645 ## core/src/main/scala/kafka/server/metadata/KRaftMetadataCache.scala: ## @@ -174,14 +174,25 @@ class KRaftMetadataCache(val brokerId: Int) extends

[GitHub] [kafka] C0urante merged pull request #13646: KAFKA-14938: Fixing flaky test testConnectorBoundary

C0urante merged PR #13646: URL: https://github.com/apache/kafka/pull/13646 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

[jira] [Updated] (KAFKA-14938) Flaky test org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest#testConnectorBoundary

[ https://issues.apache.org/jira/browse/KAFKA-14938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-14938: -- Fix Version/s: 3.6.0 > Flaky test >

[GitHub] [kafka] nicktelford commented on a diff in pull request #13993: KAFKA-15178: Improve ConsumerCoordinator.poll perf

nicktelford commented on code in PR #13993: URL: https://github.com/apache/kafka/pull/13993#discussion_r1261346057 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java: ## @@ -119,6 +119,7 @@ public final class ConsumerCoordinator

[GitHub] [kafka] lucasbru opened a new pull request, #14001: Kafka Streams Threading: Exception handling

lucasbru opened a new pull request, #14001: URL: https://github.com/apache/kafka/pull/14001 Implements punctuation inside processing threads. The scheduler algorithm checks if a task that is not assigned currently can be punctuated, and returns it when a worker thread asks for the

[GitHub] [kafka] divijvaidya commented on a diff in pull request #13956: MINOR: Remove thread leak from ConsumerBounceTest

divijvaidya commented on code in PR #13956: URL: https://github.com/apache/kafka/pull/13956#discussion_r1261325930 ## core/src/test/scala/integration/kafka/api/AbstractConsumerTest.scala: ## @@ -346,6 +346,7 @@ abstract class AbstractConsumerTest extends BaseRequestTest {

[GitHub] [kafka] fvaleri commented on pull request #13158: KAFKA-14647: Moving TopicFilter to server-common/utils

fvaleri commented on PR #13158: URL: https://github.com/apache/kafka/pull/13158#issuecomment-1632708004 Yes, once the MirrorMaker1 dependency will be gone in 4.0.0, we can move them to tools. I added a note in [KAFKA-14705](https://issues.apache.org/jira/browse/KAFKA-14705) -- This is

[jira] [Updated] (KAFKA-14705) Tools cleanup for the next major release

[ https://issues.apache.org/jira/browse/KAFKA-14705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Federico Valeri updated KAFKA-14705: Description: We can use this task to track tools cleanup for the next major release

[GitHub] [kafka] nicktelford commented on a diff in pull request #13993: KAFKA-15178: Improve ConsumerCoordinator.poll perf

nicktelford commented on code in PR #13993: URL: https://github.com/apache/kafka/pull/13993#discussion_r1261318387 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java: ## @@ -119,6 +119,7 @@ public final class ConsumerCoordinator

[jira] [Updated] (KAFKA-14705) Tools cleanup for the next major release

[ https://issues.apache.org/jira/browse/KAFKA-14705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Federico Valeri updated KAFKA-14705: Description: We can use this task to track tools cleanup for the next major release

[GitHub] [kafka] mimaison commented on pull request #13158: KAFKA-14647: Moving TopicFilter to server-common/utils

mimaison commented on PR #13158: URL: https://github.com/apache/kafka/pull/13158#issuecomment-1632638912 I took a look at the changes and they look fine. If I understand correctly we're moving these classes to server-common while we have to keep MirrorMaker. Then in Kafka 4.0 (and assuming

[GitHub] [kafka] mumrah commented on pull request #13988: [KAFKA-15137] Do not log entire request payload in KRaftControllerChannelManager

mumrah commented on PR #13988: URL: https://github.com/apache/kafka/pull/13988#issuecomment-1632618714 Thanks @divijvaidya! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

[GitHub] [kafka] divijvaidya commented on pull request #13956: MINOR: Remove thread leak from ConsumerBounceTest

divijvaidya commented on PR #13956: URL: https://github.com/apache/kafka/pull/13956#issuecomment-1632593879 >Did you verify the thread leak by checking the threat dump? No, I found the leak from a very useful line in the tests I added recently. It prints the names of the leaked

[jira] [Created] (KAFKA-15181) Race condition on partition assigned to TopicBasedRemoteLogMetadataManager

Jorge Esteban Quilcate Otoya created KAFKA-15181: Summary: Race condition on partition assigned to TopicBasedRemoteLogMetadataManager Key: KAFKA-15181 URL:

[GitHub] [kafka] OmniaGM commented on a diff in pull request #13585: KAFKA-14737: Move kafka.utils.json to server-common

OmniaGM commented on code in PR #13585: URL: https://github.com/apache/kafka/pull/13585#discussion_r1261125892 ## server-common/src/main/java/org/apache/kafka/server/util/json/DecodeJson.java: ## @@ -0,0 +1,135 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] [kafka] divijvaidya commented on a diff in pull request #13944: KAFKA-14953: Add tiered storage related metrics

divijvaidya commented on code in PR #13944: URL: https://github.com/apache/kafka/pull/13944#discussion_r1261124589 ## core/src/main/scala/kafka/server/KafkaRequestHandler.scala: ## @@ -277,6 +277,12 @@ class BrokerTopicMetrics(name: Option[String]) {

[jira] [Commented] (KAFKA-15132) Implement disable & re-enablement for Tiered Storage

[ https://issues.apache.org/jira/browse/KAFKA-15132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742409#comment-17742409 ] Divij Vaidya commented on KAFKA-15132: -- Update - need more time. new ETA 10th July. > Implement

[jira] [Updated] (KAFKA-14915) Option to consume multiple partitions that have their data in remote storage for the target offsets.

[ https://issues.apache.org/jira/browse/KAFKA-14915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Divij Vaidya updated KAFKA-14915: - Description: Context: https://github.com/apache/kafka/pull/13535#discussion_r1171250580 >

[GitHub] [kafka] jeqo commented on a diff in pull request #13944: KAFKA-14953: Add tiered storage related metrics

jeqo commented on code in PR #13944: URL: https://github.com/apache/kafka/pull/13944#discussion_r1261114941 ## core/src/main/scala/kafka/server/KafkaRequestHandler.scala: ## @@ -277,6 +277,12 @@ class BrokerTopicMetrics(name: Option[String]) {

[jira] [Commented] (KAFKA-14912) Introduce a configuration for remote index cache size, preferably a dynamic config.

[ https://issues.apache.org/jira/browse/KAFKA-14912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17742407#comment-17742407 ] Divij Vaidya commented on KAFKA-14912: -- [~ckamal] if you aren't actively working on this, can we

[GitHub] [kafka] mimaison commented on a diff in pull request #13585: KAFKA-14737: Move kafka.utils.json to server-common

mimaison commented on code in PR #13585: URL: https://github.com/apache/kafka/pull/13585#discussion_r1261096752 ## server-common/src/main/java/org/apache/kafka/server/util/json/DecodeJson.java: ## @@ -0,0 +1,135 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

  1   2   >