Re: [PR] MINOR: Expose earliest local timestamp via the GetOffsetShell [kafka]

2023-11-17 Thread via GitHub
divijvaidya commented on PR #14788: URL: https://github.com/apache/kafka/pull/14788#issuecomment-1817397316 This is a public facing change. Is this part of a KIP? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] KAFKA-15174: Ensure CommitAsync propagate the exception to the user [kafka]

2023-11-17 Thread via GitHub
philipnee commented on PR #14680: URL: https://github.com/apache/kafka/pull/14680#issuecomment-1817393641 Thanks @lucasbru -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment.

Re: [PR] KAFKA-15629: Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery [kafka]

2023-11-17 Thread via GitHub
hanyuzheng7 commented on PR #14570: URL: https://github.com/apache/kafka/pull/14570#issuecomment-1817349035 @mjsax I have already updated the javadoc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] KAFKA-15836: KafkaConsumer subscribes to multiple topics does not respect max.poll.records [kafka]

2023-11-17 Thread via GitHub
jolshan commented on code in PR #14789: URL: https://github.com/apache/kafka/pull/14789#discussion_r1398026217 ## clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java: ## @@ -1124,6 +1126,62 @@ public void testFetchMaxPollRecords() { assert

Re: [PR] KAFKA-15836: KafkaConsumer subscribes to multiple topics does not respect max.poll.records [kafka]

2023-11-17 Thread via GitHub
jolshan commented on code in PR #14789: URL: https://github.com/apache/kafka/pull/14789#discussion_r1398026217 ## clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java: ## @@ -1124,6 +1126,62 @@ public void testFetchMaxPollRecords() { assert

Re: [PR] KAFKA-15836: KafkaConsumer subscribes to multiple topics does not respect max.poll.records [kafka]

2023-11-17 Thread via GitHub
jolshan commented on code in PR #14789: URL: https://github.com/apache/kafka/pull/14789#discussion_r1398026217 ## clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java: ## @@ -1124,6 +1126,62 @@ public void testFetchMaxPollRecords() { assert

[PR] MINOR: Change test logging capture to per-test, reducing jenkins truncation [kafka]

2023-11-17 Thread via GitHub
gharris1727 opened a new pull request, #14795: URL: https://github.com/apache/kafka/pull/14795 Jenkins truncates stdout/stderr from tests which exceed 100,000 bytes. This truncation is computed once per-suite, meaning that each suite gets a 100kb budget for logs, and suites that log too muc

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1398012157 ## streams/src/main/java/org/apache/kafka/streams/kstream/internals/OuterStreamJoinStoreFactory.java: ## @@ -0,0 +1,188 @@ +/* + * Licensed to the Apache Software

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1398006947 ## streams/src/test/java/org/apache/kafka/streams/StreamsConfigTest.java: ## @@ -1038,13 +1038,15 @@ public void shouldThrowConfigExceptionWhenOptimizationConfigN

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1398005349 ## streams/src/main/java/org/apache/kafka/streams/state/DslWindowParams.java: ## @@ -0,0 +1,129 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1398004582 ## streams/src/main/java/org/apache/kafka/streams/state/DslWindowParams.java: ## @@ -0,0 +1,129 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1398003882 ## streams/src/main/java/org/apache/kafka/streams/state/DslStoreSuppliers.java: ## @@ -0,0 +1,41 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1398003330 ## streams/src/main/java/org/apache/kafka/streams/state/DslSessionParams.java: ## @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1398002738 ## streams/src/main/java/org/apache/kafka/streams/state/DslSessionParams.java: ## @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1398002400 ## streams/src/main/java/org/apache/kafka/streams/state/BuiltInDslStoreSuppliers.java: ## @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1397992729 ## streams/src/main/java/org/apache/kafka/streams/state/BuiltInDslStoreSuppliers.java: ## @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Software Foundation (ASF)

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1397987652 ## streams/src/main/java/org/apache/kafka/streams/kstream/internals/StreamJoinedStoreFactory.java: ## @@ -0,0 +1,163 @@ +/* + * Licensed to the Apache Software Fou

Re: [PR] KAFKA-15215: [KIP-954] support custom DSL store providers [kafka]

2023-11-17 Thread via GitHub
ableegoldman commented on code in PR #14648: URL: https://github.com/apache/kafka/pull/14648#discussion_r1397982732 ## streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreFactory.java: ## @@ -66,6 +66,10 @@ default void configure(final StreamsConfig config) {

Re: [PR] KAFKA-15618: Kafka metrics collector and supporting classes (KIP-714) [kafka]

2023-11-17 Thread via GitHub
xvrl commented on code in PR #14620: URL: https://github.com/apache/kafka/pull/14620#discussion_r1397923196 ## clients/src/main/java/org/apache/kafka/common/telemetry/internals/KafkaMetricsCollector.java: ## @@ -0,0 +1,334 @@ +/* + * Licensed to the Apache Software Foundation (A

Re: [PR] KAFKA-15778 & KAFKA-15779: Implement metrics manager (KIP-714) [kafka]

2023-11-17 Thread via GitHub
junrao commented on code in PR #14699: URL: https://github.com/apache/kafka/pull/14699#discussion_r1397751986 ## core/src/main/java/kafka/server/ClientMetricsManager.java: ## @@ -30,17 +69,376 @@ public class ClientMetricsManager implements Closeable { private static fina

Re: [PR] KAFKA-15618: Kafka metrics collector and supporting classes (KIP-714) [kafka]

2023-11-17 Thread via GitHub
xvrl commented on code in PR #14620: URL: https://github.com/apache/kafka/pull/14620#discussion_r1397895468 ## clients/src/main/java/org/apache/kafka/common/telemetry/internals/KafkaMetricsCollector.java: ## @@ -0,0 +1,334 @@ +/* + * Licensed to the Apache Software Foundation (A

Re: [PR] KAFKA-15618: Kafka metrics collector and supporting classes (KIP-714) [kafka]

2023-11-17 Thread via GitHub
xvrl commented on code in PR #14620: URL: https://github.com/apache/kafka/pull/14620#discussion_r1397895468 ## clients/src/main/java/org/apache/kafka/common/telemetry/internals/KafkaMetricsCollector.java: ## @@ -0,0 +1,334 @@ +/* + * Licensed to the Apache Software Foundation (A

Re: [PR] KAFKA-15778 & KAFKA-15779: Implement metrics manager (KIP-714) [kafka]

2023-11-17 Thread via GitHub
apoorvmittal10 commented on PR #14699: URL: https://github.com/apache/kafka/pull/14699#issuecomment-1817091571 Thanks @junrao for leaving the comments, I have tried to address them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to Git

Re: [PR] KAFKA-15778 & KAFKA-15779: Implement metrics manager (KIP-714) [kafka]

2023-11-17 Thread via GitHub
apoorvmittal10 commented on code in PR #14699: URL: https://github.com/apache/kafka/pull/14699#discussion_r1397875576 ## core/src/test/java/kafka/metrics/ClientMetricsInstanceTest.java: ## @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or mor

Re: [PR] KAFKA-15778 & KAFKA-15779: Implement metrics manager (KIP-714) [kafka]

2023-11-17 Thread via GitHub
apoorvmittal10 commented on code in PR #14699: URL: https://github.com/apache/kafka/pull/14699#discussion_r1397873419 ## core/src/main/java/kafka/metrics/ClientMetricsReceiverPlugin.java: ## @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or m

Re: [PR] KAFKA-15778 & KAFKA-15779: Implement metrics manager (KIP-714) [kafka]

2023-11-17 Thread via GitHub
apoorvmittal10 commented on code in PR #14699: URL: https://github.com/apache/kafka/pull/14699#discussion_r1397871398 ## clients/src/main/java/org/apache/kafka/common/requests/PushTelemetryRequest.java: ## @@ -60,17 +62,47 @@ public PushTelemetryRequest(PushTelemetryRequestData

Re: [PR] MINOR: Fix unstable sorting in AssignmentsManagerTest [kafka]

2023-11-17 Thread via GitHub
soarez commented on PR #14794: URL: https://github.com/apache/kafka/pull/14794#issuecomment-1817065152 @dajac please take a look -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

[PR] MINOR: Fix unstable sorting in AssignmentsManagerTest [kafka]

2023-11-17 Thread via GitHub
soarez opened a new pull request, #14794: URL: https://github.com/apache/kafka/pull/14794 Building AssignReplicasToDirsRequestData relies on iteration over Map entries, which can result in different sorting order. The order does not matter to the semantics of the request, but it can cause i

Re: [PR] KAFKA-15629: Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery [kafka]

2023-11-17 Thread via GitHub
hanyuzheng7 commented on code in PR #14570: URL: https://github.com/apache/kafka/pull/14570#discussion_r1397839267 ## streams/src/main/java/org/apache/kafka/streams/query/TimestampedRangeQuery.java: ## @@ -0,0 +1,130 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] KAFKA-15778 & KAFKA-15779: Implement metrics manager (KIP-714) [kafka]

2023-11-17 Thread via GitHub
junrao commented on code in PR #14699: URL: https://github.com/apache/kafka/pull/14699#discussion_r1397714746 ## clients/src/main/java/org/apache/kafka/common/requests/PushTelemetryRequest.java: ## @@ -60,17 +62,47 @@ public PushTelemetryRequest(PushTelemetryRequestData data, s

Re: [PR] KAFKA-15778 & KAFKA-15779: Implement metrics manager (KIP-714) [kafka]

2023-11-17 Thread via GitHub
junrao commented on code in PR #14699: URL: https://github.com/apache/kafka/pull/14699#discussion_r1397714746 ## clients/src/main/java/org/apache/kafka/common/requests/PushTelemetryRequest.java: ## @@ -60,17 +62,47 @@ public PushTelemetryRequest(PushTelemetryRequestData data, s

Re: [PR] KAFKA-15778 & KAFKA-15779: Implement metrics manager (KIP-714) [kafka]

2023-11-17 Thread via GitHub
junrao commented on code in PR #14699: URL: https://github.com/apache/kafka/pull/14699#discussion_r1397714746 ## clients/src/main/java/org/apache/kafka/common/requests/PushTelemetryRequest.java: ## @@ -60,17 +62,47 @@ public PushTelemetryRequest(PushTelemetryRequestData data, s

Re: [PR] KAFKA-15778 & KAFKA-15779: Implement metrics manager (KIP-714) [kafka]

2023-11-17 Thread via GitHub
apoorvmittal10 commented on code in PR #14699: URL: https://github.com/apache/kafka/pull/14699#discussion_r1397795152 ## core/src/main/java/kafka/metrics/ClientMetricsReceiverPlugin.java: ## @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or m

Re: [PR] KAFKA-15417 flip joinSpuriousLookBackTimeMs and emit non-joined items [kafka]

2023-11-17 Thread via GitHub
lihaosky commented on code in PR #14426: URL: https://github.com/apache/kafka/pull/14426#discussion_r1397715022 ## streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamKStreamJoin.java: ## @@ -104,20 +103,16 @@ public void init(final ProcessorContext context)

Re: [PR] KAFKA-15778 & KAFKA-15779: Implement metrics manager (KIP-714) [kafka]

2023-11-17 Thread via GitHub
junrao commented on code in PR #14699: URL: https://github.com/apache/kafka/pull/14699#discussion_r1397714746 ## clients/src/main/java/org/apache/kafka/common/requests/PushTelemetryRequest.java: ## @@ -60,17 +62,47 @@ public PushTelemetryRequest(PushTelemetryRequestData data, s

Re: [PR] KAFKA-15836: KafkaConsumer subscribes to multiple topics does not respect max.poll.records [kafka]

2023-11-17 Thread via GitHub
AndrewJSchofield commented on PR #14789: URL: https://github.com/apache/kafka/pull/14789#issuecomment-1816929338 Test failures reviewed and unrelated to the changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] [LI-HOTFIX] Enable register watch when getting children for federated topic namespaces [kafka]

2023-11-17 Thread via GitHub
kehuum closed pull request #14793: [LI-HOTFIX] Enable register watch when getting children for federated topic namespaces URL: https://github.com/apache/kafka/pull/14793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[PR] [LI-HOTFIX] Enable register watch when getting children for federated topic namespaces [kafka]

2023-11-17 Thread via GitHub
kehuum opened a new pull request, #14793: URL: https://github.com/apache/kafka/pull/14793 *More detailed description of your change, if necessary. The PR title and PR message become the squashed commit message, so use a separate comment to ping reviewers.* This pr sets registerWat

Re: [PR] KAFKA-15826: Close consumer when sink task is cancelled [kafka]

2023-11-17 Thread via GitHub
gharris1727 commented on PR #14762: URL: https://github.com/apache/kafka/pull/14762#issuecomment-1816905171 > What was the conclusion here? Is triggering wakeup on a separate thread (through WorkerSinkTask::stop) before also eventually closing the consumer on a separate thread (through Work

Re: [PR] KAFKA-15484: General Rack Aware Assignor [kafka]

2023-11-17 Thread via GitHub
rreddy-22 commented on code in PR #14481: URL: https://github.com/apache/kafka/pull/14481#discussion_r1397683148 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GeneralUniformAssignmentBuilder.java: ## @@ -14,17 +14,891 @@ * See the License for t

[PR] MINOR: Log the ZK dual-write time [kafka]

2023-11-17 Thread via GitHub
mumrah opened a new pull request, #14792: URL: https://github.com/apache/kafka/pull/14792 We already capture the ZK dual write timings as metrics. This patch includes those timings in the existing log messages. -- This is an automated message from the Apache Git Service. To respond to the

[jira] [Assigned] (KAFKA-15365) Broker-side replica management changes

2023-11-17 Thread Omnia Ibrahim (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omnia Ibrahim reassigned KAFKA-15365: - Assignee: Omnia Ibrahim > Broker-side replica management changes >

[PR] MINOR: Do not check whether updating tasks exist in the waiting loop [kafka]

2023-11-17 Thread via GitHub
cadonna opened a new pull request, #14791: URL: https://github.com/apache/kafka/pull/14791 The state updater waits on a condition variable if no tasks exist that need to be updated. The condition variable is wrapped by a loop to account for spurious wake-ups. The check whether updating task

[PR] KAFKA-15363: Broker log directory failure changes [kafka]

2023-11-17 Thread via GitHub
OmniaGM opened a new pull request, #14790: URL: https://github.com/apache/kafka/pull/14790 - Extend the ReplicaAlterLogDirsThread to send an AssignReplicasToDirsRequestData RPC before promoting the future replica in JBOD mode with KRAFT. - Extend AssignmentsManager and DirectoryEventHan

Re: [PR] KAFKA-14516: [1/N] Static Member leave, join, re-join request using ConsumerGroupHeartbeats [kafka]

2023-11-17 Thread via GitHub
dajac commented on code in PR #14432: URL: https://github.com/apache/kafka/pull/14432#discussion_r1397287511 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/GroupMetadataManager.java: ## @@ -829,21 +902,50 @@ private CoordinatorResult consumerGr

Re: [PR] KAFKA-15578: Migrating other system tests to use the group coordinator [kafka]

2023-11-17 Thread via GitHub
dajac commented on code in PR #14582: URL: https://github.com/apache/kafka/pull/14582#discussion_r1397610751 ## tests/kafkatest/tests/core/transactions_test.py: ## @@ -246,8 +246,9 @@ def setup_topics(self): @matrix(failure_mode=["hard_bounce", "clean_bounce"],

Re: [PR] KAFKA-15484: General Rack Aware Assignor [kafka]

2023-11-17 Thread via GitHub
dajac commented on code in PR #14481: URL: https://github.com/apache/kafka/pull/14481#discussion_r1397600750 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GeneralUniformAssignmentBuilder.java: ## @@ -14,17 +14,891 @@ * See the License for the s

Re: [PR] KAFKA-15484: General Rack Aware Assignor [kafka]

2023-11-17 Thread via GitHub
dajac commented on code in PR #14481: URL: https://github.com/apache/kafka/pull/14481#discussion_r1397596576 ## group-coordinator/src/main/java/org/apache/kafka/coordinator/group/assignor/GeneralUniformAssignmentBuilder.java: ## @@ -14,17 +14,891 @@ * See the License for the s

Re: [PR] KAFKA-15362: Resolve offline replicas in metadata cache [kafka]

2023-11-17 Thread via GitHub
soarez commented on code in PR #14737: URL: https://github.com/apache/kafka/pull/14737#discussion_r1397572189 ## core/src/main/scala/kafka/server/metadata/KRaftMetadataCache.scala: ## @@ -143,21 +143,30 @@ class KRaftMetadataCache(val brokerId: Int) extends MetadataCache with L

Re: [PR] KAFKA-15357: Aggregate and propagate assignments [kafka]

2023-11-17 Thread via GitHub
dajac commented on PR #14369: URL: https://github.com/apache/kafka/pull/14369#issuecomment-1816721578 I've also seen `Build / JDK 8 and Scala 2.12 / testAssignmentAggregation() – kafka.server.AssignmentsManagerTest` failing consistently in a few builds ([example](https://ci-builds.apache.or

Re: [PR] KAFKA-15362: Resolve offline replicas in metadata cache [kafka]

2023-11-17 Thread via GitHub
pprovenzano commented on code in PR #14737: URL: https://github.com/apache/kafka/pull/14737#discussion_r1397547750 ## core/src/main/scala/kafka/server/metadata/KRaftMetadataCache.scala: ## @@ -143,21 +143,30 @@ class KRaftMetadataCache(val brokerId: Int) extends MetadataCache w

Re: [PR] KAFKA-15836: KafkaConsumer subscribes to multiple topics does not respect max.poll.records [kafka]

2023-11-17 Thread via GitHub
AndrewJSchofield commented on code in PR #14789: URL: https://github.com/apache/kafka/pull/14789#discussion_r1397505461 ## clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java: ## @@ -1124,6 +1126,62 @@ public void testFetchMaxPollRecords() {

Re: [PR] KAFKA-15802: Validate remote segment state before fetching index [kafka]

2023-11-17 Thread via GitHub
mimaison commented on PR #14759: URL: https://github.com/apache/kafka/pull/14759#issuecomment-1816625357 Is the work complete? If so can we close [KAFKA-15802](https://issues.apache.org/jira/browse/KAFKA-15802)? -- This is an automated message from the Apache Git Service. To respond to th

Re: [PR] KAFKA-15836: KafkaConsumer subscribes to multiple topics does not respect max.poll.records [kafka]

2023-11-17 Thread via GitHub
kirktrue commented on PR #14789: URL: https://github.com/apache/kafka/pull/14789#issuecomment-1816615350 🤦‍♂️ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [PR] KAFKA-15836: KafkaConsumer subscribes to multiple topics does not respect max.poll.records [kafka]

2023-11-17 Thread via GitHub
kirktrue commented on code in PR #14789: URL: https://github.com/apache/kafka/pull/14789#discussion_r1397472048 ## clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java: ## @@ -1124,6 +1126,62 @@ public void testFetchMaxPollRecords() { asser

[jira] [Commented] (KAFKA-15836) KafkaConsumer subscribes to multiple topics does not respect max.poll.records

2023-11-17 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787243#comment-17787243 ] Andrew Schofield commented on KAFKA-15836: -- I would say that it's not applicabl

Re: [PR] KAFKA-15362: Resolve offline replicas in metadata cache [kafka]

2023-11-17 Thread via GitHub
soarez commented on PR #14737: URL: https://github.com/apache/kafka/pull/14737#issuecomment-1816570498 Thanks for the review @cmccabe. I think I have incorporated all your suggestions. PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log

[PR] KAFKA-15836: KafkaConsumer subscribes to multiple topics does not respect max.poll.records [kafka]

2023-11-17 Thread via GitHub
AndrewJSchofield opened a new pull request, #14789: URL: https://github.com/apache/kafka/pull/14789 When returning fetched records, the code was not properly honouring `max.poll.records`. When it had fetched records for multiple topic-partitions, it was intended to accumulate records up

Re: [PR] KAFKA-15327: ensure the commit manager commit on close [kafka]

2023-11-17 Thread via GitHub
lucasbru commented on code in PR #14710: URL: https://github.com/apache/kafka/pull/14710#discussion_r1397321134 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/CommitRequestManager.java: ## @@ -180,6 +180,18 @@ public void maybeAutoCommit(final Map offsets

[jira] [Commented] (KAFKA-15836) KafkaConsumer subscribes to multiple topics does not respect max.poll.records

2023-11-17 Thread Mickael Maison (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787217#comment-17787217 ] Mickael Maison commented on KAFKA-15836: Awesome, thanks [~schofielaj]  > Kafka

[jira] [Resolved] (KAFKA-15793) Flaky test ZkMigrationIntegrationTest.testMigrateTopicDeletions

2023-11-17 Thread Mickael Maison (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mickael Maison resolved KAFKA-15793. Resolution: Fixed > Flaky test ZkMigrationIntegrationTest.testMigrateTopicDeletions >

[jira] [Commented] (KAFKA-15793) Flaky test ZkMigrationIntegrationTest.testMigrateTopicDeletions

2023-11-17 Thread Mickael Maison (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787216#comment-17787216 ] Mickael Maison commented on KAFKA-15793: Since KAFKA-15799 is now done, marking

Re: [PR] KAFKA-15201: Allow git push to fail gracefully [kafka]

2023-11-17 Thread via GitHub
Owen-CH-Leung commented on PR #14645: URL: https://github.com/apache/kafka/pull/14645#issuecomment-1816422275 @divijvaidya shall we merge this PR ? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] KAFKA-15799 Handle full metadata updates on ZK brokers [kafka]

2023-11-17 Thread via GitHub
mumrah commented on PR #14719: URL: https://github.com/apache/kafka/pull/14719#issuecomment-1816420476 @mimaison yup, I closed it just now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the spec

Re: [PR] KAFKA-15038: Add metadatacache into RemoteLogManager, and refactor all relevant codes [kafka]

2023-11-17 Thread via GitHub
Owen-CH-Leung commented on PR #14136: URL: https://github.com/apache/kafka/pull/14136#issuecomment-1816419916 @kamalcph Thanks. Just rebased. Btw when I rebased, I notice that there's a failing test `kafka.server.AssignmentsManagerTest.testAssignmentAggregation` at trunk. May

[jira] [Updated] (KAFKA-15799) ZK brokers incorrectly handle KRaft metadata snapshots

2023-11-17 Thread David Arthur (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur updated KAFKA-15799: - Fix Version/s: 3.7.0 > ZK brokers incorrectly handle KRaft metadata snapshots >

[jira] [Commented] (KAFKA-14507) Add ConsumerGroupPrepareAssignment API

2023-11-17 Thread Zihao Lin (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787193#comment-17787193 ] Zihao Lin commented on KAFKA-14507: --- Hi team, If this ticket is open I will pick it up

[jira] [Resolved] (KAFKA-15849) Fix ListGroups API when runtime partition size is zero

2023-11-17 Thread David Jacot (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-15849. - Fix Version/s: 3.7.0 Reviewer: David Jacot Resolution: Fixed > Fix ListGroups AP

Re: [PR] KAFKA-15849: Fix ListGroups API when runtime partition size is zero [kafka]

2023-11-17 Thread via GitHub
dajac merged PR #14785: URL: https://github.com/apache/kafka/pull/14785 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.or

Re: [PR] KAFKA-15555: Ensure wakeups are handled correctly in poll() [kafka]

2023-11-17 Thread via GitHub
cadonna commented on code in PR #14746: URL: https://github.com/apache/kafka/pull/14746#discussion_r1397156239 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchBuffer.java: ## @@ -185,6 +188,16 @@ void awaitNotEmpty(Timer timer) { } } +

Re: [PR] KAFKA-15555: Ensure wakeups are handled correctly in poll() [kafka]

2023-11-17 Thread via GitHub
cadonna commented on code in PR #14746: URL: https://github.com/apache/kafka/pull/14746#discussion_r1397154783 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchBuffer.java: ## @@ -52,6 +53,8 @@ public class FetchBuffer implements AutoCloseable { pri

Re: [PR] KAFKA-14133: Migrate ProcessorStateManagerTest and StreamThreadTest to Mockito [kafka]

2023-11-17 Thread via GitHub
cadonna commented on code in PR #13932: URL: https://github.com/apache/kafka/pull/13932#discussion_r1396975782 ## streams/src/test/java/org/apache/kafka/streams/processor/internals/ProcessorStateManagerTest.java: ## @@ -125,11 +122,11 @@ public class ProcessorStateManagerTest {

Re: [PR] KAFKA-15833: Restrict Consumer API to be used from one thread [kafka]

2023-11-17 Thread via GitHub
lucasbru commented on code in PR #14779: URL: https://github.com/apache/kafka/pull/14779#discussion_r1397139312 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java: ## @@ -1102,46 +1207,94 @@ public void subscribe(Pattern pattern, Consum

[jira] [Assigned] (KAFKA-15815) JsonRestServer leaks sockets via HttpURLConnection when keep-alive enabled

2023-11-17 Thread Ashwin Pankaj (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Pankaj reassigned KAFKA-15815: - Assignee: Ashwin Pankaj > JsonRestServer leaks sockets via HttpURLConnection when keep-

Re: [PR] KAFKA-15833: Restrict Consumer API to be used from one thread [kafka]

2023-11-17 Thread via GitHub
lucasbru commented on code in PR #14779: URL: https://github.com/apache/kafka/pull/14779#discussion_r1397125331 ## clients/src/test/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumerTest.java: ## @@ -112,6 +112,12 @@ public void testInvalidGroupId() { a

Re: [PR] KAFKA-15833: Restrict Consumer API to be used from one thread [kafka]

2023-11-17 Thread via GitHub
lucasbru commented on code in PR #14779: URL: https://github.com/apache/kafka/pull/14779#discussion_r1397125030 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/AsyncKafkaConsumer.java: ## @@ -1068,12 +1168,17 @@ public void onComplete(Map offsets, Exception

[jira] [Updated] (KAFKA-15851) broker under replicated due to error java.nio.BufferOverflowException

2023-11-17 Thread wangliucheng (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangliucheng updated KAFKA-15851: - Attachment: (was: server.log) > broker under replicated due to error java.nio.BufferOverflow

Re: [PR] KAFKA-15555: Ensure wakeups are handled correctly in poll() [kafka]

2023-11-17 Thread via GitHub
lucasbru commented on code in PR #14746: URL: https://github.com/apache/kafka/pull/14746#discussion_r1397061812 ## clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetchBuffer.java: ## @@ -52,6 +53,8 @@ public class FetchBuffer implements AutoCloseable { pr

[PR] MINOR: Expose earliest local timestamp via the GetOffsetShell [kafka]

2023-11-17 Thread via GitHub
clolov opened a new pull request, #14788: URL: https://github.com/apache/kafka/pull/14788 With the introduction of tiered storage the ListOffsets API can now return the earliest local timestamp. The purpose of this pull request is to enable the Kafka tools to get this information in

[jira] [Commented] (KAFKA-15836) KafkaConsumer subscribes to multiple topics does not respect max.poll.records

2023-11-17 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787128#comment-17787128 ] Andrew Schofield commented on KAFKA-15836: -- Yes, I would say so. I think I'll h

Re: [PR] KAFKA-15826: Close consumer when sink task is cancelled [kafka]

2023-11-17 Thread via GitHub
yashmayya commented on code in PR #14762: URL: https://github.com/apache/kafka/pull/14762#discussion_r1397042412 ## connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java: ## @@ -163,6 +163,12 @@ public void initialize(TaskConfig taskConfig) {

[jira] [Assigned] (KAFKA-15836) KafkaConsumer subscribes to multiple topics does not respect max.poll.records

2023-11-17 Thread Andrew Schofield (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Schofield reassigned KAFKA-15836: Assignee: Andrew Schofield (was: Kirk True) > KafkaConsumer subscribes to multip

[jira] [Assigned] (KAFKA-14877) refactor InMemoryLeaderEpochCheckpoint

2023-11-17 Thread Kamal Chandraprakash (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kamal Chandraprakash reassigned KAFKA-14877: Assignee: (was: Kamal Chandraprakash) > refactor InMemoryLeaderEpochC

Re: [PR] KAFKA-15241: Compute tiered copied offset by keeping the respective epochs in scope [kafka]

2023-11-17 Thread via GitHub
clolov commented on PR #14787: URL: https://github.com/apache/kafka/pull/14787#issuecomment-1816112742 I will aim to provide a review today! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the sp

Re: [PR] KAFKA-15241: Compute tiered copied offset by keeping the respective epochs in scope [kafka]

2023-11-17 Thread via GitHub
kamalcph commented on code in PR #14787: URL: https://github.com/apache/kafka/pull/14787#discussion_r1397045499 ## core/src/main/java/kafka/log/remote/RemoteLogManager.java: ## @@ -625,9 +626,10 @@ private void maybeUpdateCopiedOffset(UnifiedLog log) throws RemoteStorageExcepti

[PR] KAFKA-15241: Compute tiered copied offset by keeping the respective epochs in scope [kafka]

2023-11-17 Thread via GitHub
kamalcph opened a new pull request, #14787: URL: https://github.com/apache/kafka/pull/14787 `findHighestRemoteOffset` does not take into account the leader-epoch end offset. This can cause log divergence between the local and remote log segments when there is unclean leader election.

Re: [PR] KAFKA-15799 Handle full metadata updates on ZK brokers [kafka]

2023-11-17 Thread via GitHub
mimaison commented on PR #14719: URL: https://github.com/apache/kafka/pull/14719#issuecomment-1816083065 Can we now close https://issues.apache.org/jira/browse/KAFKA-15799? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and u

[jira] [Commented] (KAFKA-15836) KafkaConsumer subscribes to multiple topics does not respect max.poll.records

2023-11-17 Thread Mickael Maison (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787118#comment-17787118 ] Mickael Maison commented on KAFKA-15836: This is marked as a blocker for 3.6.1.

Re: [PR] KAFKA-15327: ensure the commit manager commit on close [kafka]

2023-11-17 Thread via GitHub
philipnee commented on PR #14710: URL: https://github.com/apache/kafka/pull/14710#issuecomment-1816006917 Hi @lucasbru @cadonna - If you get a chance, could you take a look at the PR? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please lo

Re: [PR] KAFKA-15555: Ensure wakeups are handled correctly in poll() [kafka]

2023-11-17 Thread via GitHub
cadonna commented on PR #14746: URL: https://github.com/apache/kafka/pull/14746#issuecomment-1816006382 > Do you think it is worth adding a ticket about using the BackgroundEventQueue for the fetches? Let me know if you have a concrete idea about how to implement this. Let me think a

Re: [PR] KAFKA-15555: Ensure wakeups are handled correctly in poll() [kafka]

2023-11-17 Thread via GitHub
cadonna commented on PR #14746: URL: https://github.com/apache/kafka/pull/14746#issuecomment-1816005251 > I wonder if we could just use a BlockingQueue for the fetchBuffer because fetchBuffer.poll(time) blocks until non-empty or timeout. As far as I see from the javadocs of a `Blockin

Re: [PR] KAFKA-15833: Restrict Consumer API to be used from one thread [kafka]

2023-11-17 Thread via GitHub
cadonna commented on PR #14779: URL: https://github.com/apache/kafka/pull/14779#issuecomment-1815989094 I agree with @philipnee that testing the locking behavior would be great. However, I also agree with @lucasbru that testing the locking behavior is quite hard. Given the tight deadline to

Re: [PR] HOTFIX: Fix compilation error in ReplicaManagerConcurrencyTest for Scala 2.12 [kafka]

2023-11-17 Thread via GitHub
dajac merged PR #14786: URL: https://github.com/apache/kafka/pull/14786 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.or

Re: [PR] HOTFIX: Fix compilation error in ReplicaManagerConcurrencyTest for Scala 2.12 [kafka]

2023-11-17 Thread via GitHub
dajac commented on PR #14786: URL: https://github.com/apache/kafka/pull/14786#issuecomment-1815931886 Thanks @lucasbru! The compilation step has succeeded for all build and I have also confirmed that the test passes locally with Scala 2.12 and 2.13. Therefore, I will go ahead and merge it t

Re: [PR] KAFKA-15174: Ensure CommitAsync propagate the exception to the user [kafka]

2023-11-17 Thread via GitHub
lucasbru commented on PR #14680: URL: https://github.com/apache/kafka/pull/14680#issuecomment-1815926638 Scala compilation in 2.12 is broken on master, which is fixed in https://github.com/apache/kafka/pull/14786, otherwise this is looking good -- This is an automated message from the Apa

[jira] [Assigned] (KAFKA-14507) Add ConsumerGroupPrepareAssignment API

2023-11-17 Thread Zihao Lin (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zihao Lin reassigned KAFKA-14507: - Assignee: Zihao Lin > Add ConsumerGroupPrepareAssignment API >