[GitHub] [kafka] ben-manes commented on a diff in pull request #13850: KAFKA-15084: Remove lock contention from RemoteIndexCache

2023-06-13 Thread via GitHub
ben-manes commented on code in PR #13850: URL: https://github.com/apache/kafka/pull/13850#discussion_r1229087123 ## core/src/main/scala/kafka/log/remote/RemoteIndexCache.scala: ## @@ -266,13 +309,21 @@ class RemoteIndexCache(maxSize: Int = 1024, remoteStorageManager: RemoteStor

[GitHub] [kafka] ben-manes commented on a diff in pull request #13850: KAFKA-15084: Remove lock contention from RemoteIndexCache

2023-06-13 Thread via GitHub
ben-manes commented on code in PR #13850: URL: https://github.com/apache/kafka/pull/13850#discussion_r1229083529 ## core/src/main/scala/kafka/log/remote/RemoteIndexCache.scala: ## @@ -152,8 +194,7 @@ class RemoteIndexCache(maxSize: Int = 1024, remoteStorageManager: RemoteStorag

[GitHub] [kafka] ben-manes commented on a diff in pull request #13850: KAFKA-15084: Remove lock contention from RemoteIndexCache

2023-06-13 Thread via GitHub
ben-manes commented on code in PR #13850: URL: https://github.com/apache/kafka/pull/13850#discussion_r1229083529 ## core/src/main/scala/kafka/log/remote/RemoteIndexCache.scala: ## @@ -152,8 +194,7 @@ class RemoteIndexCache(maxSize: Int = 1024, remoteStorageManager: RemoteStorag

[GitHub] [kafka] ben-manes commented on a diff in pull request #13850: KAFKA-15084: Remove lock contention from RemoteIndexCache

2023-06-13 Thread via GitHub
ben-manes commented on code in PR #13850: URL: https://github.com/apache/kafka/pull/13850#discussion_r1229081529 ## core/src/main/scala/kafka/log/remote/RemoteIndexCache.scala: ## @@ -136,12 +176,14 @@ class RemoteIndexCache(maxSize: Int = 1024, remoteStorageManager: RemoteStor

[GitHub] [kafka] vamossagar12 commented on pull request #13453: KAFKA-12525: Ignoring Stale status statuses when reading from Status …

2023-06-13 Thread via GitHub
vamossagar12 commented on PR #13453: URL: https://github.com/apache/kafka/pull/13453#issuecomment-159051 @enzo-cappa , thanks.. yeah waiting for a round of review on this one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [kafka] sagarrao12 commented on pull request #13453: KAFKA-12525: Ignoring Stale status statuses when reading from Status …

2023-06-13 Thread via GitHub
sagarrao12 commented on PR #13453: URL: https://github.com/apache/kafka/pull/13453#issuecomment-1590511123 @enzo-cappa , thanks.. yeah waiting for a round of review on this one... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHu

[GitHub] [kafka] satishd commented on pull request #13828: KAFKA-15066: add "remote.log.metadata.manager.listener.name" config to rlmm

2023-06-13 Thread via GitHub
satishd commented on PR #13828: URL: https://github.com/apache/kafka/pull/13828#issuecomment-1590492505 > I think we don't have this implemented. We should pass remote.log.metadata.* into RLMM based on KIP-405. Created [KAFKA-15083](https://issues.apache.org/jira/browse/KAFKA-15083) for thi

[GitHub] [kafka] dengziming commented on a diff in pull request #13845: KAFKA-15078; KRaft leader replys with snapshot for offset 0

2023-06-13 Thread via GitHub
dengziming commented on code in PR #13845: URL: https://github.com/apache/kafka/pull/13845#discussion_r1228931662 ## raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java: ## @@ -1017,7 +1017,16 @@ private FetchResponseData tryCompleteFetchRequest( long fetc

[GitHub] [kafka] lihaosky opened a new pull request, #13851: check enable rack

2023-06-13 Thread via GitHub
lihaosky opened a new pull request, #13851: URL: https://github.com/apache/kafka/pull/13851 ## Description Initial implementation to check if rack aware assignment can be enabled ## Test TODO -- This is an automated message from the Apache Git Service. To respond to the me

[GitHub] [kafka] jolshan commented on pull request #13798: KAFKA-15028: AddPartitionsToTxnManager metrics

2023-06-13 Thread via GitHub
jolshan commented on PR #13798: URL: https://github.com/apache/kafka/pull/13798#issuecomment-1590237911 unit.kafka.server.AddPartitionsToTxnManagerTest.testAddPartitionsToTxnManagerMetrics() is failing. I was a bit worried this could be flaky. Will investigate. -- This is an automated me

[GitHub] [kafka] jolshan commented on pull request #13796: KAFKA-14034 Idempotent producer should wait for preceding in-flight b…

2023-06-13 Thread via GitHub
jolshan commented on PR #13796: URL: https://github.com/apache/kafka/pull/13796#issuecomment-1590186665 With respect to the retries. There is currently a known error where if we produce using an idempotent producer we can get stuck in a retry loop. 1. Request A with sequence 0 fails w

[GitHub] [kafka] jolshan commented on a diff in pull request #13798: KAFKA-15028: AddPartitionsToTxnManager metrics

2023-06-13 Thread via GitHub
jolshan commented on code in PR #13798: URL: https://github.com/apache/kafka/pull/13798#discussion_r1228732870 ## core/src/main/scala/kafka/network/RequestChannel.scala: ## @@ -240,16 +240,17 @@ object RequestChannel extends Logging { val responseSendTimeMs = nanosToMs(en

[GitHub] [kafka] jolshan commented on a diff in pull request #13591: KAFKA-14831: Illegal state errors should be fatal in transactional producer

2023-06-13 Thread via GitHub
jolshan commented on code in PR #13591: URL: https://github.com/apache/kafka/pull/13591#discussion_r1228729149 ## clients/src/test/java/org/apache/kafka/clients/producer/internals/TransactionManagerTest.java: ## @@ -3405,6 +3405,59 @@ MAX_REQUEST_SIZE, ACKS_ALL, MAX_RETRIES, new

[GitHub] [kafka] jolshan commented on a diff in pull request #13798: KAFKA-15028: AddPartitionsToTxnManager metrics

2023-06-13 Thread via GitHub
jolshan commented on code in PR #13798: URL: https://github.com/apache/kafka/pull/13798#discussion_r1228731391 ## core/src/main/scala/kafka/server/AddPartitionsToTxnManager.scala: ## @@ -85,12 +93,14 @@ class AddPartitionsToTxnManager(config: KafkaConfig, client: NetworkClient,

[GitHub] [kafka] jolshan commented on pull request #13591: KAFKA-14831: Illegal state errors should be fatal in transactional producer

2023-06-13 Thread via GitHub
jolshan commented on PR #13591: URL: https://github.com/apache/kafka/pull/13591#issuecomment-1590069645 Thanks @kirktrue, this looks reasonable. Let's fix the conflict and let tests run one more time. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [kafka] vcrfxia commented on a diff in pull request #13756: KAFKA-14936: Add On Disk Time Ordered Buffer (1/N)

2023-06-13 Thread via GitHub
vcrfxia commented on code in PR #13756: URL: https://github.com/apache/kafka/pull/13756#discussion_r1228495958 ## streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedKeyValueBuffer.java: ## @@ -0,0 +1,171 @@ +/* + * Licensed to the Apache Software Fo

[GitHub] [kafka] divijvaidya commented on pull request #13820: MINOR: Move Timer/TimingWheel to server-common

2023-06-13 Thread via GitHub
divijvaidya commented on PR #13820: URL: https://github.com/apache/kafka/pull/13820#issuecomment-1589745577 Hi @dajac , yes let me take a look first thing tomorrow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the U

[GitHub] [kafka] dajac commented on pull request #13820: MINOR: Move Timer/TimingWheel to server-common

2023-06-13 Thread via GitHub
dajac commented on PR #13820: URL: https://github.com/apache/kafka/pull/13820#issuecomment-1589741122 @divijvaidya As you started reviewing it, are you interested in finishing the review? That would unblock me. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [kafka] wcarlson5 commented on a diff in pull request #13756: KAFKA-14936: Add On Disk Time Ordered Buffer (1/N)

2023-06-13 Thread via GitHub
wcarlson5 commented on code in PR #13756: URL: https://github.com/apache/kafka/pull/13756#discussion_r1228430026 ## streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedKeyValueBuffer.java: ## @@ -0,0 +1,171 @@ +/* + * Licensed to the Apache Software

[GitHub] [kafka] wcarlson5 commented on pull request #13756: KAFKA-14936: Add On Disk Time Ordered Buffer (1/N)

2023-06-13 Thread via GitHub
wcarlson5 commented on PR #13756: URL: https://github.com/apache/kafka/pull/13756#issuecomment-1589678126 @cadonna I also hid the segments as you proposed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[GitHub] [kafka] dajac commented on a diff in pull request #13848: MINOR: Use admin client to create offsets topic in tests

2023-06-13 Thread via GitHub
dajac commented on code in PR #13848: URL: https://github.com/apache/kafka/pull/13848#discussion_r1228424644 ## core/src/test/scala/unit/kafka/integration/KafkaServerTestHarness.scala: ## @@ -150,12 +150,8 @@ abstract class KafkaServerTestHarness extends QuorumTestHarness {

[GitHub] [kafka] wcarlson5 commented on a diff in pull request #13756: KAFKA-14936: Add On Disk Time Ordered Buffer (1/N)

2023-06-13 Thread via GitHub
wcarlson5 commented on code in PR #13756: URL: https://github.com/apache/kafka/pull/13756#discussion_r1228419948 ## streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedKeyValueBuffer.java: ## @@ -0,0 +1,171 @@ +/* + * Licensed to the Apache Software

[GitHub] [kafka] wcarlson5 commented on a diff in pull request #13756: KAFKA-14936: Add On Disk Time Ordered Buffer (1/N)

2023-06-13 Thread via GitHub
wcarlson5 commented on code in PR #13756: URL: https://github.com/apache/kafka/pull/13756#discussion_r1228360963 ## streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedKeyValueBuffer.java: ## @@ -0,0 +1,171 @@ +/* + * Licensed to the Apache Software

[GitHub] [kafka] divijvaidya opened a new pull request, #13850: KAFKA-15084: Remove lock contention from RemoteIndexCache

2023-06-13 Thread via GitHub
divijvaidya opened a new pull request, #13850: URL: https://github.com/apache/kafka/pull/13850 ## Problem RemoteIndexCache cache is accessed from multiple threads concurrently in the fetch from consumer code path [1]. Currently, the RemoteIndexCache uses LinkedHashMap as the cache

[jira] [Created] (KAFKA-15084) Remove lock contention in RemoteIndexCache

2023-06-13 Thread Divij Vaidya (Jira)
Divij Vaidya created KAFKA-15084: Summary: Remove lock contention in RemoteIndexCache Key: KAFKA-15084 URL: https://issues.apache.org/jira/browse/KAFKA-15084 Project: Kafka Issue Type: Sub-ta

[GitHub] [kafka] mimaison commented on pull request #13849: Add 3.5.0 and 3.4.1 to system tests

2023-06-13 Thread via GitHub
mimaison commented on PR #13849: URL: https://github.com/apache/kafka/pull/13849#issuecomment-1589399101 @mjsax @vvcephei Can you upload the 3.5.0 artifacts to the Confluent S3 bucket as per the [release instructions](https://cwiki.apache.org/confluence/display/KAFKA/Release+Process#Release

[jira] [Resolved] (KAFKA-15080) Fetcher's lag never set when partition is idle

2023-06-13 Thread David Jacot (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot resolved KAFKA-15080. - Fix Version/s: 3.5.1 3.6.0 Resolution: Fixed > Fetcher's lag never set

[GitHub] [kafka] mumrah commented on a diff in pull request #13802: MINOR: Improve KRaftMigrationZkWriter test coverage

2023-06-13 Thread via GitHub
mumrah commented on code in PR #13802: URL: https://github.com/apache/kafka/pull/13802#discussion_r1228123874 ## metadata/src/test/java/org/apache/kafka/metadata/migration/KRaftMigrationZkWriterTest.java: ## @@ -195,9 +235,658 @@ public void iterateTopics(EnumSet interests, Top

[GitHub] [kafka] dajac merged pull request #13843: KAFKA-15080; Fetcher's lag never set when partition is idle

2023-06-13 Thread via GitHub
dajac merged PR #13843: URL: https://github.com/apache/kafka/pull/13843 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.or

[GitHub] [kafka] showuon commented on a diff in pull request #13760: KAFKA-8982: Add retry of fetching metadata to Admin.deleteRecords

2023-06-13 Thread via GitHub
showuon commented on code in PR #13760: URL: https://github.com/apache/kafka/pull/13760#discussion_r1228079251 ## clients/src/main/java/org/apache/kafka/clients/admin/KafkaAdminClient.java: ## @@ -2925,123 +2919,11 @@ void handleFailure(Throwable throwable) { @Override

[GitHub] [kafka] dajac merged pull request #13844: MINOR: Make sure replicas will not be removed in initial ISR

2023-06-13 Thread via GitHub
dajac merged PR #13844: URL: https://github.com/apache/kafka/pull/13844 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.or

[jira] [Commented] (KAFKA-12679) Rebalancing a restoring or running task may cause directory livelocking with newly created task

2023-06-13 Thread Bruno Cadonna (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732032#comment-17732032 ] Bruno Cadonna commented on KAFKA-12679: --- [~Andras Hatvani] Lucas' PR solves the is

[jira] [Comment Edited] (KAFKA-12679) Rebalancing a restoring or running task may cause directory livelocking with newly created task

2023-06-13 Thread Andras Hatvani (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732025#comment-17732025 ] Andras Hatvani edited comment on KAFKA-12679 at 6/13/23 11:08 AM:

[jira] [Commented] (KAFKA-12679) Rebalancing a restoring or running task may cause directory livelocking with newly created task

2023-06-13 Thread Andras Hatvani (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-12679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732025#comment-17732025 ] Andras Hatvani commented on KAFKA-12679: [~lbrutschy] [~mjsax] This is a pressin

[GitHub] [kafka] cadonna commented on pull request #12875: KAFKA-12679: Handle lock exceptions in state updater

2023-06-13 Thread via GitHub
cadonna commented on PR #12875: URL: https://github.com/apache/kafka/pull/12875#issuecomment-1589078205 @andrashatvani Do you mean the state updater? I am currently working on enabling the state updater by default on trunk. We plan to officially release it in 3.6. However, you can alre

[GitHub] [kafka] mimaison opened a new pull request, #13849: Add 3.5.0 and 3.4.1 to system tests

2023-06-13 Thread via GitHub
mimaison opened a new pull request, #13849: URL: https://github.com/apache/kafka/pull/13849 ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgr

[jira] [Commented] (KAFKA-14257) Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response

2023-06-13 Thread Jira
[ https://issues.apache.org/jira/browse/KAFKA-14257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732020#comment-17732020 ] çağla boynueğri commented on KAFKA-14257: - all the brokers must have the same CL

[jira] (KAFKA-14257) Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response

2023-06-13 Thread Jira
[ https://issues.apache.org/jira/browse/KAFKA-14257 ] çağla boynueğri deleted comment on KAFKA-14257: - was (Author: JIRAUSER300861): Hi, I am getting the same error, can you remember how did you solve the issue ? > Unexpected error INCONSIS

[jira] [Commented] (KAFKA-14257) Unexpected error INCONSISTENT_CLUSTER_ID in VOTE response

2023-06-13 Thread Jira
[ https://issues.apache.org/jira/browse/KAFKA-14257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17732014#comment-17732014 ] çağla boynueğri commented on KAFKA-14257: - Hi, I am getting the same error, can

[GitHub] [kafka] cadonna commented on a diff in pull request #13756: KAFKA-14936: Add On Disk Time Ordered Buffer (1/N)

2023-06-13 Thread via GitHub
cadonna commented on code in PR #13756: URL: https://github.com/apache/kafka/pull/13756#discussion_r1227894014 ## streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedKeyValueBuffer.java: ## @@ -0,0 +1,171 @@ +/* + * Licensed to the Apache Software Fo

[GitHub] [kafka] showuon commented on pull request #13828: KAFKA-15066: add "remote.log.metadata.manager.listener.name" config to rlmm

2023-06-13 Thread via GitHub
showuon commented on PR #13828: URL: https://github.com/apache/kafka/pull/13828#issuecomment-1588961245 @divijvaidya @satishd , PR updated. Thanks. > 1. We probably want to update the KIP-405 here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP40

[GitHub] [kafka] showuon commented on a diff in pull request #13828: KAFKA-15066: add "remote.log.metadata.manager.listener.name" config to rlmm

2023-06-13 Thread via GitHub
showuon commented on code in PR #13828: URL: https://github.com/apache/kafka/pull/13828#discussion_r1227858635 ## core/src/main/scala/kafka/server/BrokerServer.scala: ## @@ -474,7 +474,14 @@ class BrokerServer( new KafkaConfig(config.originals(), true) // Start R

[jira] [Created] (KAFKA-15083) Passing "remote.log.metadata.*" configs into RLMM

2023-06-13 Thread Luke Chen (Jira)
Luke Chen created KAFKA-15083: - Summary: Passing "remote.log.metadata.*" configs into RLMM Key: KAFKA-15083 URL: https://issues.apache.org/jira/browse/KAFKA-15083 Project: Kafka Issue Type: Sub-t

[GitHub] [kafka] clolov commented on pull request #13711: KAFKA-14133: Migrate StandbyTaskCreator mock in TaskManagerTest to Mockito

2023-06-13 Thread via GitHub
clolov commented on PR #13711: URL: https://github.com/apache/kafka/pull/13711#issuecomment-1588950243 Okay, the name has been changed and this has been rebased. I ran checkstyleTest and spotbugsTest and they are passing locally. If everything passes in the automated tests you should be abl

[GitHub] [kafka] clolov commented on pull request #13712: KAFKA-14133: Migrate Admin mock in TaskManagerTest to Mockito

2023-06-13 Thread via GitHub
clolov commented on PR #13712: URL: https://github.com/apache/kafka/pull/13712#issuecomment-1588898530 Thank you! I will now address the comments on the other one! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[GitHub] [kafka] cadonna merged pull request #13712: KAFKA-14133: Migrate Admin mock in TaskManagerTest to Mockito

2023-06-13 Thread via GitHub
cadonna merged PR #13712: URL: https://github.com/apache/kafka/pull/13712 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.

[GitHub] [kafka] clolov commented on pull request #13712: KAFKA-14133: Migrate Admin mock in TaskManagerTest to Mockito

2023-06-13 Thread via GitHub
clolov commented on PR #13712: URL: https://github.com/apache/kafka/pull/13712#issuecomment-1588826720 Ah, sorry, yeah, I have caused the confusion. Technically the order of merging between this and https://github.com/apache/kafka/pull/13711 doesn't matter as long as one is rebased on top o

[GitHub] [kafka] dajac commented on a diff in pull request #13848: MINOR: Use admin client to create offsets topic in tests

2023-06-13 Thread via GitHub
dajac commented on code in PR #13848: URL: https://github.com/apache/kafka/pull/13848#discussion_r1227742933 ## core/src/test/scala/unit/kafka/integration/KafkaServerTestHarness.scala: ## @@ -150,12 +150,8 @@ abstract class KafkaServerTestHarness extends QuorumTestHarness {

[GitHub] [kafka] dajac opened a new pull request, #13848: MINOR: Use admin client to create offsets topic

2023-06-13 Thread via GitHub
dajac opened a new pull request, #13848: URL: https://github.com/apache/kafka/pull/13848 I have see failures like the following ones in a few builds: ``` Build / JDK 11 and Scala 2.13 / testDescribeSimpleConsumerGroup() – kafka.admin.DescribeConsumerGroupTest org.apache.kafka.co

[GitHub] [kafka] dajac merged pull request #13795: KAFKA-14462; [17/N] Add CoordinatorRuntime

2023-06-13 Thread via GitHub
dajac merged PR #13795: URL: https://github.com/apache/kafka/pull/13795 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.or