Re: [PR] KAFKA-16052: Create a real dummy replicaManager instance to save heap memory [kafka]
divijvaidya commented on PR #15094: URL: https://github.com/apache/kafka/pull/15094#issuecomment-1872927839 backported to 3.7 branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16052: Create a real dummy replicaManager instance to save heap memory [kafka]
divijvaidya merged PR #15094: URL: https://github.com/apache/kafka/pull/15094 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16052: Create a real dummy replicaManager instance to save heap memory [kafka]
divijvaidya commented on PR #15094: URL: https://github.com/apache/kafka/pull/15094#issuecomment-1872924038 Both the modified tests are successful in CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16052: Create a real dummy replicaManager instance to save heap memory [kafka]
divijvaidya commented on code in PR #15094: URL: https://github.com/apache/kafka/pull/15094#discussion_r1438567614 ## core/src/test/scala/unit/kafka/coordinator/AbstractCoordinatorConcurrencyTest.scala: ## @@ -148,25 +156,34 @@ abstract class AbstractCoordinatorConcurrencyTest[M <: CoordinatorMember] { } object AbstractCoordinatorConcurrencyTest { - trait Action extends Runnable { def await(): Unit } trait CoordinatorMember { } - class TestReplicaManager extends ReplicaManager( -null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, None, null) { + class TestReplicaManager(config: KafkaConfig, + mtime: Time, + scheduler: Scheduler, + logManager: LogManager, + quotaManagers: QuotaManagers, + val watchKeys: mutable.Set[TopicPartitionOperationKey], + val producePurgatory: DelayedOperationPurgatory[DelayedProduce]) +extends ReplicaManager( + config, + metrics = null, + mtime, + scheduler, + logManager, + None, + quotaManagers, + null, + null, + null, + delayedProducePurgatoryParam = Some(producePurgatory)) { Review Comment: That's a good idea. Updated in latest commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16052: Create a real dummy replicaManager instance to save heap memory [kafka]
showuon commented on PR #15094: URL: https://github.com/apache/kafka/pull/15094#issuecomment-1872473260 > I noticed this build had more failures than usual. Not sure if its because we actually got to run all the tests without OOMing or if something new has come up. Had a quick check, they look unrelated. Just re-triggering it just in case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16052: Create a real dummy replicaManager instance to save heap memory [kafka]
showuon commented on code in PR #15094: URL: https://github.com/apache/kafka/pull/15094#discussion_r1438489270 ## core/src/test/scala/unit/kafka/coordinator/AbstractCoordinatorConcurrencyTest.scala: ## @@ -148,25 +156,34 @@ abstract class AbstractCoordinatorConcurrencyTest[M <: CoordinatorMember] { } object AbstractCoordinatorConcurrencyTest { - trait Action extends Runnable { def await(): Unit } trait CoordinatorMember { } - class TestReplicaManager extends ReplicaManager( -null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, None, null) { + class TestReplicaManager(config: KafkaConfig, + mtime: Time, + scheduler: Scheduler, + logManager: LogManager, + quotaManagers: QuotaManagers, + val watchKeys: mutable.Set[TopicPartitionOperationKey], + val producePurgatory: DelayedOperationPurgatory[DelayedProduce]) +extends ReplicaManager( + config, + metrics = null, + mtime, + scheduler, + logManager, + None, + quotaManagers, + null, + null, + null, + delayedProducePurgatoryParam = Some(producePurgatory)) { Review Comment: I found while running the profiler that after we creating the real replicaManager, we'll also create `ExpirationReaper` threads for Fetch, RemoteFetch, ... etc. Although we will close them after the tests, we can actually avoid creating them beforehand like what we did in replicaManagerTest. (ref: https://github.com/apache/kafka/pull/15077/files#diff-5ace24a01ad7792f3c168abe80e4c23674d6ce8295f5a54a94933c4e4d75e7a5R2893) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16052: Create a real dummy replicaManager instance to save heap memory [kafka]
jolshan commented on PR #15094: URL: https://github.com/apache/kafka/pull/15094#issuecomment-1872386326 Thanks Divij. Were we able to get the heap usage for this branch too? (Is it the one in the ticket?) Just wanted to confirm we are still seeing the drop in heap usage. I noticed this build had more failures than usual. Not sure if its because we actually got to run all the tests without OOMing or if something new has come up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16052: Create a real dummy replicaManager instance to save heap memory [kafka]
divijvaidya commented on PR #15094: URL: https://github.com/apache/kafka/pull/15094#issuecomment-1872122325 @showuon @jolshan please review when you get a chance -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org