Re: [PR] KAFKA-16052: Create a real dummy replicaManager instance to save heap memory [kafka]
divijvaidya commented on PR #15094: URL: https://github.com/apache/kafka/pull/15094#issuecomment-1872927839 backported to 3.7 branch -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16052: Create a real dummy replicaManager instance to save heap memory [kafka]
divijvaidya merged PR #15094: URL: https://github.com/apache/kafka/pull/15094 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16052: Create a real dummy replicaManager instance to save heap memory [kafka]
divijvaidya commented on PR #15094: URL: https://github.com/apache/kafka/pull/15094#issuecomment-1872924038 Both the modified tests are successful in CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16052: Create a real dummy replicaManager instance to save heap memory [kafka]
divijvaidya commented on code in PR #15094:
URL: https://github.com/apache/kafka/pull/15094#discussion_r1438567614
##
core/src/test/scala/unit/kafka/coordinator/AbstractCoordinatorConcurrencyTest.scala:
##
@@ -148,25 +156,34 @@ abstract class AbstractCoordinatorConcurrencyTest[M <:
CoordinatorMember] {
}
object AbstractCoordinatorConcurrencyTest {
-
trait Action extends Runnable {
def await(): Unit
}
trait CoordinatorMember {
}
- class TestReplicaManager extends ReplicaManager(
-null, null, null, null, null, null, null, null, null, null, null, null,
null, null, null, None, null) {
+ class TestReplicaManager(config: KafkaConfig,
+ mtime: Time,
+ scheduler: Scheduler,
+ logManager: LogManager,
+ quotaManagers: QuotaManagers,
+ val watchKeys:
mutable.Set[TopicPartitionOperationKey],
+ val producePurgatory:
DelayedOperationPurgatory[DelayedProduce])
+extends ReplicaManager(
+ config,
+ metrics = null,
+ mtime,
+ scheduler,
+ logManager,
+ None,
+ quotaManagers,
+ null,
+ null,
+ null,
+ delayedProducePurgatoryParam = Some(producePurgatory)) {
Review Comment:
That's a good idea. Updated in latest commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16052: Create a real dummy replicaManager instance to save heap memory [kafka]
showuon commented on PR #15094: URL: https://github.com/apache/kafka/pull/15094#issuecomment-1872473260 > I noticed this build had more failures than usual. Not sure if its because we actually got to run all the tests without OOMing or if something new has come up. Had a quick check, they look unrelated. Just re-triggering it just in case. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16052: Create a real dummy replicaManager instance to save heap memory [kafka]
showuon commented on code in PR #15094:
URL: https://github.com/apache/kafka/pull/15094#discussion_r1438489270
##
core/src/test/scala/unit/kafka/coordinator/AbstractCoordinatorConcurrencyTest.scala:
##
@@ -148,25 +156,34 @@ abstract class AbstractCoordinatorConcurrencyTest[M <:
CoordinatorMember] {
}
object AbstractCoordinatorConcurrencyTest {
-
trait Action extends Runnable {
def await(): Unit
}
trait CoordinatorMember {
}
- class TestReplicaManager extends ReplicaManager(
-null, null, null, null, null, null, null, null, null, null, null, null,
null, null, null, None, null) {
+ class TestReplicaManager(config: KafkaConfig,
+ mtime: Time,
+ scheduler: Scheduler,
+ logManager: LogManager,
+ quotaManagers: QuotaManagers,
+ val watchKeys:
mutable.Set[TopicPartitionOperationKey],
+ val producePurgatory:
DelayedOperationPurgatory[DelayedProduce])
+extends ReplicaManager(
+ config,
+ metrics = null,
+ mtime,
+ scheduler,
+ logManager,
+ None,
+ quotaManagers,
+ null,
+ null,
+ null,
+ delayedProducePurgatoryParam = Some(producePurgatory)) {
Review Comment:
I found while running the profiler that after we creating the real
replicaManager, we'll also create `ExpirationReaper` threads for Fetch,
RemoteFetch, ... etc. Although we will close them after the tests, we can
actually avoid creating them beforehand like what we did in replicaManagerTest.
(ref:
https://github.com/apache/kafka/pull/15077/files#diff-5ace24a01ad7792f3c168abe80e4c23674d6ce8295f5a54a94933c4e4d75e7a5R2893)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] KAFKA-16052: Create a real dummy replicaManager instance to save heap memory [kafka]
jolshan commented on PR #15094: URL: https://github.com/apache/kafka/pull/15094#issuecomment-1872386326 Thanks Divij. Were we able to get the heap usage for this branch too? (Is it the one in the ticket?) Just wanted to confirm we are still seeing the drop in heap usage. I noticed this build had more failures than usual. Not sure if its because we actually got to run all the tests without OOMing or if something new has come up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] KAFKA-16052: Create a real dummy replicaManager instance to save heap memory [kafka]
divijvaidya commented on PR #15094: URL: https://github.com/apache/kafka/pull/15094#issuecomment-1872122325 @showuon @jolshan please review when you get a chance -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] KAFKA-16052: Create a real dummy replicaManager instance to save heap memory [kafka]
divijvaidya opened a new pull request, #15094: URL: https://github.com/apache/kafka/pull/15094 For more detail, please check [KAFKA-16052](https://issues.apache.org/jira/browse/KAFKA-16052). The mockito will keep the invocation history in the test suite and cause the huge heap usage. Since the mock replicaManager is only used to bypass the replicaManager constructor without verifying/mocking anything, we can create a real dummy replicaManager to avoid the mockito invocation history in memory. Co-authored-by: Luke Chen -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
