[
https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934195#comment-16934195
]
Arunkumar Velusamy edited comment on KAFKA-7500 at 9/20/19 8:35 AM:
--------------------------------------------------------------------
Hello Ryanne, I have executed few performance tests for MM2 as part of
evaluation and sharing results here. Tests are specifically focused towards
1. Over all latency in replication
2. System resource usage - CPU and Memory
*Hardware*
All the tests are executed in nodes with below hardware configurations
44 core CPU
256 GB RAM
1.25 GBps (10GBps shared link)
*Kafka Cluster:*
Two 3 node kafka clusters (namely primary, backup)
Replication factor is 3 by default
Keberos authentication enabled
*MM2 setup:*
3 instances of MM2
8 GB RAM
||Test 1||Test 2||Test 3|| ||
|one topic, 1000 partitions
duration: 1 hour
Message Size - 10KB
MM2 conf: tasks.max = 100
Replication:
primary -> backup cluster
60000 messages/2 second
600 MB/2 second
* Latency: Avg. 4 seconds
* MM2 process CPU usage took 80-100% on avg throughout the run duration
* Memory usage is between 3-3.5 GB|two topics, 1000 partitions
duration: 1 hour
Message Size - 1KB
MM2 conf: tasks.max = 100
Replication:
primary -> backup
backup -> primary
120000 messages/2 second (to each topic)
120 MB/2 second (to each topic)
* Latency: Avg. 8 seconds
* MM2 process CPU usage took 100-140% on avg throughout the execution
* Memory usage is between 3-4 GB|6 topics, 6 partitions (each)
duration: 1 hour
Message Size - 1KB
MM2 conf: tasks.max = 20
Replication:
primary -> backup
backup -> primary
20000 messages/2 second (to each topic)
20 MB/2 second (to each topic) * Latency: Avg. 150 milliseconds
* Memory usage is between 3-4 GB| |
Based on the test results,
* Number of tasks has direct impact on latency. More tasks -> less latency
* Message size does not have any impacts on replication latency
* In case of shared cluster (other services are also running), CPU cores >=32
yielded good results (latency in seconds)
* It is good to have >= 8GB memory dedicated to MM2 instance
* Higher Number of messages per time interval to one topic causes more latency
than the same number of messages to multiple topics
* When number of partitions increases (also tasks increased), latency
decreases for that topic
Thanks for all your efforts on MM2. Happy to provide more details if required.
was (Author: aun):
Hello Ryanne, I have executed few performance tests for MM2 as part of
evaluation and sharing results here. Tests are specifically focused towards
1. Over all latency in replication
2. System resource usage - CPU and Memory
*Hardware*
All the tests are executed in nodes with below hardware configurations
44 core CPU
256 GB RAM
1.25 GBps (10GBps shared link)
*Kafka Cluster:*
Two 3 node kafka clusters (namely primary, backup)
Replication factor is 3 by default
Keberos authentication enabled
*MM2 setup:*
3 instances of MM2
8 GB RAM
||Test 1||Test 2||Test 3||
|one topic, 1000 partitions
duration: 1 hour
Message Size - 10KB
MM2 conf: tasks.max = 100
Replication:
primary -> backup cluster
60000 messages/2 second
600 MB/2 second
* Latency: Avg. 4 seconds
* MM2 process CPU usage took 80-100% on avg throughout the run duration
* Memory usage is between 3-3.5 GB|two topics, 1000 partitions
duration: 1 hour
Message Size - 1KB
MM2 conf: tasks.max = 100
Replication:
primary -> backup
backup -> primary
120000 messages/2 second (to each topic)
120 MB/2 second (to each topic)
* Latency: Avg. 8 seconds
* MM2 process CPU usage took 100-140% on avg throughout the execution
* Memory usage is between 3-4 GB|6 topics, 6 partitions (each)
duration: 1 hour
Message Size - 1KB
MM2 conf: tasks.max = 20
Replication:
primary -> backup
backup -> primary
20000 messages/2 second (to each topic)
20 MB/2 second (to each topic)
* Latency: Avg. 150 milliseconds
* Memory usage is between 3-4 GB|
Based on the test results,
* Number of tasks has direct impact on latency. More tasks -> less latency
* Message size does not have any impacts on replication latency
* In case of shared cluster (other services are also running), CPU cores >=32
yielded good results (latency in seconds)
* It is good to have >= 8GB memory dedicated to MM2 instance
* Higher Number of messages per time interval to one topic causes more latency
than the same number of messages to multiple topics
* When number of partitions increases (also tasks increased), latency
decreases for that topic
Thanks for all your efforts on MM2. Happy to provide more details if required.
> MirrorMaker 2.0 (KIP-382)
> -------------------------
>
> Key: KAFKA-7500
> URL: https://issues.apache.org/jira/browse/KAFKA-7500
> Project: Kafka
> Issue Type: New Feature
> Components: KafkaConnect, mirrormaker
> Affects Versions: 2.4.0
> Reporter: Ryanne Dolan
> Assignee: Manikumar
> Priority: Major
> Labels: pull-request-available, ready-to-commit
> Fix For: 2.4.0
>
> Attachments: Active-Active XDCR setup.png
>
>
> Implement a drop-in replacement for MirrorMaker leveraging the Connect
> framework.
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0]
> [https://github.com/apache/kafka/pull/6295]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
