[ https://issues.apache.org/jira/browse/KAFKA-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934195#comment-16934195 ]
Arunkumar Velusamy edited comment on KAFKA-7500 at 9/20/19 8:35 AM: -------------------------------------------------------------------- Hello Ryanne, I have executed few performance tests for MM2 as part of evaluation and sharing results here. Tests are specifically focused towards 1. Over all latency in replication 2. System resource usage - CPU and Memory *Hardware* All the tests are executed in nodes with below hardware configurations 44 core CPU 256 GB RAM 1.25 GBps (10GBps shared link) *Kafka Cluster:* Two 3 node kafka clusters (namely primary, backup) Replication factor is 3 by default Keberos authentication enabled *MM2 setup:* 3 instances of MM2 8 GB RAM ||Test 1||Test 2||Test 3|| || |one topic, 1000 partitions duration: 1 hour Message Size - 10KB MM2 conf: tasks.max = 100 Replication: primary -> backup cluster 60000 messages/2 second 600 MB/2 second * Latency: Avg. 4 seconds * MM2 process CPU usage took 80-100% on avg throughout the run duration * Memory usage is between 3-3.5 GB|two topics, 1000 partitions duration: 1 hour Message Size - 1KB MM2 conf: tasks.max = 100 Replication: primary -> backup backup -> primary 120000 messages/2 second (to each topic) 120 MB/2 second (to each topic) * Latency: Avg. 8 seconds * MM2 process CPU usage took 100-140% on avg throughout the execution * Memory usage is between 3-4 GB|6 topics, 6 partitions (each) duration: 1 hour Message Size - 1KB MM2 conf: tasks.max = 20 Replication: primary -> backup backup -> primary 20000 messages/2 second (to each topic) 20 MB/2 second (to each topic) * Latency: Avg. 150 milliseconds * Memory usage is between 3-4 GB| | Based on the test results, * Number of tasks has direct impact on latency. More tasks -> less latency * Message size does not have any impacts on replication latency * In case of shared cluster (other services are also running), CPU cores >=32 yielded good results (latency in seconds) * It is good to have >= 8GB memory dedicated to MM2 instance * Higher Number of messages per time interval to one topic causes more latency than the same number of messages to multiple topics * When number of partitions increases (also tasks increased), latency decreases for that topic Thanks for all your efforts on MM2. Happy to provide more details if required. was (Author: aun): Hello Ryanne, I have executed few performance tests for MM2 as part of evaluation and sharing results here. Tests are specifically focused towards 1. Over all latency in replication 2. System resource usage - CPU and Memory *Hardware* All the tests are executed in nodes with below hardware configurations 44 core CPU 256 GB RAM 1.25 GBps (10GBps shared link) *Kafka Cluster:* Two 3 node kafka clusters (namely primary, backup) Replication factor is 3 by default Keberos authentication enabled *MM2 setup:* 3 instances of MM2 8 GB RAM ||Test 1||Test 2||Test 3|| |one topic, 1000 partitions duration: 1 hour Message Size - 10KB MM2 conf: tasks.max = 100 Replication: primary -> backup cluster 60000 messages/2 second 600 MB/2 second * Latency: Avg. 4 seconds * MM2 process CPU usage took 80-100% on avg throughout the run duration * Memory usage is between 3-3.5 GB|two topics, 1000 partitions duration: 1 hour Message Size - 1KB MM2 conf: tasks.max = 100 Replication: primary -> backup backup -> primary 120000 messages/2 second (to each topic) 120 MB/2 second (to each topic) * Latency: Avg. 8 seconds * MM2 process CPU usage took 100-140% on avg throughout the execution * Memory usage is between 3-4 GB|6 topics, 6 partitions (each) duration: 1 hour Message Size - 1KB MM2 conf: tasks.max = 20 Replication: primary -> backup backup -> primary 20000 messages/2 second (to each topic) 20 MB/2 second (to each topic) * Latency: Avg. 150 milliseconds * Memory usage is between 3-4 GB| Based on the test results, * Number of tasks has direct impact on latency. More tasks -> less latency * Message size does not have any impacts on replication latency * In case of shared cluster (other services are also running), CPU cores >=32 yielded good results (latency in seconds) * It is good to have >= 8GB memory dedicated to MM2 instance * Higher Number of messages per time interval to one topic causes more latency than the same number of messages to multiple topics * When number of partitions increases (also tasks increased), latency decreases for that topic Thanks for all your efforts on MM2. Happy to provide more details if required. > MirrorMaker 2.0 (KIP-382) > ------------------------- > > Key: KAFKA-7500 > URL: https://issues.apache.org/jira/browse/KAFKA-7500 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect, mirrormaker > Affects Versions: 2.4.0 > Reporter: Ryanne Dolan > Assignee: Manikumar > Priority: Major > Labels: pull-request-available, ready-to-commit > Fix For: 2.4.0 > > Attachments: Active-Active XDCR setup.png > > > Implement a drop-in replacement for MirrorMaker leveraging the Connect > framework. > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0] > [https://github.com/apache/kafka/pull/6295] -- This message was sent by Atlassian Jira (v8.3.4#803005)