Hello, Following our experimenting with MM2, we seem to have a hard time recovering from lags created by increased traffic.
Since we're running MM2 on kubernetes, we can easily add more resources/instances to our MM2 cluster, which should in theory increase the clusters performance and allow us to eliminate accumulated lag faster. In practice, we see that increasing the number of MM2 instances we have does not drastically change the throughput per machine - newly created instances have much lower throughput than instances which existed before the increase. This made me think about the partition assignment strategy used by MM2. If I understand correctly, the default seems to be partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor] I believe changing this to round robin might help this? I tried setting it up in my /config/connect-mirror-maker.properties file as one of each of the following (assuming source / dest A / B respectively) A.consumer.partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor B.consumer.partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor A.partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor B.partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor consumer.partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor None of these seem to work. Am I doing it wrong? Is this setting even applicable to MM2? Thanks a lot, Iftach -- The above terms reflect a potential business arrangement, are provided solely as a basis for further discussion, and are not intended to be and do not constitute a legally binding obligation. No legally binding obligations will be created, implied, or inferred until an agreement in final form is executed in writing by all parties involved. This email and any attachments hereto may be confidential or privileged. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person. Thanks.
