did you try setting these values in the distributor.properties file? On Mon, Sep 20, 2021 at 12:43 AM Jamie <jamied...@aol.co.uk.invalid> wrote:
> Hi All, > Has anyone been able to override the producer and consumer values of the > MM2 MirrorSourceConnector as described below? > Many Thanks, > Jamie > > > -----Original Message----- > From: Jamie <jamied...@aol.co.uk> > To: users@kafka.apache.org <users@kafka.apache.org> > Sent: Thu, 16 Sep 2021 15:52 > Subject: MM2 - Overriding MirrorSourceConnector Consumer & Producer values > > Hi All, > I've trying to override the properties of the consumer and producer in MM2 > to tune them for high throughput. For example, I'm trying to set the > consumers fetch.min.bytes to 100000. > I'm running MM2 in a dedicated mirror maker cluster ( > https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP382:MirrorMaker2.0-RunningadedicatedMirrorMakercluster) > and using version 2.7.1 of Kafka. > I have the equivalent of the following in my mirror maker properties file: > clusters = A, B > A.bootstrap.servers = A_host1:9092, A_host2:9092, A_host3:9092 > B.bootstrap.servers = B_host1:9092, B_host2:9092, B_host3:9092 > # enable and configure individual replication flows A->B.enabled = > true > # regex which defines which topics gets replicated. For eg "foo-.*" > A->B.topics = .test-topic > > I'm trying to override the properties of the consumer which fetches > records from cluster "A" and the producer that sends records to cluster > "B". > I've tried the following in the config file: > A.consumer.fetch.min.bytes = 100000 > A->B.consumer.fetch.min.bytes = 100000 > A.fetch.min.bytes = 100000 > B.consumer.fetch.min.bytes = 100000 > B.fetch.min.bytes = 100000 > None of which seem to work, when I start MM2 and go into the logs and look > at the value using the the MirrorSourceConnector tasks consumer and > producer config I still see the default value for fetch.min.bytes (1) being > used. > Am I trying to override the values of the consumer incorrectly or do I > need to set these in a different place? > Many Thanks, > Jamie > > > >
