Iftach,
This is a very useful finding. While I don't know the answer to your
question below, I would like to take this opportunity to encourage you
to write a blog about this finding =)
Thanks,
-- Ricardo
On 7/7/20 2:48 AM, Iftach Ben-Yosef wrote:
I believe I got it to work with
"source->d
I believe I got it to work with "source->dest.producer.compression.type =
gzip"
Is there a way to set this globally for the mm2 process and not to do it
per mirroring flow?
Thanks,
Iftach
On Tue, Jul 7, 2020 at 9:34 AM Iftach Ben-Yosef
wrote:
> Upon further investigation, it the issue is indee
Upon further investigation, it the issue is indeed compression as in the
logs i see 'ompression.type = none'
Does anyone know how to configure gzip compression for
the connect-mirror-maker.properties file?
I tried "producer.override.compression.type = gzip" but that doesnt seem to
work.
Thanks,
I
Ricardo,
Thanks for the reply. I did some more testing. I tried mirroring a
different topic from 1 of the 3 source clusters used from the previous
test, into the same destination cluster. Again, the result topic on the
dest cluster is about 2 times larger than the source, same config and
retention
Iftach,
I think you should try observe if this happens with other topics. Maybe
something unrelated might have happened already in the case of the topic
that currently has ~3TB of data -- making things even harder to
troubleshoot.
I would recommend creating a new topic with few partitions an
Hi Ryanne, thanks for the quick reply.
I had the thought it might be compression. I see that the topics have the
following config "compression.type=producer". This is for both the source
and destination topics. Should I check something else regarding compression?
Also, the destination topics are
Iftach, is it possible the source topic is compressed?
Ryanne
On Wed, Jul 1, 2020, 8:39 AM Iftach Ben-Yosef
wrote:
> Hello everyone.
>
> I'm testing mm2 for our cross dc topic replication. We used to do it using
> mm1 but faced various issues.
>
> So far, mm2 is working well, but I have 1 issue
Hello everyone.
I'm testing mm2 for our cross dc topic replication. We used to do it using
mm1 but faced various issues.
So far, mm2 is working well, but I have 1 issue which I can't really
explain; the destination topic is larger than the source topic.
For example, We have 1 topic which on the
