Re: destination topics in mm2 larger than source topic

2020-07-06 Thread Iftach Ben-Yosef
I believe I got it to work with "source->dest.producer.compression.type = gzip" Is there a way to set this globally for the mm2 process and not to do it per mirroring flow? Thanks, Iftach On Tue, Jul 7, 2020 at 9:34 AM Iftach Ben-Yosef wrote: > Upon further investigation, it the issue is indee

Re: destination topics in mm2 larger than source topic

2020-07-06 Thread Iftach Ben-Yosef
Upon further investigation, it the issue is indeed compression as in the logs i see 'ompression.type = none' Does anyone know how to configure gzip compression for the connect-mirror-maker.properties file? I tried "producer.override.compression.type = gzip" but that doesnt seem to work. Thanks, I

Keys and partitions

2020-07-06 Thread Victoria Zuberman
Hi, I have userId as a key. Many users have moderate amounts of data but some users have more and some users have huge amount of data. I have been thinking about the following aspects of partitioning: 1. If two or more large users will fall into same partition I might end up with large part

Re: Latest Logstash 7.8 and compatibility with latest Kafka 2.5.0

2020-07-06 Thread allen chan
Best is to read the changelog of the plugin https://github.com/logstash-plugins/logstash-integration-kafka/blob/master/CHANGELOG.md they are up to 2.4.1 per 10.1.0 notes and you have to see what version is packaged with the release. If it is not the right version, you need to use automation or man

Re: Latest Logstash 7.8 and compatibility with latest Kafka 2.5.0

2020-07-06 Thread Sebastjan Vodušek
Hi, thank you for replay. I have read this on elastic site but I had some problems in production with latest Logstash 7.8 and Kafka 2.2 in production so I had to revert Logstash to one of previous versions. That is why I wanna be extra careful before upgrade. Br, Sebastjan On Jul 6 2020, at 3:

Problem with replication?!

2020-07-06 Thread s.schabbach
Hi there, I just have a problem with my kafka brokers, maybe a firewall issue, but I don’t know. I have got 3 Brokers at three different Servers (each with another IP) and on the first server running zookeeper: Server1:9092 (zookeeper:2182) Server2:9092 Server3:9092 And I have got

Re: Latest Logstash 7.8 and compatibility with latest Kafka 2.5.0

2020-07-06 Thread Pere Urbón Bayes
HI Sebastian, I would check for that https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html, where it says: This plugin uses Kafka Client 2.1.0. For broker compatibility, see the official Kafka compatibility reference

Latest Logstash 7.8 and compatibility with latest Kafka 2.5.0

2020-07-06 Thread Sebastjan Vodušek
Hello, does anyone know if latest Logstash 7.8 input plugin is compatible with latest Kafka 2.5.0? Thank you and best regards, Sebastjan