Re: Mirror Maker 2.0 Queries

Thank you Ryanne for the quick response. I further want to clarify a few points. The mirror maker 2.0 is based on the Kafka Connect framework. In Kafka connect we have multiple workers and each worker has some assigned task. To map this to Mirror Maker 2.0, A mirror Maker will driver have some

Re: Kafka topic partition distributing evenly on disks

Or manually you can move data dir . I'm assuming you have replica >1 Stop the kafka process on broker 1 Move 1 or 2 dir log from Disk 1 to disk 2 And start the kafka process Wait for ISR sync Then you can repeate this step again . On 8/7/20, 6:45 AM, "William Reynolds" wrote: [Externa

anonymization of Avro messages

Hello kafka community, We have Avro topics (with Avro values). We think of annonymizing the data to use it on UAT. Do you know of any tools that can be used or bought? Thanks, -- Dumitru-Nicolae Marasoui Software Engineer w kaluza.com LinkedIn

Re: Kafka topic partition distributing evenly on disks

Hmm, that's odd, I am sure it was in the docs previously. Here is the KIP on it https://cwiki.apache.org/confluence/display/KAFKA/KIP-113%3A+Support+replicas+movement+between+log+directories Basically the reassignment json that you get looks like this from the initial generation and if you already

Re: Kafka topic partition distributing evenly on disks

Thank you William, I checked the doc and don't see any instructions regarding disks. Should I simply "move around" the topics and Kafka will assign the topics evenly on the two disks (per broker)? The current setup looks like this (for the topic in question, 15 primary, replica partitions): Broke

Re: Kafka topic partition distributing evenly on disks

Hi Péter, Sounds like time to reassign the partitions you have across all the brokers/data dirs using the instructions from here https://kafka.apache.org/documentation/#basic_ops_automigrate. That assumes that your partition strategy has somewhat evenly filled your partitions and given it may move

Re: Kafka topic partition distributing evenly on disks

Hello everybody, Thank you for the detailed answers. My issue is partly answered here: *This rule also applies to disk-level, which means that when a set ofpartitions assigned to a specific broker, each of the disks will get thesame number of partitions without considering the load of disks at

Re: Kafka topic partition distributing evenly on disks

Hi Peter, Agreed with Manoj and Vinicius, i think those rules led to this result : 1）the partitions of a topic - N and replication number - R determine the real partition-replica count of this topic, which is N * R; 2) kafka can distribute partitions evenly among brokers, but it is based on the

Kafka-client 2.5.0 connection to Azure Event Hub authentication failure

Hello, When trying to connect our Kafka client to an Azure Event Hub via SASL_SSL we encounter an error in the authentication process. IllegalSaslStateException: Invalid SASL mechanism response, server may be expecting a different protocol at 2020-08-07T05:03:16.072637487Z, trace: org.apache.ka