Or manually you can move data dir . I'm assuming you have replica >1
Stop the kafka process on broker 1
Move 1 or 2 dir log from Disk 1 to disk 2
And start the kafka process
Wait for ISR sync
Then you can repeate this step again .
On 8/7/20, 6:45 AM, "William Reynolds"
wrote:
[Externa
Hmm, that's odd, I am sure it was in the docs previously. Here is the
KIP on it
https://cwiki.apache.org/confluence/display/KAFKA/KIP-113%3A+Support+replicas+movement+between+log+directories
Basically the reassignment json that you get looks like this from the
initial generation and if you already
Thank you William,
I checked the doc and don't see any instructions regarding disks. Should I
simply "move around" the topics and Kafka will assign the topics evenly on
the two disks (per broker)? The current setup looks like this (for the
topic in question, 15 primary, replica partitions):
Broke
Hi Péter,
Sounds like time to reassign the partitions you have across all the
brokers/data dirs using the instructions from here
https://kafka.apache.org/documentation/#basic_ops_automigrate. That
assumes that your partition strategy has somewhat evenly filled your
partitions and given it may move
Hello everybody,
Thank you for the detailed answers. My issue is partly answered here:
*This rule also applies to disk-level, which means that when a set
ofpartitions assigned to a specific broker, each of the disks will get
thesame number of partitions without considering the load of disks at
Hi Peter,
Agreed with Manoj and Vinicius, i think those rules led to this result :
1）the partitions of a topic - N and replication number - R determine the
real partition-replica count of this topic, which is N * R;
2) kafka can distribute partitions evenly among brokers, but it is based
on the
Hi Peter,
AFAIK, everything depends on:
1) How you have configured your topic
a) number of partitions (here I understand you have 15 partitions)
b) partition replication configuration (each partition necessarily has a
leader - primary responsible to hold the data - and for reads and writes)
y
I initially started with one data disk (mounted solely to hold Kafka data)
and recently added a new one.
On Thu, Aug 6, 2020 at 10:13 PM wrote:
> What do you mean older disk ?
>
> On 8/6/20, 12:05 PM, "Péter Nagykátai" wrote:
>
> [External]
>
>
> Yeah, but it doesn't do that. My "older
What do you mean older disk ?
On 8/6/20, 12:05 PM, "Péter Nagykátai" wrote:
[External]
Yeah, but it doesn't do that. My "older" disks have ~70 partitions, the
newer ones ~5 partitions. That's why I'm asking what went wrong.
On Thu, Aug 6, 2020 at 8:35 PM wrote:
> Kafka
Yeah, but it doesn't do that. My "older" disks have ~70 partitions, the
newer ones ~5 partitions. That's why I'm asking what went wrong.
On Thu, Aug 6, 2020 at 8:35 PM wrote:
> Kafka evenly distributed number of partition on each disk so in your case
> every disk should have 3/2 topic partition
Kafka evenly distributed number of partition on each disk so in your case
every disk should have 3/2 topic partitions .
It is producer job to evenly produce data by partition key to topic partition .
How it partition key , it is auto generated or producer sending key along with
message .
On
Hello,
I have a Kafka cluster with 3 brokers (v2.3.0) and each broker has 2 disks
attached. I added a new topic (heavyweight) and was surprised that even if
the topic has 15 partitions, those weren't distributed evenly on the disks.
Thus I got one disk that's almost empty and the other almost fill
12 matches
Mail list logo
