We have 5 kafka brokers available, and created a topic with replication
factor of 3. After a few broker issues (e.g. went out of file descriptors),
running kafkacat on the producer node shows the following:
Command:
kafkacat-CentOS-6.5-x86_64 -L -b "kafka01-east.manage.com,
kafka02-east.manage.com,kafka03-east.manage.com,kafka04-east.manage.com,
kafka05-east.manage.com"
Output:
5 brokers:
broker 385 at kafka04-east.manage.com:9092
broker 389 at kafka03-east.manage.com:9092
broker 381 at kafka01-east.manage.com:9092
broker 387 at kafka05-east.manage.com:9092
broker 383 at kafka02-east.manage.com:9092
...
topic "raw-events" with 32 partitions:
partition 23, leader 387, replicas: 389,387,381, isrs: 387,389
partition 8, leader 389, replicas: 381,389,383, isrs: 389,381
partition 17, leader 389, replicas: 383,389,381, isrs: 389,381
partition 26, leader 387, replicas: 387,389,381, isrs: 387,389
partition 11, leader 387, replicas: 389,387,381, isrs: 387,389
partition 29, leader 389, replicas: 383,389,381, isrs: 389,381
partition 20, leader 389, replicas: 381,389,383, isrs: 389,381
partition 2, leader 387, replicas: 387,389,381, isrs: 387
partition 5, leader 389, replicas: 383,389,381, isrs: 389,381
partition 14, leader 387, replicas: 387,389,381, isrs: 387,389
partition 4, leader 387, replicas: 381,387,389, isrs: 387,389
partition 13, leader 387, replicas: 383,387,389, isrs: 387,389
partition 22, leader 389, replicas: 387,383,389, isrs: 389,387
partition 31, leader 387, replicas: 389,383,387, isrs: 387,389
partition 7, leader 387, replicas: 389,383,387, isrs: 387,389
partition 16, leader 387, replicas: 381,387,389, isrs: 387
partition 25, leader 387, replicas: 383,387,389, isrs: 387,389
partition 10, leader 387, replicas: 387,383,389, isrs: 387,389
partition 1, leader 387, replicas: 383,387,389, isrs: 387,389
partition 28, leader 387, replicas: 381,387,389, isrs: 387
partition 19, leader 387, replicas: 389,383,387, isrs: 387,389
partition 18, leader 387, replicas: 387,381,383, isrs: 387,381
partition 9, leader 387, replicas: 383,381,387, isrs: 387,381
partition 27, leader 389, replicas: 389,381,383, isrs: 389,381
partition 12, leader 387, replicas: 381,383,387, isrs: 387,381
partition 21, leader 387, replicas: 383,381,387, isrs: 387,381
partition 3, leader 389, replicas: 389,381,383, isrs: 389,381
partition 30, leader 387, replicas: 387,381,383, isrs: 387,381
partition 15, leader 389, replicas: 389,381,383, isrs: 389,381
partition 6, leader 387, replicas: 387,381,383, isrs: 387,381
partition 24, leader 387, replicas: 381,383,387, isrs: 387,381
partition 0, leader 387, replicas: 381,383,387, isrs: 387,381
I notice that some partition (partition #2 for example) only has 1 node
under isrs. From what I read, isrs shows a list of brokers that have data
that is in-sync.
My question is - now some partitions are out of sync. What do I do to get
them in sync again?
thanks
thomas
