Venkatesh created KAFKA-7357:
--------------------------------
Summary: Reassigned partitions but latest data is still going to
old broker
Key: KAFKA-7357
URL: https://issues.apache.org/jira/browse/KAFKA-7357
Project: Kafka
Issue Type: Bug
Reporter: Venkatesh
Kafka version: kafka_2.11-0.10.0.0.jar
I reassigned a topic with 12 partitions, replicationFactor 1, across 6 brokers
using kafka manager. According to the kafka manager, reassignment successfully
completed.
Then I observed that new data was NOT being written on some of the partitions
on the new broker for the partition.
Ex: {code:java}
Topic Name: db-_-metric-_-vm-_-_default_
Partitions not receiving new data: 5
OldBroker for Partition 5, before reassignment: 11
NewBroker for Partition 5, after reassignment: 9
{code}
Observations:
- Both brokers have directory db-_-metric-_-vm-_-_default_5 in their disk
partitions. If the reassignment was successful, it should have been deleted on
the old broker
- New data being produced for partition 5 is being written on the old broker,
confirmed by looking at the corresponding directory on the old broker 11
- New broker 9 has some files in the directory corresponding to the partition
and they are not being updated, this confirms that new data is not reaching the
partition on the new broker after reassignment.
- Ran following command observed that the offset is not changing for partition 5
{code:java}
./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9446
--topic db-_-metric-_-vm-_-_default_ --time -1
Output:
db-_-metric-_-vm-_-_default_:8:23686375
db-_-metric-_-vm-_-_default_:11:2418867
db-_-metric-_-vm-_-_default_:2:149930053
db-_-metric-_-vm-_-_default_:5:2362103
db-_-metric-_-vm-_-_default_:4:64402526
db-_-metric-_-vm-_-_default_:7:23178223
db-_-metric-_-vm-_-_default_:10:4122808
db-_-metric-_-vm-_-_default_:1:78642586
db-_-metric-_-vm-_-_default_:9:1049046
db-_-metric-_-vm-_-_default_:3:62158391
db-_-metric-_-vm-_-_default_:6:22640686
db-_-metric-_-vm-_-_default_:0:84008474
{code}
Questions:
- Is this a known issue with this version of kafka ? if so, is it fixed ? in
which version ?
- How to recover the partitions from the current state so that no data is lost
?
Thanks for all your help.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
