[
https://issues.apache.org/jira/browse/KAFKA-1120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110002#comment-16110002
]
James Cheng commented on KAFKA-1120:
------------------------------------
[~onurkaraman], try this? I made it as copy/pasteable as possible.
{code}
wget http://apache.ip-guide.com/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz
tar xvfz kafka_2.11-0.11.0.0.tgz
cp -r kafka_2.11-0.11.0.0 kafka_2.11-0.11.0.0_2
KAFKA_DIR=kafka_2.11-0.11.0.0
( cd ${KAFKA_DIR};
cp config/server.properties config/server1.properties;
echo >> config/server1.properties;
echo broker.id=1 >> config/server1.properties;
echo delete.topic.enable=true >> config/server1.properties;
echo listeners=PLAINTEXT://:9092 >> config/server1.properties;
echo log.dirs=/tmp/kafka-logs1 >> config/server1.properties;
echo log.index.size.max.bytes=100000 >> config/server1.properties;
echo controlled.shutdown.max.retries=1 >> config/server1.properties;
)
KAFKA_DIR=kafka_2.11-0.11.0.0_2
( cd ${KAFKA_DIR};
cp config/server.properties config/server2.properties;
echo >> config/server2.properties;
echo broker.id=2 >> config/server2.properties;
echo delete.topic.enable=true >> config/server2.properties;
echo listeners=PLAINTEXT://:9093 >> config/server2.properties;
echo log.dirs=/tmp/kafka-logs2 >> config/server2.properties;
echo log.index.size.max.bytes=100000 >> config/server2.properties;
echo controlled.shutdown.max.retries=1 >> config/server2.properties;
)
# Start zookeeper and kafka brokers
# In terminal 1, zookeeper
cd kafka_2.11-0.11.0.0
./bin/zookeeper-server-start.sh config/zookeeper.properties
# In terminal 2, broker 1
cd kafka_2.11-0.11.0.0
./bin/kafka-server-start.sh config/server1.properties
# In terminal 3, broker 2
cd kafka_2.11-0.11.0.0_2
./bin/kafka-server-start.sh config/server2.properties
# Create topics
cd kafka_2.11-0.11.0.0
# create 5000 partitions, all leaders on broker 1
./bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic leader1
--replica-assignment `echo -n 1:2; for i in \`seq 4999\`; do echo -n ,1:2; done`
# create 5000 partitions, all leaders on broker 2
./bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic leader2
--replica-assignment `echo -n 2:1; for i in \`seq 4999\`; do echo -n ,2:1; done`
# Wait until topics are fully created
# Stop and immediately start a broker
1. Verify that broker1 is the controller
2. Ctrl-C on broker2. Broker 2 will attempt controlled shutdown and will give
up after 30 seconds.
3. The instant it exits, restart broker2.
4. Wait until both brokers have settled down and controller has settled down.
Notice that broker2 is missing from lots of ISRs.
{code}
> Controller could miss a broker state change
> --------------------------------------------
>
> Key: KAFKA-1120
> URL: https://issues.apache.org/jira/browse/KAFKA-1120
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.1
> Reporter: Jun Rao
> Labels: reliability
> Fix For: 1.0.0
>
>
> When the controller is in the middle of processing a task (e.g., preferred
> leader election, broker change), it holds a controller lock. During this
> time, a broker could have de-registered and re-registered itself in ZK. After
> the controller finishes processing the current task, it will start processing
> the logic in the broker change listener. However, it will see no broker
> change and therefore won't do anything to the restarted broker. This broker
> will be in a weird state since the controller doesn't inform it to become the
> leader of any partition. Yet, the cached metadata in other brokers could
> still list that broker as the leader for some partitions. Client requests
> routed to that broker will then get a TopicOrPartitionNotExistException. This
> broker will continue to be in this bad state until it's restarted again.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
