Carita Ou created AMBARI-12678:
----------------------------------
Summary: Installing additional Kafka broker on node with hostname
that comes alphabetically before existing broker fails to start
Key: AMBARI-12678
URL: https://issues.apache.org/jira/browse/AMBARI-12678
Project: Ambari
Issue Type: Bug
Components: ambari-server
Affects Versions: 2.0.0, 2.1.0
Reporter: Carita Ou
Priority: Minor
1. Install Ambari with 3 nodes, example hostname01, hostname02, hostname03
2. Install Kafka service on hostname03. Kafka starts successfully
3. Go to hostname02 and add Kafka Broker. Start kafka broker and it will fail
with below error:
{code}
[2015-08-05 17:13:51,306] FATAL Fatal error during KafkaServerStartable
startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
java.lang.RuntimeException: A broker is already registered on the path
/brokers/ids/0. This probably indicates that you either have configured a
brokerid that is already in use, or else you have shutdown this broker and
restarted it faster than the zookeeper timeout so it appears to be
re-registering.
at kafka.utils.ZkUtils$.registerBrokerInZk(ZkUtils.scala:174)
at
kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:58)
at
kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:45)
at kafka.server.KafkaServer.startup(KafkaServer.scala:124)
at
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:29)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
{code}
The 2nd broker added on hostname02 failed to start because it was assigned the
same brokerid as the broker installed on hostname03. The brokerid is assigned
to a broker in kafka.py:
{code}
brokerid = str(sorted(params.kafka_hosts).index(params.hostname))
kafka_server_config['broker.id'] = brokerid
{code}
params.py gets the kafka_hosts from clusterHostInfo:
{code}
kafka_hosts = config['clusterHostInfo']['kafka_broker_hosts']
{code}
clusterHostInfo maintains the list of kafka brokers in order:
{code}
"kafka_broker_hosts": [
"hostname02",
"hostname03"
],
{code}
Since the brokerid is the index of the hostname, and the broker on hostname03
was added first, it was at index 0. After adding a new broker on hostname02, it
was inserted to the list at index 0 and hostname03 was moved to index 1.
Because the broker on hostname03 was not restarted, it was still running with
brokerid 0, so the broker on hostname02 with brokerid 0 was unable to start.
Workaround: Restart all brokers
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
