GitHub user onurkaraman opened a pull request:
https://github.com/apache/kafka/pull/2177
KAFKA-3959: enforce offsets.topic.replication.factor
Kafka brokers have a config called "offsets.topic.replication.factor" that
specify the replication factor for the "__consumer_offsets" topic. The problem
is that this config isn't being enforced. If an attempt to create the internal
topic is made when there are fewer brokers than
"offsets.topic.replication.factor", the topic ends up getting created anyway
with the current number of live brokers. The current behavior is pretty
surprising when you have clients or tooling running as the cluster is getting
setup. Even if your cluster ends up being huge, you'll find out much later that
__consumer_offsets was setup with no replication.
The cluster not meeting the "offsets.topic.replication.factor" requirement
on the internal topic is another way of saying the cluster isn't fully setup
yet.
The right behavior should be for "offsets.topic.replication.factor" to be
enforced. Topic creation of the internal topic should fail with
GROUP_COORDINATOR_NOT_AVAILABLE until the "offsets.topic.replication.factor"
requirement is met. This closely resembles the behavior of regular topic
creation when the requested replication factor exceeds the current size of the
cluster, as the request fails with error INVALID_REPLICATION_FACTOR.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/onurkaraman/kafka KAFKA-3959
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2177.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2177
----
commit ef8558215ff049f0787ae6df688fd25194b7aae2
Author: Onur Karaman <[email protected]>
Date: 2016-11-28T05:29:28Z
KAFKA-3959: enforce offsets.topic.replication.factor
Kafka brokers have a config called "offsets.topic.replication.factor" that
specify the replication factor for the "__consumer_offsets" topic. The problem
is that this config isn't being enforced. If an attempt to create the internal
topic is made when there are fewer brokers than
"offsets.topic.replication.factor", the topic ends up getting created anyway
with the current number of live brokers. The current behavior is pretty
surprising when you have clients or tooling running as the cluster is getting
setup. Even if your cluster ends up being huge, you'll find out much later that
__consumer_offsets was setup with no replication.
The cluster not meeting the "offsets.topic.replication.factor" requirement
on the internal topic is another way of saying the cluster isn't fully setup
yet.
The right behavior should be for "offsets.topic.replication.factor" to be
enforced. Topic creation of the internal topic should fail with
GROUP_COORDINATOR_NOT_AVAILABLE until the "offsets.topic.replication.factor"
requirement is met. This closely resembles the behavior of regular topic
creation when the requested replication factor exceeds the current size of the
cluster, as the request fails with error INVALID_REPLICATION_FACTOR.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
