Evan Huus created KAFKA-1998:
--------------------------------
Summary: Partitions Missing From MetadataResponse
Key: KAFKA-1998
URL: https://issues.apache.org/jira/browse/KAFKA-1998
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.2.0
Reporter: Evan Huus
It is known behaviour that when a partition is entirely offline (it has no
leader because all of its replicas are down) then that partition will not be
included in the metadata returned by other brokers. For example, if topic "foo"
has 3 partitions, but all replicas of partition 3 are offline, then requesting
metadata for "foo" will only return information about partitions 1 and 2.
This means that there is no way to reliably determine the number of partitions
for a topic via kafka's metadata API; if I receive information on partitions 1
and 2, I don't know if partition 3 is offline or if it is simply that there are
only two partitions total. (You can presumably still ask zookeeper directly,
but that is a work-around).
This ambiguity, in turn, can lead to a consistency problem with the default
partitioner, since that effectively implements `hash(key) mod #partitions`. If
a partition goes offline and is removed from the metadata response, then the
number of partitions the producer knows about will change (on its next metadata
refresh) and the mapping from keys to partitions will also change. Instead of
distributing messages among (for example) 3 partitions, and failing to produce
to the offline partition, it will distribute *all* messages among the two
online partitions. This results in messages being sent to the wrong partition.
Since kafka already returns partitions with error messages in many cases (e.g.
`LeaderNotAvailable`) I think it makes much more sense and fixes the above
partition problem if it would simply return offline partitions as well with the
appropriate error (whether that is `LeaderNotAvailable` or it would be better
to add an additional error is up to you).
CC [~guozhang]
(This issue was originally described/discussed on the kafka-users mailing list,
in the thread involving
https://mail-archives.apache.org/mod_mbox/kafka-users/201503.mbox/%3CCAA4pprAZvp2XhdNmy0%2BqVZ1UVdVxmUfz3DDArhGbwP-iiH%2BGyg%40mail.gmail.com%3E)
If there are any questions I am happy to clarify, I realize the scenario is
somewhat complex.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
