[
https://issues.apache.org/jira/browse/KAFKA-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayuresh Gharat reassigned KAFKA-1998:
--------------------------------------
Assignee: Mayuresh Gharat
> Partitions Missing From MetadataResponse
> ----------------------------------------
>
> Key: KAFKA-1998
> URL: https://issues.apache.org/jira/browse/KAFKA-1998
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.2.0
> Reporter: Evan Huus
> Assignee: Mayuresh Gharat
>
> It is known behaviour that when a partition is entirely offline (it has no
> leader because all of its replicas are down) then that partition will not be
> included in the metadata returned by other brokers. For example, if topic
> "foo" has 3 partitions, but all replicas of partition 3 are offline, then
> requesting metadata for "foo" will only return information about partitions 1
> and 2.
> This means that there is no way to reliably determine the number of
> partitions for a topic via kafka's metadata API; if I receive information on
> partitions 1 and 2, I don't know if partition 3 is offline or if it is simply
> that there are only two partitions total. (You can presumably still ask
> zookeeper directly, but that is a work-around).
> This ambiguity, in turn, can lead to a consistency problem with the default
> partitioner, since that effectively implements `hash(key) mod #partitions`.
> If a partition goes offline and is removed from the metadata response, then
> the number of partitions the producer knows about will change (on its next
> metadata refresh) and the mapping from keys to partitions will also change.
> Instead of distributing messages among (for example) 3 partitions, and
> failing to produce to the offline partition, it will distribute *all*
> messages among the two online partitions. This results in messages being sent
> to the wrong partition.
> Since kafka already returns partitions with error messages in many cases
> (e.g. `LeaderNotAvailable`) I think it makes much more sense and fixes the
> above partition problem if it would simply return offline partitions as well
> with the appropriate error (whether that is `LeaderNotAvailable` or it would
> be better to add an additional error is up to you).
> CC [~guozhang]
> (This issue was originally described/discussed on the kafka-users mailing
> list, in the thread involving
> https://mail-archives.apache.org/mod_mbox/kafka-users/201503.mbox/%3CCAA4pprAZvp2XhdNmy0%2BqVZ1UVdVxmUfz3DDArhGbwP-iiH%2BGyg%40mail.gmail.com%3E)
> If there are any questions I am happy to clarify, I realize the scenario is
> somewhat complex.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
