Dave, Currently, the broker expects each producer request to specify the exact partition id (-1 is on longer valid). The mapping from a message to a partition is done at the producer client. The producer can choose a random partition (from the existing list of partitions) or deterministically choose a partition based on the key.
Thanks, Jun On Tue, May 21, 2013 at 1:12 PM, Dave Peterson <dspeter...@tagged.com>wrote: > In my case, there is a load balancer between the producers and the > brokers, so I want the behavior described for the Java client (null key > specifies "any partition"). If the Key field of each individual message > specifies the partition to send it to, then I don't understand the purpose > of the 32-bit partition identifier that precedes each message set in a > produce request: what if a produce request specifies "partition N" for a > given message set, and then each individual message in the set > specifies a different partition in its Key field? Also, the above- > mentioned partition identifier is a 32-bit integer and the Key field of > each individual message can contain data of arbitrary length, which > seems inconsistent. Is a partition identifier a 32-bit integer, or can it > be of arbitrary length? > > Thanks, > Dave > > On Tue, May 21, 2013 at 12:30 PM, Neha Narkhede <neha.narkh...@gmail.com> > wrote: > > Dave, > > > > Colin described the producer behavior of picking the partition for a > > message before it is sent to Kafka broker correctly. However, I'm > > interested in knowing your use case a little before to see why you would > > rather have the broker decide the partition? > > > > Thanks, > > Neha > > > > > > On Tue, May 21, 2013 at 12:05 PM, Colin Blower <cblo...@barracuda.com > >wrote: > > > >> The key is used by the client to decide which partition to send the > >> message to. By the time the client is creating the produce request, it > >> should be known which partition each message is being sent to. I believe > >> Neha described the behavior of the Java client which sends messages with > >> a null key to any partition. > >> > >> The key is described in past tense because of the use case for > >> persisting keys with messages. The key is persisted through the broker > >> so that a consumer knows what key was used to partition the message on > >> the producer side. > >> > >> I don't believe that you can have the broker decide which partition a > >> message goes to. > >> > >> -- > >> Colin B. > >> > >> On 05/21/2013 11:48 AM, Dave Peterson wrote: > >> > I'm looking at the document entitled "A Guide to the Kafka Protocol" > >> > located here: > >> > > >> > https://cwiki.apache.org/KAFKA/a-guide-to-the-kafka-protocol.html > >> > > >> > It shows a produce request as containing a number of message sets, > which > >> are > >> > grouped first by topic and second by partition (a 32-bit integer). > >> > However, each > >> > message in a message set contains a Key field, which is described as > >> follows: > >> > > >> > The key is an optional message key that was used for partition > >> assignment. > >> > The key can be null. > >> > > >> > I notice the use of "was" (past tense) above. That seems to suggest > >> that the > >> > Key field was once used to specify a partition (at the granularity of > >> each > >> > individual message), but the plan for the future is to instead use the > >> 32-bit > >> > partition value preceding each message set. Is this correct? If so, > >> when I am > >> > creating a produce request for 0.8, what should I use for the 32-bit > >> partition > >> > value, and how does this relate to the Key field of each individual > >> message? > >> > Ideally, I would like to just send a produce request and let the > broker > >> choose > >> > the partition. How do I accomplish this in 0.8, and are there plans > to > >> change > >> > this after 0.8? > >> > > >> > Thanks, > >> > Dave > >> > > >> > On Tue, May 21, 2013 at 10:47 AM, Neha Narkhede < > neha.narkh...@gmail.com> > >> wrote: > >> >> No. In 0.8, if you don't specify a key for a message, it is sent to > any > >> of > >> >> the available partitions. In other words, the partition id is > selected > >> on > >> >> the partition and the server doesn't get -1 as the partition id. > >> >> > >> >> Thanks, > >> >> Neha > >> >> > >> >> > >> >> On Tue, May 21, 2013 at 9:54 AM, Dave Peterson < > dspeter...@tagged.com > >> >wrote: > >> >> > >> >>> In the version 0.8 wire format for a produce request, does a value > of > >> -1 > >> >>> still indicate "use a random partition" as it did for 0.7? > >> >>> > >> >>> Thanks, > >> >>> Dave > >> >>> > >> > >> > >> >
