Matt, Today, data from a partition is always consumed by a single consumer, unless a rebalance is triggered. A rebalance only happens when there is a broker or consumer change (e.g., going up/down).
In the near future when we have replication, a partition will never go away in theory. So, rebalance only happens when there is a consumer change. Jun On Thu, Aug 11, 2011 at 2:45 PM, Bateman, Matt <[email protected]> wrote: > Hi Jun, > > From the paper it appears that what I'm looking for is scheduled as > "future" work. I think this paragraph illustrates the functionality we're > looking for: > > "At the lowest level this is supported by semantically partitioning > messages on the join key during publishing so that all messages sent with a > particular key go to the same partition and hence arrive at a single > consumer process. This provides the foundation for processing distributed > streams across a cluster of consumer machines." > > We need to guarantee that all messages in a single partition are delivered > to the _same_ consumer process. > > Am I correct in understanding that this is not currently possible? > > One way of achieving this might be to use multiple topics and publish > messages to a particular topic which is being consumed by only one consumer > process. Am I missing something fundamental or might this work? > > Thanks for your help, > > Matt > > -----Original Message----- > From: Jun Rao [mailto:[email protected]] > Sent: Sunday, August 07, 2011 5:30 PM > To: [email protected] > Subject: Re: More Information On Partitioning > > You can take a look at the Kafka paper in NetDB ( > http://research.microsoft.com/en-us/um/people/srikanth/netdb11/netdb11papers/netdb11-final12.pdf > ). > If you have more questions after reading, feel free to ask again. > > Jun > > On Fri, Aug 5, 2011 at 6:08 PM, Bateman, Matt <[email protected]> wrote: > > > Hi All, > > > > The documentation on partitioning is a little light on the website. Is > > there somewhere I can get more detail as to how this exactly works? > > I'm interested from the deployment, load balancing and intended use > > patterns perspectives. > > > > I'm happy to read whatever docs I've missed if pointed in the right > > direction. > > > > Thanks, > > > > Matt > > >
