You can't you only get a guarantee on the order for each partition, not over partitions. Adding partitions will possible make it a lot worse, since items with the same key wll land in other partitions. For example with two partitions these will be about the hashes in each partitions: partition-0: 0,2,4,6,8 partition-1: 1,3,5,7,9 With three partitions: partition-0: 0,3,6,9 partition-1: 1,4,7 partition-2: 2,5,8 So items with a key which hashes to 2, will move from partition-0, to partition-2. I think if you really need to be able to guarantee order, you will need to add some sequential id to the messages, and buffer them when reading, but this will have all sorts of drawbacks, like lossing the messages in the buffer in case of error (or you must make the commit offset dependent on the buffer).
On Mon, Jul 18, 2016 at 9:19 PM Fumo, Vincent <[email protected]> wrote: > I want to use Kafka for notifications of changes to data in a > dataservice/database. For each object that changes, a kafka message will be > sent. This is easy and we've got that working no problem. > > Here is my use case : I want to be able to fire up a process that will > > 1) determine the current location of the kafka topic (right now we use 2 > partitions so that would be the offset for each partition) > 2) do a long running process that will copy data from the database > 3) once the process is over, put the location back into a kafka consumer > and start processing notifications in sequence > > This isn't very hard either but there is a problem that we'd face if > during step (2) partitions are added to the topic (say by our operations > team). > > I know we can set up a ConsumerRebalanceListener but I don't think that > will help because we'd need to back to a time when we had our original > number of partitions and then we'd need to know exactly when to start > reading from the new partition(s). > > for example > > start : 2 partitions (0,1) at offsets p0,100 and p1,100 > > 1) we store the offsets and partitions : p0,100 and p1,100 > 2) we run the db ingest > 3) messages are posted to p0 and p1 > 4) OPS team adds p2 and our ConsumerRebalanceListener would be notified > 5) we are done and we set our consumer to p0,100 and p1,100 (and p2,0 > thanks to the ConsumerRebalanceListener) > > how would we guarantee the order of messages received from our consumer > across all 3 partitions? > > >
