Dear all, Kafka supports adding new partition to a topic dynamically. So suppose that initially I have a topic T with two partitions P0, P1 and a key space of three keys K0, K1, K2. Suppose further that I am using some kind of hash partitioner modulo 2 (number of partitions) at the producer that maps (K0) to P0, and (K1,K2) to P1. Let’s further assume that I have two consumers C0 for P0, and C1 for P1. For simplicity, the consumers are doing some basic key-based aggregation (stored in some KV.).
After few time suppose that I added a new partition P2 into T. So now my producer will automatically (once detects the new number of partitions) uses the same hash partitioner modulo 3. so let’ assume now that the mapping keys-partitions become (K0, P0), (K1, P1), (K2, P2). 1. in such cases what would happen to K2 values already written into P1 (before the addition of the new partition P2). And what if I would like to have all same key records mapped to the same partition always, even when a new topic partition is added. That is, I do not want that my K2 records to be spread between partitions P1 and P2. Does Kafka framework provide such guarantee in some way? or the application must handle such requirement? 2. When the new partition P2 is added, and the mapping keys-partitions become (K0, P0), (K1, P1), (K2, P2), a rebalancing process will be triggered. As result of rebalancing, let’s suppose that P2 is assigned to C0. So how could C0 get the most recent aggregated value for K2 out of the KV store in C1 instead of restarting from a scratch state. In such scenario, is state reshuffling to appropriate consumers guaranteed by kafka? or it should be handled by the application if any? Thank you.
