Hi, I'm currently trying to understand how Kafka (0.8) can scale with our usage pattern and how to setup the partitioning.
We want to route the same messages belonging to the same id to the same queue, so its consumer will able to consume all the messages of that id. My questions: - From my understanding, in Kafka we would need to have a custom partitioner that routes the same messages to the same partition right? I'm trying to find examples of writing this partitioner logic, but I can't find any. Can someone point me to an example? - I see that Kafka server.properties allows one to specify the number of partitions it supports. However, when we want to scale I wonder if we add # of partitions or # of brokers, will the same partitioner start distributing the messages to different partitions? And if it does, how can that same consumer continue to read off the messages of those ids if it was interrupted in the middle? - I'd like to create a consumer per partition, and for each one to subscribe to the changes of that one. How can this be done in kafka? Thanks, Tim
