Hello All, I am working on building an application on cloud and below are the high level requirements.
- Ingest data into Kafka with say 100 partitions (Incoming rate - 100,000 msgs/sec) - Read data from Kafka and process each data (Do some computation, compare with old data, find location etc) real time I need some suggestions on how to distribute and manage kafka consumers for my scenario. - If I deploy my application with kafka consumer threads then how can I distribute load and rebalance it if I add more partitions or during failure. - Should I be using some kind of data streaming platform like storm, flink, samza to get the data from kafka and stream it realtime? Thanks a bunch and appreciate all your help. Regards, Govind