hi, Few things you have to plan for: a. Ensure that from resilience point of view, you are having sufficient follower brokers for your partitions. b. In my testing of kafka (50TB/week) so far, haven't seen much issue with CPU utilization or memory. I had 24 CPU and 32GB RAM. c. 200,000 partitions means around 1MB/week/partition. are you sure you need so many partitions?
Regards, Nitin Kumar Sharma. On Fri, Dec 19, 2014 at 9:12 AM, Achanta Vamsi Subhash < achanta.va...@flipkart.com> wrote: > > We definitely need a retention policy of a week. Hence. > > On Fri, Dec 19, 2014 at 7:40 PM, Achanta Vamsi Subhash < > achanta.va...@flipkart.com> wrote: > > > > Hi, > > > > We are using Kafka for our messaging system and we have an estimate for > > 200 TB/week in the coming months. Will it impact any performance for > Kafka? > > > > PS: We will be having greater than 2 lakh partitions. > > > > -- > > Regards > > Vamsi Subhash > > > > > -- > Regards > Vamsi Subhash >
