Hi,
We are using Kafka for our messaging system and we have an estimate for 200
TB/week in the coming months. Will it impact any performance for Kafka?
PS: We will be having greater than 2 lakh partitions.
--
Regards
Vamsi Subhash
We definitely need a retention policy of a week. Hence.
On Fri, Dec 19, 2014 at 7:40 PM, Achanta Vamsi Subhash
achanta.va...@flipkart.com wrote:
Hi,
We are using Kafka for our messaging system and we have an estimate for
200 TB/week in the coming months. Will it impact any performance for
hi,
Few things you have to plan for:
a. Ensure that from resilience point of view, you are having sufficient
follower brokers for your partitions.
b. In my testing of kafka (50TB/week) so far, haven't seen much issue with
CPU utilization or memory. I had 24 CPU and 32GB RAM.
c. 200,000 partitions
Yes. We need those many max partitions as we have a central messaging
service and thousands of topics.
On Friday, December 19, 2014, nitin sharma kumarsharma.ni...@gmail.com
wrote:
hi,
Few things you have to plan for:
a. Ensure that from resilience point of view, you are having sufficient
Subhash achanta.va...@flipkart.com
To: users@kafka.apache.org users@kafka.apache.org
Sent: Friday, December 19, 2014 9:00 AM
Subject: Re: Max. storage for Kafka and impact
Yes. We need those many max partitions as we have a central messaging
service and thousands of topics
@kafka.apache.org
Sent: Friday, December 19, 2014 9:00 AM
Subject: Re: Max. storage for Kafka and impact
Yes. We need those many max partitions as we have a central messaging
service and thousands of topics.
On Friday, December 19, 2014, nitin sharma kumarsharma.ni...@gmail.com
Subject: Re: Max. storage for Kafka and impact
Yes. We need those many max partitions as we have a central messaging
service and thousands of topics.
On Friday, December 19, 2014, nitin sharma
kumarsharma.ni...@gmail.com
wrote:
hi,
Few things you have to plan
From: Achanta Vamsi Subhash achanta.va...@flipkart.com
To: users@kafka.apache.org users@kafka.apache.org
Sent: Friday, December 19, 2014 9:00 AM
Subject: Re: Max. storage for Kafka and impact
Yes. We need those many max partitions as we have a central
messaging
service
