Hi everyone, I am new here, recently join the group. I faced a problem in Kafka Cluster, a problem is described below.
I am using Kafka version 0.9.0.0 We have established a Kafka Cluster of 3 machines where 2 machines are utilized for Kafka broker and same 3 machines utilized for zookeeper. When we start our consumer written in Kafka-Python, Kafka brokers create a new topic name __consumer_offsets with 50 partitions and spread over 100 % in Kafka broker with replica 2. I know why topic is get created it is because offsets are stored in Kafka broker instead of zookeeper but problem is each partition of __consumer_offsets take a lot of space as compared to topic data and yesterday I have restarted my cluster gain due to lack of space. Can you exactly help me why __consumer_offsets are taking a lot of space? and how would I get rid of this. Topic data space is 2 GB and while __consumer_offsets are taking 92 GB. My server size is 98 GB with 4 GB RAM. *Thanks, Kunal* *+91-9958189589* *Data Analyst* *First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798 <http://dl.acm.org/citation.cfm?id=2790798> * *Blog:- **http://learnhardwithkunalgupta.blogspot.in <http://learnhardwithkunalgupta.blogspot.in> *
