Re: questions on number of partitions
You may need to consider replication factor as well? Here you increased 4 times of partitions but if you are having 3 replication factor then it will be (4 * 3) times of partition copies in the cluster. On Sat, Nov 21, 2015 at 12:43 AM, Chen Song wrote: > Any thoughts on this topic. Not sure if others have seen the same spike as > us. > > On Tue, Nov 17, 2015 at 3:51 PM, Chen Song wrote: > > > BTW, we are running Kafka 0.8.2.2. > > > > On Tue, Nov 17, 2015 at 3:48 PM, Chen Song > wrote: > > > >> We have a cluster of 32 nodes cluster, with a few topics. When we set > 100 > >> partitions for each topic, the overall CPU usage is 30%, and when we > >> increase to 400 partitions, the CPU goes up over 65% - 70% sometimes. > The # > >> of producers are not changed. > >> > >> I understand that more partitions will drive up broker CPU usage. > >> However, this seems to be a limiting factor to parallelism (mostly for > >> consumer perspective) because all I test is just 400 partitions. > >> > >> Not sure if this is being addressed in an way. Any thoughts on this? > >> > >> -- > >> Chen Song > >> > >> > > > > > > -- > > Chen Song > > > > > > > -- > Chen Song >
Re: questions on number of partitions
Any thoughts on this topic. Not sure if others have seen the same spike as us. On Tue, Nov 17, 2015 at 3:51 PM, Chen Song wrote: > BTW, we are running Kafka 0.8.2.2. > > On Tue, Nov 17, 2015 at 3:48 PM, Chen Song wrote: > >> We have a cluster of 32 nodes cluster, with a few topics. When we set 100 >> partitions for each topic, the overall CPU usage is 30%, and when we >> increase to 400 partitions, the CPU goes up over 65% - 70% sometimes. The # >> of producers are not changed. >> >> I understand that more partitions will drive up broker CPU usage. >> However, this seems to be a limiting factor to parallelism (mostly for >> consumer perspective) because all I test is just 400 partitions. >> >> Not sure if this is being addressed in an way. Any thoughts on this? >> >> -- >> Chen Song >> >> > > > -- > Chen Song > > -- Chen Song
Re: questions on number of partitions
BTW, we are running Kafka 0.8.2.2. On Tue, Nov 17, 2015 at 3:48 PM, Chen Song wrote: > We have a cluster of 32 nodes cluster, with a few topics. When we set 100 > partitions for each topic, the overall CPU usage is 30%, and when we > increase to 400 partitions, the CPU goes up over 65% - 70% sometimes. The # > of producers are not changed. > > I understand that more partitions will drive up broker CPU usage. However, > this seems to be a limiting factor to parallelism (mostly for consumer > perspective) because all I test is just 400 partitions. > > Not sure if this is being addressed in an way. Any thoughts on this? > > -- > Chen Song > > -- Chen Song
questions on number of partitions
We have a cluster of 32 nodes cluster, with a few topics. When we set 100 partitions for each topic, the overall CPU usage is 30%, and when we increase to 400 partitions, the CPU goes up over 65% - 70% sometimes. The # of producers are not changed. I understand that more partitions will drive up broker CPU usage. However, this seems to be a limiting factor to parallelism (mostly for consumer perspective) because all I test is just 400 partitions. Not sure if this is being addressed in an way. Any thoughts on this? -- Chen Song
