We normally run over 1,000 partitions per broker, and I know of a major company with 30+ kafka clusters that averages 1,100 partitions per broker across all clusters.
So 300 shouldn't be an issue as long as the throughput per partition isn't too high. Given that disk and cpu are so low, I'd guess this isn't the issue. On Wed, Mar 1, 2017 at 10:54 PM, Vijay Patil <vijay2110.t...@gmail.com> wrote: > 300 partitions is at lower side, surely that won't be the root cause. > How about usage of network bandwidth for your nodes? > Are they reachable to zookeeper? > Are you executing some partition-rebalancing jobs (or re-assigning > partitions) in parallel? > > On 2 March 2017 at 10:57, Jun MA <mj.saber1...@gmail.com> wrote: > > > Hi, > > > > I’m curious what’s the recommended number of partitions running on each > > individual broker? We have a 3 nodes clusters running 0.9.0.1, each one > has > > 24 cores, 1.1T ssd, 48G ram, 10G NIC. There’s about 300 partitions > running > > on each broker and the resource usage is pretty low (5% cpu, 50% ram, 2% > > disk). But we see a lot random ISR expand/shrink happens to the cluster. > > Our networking is pretty stable, so I’m wondering if that’s because more > > partitions than kafka could handle? Is there any kafka parameters that we > > could tuning? > > > > Thanks, > > Jun >
