The number of brokers doesn't really matter here, as far as I can tell, because the question is about what a single broker can handle. The number of partitions in the cluster is governed by the ability of the controller to manage the list of partitions for the cluster, and the ability of each broker to keep that list (to serve metadata requests). The number of partitions on a single broker is governed by that broker's ability to handle the messages and files on disk. That's a much more limiting factor than what the controller can do.
-Todd On Tue, Oct 21, 2014 at 2:52 PM, Neil Harkins <nhark...@gmail.com> wrote: > On Tue, Oct 21, 2014 at 2:10 PM, Todd Palino <tpal...@gmail.com> wrote: > > As far as the number of partitions a single broker can handle, we've set > > our cap at 4000 partitions (including replicas). Above that we've seen > some > > performance and stability issues. > > How many brokers? I'm curious: what kinds of problems would affect > a single broker with a large number of partitions, but not affect the > entire cluster with even more partitions? >
