RAID-10? Interesting choice for a system where the data is already replicated between nodes. Is it to avoid the cost of large replication over the network? how large are these disks?
On Wed, Oct 22, 2014 at 10:00 AM, Todd Palino <tpal...@gmail.com> wrote: > In fact there are many more than 4000 open files. Many of our brokers run > with 28,000+ open files (regular file handles, not network connections). In > our case, we're beefing up the disk performance as much as we can by > running in a RAID-10 configuration with 14 disks. > > -Todd > > On Tue, Oct 21, 2014 at 7:58 PM, Xiaobin She <xiaobin...@gmail.com> wrote: > >> Todd, >> >> Actually I'm wondering how kafka handle so much partition, with one >> partition there is at least one file on disk, and with 4000 partition, >> there will be at least 4000 files. >> >> When all these partitions have write request, how did Kafka make the write >> operation on the disk to be sequential (which is emphasized in the design >> document of Kafka) and make sure the disk access is effective? >> >> Thank you for your reply. >> >> xiaobinshe >> >> >> >> 2014-10-22 5:10 GMT+08:00 Todd Palino <tpal...@gmail.com>: >> >> > As far as the number of partitions a single broker can handle, we've set >> > our cap at 4000 partitions (including replicas). Above that we've seen >> some >> > performance and stability issues. >> > >> > -Todd >> > >> > On Tue, Oct 21, 2014 at 12:15 AM, Xiaobin She <xiaobin...@gmail.com> >> > wrote: >> > >> > > hello, everyone >> > > >> > > I'm new to kafka, I'm wondering what's the max num of partition can one >> > > siggle machine handle in Kafka? >> > > >> > > Is there an sugeest num? >> > > >> > > Thanks. >> > > >> > > xiaobinshe >> > > >> > >>
