There are various costs when a broker fails, including broker leader election for each partition, etc., as well as exposing possible issues for in-flight messages, and client rebalancing etc.
So even though replication provides partition redundancy, RAID 10 on each broker is usually a good tradeoff to prevent the typical most common cause of broker server failure (e.g. disk failure) as well, and overall smoother operation. Best Regards, -Jonathan On Oct 22, 2014, at 11:01 AM, Gwen Shapira <gshap...@cloudera.com> wrote: > RAID-10? > Interesting choice for a system where the data is already replicated > between nodes. Is it to avoid the cost of large replication over the > network? how large are these disks? > > On Wed, Oct 22, 2014 at 10:00 AM, Todd Palino <tpal...@gmail.com> wrote: >> In fact there are many more than 4000 open files. Many of our brokers run >> with 28,000+ open files (regular file handles, not network connections). In >> our case, we're beefing up the disk performance as much as we can by >> running in a RAID-10 configuration with 14 disks. >> >> -Todd >> >> On Tue, Oct 21, 2014 at 7:58 PM, Xiaobin She <xiaobin...@gmail.com> wrote: >> >>> Todd, >>> >>> Actually I'm wondering how kafka handle so much partition, with one >>> partition there is at least one file on disk, and with 4000 partition, >>> there will be at least 4000 files. >>> >>> When all these partitions have write request, how did Kafka make the write >>> operation on the disk to be sequential (which is emphasized in the design >>> document of Kafka) and make sure the disk access is effective? >>> >>> Thank you for your reply. >>> >>> xiaobinshe >>> >>> >>> >>> 2014-10-22 5:10 GMT+08:00 Todd Palino <tpal...@gmail.com>: >>> >>>> As far as the number of partitions a single broker can handle, we've set >>>> our cap at 4000 partitions (including replicas). Above that we've seen >>> some >>>> performance and stability issues. >>>> >>>> -Todd >>>> >>>> On Tue, Oct 21, 2014 at 12:15 AM, Xiaobin She <xiaobin...@gmail.com> >>>> wrote: >>>> >>>>> hello, everyone >>>>> >>>>> I'm new to kafka, I'm wondering what's the max num of partition can one >>>>> siggle machine handle in Kafka? >>>>> >>>>> Is there an sugeest num? >>>>> >>>>> Thanks. >>>>> >>>>> xiaobinshe >>>>> >>>> >>>
