I have a possibly related question: In a batch sending mode it's possible that messages are lost if producer dies while accumulating the batch. What is the performance overhead if we don't use batching?
Thanks. On Wed, Mar 26, 2014 at 9:24 AM, Jagbir <jsho...@hotmail.com> wrote: > Hi Neha, > > Thank you for the numbers as this information helped me to size my project > as well. While we are on this topic, do you mind commenting on the maximum > number of topics that can be hosted on a 8-10 node cluster. We have a need > for about 100k topics and similar number of producers and consumers and I'm > looking for some rough estimates. > > Thanks, > Jagbir > > On March 25, 2014 8:59:21 PM PDT, Neha Narkhede <neha.narkh...@gmail.com> > wrote: >>You shouldn't have any problem with that. We frequently have 10s of >>thousands of producer connections to a 8-10 node cluster at all times. >>You >>might have to bump up the limit for the number of open file handles per >>broker though. >> >>Thanks, >>Neha >> >> >>On Tue, Mar 25, 2014 at 3:41 PM, Kane Kane <kane.ist...@gmail.com> >>wrote: >> >>> Is there a recommended cap for the concurrent producers threads? >>> We plan to have around 4000 connections across cluster writing to >>> kafka, i assume there shouldn't be any performance implications >>> related to that? >>> >>> Thanks. >>> > > -- > Sent from my Android phone with K-9 Mail. Please excuse my brevity.
