Dear Jens Thank you for the reply!
It's really hard to decide how many brokers/partitions are optimal for a system. Is there any good reports or documents about that? I'd like to know some examples related to the optimization, especially on product level environment. Thank you in advance. Best regards Kim 2016-03-02 21:19 GMT+09:00 Jens Rantil <jens.ran...@tink.se>: > Hi Kim, > > You are correct in that the number of partitions sets the upper limit on > consumer parallelization. That is, a single consumer in a group can consume > multiple partitions, however multiple consumers in a group can't consume a > single partition. > > Also, since partitions are spread across your brokers, really it's the > ratio nPartitions/nBrokers that you want to optimize for. > > Given the above parallelization limit, it would make sense to have a very > large ratio. This would have other implications: > > - Your brokers will have a lot of smaller files to they will have to > flush periodically. This can incur a lot of overhead and introduce > latencies. Especially on a spinning disk where seeks are expensive. > - Brokers are generally set to rotate their logs at a certain size. It > could be hard to tune rotation with many small files. > > Given this, really you need to benchmark for your use case with your > message sizes etc. > > Side-note: Note that for autoscaling you will have to overprovision your > partitions somewhat to not hit the parallelization limit. > > Cheers, > Jens > > On Wed, Mar 2, 2016 at 1:11 AM, BYEONG-GI KIM <bg...@bluedigm.com> wrote: > > > Hello. > > > > I have questions about how many partitions are optimal while using kafka. > > As far as I know, even if there are multiple consumers that belong to a > > consumer group, say *group_A*, only one consumer can receive a kafka > > message produced by a producer if there is a partition. So, as a result, > > multiple partitions are required in order to distribute the message to > all > > the consumers in group_A if I want the consumers to get the message. > > > > Is it right? > > > > I'm considering developing several kafka consumer applications, e.g., > > message saver, message analyzer, etc., so a message from a producer must > be > > consumed by those kinds of consumers. > > > > Any advice and help would be really appreciated. > > > > Thanks in advance! > > > > Best regards > > > > Kim > > > > > > -- > Jens Rantil > Backend engineer > Tink AB > > Email: jens.ran...@tink.se > Phone: +46 708 84 18 32 > Web: www.tink.se > > Facebook <https://www.facebook.com/#!/tink.se> Linkedin > < > http://www.linkedin.com/company/2735919?trk=vsrp_companies_res_photo&trkInfo=VSRPsearchId%3A1057023381369207406670%2CVSRPtargetId%3A2735919%2CVSRPcmpt%3Aprimary > > > Twitter <https://twitter.com/tink> >
