Hi, I am now in the process of deciding partitions and replicas for my cluster. I am making use of perf test utilities and it really helps a lot. Just measure perf by creating multiple topics with same number of recodrs with diff partitions and replicas. Then compare the througput and also look at latency. If possible change network and io threads at broker level and then again do perf testing for above scenarios.
Thanks, Sunil On Fri, 28 Feb 2020 at 11:40 AM, 张祥 <xiangzhang1...@gmail.com> wrote: > Thanks, it helps a lot. > > Peter Bukowinski <pmb...@gmail.com> 于2020年2月28日周五 上午5:18写道： > > > No, it’s not bad. Kafka is designed to serve data to many consumers at > the > > same time, whether they are independent of each other or in the same > > consumer group. > > > > I would encourage you to play with different partition counts and use > > kafka’s performance testing tools (kafka-producer-perf-test.sh and > > kafka-consumer-perf-test.sh) to test throughput in different scenarios > and > > see the results for yourself. > > > > — > > Peter > > > > > On Feb 27, 2020, at 1:28 AM, 张祥 <xiangzhang1...@gmail.com> wrote: > > > > > > I believe no matter the partition count exceeds the broker count, we > can > > > always have the same number of consumer instances as the partition > count. > > > > > > So what I want to know is when two partition exists on the same broker, > > two > > > consumer instances will be talking to same broker, is that bad ? > > > > > > 张祥 <xiangzhang1...@gmail.com> 于2020年2月27日周四 下午2:20写道： > > > > > >> Thanks. What influence does it have for consumers and producers when > > >> partition number is more than broker number, which means at least one > > >> broker serves two partitions for one topic ? performance wise. > > >> > > >> Peter Bukowinski <pmb...@gmail.com> 于2020年2月26日周三 下午11:02写道： > > >> > > >>> Disk usage is one reason to expand. Another reason is if you need > more > > >>> ingest or output throughout for your topic data. If your producers > > aren’t > > >>> able to send data to kafka fast enough or your consumers are lagging, > > you > > >>> might benefit from more brokers and more partitions. > > >>> > > >>> -- Peter > > >>> > > >>>> On Feb 26, 2020, at 12:56 AM, 张祥 <xiangzhang1...@gmail.com> wrote: > > >>>> > > >>>> In documentation, it is described how to expand cluster: > > >>>> > > >>> > > > https://kafka.apache.org/20/documentation.html#basic_ops_cluster_expansion > > >>> . > > >>>> But I am wondering what the criteria for expand is. I can only think > > of > > >>>> disk usage threshold. For example, suppose several disk usage exceed > > >>> 80%. > > >>>> Is this correct and is there more ? > > >>> > > >> > > > > >
