Re: Apache Kafka blog on more partitions support
The blog is now published to https://blogs.apache.org/kafka/entry/apache-kafka-supports-more-partitions Thanks, Jun On Fri, Nov 2, 2018 at 2:36 PM, Jun Rao wrote: > Hi, Mayuresh, > > Most of the controlled shutdown time is in leader election. The controller > currently doesn't wait for LeaderAndIsrRequest to be sent out before > responding to the controlled shutdown request. > > Thanks, > > Jun > > On Fri, Nov 2, 2018 at 1:58 PM, Mayuresh Gharat < > gharatmayures...@gmail.com> wrote: > >> Thanks Jun for sharing this. Looks nice ! >> >> Do we intend to shed light on how much time is required, on an average, >> for >> new Leader election. Also would it be good to add "if the controller waits >> for the LeaderAndIsrResponses before sending shutDown_OK to the shutting >> down broker". >> >> Thanks, >> >> Mayuresh >> >> On Fri, Nov 2, 2018 at 12:07 PM wrote: >> >> > Thanks Jun for sharing the post. >> > Minor Nit: Date says December 16, 2019. >> > >> > Did this test measured the replication affects on the overall cluster >> > health and performance? >> > It looks like we are suggesting with 200k partitions and 4k per broker >> max >> > size of a cluster should be around 50 brokers? >> > >> > Thanks, >> > Harsha >> > On Nov 2, 2018, 11:50 AM -0700, Jun Rao , wrote: >> > > Hi, Everyone, >> > > >> > > The follow is the preview of a blog on Kafka supporting more >> partitions. >> > > >> > > https://drive.google.com/file/d/122TK0oCoforc2cBWfW_yaEBjTMoX6yMt >> > > >> > > Please let me know if you have any comments by Tuesday. >> > > >> > > Thanks, >> > > >> > > Jun >> > >> >> >> -- >> -Regards, >> Mayuresh R. Gharat >> (862) 250-7125 >> > >
Re: Apache Kafka blog on more partitions support
Hi, Mayuresh, Most of the controlled shutdown time is in leader election. The controller currently doesn't wait for LeaderAndIsrRequest to be sent out before responding to the controlled shutdown request. Thanks, Jun On Fri, Nov 2, 2018 at 1:58 PM, Mayuresh Gharat wrote: > Thanks Jun for sharing this. Looks nice ! > > Do we intend to shed light on how much time is required, on an average, for > new Leader election. Also would it be good to add "if the controller waits > for the LeaderAndIsrResponses before sending shutDown_OK to the shutting > down broker". > > Thanks, > > Mayuresh > > On Fri, Nov 2, 2018 at 12:07 PM wrote: > > > Thanks Jun for sharing the post. > > Minor Nit: Date says December 16, 2019. > > > > Did this test measured the replication affects on the overall cluster > > health and performance? > > It looks like we are suggesting with 200k partitions and 4k per broker > max > > size of a cluster should be around 50 brokers? > > > > Thanks, > > Harsha > > On Nov 2, 2018, 11:50 AM -0700, Jun Rao , wrote: > > > Hi, Everyone, > > > > > > The follow is the preview of a blog on Kafka supporting more > partitions. > > > > > > https://drive.google.com/file/d/122TK0oCoforc2cBWfW_yaEBjTMoX6yMt > > > > > > Please let me know if you have any comments by Tuesday. > > > > > > Thanks, > > > > > > Jun > > > > > -- > -Regards, > Mayuresh R. Gharat > (862) 250-7125 >
Re: Apache Kafka blog on more partitions support
Hi, Harsha, The recommendation is the upper bound for partitions. One can certainly have more than 50 brokers in a cluster if there are fewer partitions per broker. I will clarify this in the blog. Thanks, Jun On Fri, Nov 2, 2018 at 12:07 PM, wrote: > Thanks Jun for sharing the post. > Minor Nit: Date says December 16, 2019. > > Did this test measured the replication affects on the overall cluster > health and performance? > It looks like we are suggesting with 200k partitions and 4k per broker max > size of a cluster should be around 50 brokers? > > Thanks, > Harsha > On Nov 2, 2018, 11:50 AM -0700, Jun Rao , wrote: > > Hi, Everyone, > > > > The follow is the preview of a blog on Kafka supporting more partitions. > > > > https://drive.google.com/file/d/122TK0oCoforc2cBWfW_yaEBjTMoX6yMt > > > > Please let me know if you have any comments by Tuesday. > > > > Thanks, > > > > Jun >
Re: Apache Kafka blog on more partitions support
Thanks Jun for sharing this. Looks nice ! Do we intend to shed light on how much time is required, on an average, for new Leader election. Also would it be good to add "if the controller waits for the LeaderAndIsrResponses before sending shutDown_OK to the shutting down broker". Thanks, Mayuresh On Fri, Nov 2, 2018 at 12:07 PM wrote: > Thanks Jun for sharing the post. > Minor Nit: Date says December 16, 2019. > > Did this test measured the replication affects on the overall cluster > health and performance? > It looks like we are suggesting with 200k partitions and 4k per broker max > size of a cluster should be around 50 brokers? > > Thanks, > Harsha > On Nov 2, 2018, 11:50 AM -0700, Jun Rao , wrote: > > Hi, Everyone, > > > > The follow is the preview of a blog on Kafka supporting more partitions. > > > > https://drive.google.com/file/d/122TK0oCoforc2cBWfW_yaEBjTMoX6yMt > > > > Please let me know if you have any comments by Tuesday. > > > > Thanks, > > > > Jun > -- -Regards, Mayuresh R. Gharat (862) 250-7125
Re: Apache Kafka blog on more partitions support
Thanks Jun for sharing the post. Minor Nit: Date says December 16, 2019. Did this test measured the replication affects on the overall cluster health and performance? It looks like we are suggesting with 200k partitions and 4k per broker max size of a cluster should be around 50 brokers? Thanks, Harsha On Nov 2, 2018, 11:50 AM -0700, Jun Rao , wrote: > Hi, Everyone, > > The follow is the preview of a blog on Kafka supporting more partitions. > > https://drive.google.com/file/d/122TK0oCoforc2cBWfW_yaEBjTMoX6yMt > > Please let me know if you have any comments by Tuesday. > > Thanks, > > Jun
