@Nick That report does not necessarily mean we need to add a new broker for every 30 topics. It depends on many factors such as the number of partitions and the number of messages per topic, etc.
I just refered that to let you give a hint about how big Kafka cluster could be. @Chetan Thank you for comments. I would very appreciate if you look into it : ) Yes agree. But I think we need to define the meaning of "small" or "big" cluster. At least we can assume we can have 350,000 partitions(same with Linkedin) when we have enough number of brokers. And accordingly we can assume 10.5 Million messages/sec. Since one message is one activation in OpenWhisk, 10.5 Million messages mean 10.5 Million activation. Since these message are handled by containers, if we utilize such a number of messages, we need a huge number of invokers. When I did brief benchmarking with new scheduler, I observed about 12,000 Invocations/sec with 10 invokers on commodity servers(8 cores, 16GB memory). Based on this, if we need to utilize 10.5M messages, we need about 8,750 invokers(10500000/12000*10). I think it's quite big enough and other components would be saturated before Kafka would be saturated. Since this is a just arithmatic calculation there would be differences.(may be much less than this) But I think the scalability of Kafka might not become a problem in many cases. One more thing is, in the recent Kafka(>=2.0.0), the zookeeper-based old consumer is not being used anymore. Now brokers manage consumer metadata and a new consumer directly connects to brokers. It fixes many issues in zookeeper-based consumer and generally more scalable than old consumer. But these are mostly based on assumption and there are a few obvious shortcomings in the scheduler. I will address them more details and do more benchmarking to have more grounds. Thanks Regards Dominic 2018년 11월 29일 (목) 오후 1:10, Chetan Mehrotra <[email protected]>님이 작성: > Hi Dominic, > > From various docs it appears that there are 2 aspects which are impacted by > large number of topics (or partitions) 1) #partitions/#brokers and 2) > zookeeper. With lots of topic Zookeeper may become a bottleneck. > > However I like the better throughput guarantees with such an approach and > it might be good viable option for smaller setups. Good write up on the > proposal. It would take some time to go over that! > > Chetan Mehrotra > [1] https://www.quora.com/How-many-topics-can-be-created-in-Apache-Kafka > [2] > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowmanytopicscanIhave > ? > > > On Thu, Nov 29, 2018 at 9:10 AM Nick Mitchell <[email protected]> wrote: > > > > > > > 1100+ Kafka brokers, over 31,000 topics > > > > > > am i reading this right, as this seems like an unhealthy ratio. a new > > broker for every 30 topics? so i must be misinterpreting the numbers... > > > > On Wed, Nov 28, 2018 at 10:18 PM Dominic Kim <[email protected]> > wrote: > > > > > Dear Rodric. > > > > > > We`ve just started benchmarking of the scheduler with a variety of > > > scenarios. > > > I will come with more results. > > > > > > If you ask that because of scalability of Kafka, I have considered > Kafka > > as > > > a very scalable solution. > > > > > > Linkedin announced that they are utilizing a huge Kafka cluster. > > > > > > > > > https://events.static.linuxfound.org/sites/events/files/slides/Kafka%20At%20Scale.pdf > > > > > > As per the document, they are utilizing 1100+ Kafka brokers, over > 31,000 > > > topics, 350,000+ partitions. > > > > > > And as per my understanding, only a limited number of brokers > participate > > > in message delivery at a time for a topic. > > > Since partitions are the unit of parallelism, if we have 5 partitions > in > > a > > > topic, only maximum 5 brokers can deliver the message at the same time. > > > (Actually, consumers fetch the messages instead brokers deliver them, > but > > > anyway.) > > > > > > Since it's not required for all brokers to participate in all message > > > processing, when we need to scale out Kafka, we can just add more > > brokers. > > > > > > For example, if we have 5 topics with 5 partitions each in 5 brokers. > > > There will be 1 partition for each topic in each broker. As more and > more > > > topics are created, there will be more partitions in one broker. > > > Let's say the maximum number of partitions that one broker properly > > handles > > > in time is 10, then if there are 10 topics, message processing will > start > > > becoming slow. > > > Then we can just add more brokers. If we add 1 broker, it can handle 10 > > > partitions and it is equal to 2 topics in this example. > > > So whenever 2 new topics are created we can add a broker so that we can > > > keep the number of partitions that one broker holds same. > > > If we can keep the number of partitions in one broker at the same > level, > > we > > > can guarantee the performance of Kafka as one broker will only > > participate > > > in the limited number of partition processing. > > > > > > In this scene, metadata sharing for partitions might be considered as a > > > bottleneck. > > > But once partitions are assigned to brokers, it does not change over > > > time(if we don't explicitly rebalance them). > > > Then consumers can just statically read the location of partitions and > > > fetch data from target brokers. > > > > > > Partitions are evenly distributed among brokers and each consumer will > > > access to a few of them only at a time, we can achieve massive > > scalability > > > with Kafka. > > > > > > Since this is based on my understanding about Kafka rather than real > > > experience, if I observe any issues during benchmarking I will share > them > > > as well. > > > > > > > > > > > > Thanks > > > Regards > > > Dominic > > > > > > > > > > > > > > > > > > > > > 2018년 11월 29일 (목) 오전 9:54, Rodric Rabbah <[email protected]>님이 작성: > > > > > > > Thanks for sharing Dominic. I'll need to read this slowly to > understand > > > it. > > > > It does look like a bit of the proposal you've discussed in the past > > > with 1 > > > > topic per action name (per namespace). Have you seen this scale in > your > > > > production to thousands of topics? > > > > > > > > -r > > > > > > > > On Wed, Nov 28, 2018 at 5:59 AM Dominic Kim <[email protected]> > > wrote: > > > > > > > > > Dear whiskers. > > > > > > > > > > I`ve just submitted the proposal for the next version of Autonomous > > > > > Container Scheduler. > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/OPENWHISK/Autonomous+Container+Scheduler+v2 > > > > > > > > > > We(Naver), already implemented the scheduler based on that proposal > > and > > > > > it's almost finished. > > > > > It's not all-round scheduler but we could observe many performance > > > > benefits > > > > > in a few cases. > > > > > > > > > > We(Naver) are willing to contribute it and preparing for the > > > > contribution. > > > > > I hope the proposal would help you guys understand the > > implementation. > > > > > > > > > > To contribute the scheduler, it is required to contribute SPI for > > > > > ContainerPool and ContainerProxy first. > > > > > I will start working on this. > > > > > And I will describe the details about each component in each > subpage. > > > > > Currently, they are blank. > > > > > > > > > > Please be noted again that the scheduler is not for all cases. > > > > > But it would be useful in many cases until new implementation based > > on > > > > the > > > > > future architecture of OW would be landed. > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/OPENWHISK/OpenWhisk+future+architecture?src=contextnavpagetreemode > > > > > > > > > > Any comments and feedback would be very grateful. > > > > > > > > > > Thanks > > > > > Regards > > > > > Dominic > > > > > > > > > > > > > > >
