Thanks Jeyhun. Yes http server would be problematic here w.r.t network , memory ..
Hi Dave , The problem is not with Kafka , it's all about how do you handle huge data before kafka. I did a simple test with 5 node Kafka Cluster which gives good result ( ~950 MB/s ) ..So Kafka side i dont see a scaling issue ... All we are trying is before kafka how do we handle messages from different servers ... Webservers can send fast to kafka but still i can handle only 50k events per second which is less for my use case.. also i can't deploy 20 webservers to handle this load. I'm looking for an option what could be the best candidate before kafka , it should be super fast in getting all and send it to kafka producer .. --Senthil On Wed, Jun 21, 2017 at 6:53 PM, Tauzell, Dave <dave.tauz...@surescripts.com > wrote: > What are your configurations? > > - production > - brokers > - consumers > > Is the problem that web servers cannot send to Kafka fast enough or your > consumers cannot process messages off of kafka fast enough? > What is the average size of these messages? > > -Dave > > -----Original Message----- > From: SenthilKumar K [mailto:senthilec...@gmail.com] > Sent: Wednesday, June 21, 2017 7:58 AM > To: users@kafka.apache.org > Cc: senthilec...@apache.org; Senthil kumar; d...@kafka.apache.org > Subject: Handling 2 to 3 Million Events before Kafka > > Hi Team , Sorry if this question is irrelevant to Kafka Group ... > > I have been trying to solve problem of handling 5 GB/sec ingestion. Kafka > is really good candidate for us to handle this ingestion rate .. > > > 100K machines ----> { Http Server (Jetty/Netty) } --> Kafka Cluster.. > > I see the problem in Http Server where it can't handle beyond 50K events > per instance .. I'm thinking some other solution would be right choice > before Kafka .. > > Anyone worked on similar use case and similar load ? Suggestions/Thoughts ? > > --Senthil > This e-mail and any files transmitted with it are confidential, may > contain sensitive information, and are intended solely for the use of the > individual or entity to whom they are addressed. If you have received this > e-mail in error, please notify the sender by reply e-mail immediately and > destroy all copies of the e-mail and any attachments. >
