Hi, With kafka you can increase overall throughput by increasing the number of nodes in a cluster. I had a similar issue, where we needed to ingest vast amounts of data to streaming system. In our case, kafka was a bottleneck, because of disk I/O. To solve it, we implemented (simple) distributed pub-sub system with C which reside data in memory. Also you should take account your network bandwidth and the (upper-bound) capability of your processing engine or http server.
Cheers, Jeyhun On Wed, Jun 21, 2017 at 2:58 PM SenthilKumar K <senthilec...@gmail.com> wrote: > Hi Team , Sorry if this question is irrelevant to Kafka Group ... > > I have been trying to solve problem of handling 5 GB/sec ingestion. Kafka > is really good candidate for us to handle this ingestion rate .. > > > 100K machines ----> { Http Server (Jetty/Netty) } --> Kafka Cluster.. > > I see the problem in Http Server where it can't handle beyond 50K events > per instance .. I'm thinking some other solution would be right choice > before Kafka .. > > Anyone worked on similar use case and similar load ? Suggestions/Thoughts ? > > --Senthil > -- -Cheers Jeyhun