I'm interested in this topic as well. If you put kafka brokers inside an autoscaling group, then AWS will automatically add brokers if demand increases, and the ELB will automatically round-robin across all of your kafka instances. So in your config files and code, you only need to provide a single DNS name (the load balancer). You don't need to specify all your kafka brokers inside your config file. If a broker dies, the ELB will only route to healthy nodes.
So you get a lot of robustness, scalability, and fault-tolerance by using the AWS services. Kafka Brokers will automatically load balance, but the question is whether it is ok to put all your brokers behind an ELB and expect the system to work properly. What alternatives are there to dynamic/scalable broker clusters? I don't want to have to modify my config files or code if I add more brokers, and I want to be able to handle a broker going down. So these are the reasons AWS questions like this come up. Thanks for any comments too. :) On Mon, May 4, 2015 at 9:03 AM, Mayuresh Gharat <gharatmayures...@gmail.com> wrote: > Ok. You can deploy kafka in AWS. You can have brokers on AWS servers. > Kafka is not a push system. So you will need someone writing to kafka and > consuming from kafka. It will work. My suggestion will be to try it out on > a smaller instance in AWS and see the effects. > > As I do not know the actual use case about why you want to use kafka for, I > cannot comment on whether it will work for you personalized use case. > > Thanks, > > Mayuresh > > On Mon, May 4, 2015 at 8:55 AM, Chandrashekhar Kotekar < > shekhar.kote...@gmail.com> wrote: > > > I am sorry but I cannot reveal those details due to confidentiality > issues. > > I hope you understand. > > > > > > Regards, > > Chandrash3khar Kotekar > > Mobile - +91 8600011455 > > > > On Mon, May 4, 2015 at 9:18 PM, Mayuresh Gharat < > > gharatmayures...@gmail.com> > > wrote: > > > > > Hi Chandrashekar, > > > > > > Can you please elaborate the use case for Kafka here, like how you are > > > planning to use it. > > > > > > > > > Thanks, > > > > > > Mayuresh > > > > > > On Sat, May 2, 2015 at 9:08 PM, Chandrashekhar Kotekar < > > > shekhar.kote...@gmail.com> wrote: > > > > > > > Hi, > > > > > > > > I am new to Apache Kafka. I have played with it on my laptop. > > > > > > > > I want to use Kafka in AWS. Currently we have tomcat web servers > based > > > REST > > > > API. We want to replace REST API with Apache Kafka, web servers are > > > behind > > > > ELB. > > > > > > > > I would like to know if we can keep Kafka brokers behind ELB? Will it > > > work? > > > > > > > > Regards, > > > > Chandrash3khar Kotekar > > > > Mobile - +91 8600011455 > > > > > > > > > > > > > > > > -- > > > -Regards, > > > Mayuresh R. Gharat > > > (862) 250-7125 > > > > > > > > > -- > -Regards, > Mayuresh R. Gharat > (862) 250-7125 >
