By looking at the responses: It looks like there is *NO *standard solution, we will have to come up with some custom/manual script..... any one has any other suggestions?
On Fri, Nov 16, 2018 at 11:18 AM Srinivas Rapolu <cnu.t...@gmail.com> wrote: > Yes, that is the whole point of discussion, replication factor is more > than 1. So keeping it in instance storage, we will lose the one copy of > data, but retaining broker-id will auto-replicate the topic/partitions to > new instance as zookeeper has brokerid-topic-partition assignment mapping. > > By looking at the responses: It looks like there is standard solution, we > will have to come up with some manual script..... any one has any other > suggestions? > > On Fri, Nov 16, 2018 at 10:08 AM Kaufman Ng <kauf...@confluent.io> wrote: > >> Yes data will be lost when you use instance storage. But keeping the >> *same* >> broker id will allow the new broker instance to easily replicate data from >> other brokers, therefore restoring back to the previous fully replicated >> state. >> >> Srinivas, do your topics have replicator factor > 1? If yes, data can be >> recovered from other brokers. >> >> On Fri, Nov 16, 2018 at 10:53 AM Ryanne Dolan <ryannedo...@gmail.com> >> wrote: >> >> > > if we are not using EBS >> > >> > In that case, what's the point of keeping the broker id? The data will >> be >> > lost anyway right? >> > >> > >> > On Thu, Nov 15, 2018, 11:40 AM Srinivas Rapolu <cnu.t...@gmail.com> >> wrote: >> > >> > > Yes, I understand we need to specify the required broker.id in >> > > server.properties/meta.properties file in-order to retain the >> broker.id >> > > which was failed. >> > > >> > > We can write custom script to query the zookeeper on launch process to >> > get >> > > the broker.id needs to be set. >> > > >> > > Do we have any standards/script/way defined/preferred for doing this >> or >> > > suggested by Kafka experts if we are not using EBS? >> > > >> > > Thanks and Regards, >> > > Srinivas >> > > >> > > On Thu, Nov 15, 2018 at 7:10 AM Kaufman Ng <kauf...@confluent.io> >> wrote: >> > > >> > > > Srinivas, >> > > > >> > > > You might want to check out the AWS best practices from this blog >> post: >> > > > >> > > > >> > > >> > >> https://www.confluent.io/blog/design-and-deployment-considerations-for-deploying-apache-kafka-on-aws/ >> > > > >> > > > Kafka broker ids by default are auto-generated, but you can also >> > specify >> > > > values (in server.properties file). By having static broker ids, >> it's >> > > > easier to recover in general. The blog post above mentions it in >> more >> > > > details. >> > > > >> > > > Good luck. >> > > > >> > > > On Thu, Nov 15, 2018 at 8:09 AM amit pal <amit5...@gmail.com> >> wrote: >> > > > >> > > > > If at all you were hell bent on doing this, you could use >> zookeeper >> > to >> > > > find >> > > > > out the health of current brokers along with their broker id. That >> > > should >> > > > > help re-spin/start the unhealthy instance with same instance id. >> > > > > >> > > > > On Thu, Nov 15, 2018 at 6:08 PM Srinivas Rapolu < >> cnu.t...@gmail.com> >> > > > > wrote: >> > > > > >> > > > > > having all this stored in DB is getting too complicated, >> especially >> > > > with >> > > > > > instance level storage and not EBS. >> > > > > > >> > > > > > I am sure there should be easy way to retain the old broker.id >> for >> > > new >> > > > > AWS >> > > > > > instance spun-up for auto replication. >> > > > > > >> > > > > > Any other ideas/help is appreciated. >> > > > > > >> > > > > > On Thu, Nov 15, 2018 at 2:27 AM Eno Thereska < >> > eno.there...@gmail.com >> > > > >> > > > > > wrote: >> > > > > > >> > > > > > > The general answer depends on what control plane software is >> > taking >> > > > > care >> > > > > > of >> > > > > > > your Kafka deployment. You probably have a layer that launches >> > > Kafka >> > > > > > > instances and monitors their health, right? If so, that layer >> > > should >> > > > > take >> > > > > > > care of the mapping between instances and broker IDs and keep >> > that >> > > > in a >> > > > > > > table persisted somewhere (e.g., DynamoDB). >> > > > > > > >> > > > > > > Eno >> > > > > > > >> > > > > > > On Wed, Nov 14, 2018 at 7:38 PM Srinivas Rapolu < >> > > cnu.t...@gmail.com> >> > > > > > > wrote: >> > > > > > > >> > > > > > > > EBS is one of the option. But we use instance level storage >> > where >> > > > we >> > > > > > > loose >> > > > > > > > all data as soon as we have a broker failed in AWS. >> > > > > > > > >> > > > > > > > In such scenario, anyone has better launch script or >> > cofiguration >> > > > can >> > > > > > be >> > > > > > > > executed on new broker to retain the old id not conflicting >> > with >> > > > > > existing >> > > > > > > > broker ids. >> > > > > > > > >> > > > > > > > On Wed, Nov 14, 2018, 11:58 AM Andrey Dyachkov < >> > > > > > > andrey.dyach...@gmail.com >> > > > > > > > wrote: >> > > > > > > > >> > > > > > > > > You can attach EBS volume, which will store data and >> > > > metadata(e.g. >> > > > > > > broker >> > > > > > > > > id), and then attach it to the new AWS instance and start >> > > Kafka, >> > > > it >> > > > > > > will >> > > > > > > > > pick the broker id plus you won’t need to rebalance the >> > > cluster. >> > > > > > > > > >> > > > > > > > > On Wed 14. Nov 2018 at 19:48, naresh Goud < >> > > > > > nareshgoud.du...@gmail.com> >> > > > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > Static IP. Buying static IP may help. I am not aws >> expert >> > > > > > > > > > >> > > > > > > > > > On Wed, Nov 14, 2018 at 12:47 PM Srinivas Rapolu < >> > > > > > cnu.t...@gmail.com >> > > > > > > > >> > > > > > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > > Hello Kafka experts, >> > > > > > > > > > > >> > > > > > > > > > > We are running Kafka on AWS, main question is what is >> the >> > > > best >> > > > > > way >> > > > > > > to >> > > > > > > > > > > retain broker.id on new instance spun-up in-place of >> > > > > > > instance/broker >> > > > > > > > > > > failed. >> > > > > > > > > > > >> > > > > > > > > > > We are currently running Kafka in AWS with broker.id >> > gets >> > > > auto >> > > > > > > > > > generated. >> > > > > > > > > > > But we are having issues when a broker is failed, new >> > > > > > > broker/instance >> > > > > > > > > > > spun-up in AWS get assigned with new broker.id. The >> > issue >> > > > is, >> > > > > > with >> > > > > > > > > this >> > > > > > > > > > > approach, we need to re-assign the >> topics/replications on >> > > to >> > > > > the >> > > > > > > new >> > > > > > > > > > broker >> > > > > > > > > > > manually. >> > > > > > > > > > > >> > > > > > > > > > > We learned that, replication can be auto resolved by >> > Kafka, >> > > > if >> > > > > we >> > > > > > > can >> > > > > > > > > > > manage to get the same broker.id on the new AWS >> instance >> > > > > spun-up >> > > > > > > > > > in-place >> > > > > > > > > > > of failed broker/instance. >> > > > > > > > > > > >> > > > > > > > > > > I have read, we can set broker.id.generation.enable= >> > false, >> > > > but >> > > > > > > what >> > > > > > > > is >> > > > > > > > > > the >> > > > > > > > > > > best way to identify and retain the broker.id? Any >> > > > links/help >> > > > > is >> > > > > > > > > > > appreciated. >> > > > > > > > > > > Thanks and Regards, >> > > > > > > > > > > Cnu >> > > > > > > > > > > >> > > > > > > > > > -- >> > > > > > > > > > Thanks, >> > > > > > > > > > Naresh >> > > > > > > > > > www.linkedin.com/in/naresh-dulam >> > > > > > > > > > http://hadoopandspark.blogspot.com/ >> > > > > > > > > > >> > > > > > > > > -- >> > > > > > > > > Thanks, >> > > > > > > > > Andrey >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > >> > > > > -- >> > > > > Thanks and Regards, >> > > > > Amit Pal >> > > > > >> > > > >> > > > >> > > > -- >> > > > Kaufman Ng >> > > > +1 646 961 8063 >> > > > Solutions Architect | Confluent | www.confluent.io >> > > > >> > > >> > >> >> >> -- >> Kaufman Ng >> +1 646 961 8063 >> Solutions Architect | Confluent | www.confluent.io >> >
