The general answer depends on what control plane software is taking care of your Kafka deployment. You probably have a layer that launches Kafka instances and monitors their health, right? If so, that layer should take care of the mapping between instances and broker IDs and keep that in a table persisted somewhere (e.g., DynamoDB).
Eno On Wed, Nov 14, 2018 at 7:38 PM Srinivas Rapolu <cnu.t...@gmail.com> wrote: > EBS is one of the option. But we use instance level storage where we loose > all data as soon as we have a broker failed in AWS. > > In such scenario, anyone has better launch script or cofiguration can be > executed on new broker to retain the old id not conflicting with existing > broker ids. > > On Wed, Nov 14, 2018, 11:58 AM Andrey Dyachkov <andrey.dyach...@gmail.com > wrote: > > > You can attach EBS volume, which will store data and metadata(e.g. broker > > id), and then attach it to the new AWS instance and start Kafka, it will > > pick the broker id plus you won’t need to rebalance the cluster. > > > > On Wed 14. Nov 2018 at 19:48, naresh Goud <nareshgoud.du...@gmail.com> > > wrote: > > > > > Static IP. Buying static IP may help. I am not aws expert > > > > > > On Wed, Nov 14, 2018 at 12:47 PM Srinivas Rapolu <cnu.t...@gmail.com> > > > wrote: > > > > > > > Hello Kafka experts, > > > > > > > > We are running Kafka on AWS, main question is what is the best way to > > > > retain broker.id on new instance spun-up in-place of instance/broker > > > > failed. > > > > > > > > We are currently running Kafka in AWS with broker.id gets auto > > > generated. > > > > But we are having issues when a broker is failed, new broker/instance > > > > spun-up in AWS get assigned with new broker.id. The issue is, with > > this > > > > approach, we need to re-assign the topics/replications on to the new > > > broker > > > > manually. > > > > > > > > We learned that, replication can be auto resolved by Kafka, if we can > > > > manage to get the same broker.id on the new AWS instance spun-up > > > in-place > > > > of failed broker/instance. > > > > > > > > I have read, we can set broker.id.generation.enable= false, but what > is > > > the > > > > best way to identify and retain the broker.id? Any links/help is > > > > appreciated. > > > > Thanks and Regards, > > > > Cnu > > > > > > > -- > > > Thanks, > > > Naresh > > > www.linkedin.com/in/naresh-dulam > > > http://hadoopandspark.blogspot.com/ > > > > > -- > > Thanks, > > Andrey > > >
