Re: Change kafka broker ids dynamically

2015-11-24 Thread SpikyHawk SpikyHawk
Excellent, We are planning to use Kafka 0.9.0 so your last point is very useful information. can you please point me to some documentation or code where I can understand how this auto-generation works? In the 0.9.0 documentation is see that the default value for broker.id is -1. That means it will

Re: Change kafka broker ids dynamically

2015-11-24 Thread Gwen Shapira
You should definitely use the same id if you still have the data - it makes life so much better. There are 3 common ways to do it: 1. Use the last 3 digits of the IP as the broker ID (assuming Docker gives you the same IP when the container relaunches) 2. Use a deployment manager that can register

Re: Change kafka broker ids dynamically

2015-11-24 Thread SpikyHawk SpikyHawk
Hi there. I have a similar question but is related to this scenario. Docker server running in EC2 instance with an EBS volume attached to it. Kafka running in a Docker container in this host with server.properties "autogenerated" using a bootstrap script. Part of the bootstrapping process is to

Re: Change kafka broker ids dynamically

2015-11-06 Thread Todd Palino
I’m not quite sure why you would need to do this - the broker IDs are not significant outside of the internal metadata. But this is what you would have to do for each move (assuming you are running with at least replication factor 2): 1) Shut down the broker 2) Clear its partition data 3) Reconfig

Change kafka broker ids dynamically

2015-11-06 Thread Arathi Maddula
Hi, Is it possible to change the broker.ids property for a node belonging to a Kafka cluster? For example, currently if I have brokers with ids 1,2,3. If I want to stop broker 1, can I change broker.id to 0 (with current id = 1) in server.properties and meta.properties files and then restart b