Re: Problems trying to make kafka 'rack-aware'

2018-09-21 Thread Bryan Duggan
Hi Eno, many thanks for trying that. That is very helpful for me. That basic check didn't work for me but I have since discovered what my issue was. Despite using a version of kafka that supports rack-awareness we have been deliberately setting 'inter.broker.protocol.version' to an older

Re: Problems trying to make kafka 'rack-aware'

2018-09-21 Thread Eno Thereska
Hi Bryan, I did a simple check with starting a broker with no rack id and then restarting with a rack id and I can confirm I could get the rack id from zookeeper after the restart. This was on trunk. Does that basic check work for you (i.e., without reassigning partitions)? Thanks Eno On Fri,

Re: Problems trying to make kafka 'rack-aware'

2018-09-21 Thread Bryan Duggan
I didn't get a response to this, but I've been investigating more and can now frame the problem slightly differently (hopefully, more accurately). According to this document https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper Which defines broker data

Problems trying to make kafka 'rack-aware'

2018-09-20 Thread Bryan Duggan
Hi, I have a kafka cluster consisting of 3 brokers across 3 different AWS availability zones.  It hosts several topics, each of which has a replication factor of 3. The cluster is currently not 'rack-aware'. I am trying to do the following;     - add 3 additional brokers (one in each of the