Agreed. So it seems that for 0.9.1, the only option is to keep the JSON version unchanged. But as part of the PR, I can change the behavior of ZkUtils.getBrokerInfo() to make it compatible with future JSON versions.
Thanks, Allen On Tue, Jan 12, 2016 at 2:57 PM, Jun Rao <j...@confluent.io> wrote: > Hi, Allen, > > That's a good point. In 0.9.0.0, the old consumer reads broker info > directly from ZK and the code throws an exception if the version in json is > not 1 or 2. This old consumer will break when we upgrade the broker json to > version 3 in ZK in 0.9.1, which will be an issue. We overlooked this issue > in 0.9.0.0. The easiest fix is probably not to check the version in > ZkUtils.getBrokerInfo(). > This way, as long as we are only adding new fields in broker json, we can > preserve the compatibility. > > Thanks, > > Jun > > On Tue, Jan 12, 2016 at 1:52 PM, Allen Wang <allenxw...@gmail.com> wrote: > > > Hi Jun, > > > > That's a good suggestion. However, it does not solve the problem for the > > clients or thirty party tools that get broker information directly from > > ZooKeeper. > > > > Thanks, > > Allen > > > > > > On Tue, Jan 12, 2016 at 1:29 PM, Jun Rao <j...@confluent.io> wrote: > > > > > Allen, > > > > > > Another way to do this is the following. > > > > > > When inter.broker.protocol.version is set to 0.9.0, the broker will > write > > > the broker info in ZK using version 2, ignoring the rack info. > > > > > > When inter.broker.protocol.version is set to 0.9.1, the broker will > write > > > the broker info in ZK using version 3, including the rack info. > > > > > > If one follows the upgrade process, after the 2nd round of rolling > > bounces, > > > every broker is capable of parsing version 3 of broker info in ZK. This > > is > > > when the rack-aware feature will be used. > > > > > > > > > Thanks, > > > > > > Jun > > > > > > On Tue, Jan 12, 2016 at 12:19 PM, Allen Wang <allenxw...@gmail.com> > > wrote: > > > > > > > Regarding the JSON version of Broker: > > > > > > > > I don't why the ZkUtils.getBrokerInfo() restricts the JSON versions > it > > > can > > > > read. It will throw exception if version is not 1 or 2. Seems to me > > that > > > it > > > > will cause compatibility problem whenever the version needs to be > > changed > > > > and make the upgrade path difficult. > > > > > > > > One option we have is to make rack also part of version 2 and keep > the > > > > version 2 unchanged for this update. This will make the old clients > > > > compatible. During rolling upgrade, it will also avoid problems if > the > > > > controller/broker is still the old version. > > > > > > > > However, ZkUtils.getBrokerInfo() will be updated to return the Broker > > > with > > > > rack so the rack information will be available once the server/client > > is > > > > upgraded to the latest version. > > > > > > > > > > > > > > > > On Wed, Jan 6, 2016 at 6:28 PM, Allen Wang <allenxw...@gmail.com> > > wrote: > > > > > > > > > Updated KIP according to Jun's comment and included changes to TMR. > > > > > > > > > > On Tue, Jan 5, 2016 at 5:59 PM, Jun Rao <j...@confluent.io> wrote: > > > > > > > > > >> Hi, Allen, > > > > >> > > > > >> A couple of minor comments on the KIP. > > > > >> > > > > >> 1. The version of the broker JSON string says 2. It should be 3. > > > > >> > > > > >> 2. The new version of UpdateMetadataRequest should be 2, instead > of > > 1. > > > > >> Could you include the full wire protocol of version 2 of > > > > >> UpdateMetadataRequest and highlight the changed part? > > > > >> > > > > >> Thanks, > > > > >> > > > > >> Jun > > > > >> > > > > >> On Tue, Jan 5, 2016 at 3:11 PM, Allen Wang <allenxw...@gmail.com> > > > > wrote: > > > > >> > > > > >> > Jun and I had a chance to discuss it in a meeting and it is > agreed > > > to > > > > >> > change the TMR in a different patch. > > > > >> > > > > > >> > I can change the KIP to include rack in TMR. The essential > change > > is > > > > to > > > > >> add > > > > >> > rack into class BrokerEndPoint and make TMR version aware. > > > > >> > > > > > >> > > > > > >> > > > > > >> > On Tue, Jan 5, 2016 at 10:21 AM, Aditya Auradkar < > > > > >> > aaurad...@linkedin.com.invalid> wrote: > > > > >> > > > > > >> > > Jun/Allen - > > > > >> > > > > > > >> > > Did we ever actually agree on whether we should evolve the TMR > > to > > > > >> include > > > > >> > > rack info or not? > > > > >> > > I don't feel strongly about it but I if it's the right thing > to > > do > > > > we > > > > >> > > should probably do it in this KIP (can be a separate patch).. > it > > > > >> isn't a > > > > >> > > large change. > > > > >> > > > > > > >> > > Aditya > > > > >> > > > > > > >> > > On Sat, Dec 26, 2015 at 3:01 PM, Allen Wang < > > allenxw...@gmail.com > > > > > > > > >> > wrote: > > > > >> > > > > > > >> > > > Added the rolling upgrade instruction in the KIP, similar to > > > those > > > > >> in > > > > >> > > 0.9.0 > > > > >> > > > release notes. > > > > >> > > > > > > > >> > > > On Wed, Dec 16, 2015 at 11:32 AM, Allen Wang < > > > > allenxw...@gmail.com> > > > > >> > > wrote: > > > > >> > > > > > > > >> > > > > Hi Jun, > > > > >> > > > > > > > > >> > > > > The reason that TopicMetadataResponse is not included in > the > > > KIP > > > > >> is > > > > >> > > that > > > > >> > > > > it currently is not version aware . So we need to > introduce > > > > >> version > > > > >> > to > > > > >> > > it > > > > >> > > > > in order to make sure backward compatibility. It seems to > > me a > > > > big > > > > >> > > > change. > > > > >> > > > > Do we want to couple it with this KIP? Do we need to > further > > > > >> discuss > > > > >> > > what > > > > >> > > > > information to include in the new version besides rack? > For > > > > >> example, > > > > >> > > > should > > > > >> > > > > we include broker security protocol in > > TopicMetadataResponse? > > > > >> > > > > > > > > >> > > > > The other option is to make it a separate KIP to make > > > > >> > > > > TopicMetadataResponse version aware and decide what to > > > include, > > > > >> and > > > > >> > > make > > > > >> > > > > this KIP focus on the rack aware algorithm, admin tools > and > > > > >> related > > > > >> > > > > changes to inter-broker protocol . > > > > >> > > > > > > > > >> > > > > Thanks, > > > > >> > > > > Allen > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > On Mon, Dec 14, 2015 at 8:30 AM, Jun Rao < > j...@confluent.io> > > > > >> wrote: > > > > >> > > > > > > > > >> > > > >> Allen, > > > > >> > > > >> > > > > >> > > > >> Thanks for the proposal. A few comments. > > > > >> > > > >> > > > > >> > > > >> 1. Since this KIP changes the inter broker communication > > > > protocol > > > > >> > > > >> (UpdateMetadataRequest), we will need to document the > > upgrade > > > > >> path > > > > >> > > > >> (similar > > > > >> > > > >> to what's described in > > > > >> > > > >> http://kafka.apache.org/090/documentation.html#upgrade). > > > > >> > > > >> > > > > >> > > > >> 2. It might be useful to include the rack info of the > > broker > > > in > > > > >> > > > >> TopicMetadataResponse. This can be useful for > > administrative > > > > >> tasks, > > > > >> > as > > > > >> > > > >> well > > > > >> > > > >> as read affinity in the future. > > > > >> > > > >> > > > > >> > > > >> Jun > > > > >> > > > >> > > > > >> > > > >> > > > > >> > > > >> > > > > >> > > > >> On Thu, Dec 10, 2015 at 9:38 AM, Allen Wang < > > > > >> allenxw...@gmail.com> > > > > >> > > > wrote: > > > > >> > > > >> > > > > >> > > > >> > If there are no more comments I would like to call for > a > > > > vote. > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > > >> > On Sun, Nov 15, 2015 at 10:08 PM, Allen Wang < > > > > >> > allenxw...@gmail.com> > > > > >> > > > >> wrote: > > > > >> > > > >> > > > > > >> > > > >> > > KIP is updated with more details and how to handle > the > > > > >> situation > > > > >> > > > where > > > > >> > > > >> > > rack information is incomplete. > > > > >> > > > >> > > > > > > >> > > > >> > > In the situation where rack information is > incomplete, > > > but > > > > we > > > > >> > want > > > > >> > > > to > > > > >> > > > >> > > continue with the assignment, I have suggested to > > ignore > > > > all > > > > >> > rack > > > > >> > > > >> > > information and fallback to original algorithm. The > > > reason > > > > is > > > > >> > > > >> explained > > > > >> > > > >> > > below: > > > > >> > > > >> > > > > > > >> > > > >> > > The other options are to assume that the broker > without > > > the > > > > >> rack > > > > >> > > > >> belong > > > > >> > > > >> > to > > > > >> > > > >> > > its own unique rack, or they belong to one "default" > > > rack. > > > > >> > Either > > > > >> > > > way > > > > >> > > > >> we > > > > >> > > > >> > > choose, it is highly likely to result in uneven > number > > of > > > > >> > brokers > > > > >> > > in > > > > >> > > > >> > racks, > > > > >> > > > >> > > and it is quite possible that the "made up" racks > will > > > have > > > > >> much > > > > >> > > > fewer > > > > >> > > > >> > > number of brokers. As I explained in the KIP, uneven > > > number > > > > >> of > > > > >> > > > >> brokers in > > > > >> > > > >> > > racks will lead to uneven distribution of replicas > > among > > > > >> brokers > > > > >> > > > (even > > > > >> > > > >> > > though the leader distribution is still even). The > > > brokers > > > > in > > > > >> > the > > > > >> > > > rack > > > > >> > > > >> > that > > > > >> > > > >> > > has fewer number of brokers will get more replicas > per > > > > broker > > > > >> > than > > > > >> > > > >> > brokers > > > > >> > > > >> > > in other racks. > > > > >> > > > >> > > > > > > >> > > > >> > > Given this fact and the replica assignment produced > > will > > > be > > > > >> > > > incorrect > > > > >> > > > >> > > anyway from rack aware point of view, ignoring all > rack > > > > >> > > information > > > > >> > > > >> and > > > > >> > > > >> > > fallback to the original algorithm is not a bad > choice > > > > since > > > > >> it > > > > >> > > will > > > > >> > > > >> at > > > > >> > > > >> > > least have a better guarantee of replica > distribution. > > > > >> > > > >> > > > > > > >> > > > >> > > Also for command line tools it gives user a choice if > > for > > > > any > > > > >> > > reason > > > > >> > > > >> they > > > > >> > > > >> > > want to ignore rack information and fallback to the > > > > original > > > > >> > > > >> algorithm. > > > > >> > > > >> > > > > > > >> > > > >> > > > > > > >> > > > >> > > On Tue, Nov 10, 2015 at 9:04 AM, Allen Wang < > > > > >> > allenxw...@gmail.com > > > > >> > > > > > > > >> > > > >> > wrote: > > > > >> > > > >> > > > > > > >> > > > >> > >> I am busy with some time pressing issues for the > last > > > few > > > > >> > days. I > > > > >> > > > >> will > > > > >> > > > >> > >> think about how the incomplete rack information will > > > > affect > > > > >> the > > > > >> > > > >> balance > > > > >> > > > >> > and > > > > >> > > > >> > >> update the KIP by early next week. > > > > >> > > > >> > >> > > > > >> > > > >> > >> Thanks, > > > > >> > > > >> > >> Allen > > > > >> > > > >> > >> > > > > >> > > > >> > >> > > > > >> > > > >> > >> On Tue, Nov 3, 2015 at 9:03 AM, Neha Narkhede < > > > > >> > n...@confluent.io > > > > >> > > > > > > > >> > > > >> > wrote: > > > > >> > > > >> > >> > > > > >> > > > >> > >>> Few suggestions on improving the KIP > > > > >> > > > >> > >>> > > > > >> > > > >> > >>> *If some brokers have rack, and some do not, the > > > > algorithm > > > > >> > will > > > > >> > > > >> thrown > > > > >> > > > >> > an > > > > >> > > > >> > >>> > exception. This is to prevent incorrect > assignment > > > > >> caused by > > > > >> > > > user > > > > >> > > > >> > >>> error.* > > > > >> > > > >> > >>> > > > > >> > > > >> > >>> > > > > >> > > > >> > >>> In the KIP, can you clearly state the user-facing > > > > behavior > > > > >> > when > > > > >> > > > some > > > > >> > > > >> > >>> brokers have rack information and some don't. Which > > > > actions > > > > >> > and > > > > >> > > > >> > requests > > > > >> > > > >> > >>> will error out and how? > > > > >> > > > >> > >>> > > > > >> > > > >> > >>> *Even distribution of partition leadership among > > > brokers* > > > > >> > > > >> > >>> > > > > >> > > > >> > >>> > > > > >> > > > >> > >>> There is some information about arranging the > sorted > > > > broker > > > > >> > list > > > > >> > > > >> > >>> interlaced > > > > >> > > > >> > >>> with rack ids. Can you describe the changes to the > > > > current > > > > >> > > > algorithm > > > > >> > > > >> > in a > > > > >> > > > >> > >>> little more detail? How does this interlacing work > if > > > > only > > > > >> a > > > > >> > > > subset > > > > >> > > > >> of > > > > >> > > > >> > >>> brokers have the rack id configured? Does this > still > > > work > > > > >> if > > > > >> > > > uneven > > > > >> > > > >> # > > > > >> > > > >> > of > > > > >> > > > >> > >>> brokers are assigned to each rack? It might work, > I'm > > > > >> looking > > > > >> > > for > > > > >> > > > >> more > > > > >> > > > >> > >>> details on the changes, since it will affect the > > > behavior > > > > >> seen > > > > >> > > by > > > > >> > > > >> the > > > > >> > > > >> > >>> user > > > > >> > > > >> > >>> - imbalance on either the leaders or data or both. > > > > >> > > > >> > >>> > > > > >> > > > >> > >>> On Mon, Nov 2, 2015 at 6:39 PM, Aditya Auradkar < > > > > >> > > > >> > aaurad...@linkedin.com> > > > > >> > > > >> > >>> wrote: > > > > >> > > > >> > >>> > > > > >> > > > >> > >>> > I think this sounds reasonable. Anyone else have > > > > >> comments? > > > > >> > > > >> > >>> > > > > > >> > > > >> > >>> > Aditya > > > > >> > > > >> > >>> > > > > > >> > > > >> > >>> > On Tue, Oct 27, 2015 at 5:23 PM, Allen Wang < > > > > >> > > > allenxw...@gmail.com > > > > >> > > > >> > > > > > >> > > > >> > >>> wrote: > > > > >> > > > >> > >>> > > > > > >> > > > >> > >>> > > During the discussion in the hangout, it was > > > > mentioned > > > > >> > that > > > > >> > > it > > > > >> > > > >> > would > > > > >> > > > >> > >>> be > > > > >> > > > >> > >>> > > desirable that consumers know the rack > > information > > > of > > > > >> the > > > > >> > > > >> brokers > > > > >> > > > >> > so > > > > >> > > > >> > >>> that > > > > >> > > > >> > >>> > > they can consume from the broker in the same > rack > > > to > > > > >> > reduce > > > > >> > > > >> > latency. > > > > >> > > > >> > >>> As I > > > > >> > > > >> > >>> > > understand this will only be beneficial if > > consumer > > > > can > > > > >> > > > consume > > > > >> > > > >> > from > > > > >> > > > >> > >>> any > > > > >> > > > >> > >>> > > broker in ISR, which is not possible now. > > > > >> > > > >> > >>> > > > > > > >> > > > >> > >>> > > I suggest we skip the change to TMR. Once the > > > change > > > > is > > > > >> > made > > > > >> > > > to > > > > >> > > > >> > >>> consumer > > > > >> > > > >> > >>> > to > > > > >> > > > >> > >>> > > be able to consume from any broker in ISR, the > > rack > > > > >> > > > information > > > > >> > > > >> can > > > > >> > > > >> > >>> be > > > > >> > > > >> > >>> > > added to TMR. > > > > >> > > > >> > >>> > > > > > > >> > > > >> > >>> > > Another thing I want to confirm is command > line > > > > >> > behavior. I > > > > >> > > > >> think > > > > >> > > > >> > >>> the > > > > >> > > > >> > >>> > > desirable default behavior is to fail fast on > > > command > > > > >> line > > > > >> > > for > > > > >> > > > >> > >>> incomplete > > > > >> > > > >> > >>> > > rack mapping. The error message can include > > further > > > > >> > > > instruction > > > > >> > > > >> > that > > > > >> > > > >> > >>> > tells > > > > >> > > > >> > >>> > > the user to add an extra argument (like > > > > >> > > > >> "--allow-partial-rackinfo") > > > > >> > > > >> > >>> to > > > > >> > > > >> > >>> > > suppress the error and do an imperfect rack > aware > > > > >> > > assignment. > > > > >> > > > If > > > > >> > > > >> > the > > > > >> > > > >> > >>> > > default behavior is to allow incomplete > mapping, > > > the > > > > >> error > > > > >> > > can > > > > >> > > > >> > still > > > > >> > > > >> > >>> be > > > > >> > > > >> > >>> > > easily missed. > > > > >> > > > >> > >>> > > > > > > >> > > > >> > >>> > > The affected command line tools are > TopicCommand > > > and > > > > >> > > > >> > >>> > > ReassignPartitionsCommand. > > > > >> > > > >> > >>> > > > > > > >> > > > >> > >>> > > Thanks, > > > > >> > > > >> > >>> > > Allen > > > > >> > > > >> > >>> > > > > > > >> > > > >> > >>> > > > > > > >> > > > >> > >>> > > > > > > >> > > > >> > >>> > > > > > > >> > > > >> > >>> > > > > > > >> > > > >> > >>> > > On Mon, Oct 26, 2015 at 12:55 PM, Aditya > > Auradkar < > > > > >> > > > >> > >>> > aaurad...@linkedin.com> > > > > >> > > > >> > >>> > > wrote: > > > > >> > > > >> > >>> > > > > > > >> > > > >> > >>> > > > Hi Allen, > > > > >> > > > >> > >>> > > > > > > > >> > > > >> > >>> > > > For TopicMetadataResponse to understand > > version, > > > > you > > > > >> can > > > > >> > > > bump > > > > >> > > > >> up > > > > >> > > > >> > >>> the > > > > >> > > > >> > >>> > > > request version itself. Based on the version > of > > > the > > > > >> > > request, > > > > >> > > > >> the > > > > >> > > > >> > >>> > response > > > > >> > > > >> > >>> > > > can be appropriately serialized. It shouldn't > > be > > > a > > > > >> huge > > > > >> > > > >> change. > > > > >> > > > >> > For > > > > >> > > > >> > >>> > > > example: We went through something similar > for > > > > >> > > > ProduceRequest > > > > >> > > > >> > >>> recently > > > > >> > > > >> > >>> > ( > > > > >> > > > >> > >>> > > > https://reviews.apache.org/r/33378/) > > > > >> > > > >> > >>> > > > I guess the reason protocol information is > not > > > > >> included > > > > >> > in > > > > >> > > > the > > > > >> > > > >> > TMR > > > > >> > > > >> > >>> is > > > > >> > > > >> > >>> > > > because the topic itself is independent of > any > > > > >> > particular > > > > >> > > > >> > protocol > > > > >> > > > >> > >>> (SSL > > > > >> > > > >> > >>> > > vs > > > > >> > > > >> > >>> > > > Plaintext). Having said that, I'm not sure we > > > even > > > > >> need > > > > >> > > rack > > > > >> > > > >> > >>> > information > > > > >> > > > >> > >>> > > in > > > > >> > > > >> > >>> > > > TMR. What usecase were you thinking of > > initially? > > > > >> > > > >> > >>> > > > > > > > >> > > > >> > >>> > > > For 1 - I'd be fine with adding an option to > > the > > > > >> command > > > > >> > > > line > > > > >> > > > >> > tools > > > > >> > > > >> > >>> > that > > > > >> > > > >> > >>> > > > check rack assignment. For e.g. > > > > >> "--strict-assignment" or > > > > >> > > > >> > something > > > > >> > > > >> > >>> > > similar. > > > > >> > > > >> > >>> > > > > > > > >> > > > >> > >>> > > > Aditya > > > > >> > > > >> > >>> > > > > > > > >> > > > >> > >>> > > > On Thu, Oct 22, 2015 at 6:44 PM, Allen Wang < > > > > >> > > > >> > allenxw...@gmail.com> > > > > >> > > > >> > >>> > > wrote: > > > > >> > > > >> > >>> > > > > > > > >> > > > >> > >>> > > > > For 2 and 3, I have updated the KIP. Please > > > take > > > > a > > > > >> > look. > > > > >> > > > One > > > > >> > > > >> > >>> thing I > > > > >> > > > >> > >>> > > have > > > > >> > > > >> > >>> > > > > changed is removing the proposal to add > rack > > to > > > > >> > > > >> > >>> > TopicMetadataResponse. > > > > >> > > > >> > >>> > > > The > > > > >> > > > >> > >>> > > > > reason is that unlike > UpdateMetadataRequest, > > > > >> > > > >> > >>> TopicMetadataResponse > > > > >> > > > >> > >>> > does > > > > >> > > > >> > >>> > > > not > > > > >> > > > >> > >>> > > > > understand version. I don't see a way to > > > include > > > > >> rack > > > > >> > > > >> without > > > > >> > > > >> > >>> > breaking > > > > >> > > > >> > >>> > > > old > > > > >> > > > >> > >>> > > > > version of clients. That's probably why > > secure > > > > >> > protocol > > > > >> > > is > > > > >> > > > >> not > > > > >> > > > >> > >>> > included > > > > >> > > > >> > >>> > > > in > > > > >> > > > >> > >>> > > > > the TopicMetadataResponse either. I think > it > > > will > > > > >> be a > > > > >> > > > much > > > > >> > > > >> > >>> bigger > > > > >> > > > >> > >>> > > change > > > > >> > > > >> > >>> > > > > to include rack in TopicMetadataResponse. > > > > >> > > > >> > >>> > > > > > > > > >> > > > >> > >>> > > > > For 1, my concern is that doing rack aware > > > > >> assignment > > > > >> > > > >> without > > > > >> > > > >> > >>> > complete > > > > >> > > > >> > >>> > > > > broker to rack mapping will result in > > > assignment > > > > >> that > > > > >> > is > > > > >> > > > not > > > > >> > > > >> > rack > > > > >> > > > >> > >>> > aware > > > > >> > > > >> > >>> > > > and > > > > >> > > > >> > >>> > > > > fail to provide fault tolerance in the > event > > of > > > > >> rack > > > > >> > > > outage. > > > > >> > > > >> > This > > > > >> > > > >> > >>> > kind > > > > >> > > > >> > >>> > > of > > > > >> > > > >> > >>> > > > > problem will be difficult to surface. And > the > > > > cost > > > > >> of > > > > >> > > this > > > > >> > > > >> > >>> problem is > > > > >> > > > >> > >>> > > > high: > > > > >> > > > >> > >>> > > > > you have to do partition reassignment if > you > > > are > > > > >> lucky > > > > >> > > to > > > > >> > > > >> spot > > > > >> > > > >> > >>> the > > > > >> > > > >> > >>> > > > problem > > > > >> > > > >> > >>> > > > > early on or face the consequence of data > loss > > > > >> during > > > > >> > > real > > > > >> > > > >> rack > > > > >> > > > >> > >>> > outage. > > > > >> > > > >> > >>> > > > > > > > > >> > > > >> > >>> > > > > I do see the concern of fail-fast as it > might > > > > also > > > > >> > cause > > > > >> > > > >> data > > > > >> > > > >> > >>> loss if > > > > >> > > > >> > >>> > > > > producer is not able produce the message > due > > to > > > > >> topic > > > > >> > > > >> creation > > > > >> > > > >> > >>> > failure. > > > > >> > > > >> > >>> > > > Is > > > > >> > > > >> > >>> > > > > it feasible to treat dynamic topic creation > > and > > > > >> > command > > > > >> > > > >> tools > > > > >> > > > >> > >>> > > > differently? > > > > >> > > > >> > >>> > > > > We allow dynamic topic creation with > > incomplete > > > > >> > > > broker-rack > > > > >> > > > >> > >>> mapping > > > > >> > > > >> > >>> > and > > > > >> > > > >> > >>> > > > > fail fast in command line. Another option > is > > to > > > > let > > > > >> > user > > > > >> > > > >> > >>> determine > > > > >> > > > >> > >>> > the > > > > >> > > > >> > >>> > > > > behavior for command line. For example, by > > > > default > > > > >> > fail > > > > >> > > > >> fast in > > > > >> > > > >> > >>> > command > > > > >> > > > >> > >>> > > > > line but allow incomplete broker-rack > mapping > > > if > > > > >> > another > > > > >> > > > >> switch > > > > >> > > > >> > >>> is > > > > >> > > > >> > >>> > > > > provided. > > > > >> > > > >> > >>> > > > > > > > > >> > > > >> > >>> > > > > > > > > >> > > > >> > >>> > > > > > > > > >> > > > >> > >>> > > > > > > > > >> > > > >> > >>> > > > > On Tue, Oct 20, 2015 at 10:05 AM, Aditya > > > > Auradkar < > > > > >> > > > >> > >>> > > > > aaurad...@linkedin.com.invalid> wrote: > > > > >> > > > >> > >>> > > > > > > > > >> > > > >> > >>> > > > > > Hey Allen, > > > > >> > > > >> > >>> > > > > > > > > > >> > > > >> > >>> > > > > > 1. If we choose fail fast topic creation, > > we > > > > will > > > > >> > have > > > > >> > > > >> topic > > > > >> > > > >> > >>> > creation > > > > >> > > > >> > >>> > > > > > failures while upgrading the cluster. I > > > really > > > > >> doubt > > > > >> > > we > > > > >> > > > >> want > > > > >> > > > >> > >>> this > > > > >> > > > >> > >>> > > > > behavior. > > > > >> > > > >> > >>> > > > > > Ideally, this should be invisible to > > clients > > > > of a > > > > >> > > > cluster. > > > > >> > > > >> > >>> > Currently, > > > > >> > > > >> > >>> > > > > each > > > > >> > > > >> > >>> > > > > > broker is effectively its own rack. So we > > > > >> probably > > > > >> > can > > > > >> > > > use > > > > >> > > > >> > the > > > > >> > > > >> > >>> rack > > > > >> > > > >> > >>> > > > > > information whenever possible but not > make > > > it a > > > > >> hard > > > > >> > > > >> > >>> requirement. > > > > >> > > > >> > >>> > To > > > > >> > > > >> > >>> > > > > extend > > > > >> > > > >> > >>> > > > > > Gwen's example, one badly configured > broker > > > > >> should > > > > >> > not > > > > >> > > > >> > degrade > > > > >> > > > >> > >>> > topic > > > > >> > > > >> > >>> > > > > > creation for the entire cluster. > > > > >> > > > >> > >>> > > > > > > > > > >> > > > >> > >>> > > > > > 2. Upgrade scenario - Can you add a > section > > > on > > > > >> the > > > > >> > > > upgrade > > > > >> > > > >> > >>> piece to > > > > >> > > > >> > >>> > > > > confirm > > > > >> > > > >> > >>> > > > > > that old clients will not see errors? I > > > believe > > > > >> > > > >> > >>> > > > > ZookeeperConsumerConnector > > > > >> > > > >> > >>> > > > > > reads the Broker objects from ZK. I > wanted > > to > > > > >> > confirm > > > > >> > > > that > > > > >> > > > >> > this > > > > >> > > > >> > >>> > will > > > > >> > > > >> > >>> > > > not > > > > >> > > > >> > >>> > > > > > cause any problems. > > > > >> > > > >> > >>> > > > > > > > > > >> > > > >> > >>> > > > > > 3. Could you elaborate your proposed > > changes > > > to > > > > >> the > > > > >> > > > >> > >>> > > > UpdateMetadataRequest > > > > >> > > > >> > >>> > > > > > in the "Public Interfaces" section? > > > > Personally, I > > > > >> > find > > > > >> > > > >> this > > > > >> > > > >> > >>> format > > > > >> > > > >> > >>> > > easy > > > > >> > > > >> > >>> > > > > to > > > > >> > > > >> > >>> > > > > > read in terms of wire protocol changes: > > > > >> > > > >> > >>> > > > > > > > > > >> > > > >> > >>> > > > > > > > > > >> > > > >> > >>> > > > > > > > > >> > > > >> > >>> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >> > >>> > > > > > >> > > > >> > >>> > > > > >> > > > >> > > > > > >> > > > >> > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicRequest > > > > >> > > > >> > >>> > > > > > > > > > >> > > > >> > >>> > > > > > Aditya > > > > >> > > > >> > >>> > > > > > > > > > >> > > > >> > >>> > > > > > On Fri, Oct 16, 2015 at 3:45 PM, Allen > > Wang < > > > > >> > > > >> > >>> allenxw...@gmail.com> > > > > >> > > > >> > >>> > > > > wrote: > > > > >> > > > >> > >>> > > > > > > > > > >> > > > >> > >>> > > > > > > KIP is updated include rack as an > > optional > > > > >> > property > > > > >> > > > for > > > > >> > > > >> > >>> broker. > > > > >> > > > >> > >>> > > > Please > > > > >> > > > >> > >>> > > > > > take > > > > >> > > > >> > >>> > > > > > > a look and let me know if more details > > are > > > > >> needed. > > > > >> > > > >> > >>> > > > > > > > > > > >> > > > >> > >>> > > > > > > For the case where some brokers have > rack > > > and > > > > >> some > > > > >> > > do > > > > >> > > > >> not, > > > > >> > > > >> > >>> the > > > > >> > > > >> > >>> > > > current > > > > >> > > > >> > >>> > > > > > KIP > > > > >> > > > >> > >>> > > > > > > uses the fail-fast behavior. If there > are > > > > >> > concerns, > > > > >> > > we > > > > >> > > > >> can > > > > >> > > > >> > >>> > further > > > > >> > > > >> > >>> > > > > > discuss > > > > >> > > > >> > >>> > > > > > > this in the email thread or next > hangout. > > > > >> > > > >> > >>> > > > > > > > > > > >> > > > >> > >>> > > > > > > > > > > >> > > > >> > >>> > > > > > > > > > > >> > > > >> > >>> > > > > > > On Thu, Oct 15, 2015 at 10:42 AM, Allen > > > Wang > > > > < > > > > >> > > > >> > >>> > allenxw...@gmail.com > > > > >> > > > >> > >>> > > > > > > > >> > > > >> > >>> > > > > > wrote: > > > > >> > > > >> > >>> > > > > > > > > > > >> > > > >> > >>> > > > > > > > That's a good question. I can think > of > > > > three > > > > >> > > actions > > > > >> > > > >> if > > > > >> > > > >> > the > > > > >> > > > >> > >>> > rack > > > > >> > > > >> > >>> > > > > > > > information is incomplete: > > > > >> > > > >> > >>> > > > > > > > > > > > >> > > > >> > >>> > > > > > > > 1. Treat the node without rack as if > it > > > is > > > > on > > > > >> > its > > > > >> > > > >> unique > > > > >> > > > >> > >>> rack > > > > >> > > > >> > >>> > > > > > > > 2. Disregard all rack information and > > > > >> fallback > > > > >> > to > > > > >> > > > >> current > > > > >> > > > >> > >>> > > algorithm > > > > >> > > > >> > >>> > > > > > > > 3. Fail-fast > > > > >> > > > >> > >>> > > > > > > > > > > > >> > > > >> > >>> > > > > > > > Now I think about it, one and three > > make > > > > more > > > > >> > > sense. > > > > >> > > > >> The > > > > >> > > > >> > >>> reason > > > > >> > > > >> > >>> > > for > > > > >> > > > >> > >>> > > > > > > > fail-fast is that user mistake for > not > > > > >> providing > > > > >> > > the > > > > >> > > > >> rack > > > > >> > > > >> > >>> may > > > > >> > > > >> > >>> > > never > > > > >> > > > >> > >>> > > > > be > > > > >> > > > >> > >>> > > > > > > > found if we tolerate that and the > > > > assignment > > > > >> may > > > > >> > > not > > > > >> > > > >> be > > > > >> > > > >> > >>> rack > > > > >> > > > >> > >>> > > aware > > > > >> > > > >> > >>> > > > as > > > > >> > > > >> > >>> > > > > > the > > > > >> > > > >> > >>> > > > > > > > user has expected and this creates > > debug > > > > >> > problems > > > > >> > > > when > > > > >> > > > >> > >>> things > > > > >> > > > >> > >>> > > fail. > > > > >> > > > >> > >>> > > > > > > > > > > > >> > > > >> > >>> > > > > > > > What do you think? If not fail-fast, > is > > > > there > > > > >> > > anyway > > > > >> > > > >> we > > > > >> > > > >> > can > > > > >> > > > >> > >>> > make > > > > >> > > > >> > >>> > > > the > > > > >> > > > >> > >>> > > > > > user > > > > >> > > > >> > >>> > > > > > > > error standing out? > > > > >> > > > >> > >>> > > > > > > > > > > > >> > > > >> > >>> > > > > > > > > > > > >> > > > >> > >>> > > > > > > > On Thu, Oct 15, 2015 at 10:17 AM, > Gwen > > > > >> Shapira < > > > > >> > > > >> > >>> > > g...@confluent.io> > > > > >> > > > >> > >>> > > > > > > wrote: > > > > >> > > > >> > >>> > > > > > > > > > > > >> > > > >> > >>> > > > > > > >> Thanks! Just to clarify, when some > > > brokers > > > > >> have > > > > >> > > > rack > > > > >> > > > >> > >>> > assignment > > > > >> > > > >> > >>> > > > and > > > > >> > > > >> > >>> > > > > > some > > > > >> > > > >> > >>> > > > > > > >> don't, do we act like none of them > > have > > > > it? > > > > >> or > > > > >> > > like > > > > >> > > > >> > those > > > > >> > > > >> > >>> > > without > > > > >> > > > >> > >>> > > > > > > >> assignment are in their own rack? > > > > >> > > > >> > >>> > > > > > > >> > > > > >> > > > >> > >>> > > > > > > >> The first scenario is good when > first > > > > >> setting > > > > >> > up > > > > >> > > > >> > >>> > rack-awareness, > > > > >> > > > >> > >>> > > > but > > > > >> > > > >> > >>> > > > > > the > > > > >> > > > >> > >>> > > > > > > >> second makes more sense for on-going > > > > >> > maintenance > > > > >> > > (I > > > > >> > > > >> can > > > > >> > > > >> > >>> > totally > > > > >> > > > >> > >>> > > > see > > > > >> > > > >> > >>> > > > > > > >> someone > > > > >> > > > >> > >>> > > > > > > >> adding a node and forgetting to set > > the > > > > rack > > > > >> > > > >> property, > > > > >> > > > >> > we > > > > >> > > > >> > >>> > don't > > > > >> > > > >> > >>> > > > want > > > > >> > > > >> > >>> > > > > > > this > > > > >> > > > >> > >>> > > > > > > >> to change behavior for anything > except > > > the > > > > >> new > > > > >> > > > node). > > > > >> > > > >> > >>> > > > > > > >> > > > > >> > > > >> > >>> > > > > > > >> What do you think? > > > > >> > > > >> > >>> > > > > > > >> > > > > >> > > > >> > >>> > > > > > > >> Gwen > > > > >> > > > >> > >>> > > > > > > >> > > > > >> > > > >> > >>> > > > > > > >> On Thu, Oct 15, 2015 at 10:13 AM, > > Allen > > > > >> Wang < > > > > >> > > > >> > >>> > > > allenxw...@gmail.com> > > > > >> > > > >> > >>> > > > > > > >> wrote: > > > > >> > > > >> > >>> > > > > > > >> > > > > >> > > > >> > >>> > > > > > > >> > For scenario 1: > > > > >> > > > >> > >>> > > > > > > >> > > > > > >> > > > >> > >>> > > > > > > >> > - Add the rack information to > broker > > > > >> property > > > > >> > > > file > > > > >> > > > >> or > > > > >> > > > >> > >>> > > > dynamically > > > > >> > > > >> > >>> > > > > > set > > > > >> > > > >> > >>> > > > > > > >> it in > > > > >> > > > >> > >>> > > > > > > >> > the wrapper code to bootstrap > Kafka > > > > >> server. > > > > >> > You > > > > >> > > > >> would > > > > >> > > > >> > do > > > > >> > > > >> > >>> > that > > > > >> > > > >> > >>> > > > for > > > > >> > > > >> > >>> > > > > > all > > > > >> > > > >> > >>> > > > > > > >> > brokers and restart the brokers > one > > by > > > > >> one. > > > > >> > > > >> > >>> > > > > > > >> > > > > > >> > > > >> > >>> > > > > > > >> > In this scenario, the complete > > broker > > > to > > > > >> rack > > > > >> > > > >> mapping > > > > >> > > > >> > >>> may > > > > >> > > > >> > >>> > not > > > > >> > > > >> > >>> > > be > > > > >> > > > >> > >>> > > > > > > >> available > > > > >> > > > >> > >>> > > > > > > >> > until every broker is restarted. > > > During > > > > >> that > > > > >> > > time > > > > >> > > > >> we > > > > >> > > > >> > >>> fall > > > > >> > > > >> > >>> > back > > > > >> > > > >> > >>> > > > to > > > > >> > > > >> > >>> > > > > > > >> default > > > > >> > > > >> > >>> > > > > > > >> > replica assignment algorithm. > > > > >> > > > >> > >>> > > > > > > >> > > > > > >> > > > >> > >>> > > > > > > >> > For scenario 2: > > > > >> > > > >> > >>> > > > > > > >> > > > > > >> > > > >> > >>> > > > > > > >> > - Add the rack information to > broker > > > > >> property > > > > >> > > > file > > > > >> > > > >> or > > > > >> > > > >> > >>> > > > dynamically > > > > >> > > > >> > >>> > > > > > set > > > > >> > > > >> > >>> > > > > > > >> it in > > > > >> > > > >> > >>> > > > > > > >> > the wrapper code and start the > > broker. > > > > >> > > > >> > >>> > > > > > > >> > > > > > >> > > > >> > >>> > > > > > > >> > > > > > >> > > > >> > >>> > > > > > > >> > On Wed, Oct 14, 2015 at 2:36 PM, > > Gwen > > > > >> > Shapira < > > > > >> > > > >> > >>> > > > g...@confluent.io> > > > > >> > > > >> > >>> > > > > > > >> wrote: > > > > >> > > > >> > >>> > > > > > > >> > > > > > >> > > > >> > >>> > > > > > > >> > > Can you clarify the workflow for > > the > > > > >> > > following > > > > >> > > > >> > >>> scenarios: > > > > >> > > > >> > >>> > > > > > > >> > > > > > > >> > > > >> > >>> > > > > > > >> > > 1. I currently have 6 brokers > and > > > want > > > > >> to > > > > >> > add > > > > >> > > > >> rack > > > > >> > > > >> > >>> > > information > > > > >> > > > >> > >>> > > > > for > > > > >> > > > >> > >>> > > > > > > >> each > > > > >> > > > >> > >>> > > > > > > >> > > 2. I'm adding a new broker and I > > > want > > > > to > > > > >> > > > specify > > > > >> > > > >> > which > > > > >> > > > >> > >>> > rack > > > > >> > > > >> > >>> > > it > > > > >> > > > >> > >>> > > > > > > >> belongs on > > > > >> > > > >> > >>> > > > > > > >> > > while adding it. > > > > >> > > > >> > >>> > > > > > > >> > > > > > > >> > > > >> > >>> > > > > > > >> > > Thanks! > > > > >> > > > >> > >>> > > > > > > >> > > > > > > >> > > > >> > >>> > > > > > > >> > > On Tue, Oct 13, 2015 at 2:21 PM, > > > Allen > > > > >> > Wang < > > > > >> > > > >> > >>> > > > > allenxw...@gmail.com > > > > >> > > > >> > >>> > > > > > > > > > > >> > > > >> > >>> > > > > > > >> > wrote: > > > > >> > > > >> > >>> > > > > > > >> > > > > > > >> > > > >> > >>> > > > > > > >> > > > We discussed the KIP in the > > > hangout > > > > >> > today. > > > > >> > > > The > > > > >> > > > >> > >>> > > > recommendation > > > > >> > > > >> > >>> > > > > is > > > > >> > > > >> > >>> > > > > > > to > > > > >> > > > >> > >>> > > > > > > >> > make > > > > >> > > > >> > >>> > > > > > > >> > > > rack as a broker property in > > > > >> ZooKeeper. > > > > >> > For > > > > >> > > > >> users > > > > >> > > > >> > >>> with > > > > >> > > > >> > >>> > > > > existing > > > > >> > > > >> > >>> > > > > > > rack > > > > >> > > > >> > >>> > > > > > > >> > > > information stored somewhere, > > they > > > > >> would > > > > >> > > need > > > > >> > > > >> to > > > > >> > > > >> > >>> > retrieve > > > > >> > > > >> > >>> > > > the > > > > >> > > > >> > >>> > > > > > > >> > information > > > > >> > > > >> > >>> > > > > > > >> > > > at broker start up and > > dynamically > > > > set > > > > >> > the > > > > >> > > > rack > > > > >> > > > >> > >>> > property, > > > > >> > > > >> > >>> > > > > which > > > > >> > > > >> > >>> > > > > > > can > > > > >> > > > >> > >>> > > > > > > >> be > > > > >> > > > >> > >>> > > > > > > >> > > > implemented as a wrapper to > > > > bootstrap > > > > >> > > broker. > > > > >> > > > >> > There > > > > >> > > > >> > >>> will > > > > >> > > > >> > >>> > > be > > > > >> > > > >> > >>> > > > no > > > > >> > > > >> > >>> > > > > > > >> > interface > > > > >> > > > >> > >>> > > > > > > >> > > or > > > > >> > > > >> > >>> > > > > > > >> > > > pluggable implementation to > > > retrieve > > > > >> the > > > > >> > > rack > > > > >> > > > >> > >>> > information. > > > > >> > > > >> > >>> > > > > > > >> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > The assumption is that you > > always > > > > >> need to > > > > >> > > > >> restart > > > > >> > > > >> > >>> the > > > > >> > > > >> > >>> > > broker > > > > >> > > > >> > >>> > > > > to > > > > >> > > > >> > >>> > > > > > > >> make a > > > > >> > > > >> > >>> > > > > > > >> > > > change to the rack. > > > > >> > > > >> > >>> > > > > > > >> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > Once the rack becomes a broker > > > > >> property, > > > > >> > it > > > > >> > > > >> will > > > > >> > > > >> > be > > > > >> > > > >> > >>> > > possible > > > > >> > > > >> > >>> > > > > to > > > > >> > > > >> > >>> > > > > > > make > > > > >> > > > >> > >>> > > > > > > >> > rack > > > > >> > > > >> > >>> > > > > > > >> > > > part of the meta data to help > > the > > > > >> > consumer > > > > >> > > > >> choose > > > > >> > > > >> > >>> which > > > > >> > > > >> > >>> > in > > > > >> > > > >> > >>> > > > > sync > > > > >> > > > >> > >>> > > > > > > >> replica > > > > >> > > > >> > >>> > > > > > > >> > > to > > > > >> > > > >> > >>> > > > > > > >> > > > consume from as part of the > > future > > > > >> > consumer > > > > >> > > > >> > >>> enhancement. > > > > >> > > > >> > >>> > > > > > > >> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > I will update the KIP. > > > > >> > > > >> > >>> > > > > > > >> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > Thanks, > > > > >> > > > >> > >>> > > > > > > >> > > > Allen > > > > >> > > > >> > >>> > > > > > > >> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > On Thu, Oct 8, 2015 at 9:23 > AM, > > > > Allen > > > > >> > Wang > > > > >> > > < > > > > >> > > > >> > >>> > > > > > allenxw...@gmail.com> > > > > >> > > > >> > >>> > > > > > > >> > wrote: > > > > >> > > > >> > >>> > > > > > > >> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > > I attended Tuesday's KIP > > hangout > > > > but > > > > >> > this > > > > >> > > > KIP > > > > >> > > > >> > was > > > > >> > > > >> > >>> not > > > > >> > > > >> > >>> > > > > > discussed > > > > >> > > > >> > >>> > > > > > > >> due > > > > >> > > > >> > >>> > > > > > > >> > to > > > > >> > > > >> > >>> > > > > > > >> > > > > time constraint. > > > > >> > > > >> > >>> > > > > > > >> > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > > However, after hearing > > > discussion > > > > of > > > > >> > > > KIP-35, > > > > >> > > > >> I > > > > >> > > > >> > >>> have > > > > >> > > > >> > >>> > the > > > > >> > > > >> > >>> > > > > > feeling > > > > >> > > > >> > >>> > > > > > > >> that > > > > >> > > > >> > >>> > > > > > > >> > > > > incompatibility (caused by > new > > > > >> broker > > > > >> > > > >> property) > > > > >> > > > >> > >>> > between > > > > >> > > > >> > >>> > > > > > brokers > > > > >> > > > >> > >>> > > > > > > >> with > > > > >> > > > >> > >>> > > > > > > >> > > > > different versions will be > > > solved > > > > >> > there. > > > > >> > > > In > > > > >> > > > >> > >>> addition, > > > > >> > > > >> > >>> > > > > having > > > > >> > > > >> > >>> > > > > > > >> stack > > > > >> > > > >> > >>> > > > > > > >> > in > > > > >> > > > >> > >>> > > > > > > >> > > > > broker property as meta data > > may > > > > >> also > > > > >> > > help > > > > >> > > > >> > >>> consumers > > > > >> > > > >> > >>> > in > > > > >> > > > >> > >>> > > > the > > > > >> > > > >> > >>> > > > > > > >> future. > > > > >> > > > >> > >>> > > > > > > >> > So > > > > >> > > > >> > >>> > > > > > > >> > > I > > > > >> > > > >> > >>> > > > > > > >> > > > am > > > > >> > > > >> > >>> > > > > > > >> > > > > open to adding stack > property > > to > > > > >> > broker. > > > > >> > > > >> > >>> > > > > > > >> > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > > Hopefully we can discuss > this > > in > > > > the > > > > >> > next > > > > >> > > > KIP > > > > >> > > > >> > >>> hangout. > > > > >> > > > >> > >>> > > > > > > >> > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > > On Wed, Sep 30, 2015 at 2:46 > > PM, > > > > >> Allen > > > > >> > > > Wang < > > > > >> > > > >> > >>> > > > > > > allenxw...@gmail.com > > > > >> > > > >> > >>> > > > > > > >> > > > > > >> > > > >> > >>> > > > > > > >> > > > wrote: > > > > >> > > > >> > >>> > > > > > > >> > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >> Can you send me the > > information > > > > on > > > > >> the > > > > >> > > > next > > > > >> > > > >> KIP > > > > >> > > > >> > >>> > > hangout? > > > > >> > > > >> > >>> > > > > > > >> > > > >> > > > > >> > > > >> > >>> > > > > > > >> > > > >> Currently the broker-rack > > > mapping > > > > >> is > > > > >> > not > > > > >> > > > >> > cached. > > > > >> > > > >> > >>> In > > > > >> > > > >> > >>> > > > > > KafkaApis, > > > > >> > > > >> > >>> > > > > > > >> > > > >> RackLocator.getRackInfo() > is > > > > called > > > > >> > each > > > > >> > > > >> time > > > > >> > > > >> > the > > > > >> > > > >> > >>> > > mapping > > > > >> > > > >> > >>> > > > > is > > > > >> > > > >> > >>> > > > > > > >> needed > > > > >> > > > >> > >>> > > > > > > >> > > for > > > > >> > > > >> > >>> > > > > > > >> > > > >> auto topic creation. This > > will > > > > >> ensure > > > > >> > > > latest > > > > >> > > > >> > >>> mapping > > > > >> > > > >> > >>> > is > > > > >> > > > >> > >>> > > > > used > > > > >> > > > >> > >>> > > > > > at > > > > >> > > > >> > >>> > > > > > > >> any > > > > >> > > > >> > >>> > > > > > > >> > > > time. > > > > >> > > > >> > >>> > > > > > > >> > > > >> > > > > >> > > > >> > >>> > > > > > > >> > > > >> The ability to get the > > complete > > > > >> > mapping > > > > >> > > > >> makes > > > > >> > > > >> > it > > > > >> > > > >> > >>> > simple > > > > >> > > > >> > >>> > > > to > > > > >> > > > >> > >>> > > > > > > reuse > > > > >> > > > >> > >>> > > > > > > >> the > > > > >> > > > >> > >>> > > > > > > >> > > > same > > > > >> > > > >> > >>> > > > > > > >> > > > >> interface in command line > > > tools. > > > > >> > > > >> > >>> > > > > > > >> > > > >> > > > > >> > > > >> > >>> > > > > > > >> > > > >> > > > > >> > > > >> > >>> > > > > > > >> > > > >> On Wed, Sep 30, 2015 at > 11:01 > > > AM, > > > > >> > Aditya > > > > >> > > > >> > >>> Auradkar < > > > > >> > > > >> > >>> > > > > > > >> > > > >> > > aaurad...@linkedin.com.invalid > > > > > > > > >> > wrote: > > > > >> > > > >> > >>> > > > > > > >> > > > >> > > > > >> > > > >> > >>> > > > > > > >> > > > >>> Perhaps we discuss this > > during > > > > the > > > > >> > next > > > > >> > > > KIP > > > > >> > > > >> > >>> hangout? > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > >> > > > >> > >>> > > > > > > >> > > > >>> I do see that a pluggable > > rack > > > > >> > locator > > > > >> > > > can > > > > >> > > > >> be > > > > >> > > > >> > >>> useful > > > > >> > > > >> > >>> > > > but I > > > > >> > > > >> > >>> > > > > > do > > > > >> > > > >> > >>> > > > > > > >> see a > > > > >> > > > >> > >>> > > > > > > >> > > few > > > > >> > > > >> > >>> > > > > > > >> > > > >>> concerns: > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > >> > > > >> > >>> > > > > > > >> > > > >>> - The RackLocator (as > > > described > > > > in > > > > >> > the > > > > >> > > > >> > >>> document), > > > > >> > > > >> > >>> > > > implies > > > > >> > > > >> > >>> > > > > > that > > > > >> > > > >> > >>> > > > > > > >> it > > > > >> > > > >> > >>> > > > > > > >> > can > > > > >> > > > >> > >>> > > > > > > >> > > > >>> discover rack information > > for > > > > any > > > > >> > node > > > > >> > > in > > > > >> > > > >> the > > > > >> > > > >> > >>> > cluster. > > > > >> > > > >> > >>> > > > How > > > > >> > > > >> > >>> > > > > > > does > > > > >> > > > >> > >>> > > > > > > >> it > > > > >> > > > >> > >>> > > > > > > >> > > deal > > > > >> > > > >> > >>> > > > > > > >> > > > >>> with rack location > changes? > > > For > > > > >> > > example, > > > > >> > > > >> if I > > > > >> > > > >> > >>> moved > > > > >> > > > >> > >>> > > > broker > > > > >> > > > >> > >>> > > > > > id > > > > >> > > > >> > >>> > > > > > > >> (1) > > > > >> > > > >> > >>> > > > > > > >> > > from > > > > >> > > > >> > >>> > > > > > > >> > > > >>> rack > > > > >> > > > >> > >>> > > > > > > >> > > > >>> X to Y, I only have to > start > > > > that > > > > >> > > broker > > > > >> > > > >> with > > > > >> > > > >> > a > > > > >> > > > >> > >>> > newer > > > > >> > > > >> > >>> > > > rack > > > > >> > > > >> > >>> > > > > > > >> config. > > > > >> > > > >> > >>> > > > > > > >> > If > > > > >> > > > >> > >>> > > > > > > >> > > > >>> RackLocator discovers > broker > > > -> > > > > >> rack > > > > >> > > > >> > >>> information at > > > > >> > > > >> > >>> > > > start > > > > >> > > > >> > >>> > > > > up > > > > >> > > > >> > >>> > > > > > > >> time, > > > > >> > > > >> > >>> > > > > > > >> > > any > > > > >> > > > >> > >>> > > > > > > >> > > > >>> change to a broker will > > > require > > > > >> > > bouncing > > > > >> > > > >> the > > > > >> > > > >> > >>> entire > > > > >> > > > >> > >>> > > > > cluster > > > > >> > > > >> > >>> > > > > > > >> since > > > > >> > > > >> > >>> > > > > > > >> > > > >>> createTopic requests can > be > > > sent > > > > >> to > > > > >> > any > > > > >> > > > >> node > > > > >> > > > >> > in > > > > >> > > > >> > >>> the > > > > >> > > > >> > >>> > > > > cluster. > > > > >> > > > >> > >>> > > > > > > >> > > > >>> For this reason it may be > > > > simpler > > > > >> to > > > > >> > > have > > > > >> > > > >> each > > > > >> > > > >> > >>> node > > > > >> > > > >> > >>> > be > > > > >> > > > >> > >>> > > > > aware > > > > >> > > > >> > >>> > > > > > > of > > > > >> > > > >> > >>> > > > > > > >> its > > > > >> > > > >> > >>> > > > > > > >> > > own > > > > >> > > > >> > >>> > > > > > > >> > > > >>> rack and persist it in ZK > > > during > > > > >> > start > > > > >> > > up > > > > >> > > > >> > time. > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > >> > > > >> > >>> > > > > > > >> > > > >>> - A pluggable RackLocator > > > relies > > > > >> on > > > > >> > an > > > > >> > > > >> > external > > > > >> > > > >> > >>> > > service > > > > >> > > > >> > >>> > > > > > being > > > > >> > > > >> > >>> > > > > > > >> > > available > > > > >> > > > >> > >>> > > > > > > >> > > > >>> to > > > > >> > > > >> > >>> > > > > > > >> > > > >>> serve rack information. > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > >> > > > >> > >>> > > > > > > >> > > > >>> Out of curiosity, I looked > > up > > > > how > > > > >> a > > > > >> > > > couple > > > > >> > > > >> of > > > > >> > > > >> > >>> other > > > > >> > > > >> > >>> > > > > systems > > > > >> > > > >> > >>> > > > > > > deal > > > > >> > > > >> > >>> > > > > > > >> > with > > > > >> > > > >> > >>> > > > > > > >> > > > >>> zone/rack awareness. > > > > >> > > > >> > >>> > > > > > > >> > > > >>> For Cassandra some > > interesting > > > > >> modes > > > > >> > > are: > > > > >> > > > >> > >>> > > > > > > >> > > > >>> (Property File > > configuration) > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > >> > > > >> > >>> > > > > > > >> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > > > > >> > > > >> > >>> > > > > > > >> > > > > > >> > > > >> > >>> > > > > > > >> > > > > >> > > > >> > >>> > > > > > > > > > > >> > > > >> > >>> > > > > > > > > > >> > > > >> > >>> > > > > > > > > >> > > > >> > >>> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >> > >>> > > > > > >> > > > >> > >>> > > > > >> > > > >> > > > > > >> > > > >> > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > http://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architectureSnitchPFSnitch_t.html > > > > >> > > > >> > >>> > > > > > > >> > > > >>> (Dynamic inference) > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > >> > > > >> > >>> > > > > > > >> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > > > > >> > > > >> > >>> > > > > > > >> > > > > > >> > > > >> > >>> > > > > > > >> > > > > >> > > > >> > >>> > > > > > > > > > > >> > > > >> > >>> > > > > > > > > > >> > > > >> > >>> > > > > > > > > >> > > > >> > >>> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >> > >>> > > > > > >> > > > >> > >>> > > > > >> > > > >> > > > > > >> > > > >> > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > http://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architectureSnitchRackInf_c.html > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > >> > > > >> > >>> > > > > > > >> > > > >>> Voldemort does a static > node > > > -> > > > > >> zone > > > > >> > > > >> > assignment > > > > >> > > > >> > >>> > based > > > > >> > > > >> > >>> > > on > > > > >> > > > >> > >>> > > > > > > >> > > configuration. > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > >> > > > >> > >>> > > > > > > >> > > > >>> Aditya > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > >> > > > >> > >>> > > > > > > >> > > > >>> On Wed, Sep 30, 2015 at > > 10:05 > > > > AM, > > > > >> > Allen > > > > >> > > > >> Wang < > > > > >> > > > >> > >>> > > > > > > >> allenxw...@gmail.com > > > > >> > > > >> > >>> > > > > > > >> > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> wrote: > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > I would like to see if > we > > > can > > > > do > > > > >> > > both: > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > - Make RackLocator > > pluggable > > > > to > > > > >> > > > >> facilitate > > > > >> > > > >> > >>> > migration > > > > >> > > > >> > >>> > > > > with > > > > >> > > > >> > >>> > > > > > > >> > existing > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > broker-rack mapping > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > - Make rack an optional > > > > property > > > > >> > for > > > > >> > > > >> broker. > > > > >> > > > >> > >>> If > > > > >> > > > >> > >>> > rack > > > > >> > > > >> > >>> > > > is > > > > >> > > > >> > >>> > > > > > > >> available > > > > >> > > > >> > >>> > > > > > > >> > > > from > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > broker, treat it as > source > > > of > > > > >> > truth. > > > > >> > > > For > > > > >> > > > >> > users > > > > >> > > > >> > >>> > with > > > > >> > > > >> > >>> > > > > > existing > > > > >> > > > >> > >>> > > > > > > >> > > > >>> broker-rack > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > mapping somewhere else, > > they > > > > can > > > > >> > use > > > > >> > > > the > > > > >> > > > >> > >>> pluggable > > > > >> > > > >> > >>> > > way > > > > >> > > > >> > >>> > > > > or > > > > >> > > > >> > >>> > > > > > > they > > > > >> > > > >> > >>> > > > > > > >> > can > > > > >> > > > >> > >>> > > > > > > >> > > > >>> transfer > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > the mapping to the > broker > > > rack > > > > >> > > > property. > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > One thing I am not sure > is > > > > what > > > > >> > > happens > > > > >> > > > >> at > > > > >> > > > >> > >>> rolling > > > > >> > > > >> > >>> > > > > upgrade > > > > >> > > > >> > >>> > > > > > > >> when > > > > >> > > > >> > >>> > > > > > > >> > we > > > > >> > > > >> > >>> > > > > > > >> > > > have > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > rack as a broker > property. > > > For > > > > >> > > brokers > > > > >> > > > >> with > > > > >> > > > >> > >>> older > > > > >> > > > >> > >>> > > > > version > > > > >> > > > >> > >>> > > > > > of > > > > >> > > > >> > >>> > > > > > > >> > Kafka, > > > > >> > > > >> > >>> > > > > > > >> > > > >>> will it > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > cause problem for them? > If > > > so, > > > > >> is > > > > >> > > there > > > > >> > > > >> any > > > > >> > > > >> > >>> > > > workaround? > > > > >> > > > >> > >>> > > > > I > > > > >> > > > >> > >>> > > > > > > also > > > > >> > > > >> > >>> > > > > > > >> > > think > > > > >> > > > >> > >>> > > > > > > >> > > > it > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > would be better not to > > have > > > > >> rack in > > > > >> > > the > > > > >> > > > >> > >>> controller > > > > >> > > > >> > >>> > > > wire > > > > >> > > > >> > >>> > > > > > > >> protocol > > > > >> > > > >> > >>> > > > > > > >> > > but > > > > >> > > > >> > >>> > > > > > > >> > > > >>> not > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > sure if it is > achievable. > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > Thanks, > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > Allen > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > On Mon, Sep 28, 2015 at > > 4:55 > > > > PM, > > > > >> > Todd > > > > >> > > > >> > Palino < > > > > >> > > > >> > >>> > > > > > > >> tpal...@gmail.com> > > > > >> > > > >> > >>> > > > > > > >> > > > >>> wrote: > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > I tend to like the > idea > > > of a > > > > >> > > > pluggable > > > > >> > > > >> > >>> locator. > > > > >> > > > >> > >>> > > For > > > > >> > > > >> > >>> > > > > > > >> example, we > > > > >> > > > >> > >>> > > > > > > >> > > > >>> already > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > have an interface for > > > > >> discovering > > > > >> > > > >> > >>> information > > > > >> > > > >> > >>> > > about > > > > >> > > > >> > >>> > > > > the > > > > >> > > > >> > >>> > > > > > > >> > physical > > > > >> > > > >> > >>> > > > > > > >> > > > >>> location > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > of servers. I don't > > relish > > > > the > > > > >> > idea > > > > >> > > > of > > > > >> > > > >> > >>> having to > > > > >> > > > >> > >>> > > > > > maintain > > > > >> > > > >> > >>> > > > > > > >> data > > > > >> > > > >> > >>> > > > > > > >> > in > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > multiple > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > places. > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > -Todd > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > On Mon, Sep 28, 2015 > at > > > 4:48 > > > > >> PM, > > > > >> > > > Aditya > > > > >> > > > >> > >>> > Auradkar < > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > aaurad...@linkedin.com.invalid > > > > >> > > > > > >> > > > wrote: > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > Thanks for starting > > this > > > > KIP > > > > >> > > Allen. > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > I agree with Gwen > that > > > > >> having a > > > > >> > > > >> > >>> RackLocator > > > > >> > > > >> > >>> > > class > > > > >> > > > >> > >>> > > > > that > > > > >> > > > >> > >>> > > > > > > is > > > > >> > > > >> > >>> > > > > > > >> > > > pluggable > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > seems > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > to be too complex. > The > > > KIP > > > > >> > refers > > > > >> > > > to > > > > >> > > > >> > >>> > potentially > > > > >> > > > >> > >>> > > > > > non-ZK > > > > >> > > > >> > >>> > > > > > > >> > storage > > > > >> > > > >> > >>> > > > > > > >> > > > >>> for the > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > rack info which I > > don't > > > > >> think > > > > >> > is > > > > >> > > > >> > >>> necessary. > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > Perhaps we can > persist > > > > this > > > > >> > info > > > > >> > > in > > > > >> > > > >> zk > > > > >> > > > >> > >>> under > > > > >> > > > >> > >>> > > > > > > >> > > > >>> /brokers/ids/<broker_id> > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > similar to other > > broker > > > > >> > > properties > > > > >> > > > >> and > > > > >> > > > >> > >>> add a > > > > >> > > > >> > >>> > > > config > > > > >> > > > >> > >>> > > > > in > > > > >> > > > >> > >>> > > > > > > >> > > > KafkaConfig > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > called > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > "rack". > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > >> > > > >> > >>> > > > > > > > > > > >> > > > >> {"jmx_port":-1,"endpoints":[...],"host":"xxx","port":yyy, > > > > >> > > > >> > >>> > > > > > > >> > > "rack": > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > "abc"} > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > Aditya > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > On Mon, Sep 28, 2015 > > at > > > > 2:30 > > > > >> > PM, > > > > >> > > > Gwen > > > > >> > > > >> > >>> Shapira > > > > >> > > > >> > >>> > < > > > > >> > > > >> > >>> > > > > > > >> > > g...@confluent.io > > > > >> > > > >> > >>> > > > > > > >> > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > wrote: > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > Hi, > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > First, thanks for > > > > putting > > > > >> > out a > > > > >> > > > KIP > > > > >> > > > >> > for > > > > >> > > > >> > >>> > this. > > > > >> > > > >> > >>> > > > This > > > > >> > > > >> > >>> > > > > > is > > > > >> > > > >> > >>> > > > > > > >> super > > > > >> > > > >> > >>> > > > > > > >> > > > >>> important > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > for > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > production > > deployments > > > > of > > > > >> > > Kafka. > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > Few questions: > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > 1) Are we sure we > > want > > > > "as > > > > >> > many > > > > >> > > > >> racks > > > > >> > > > >> > as > > > > >> > > > >> > >>> > > > > possible"? > > > > >> > > > >> > >>> > > > > > > I'd > > > > >> > > > >> > >>> > > > > > > >> > want > > > > >> > > > >> > >>> > > > > > > >> > > to > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > balance > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > between safety > (more > > > > >> racks) > > > > >> > and > > > > >> > > > >> > network > > > > >> > > > >> > >>> > > > > utilization > > > > >> > > > >> > >>> > > > > > > >> > (traffic > > > > >> > > > >> > >>> > > > > > > >> > > > >>> within a > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > rack > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > uses the > > > high-bandwidth > > > > >> TOR > > > > >> > > > >> switch). > > > > >> > > > >> > One > > > > >> > > > >> > >>> > > replica > > > > >> > > > >> > >>> > > > > on > > > > >> > > > >> > >>> > > > > > a > > > > >> > > > >> > >>> > > > > > > >> > > different > > > > >> > > > >> > >>> > > > > > > >> > > > >>> rack > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > and > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > the rest on same > > rack > > > > (if > > > > >> > > > possible) > > > > >> > > > >> > >>> sounds > > > > >> > > > >> > >>> > > > better > > > > >> > > > >> > >>> > > > > to > > > > >> > > > >> > >>> > > > > > > me. > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > 2) Rack-locator > > class > > > > >> seems > > > > >> > > > overly > > > > >> > > > >> > >>> complex > > > > >> > > > >> > >>> > > > > compared > > > > >> > > > >> > >>> > > > > > to > > > > >> > > > >> > >>> > > > > > > >> > > adding a > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > rack.number > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > property to the > > broker > > > > >> > > properties > > > > >> > > > >> > file. > > > > >> > > > >> > >>> Why > > > > >> > > > >> > >>> > do > > > > >> > > > >> > >>> > > > we > > > > >> > > > >> > >>> > > > > > want > > > > >> > > > >> > >>> > > > > > > >> > that? > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > Gwen > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > On Mon, Sep 28, > 2015 > > > at > > > > >> 12:15 > > > > >> > > PM, > > > > >> > > > >> > Allen > > > > >> > > > >> > >>> > Wang < > > > > >> > > > >> > >>> > > > > > > >> > > > >>> allenxw...@gmail.com> > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > wrote: > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > Hello Kafka > > > > Developers, > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > I just created > > > KIP-36 > > > > >> for > > > > >> > > rack > > > > >> > > > >> aware > > > > >> > > > >> > >>> > replica > > > > >> > > > >> > >>> > > > > > > >> assignment. > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > >> > > > >> > >>> > > > > > > >> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > > > > >> > > > >> > >>> > > > > > > >> > > > > > >> > > > >> > >>> > > > > > > >> > > > > >> > > > >> > >>> > > > > > > > > > > >> > > > >> > >>> > > > > > > > > > >> > > > >> > >>> > > > > > > > > >> > > > >> > >>> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >> > >>> > > > > > >> > > > >> > >>> > > > > >> > > > >> > > > > > >> > > > >> > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > The goal is to > > > utilize > > > > >> the > > > > >> > > > >> isolation > > > > >> > > > >> > >>> > > provided > > > > >> > > > >> > >>> > > > by > > > > >> > > > >> > >>> > > > > > the > > > > >> > > > >> > >>> > > > > > > >> > racks > > > > >> > > > >> > >>> > > > > > > >> > > in > > > > >> > > > >> > >>> > > > > > > >> > > > >>> data > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > center > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > and distribute > > > > replicas > > > > >> to > > > > >> > > > racks > > > > >> > > > >> to > > > > >> > > > >> > >>> > provide > > > > >> > > > >> > >>> > > > > fault > > > > >> > > > >> > >>> > > > > > > >> > > tolerance. > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > Comments are > > > welcome. > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > Thanks, > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > Allen > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > > >> > > > >> > >>> > > > > > > >> > > > >>> > > > > >> > > > >> > >>> > > > > > > >> > > > >> > > > > >> > > > >> > >>> > > > > > > >> > > > >> > > > > >> > > > >> > >>> > > > > > > >> > > > > > > > > >> > > > >> > >>> > > > > > > >> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > > > > >> > > > >> > >>> > > > > > > >> > > > > > >> > > > >> > >>> > > > > > > >> > > > > >> > > > >> > >>> > > > > > > > > > > > >> > > > >> > >>> > > > > > > > > > > > >> > > > >> > >>> > > > > > > > > > > >> > > > >> > >>> > > > > > > > > > >> > > > >> > >>> > > > > > > > > >> > > > >> > >>> > > > > > > > >> > > > >> > >>> > > > > > > >> > > > >> > >>> > > > > > >> > > > >> > >>> > > > > >> > > > >> > >>> > > > > >> > > > >> > >>> > > > > >> > > > >> > >>> -- > > > > >> > > > >> > >>> Thanks, > > > > >> > > > >> > >>> Neha > > > > >> > > > >> > >>> > > > > >> > > > >> > >> > > > > >> > > > >> > >> > > > > >> > > > >> > > > > > > >> > > > >> > > > > > >> > > > >> > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > >
