Joel, Aditya, I believe we don't need another thread to do voting since affected items are not related to the core proposed changes. I agree people can explicitly down-vote in case of concerns about the things that changed.
Thanks, Andrii Biletskyi On Fri, Jun 12, 2015 at 8:24 PM, Aditya Auradkar < aaurad...@linkedin.com.invalid> wrote: > Aside from the things I mentioned, I don't think there were other changes. > I'll mark this as adopted since there don't appear to be any concerns. > > Aditya > > ________________________________________ > From: Joel Koshy [jjkosh...@gmail.com] > Sent: Thursday, June 11, 2015 1:28 PM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative > operations (Thread 2) > > Discussion aside, was there any significant material change besides > the additions below? If so, then we can avoid the overhead of another > vote unless someone wants to down-vote these changes. > > Joel > > On Thu, Jun 11, 2015 at 06:36:36PM +0000, Aditya Auradkar wrote: > > Andrii, > > > > Do we need a new voting thread for this KIP? The last round of votes had > 3 binding +1's but there's been a fair amount of discussion since then. > > > > Aditya > > > > ________________________________________ > > From: Aditya Auradkar > > Sent: Thursday, June 11, 2015 10:32 AM > > To: dev@kafka.apache.org > > Subject: RE: [DISCUSS] KIP-4 - Command line and centralized > administrative operations (Thread 2) > > > > I've made two changes to the document: > > - Removed the TMR evolution piece since we agreed to retain this. > > - Added two new API's to the admin client spec. (Alter and Describe > config). > > > > Please review. > > > > Aditya > > > > ________________________________________ > > From: Ashish Singh [asi...@cloudera.com] > > Sent: Friday, May 29, 2015 8:36 AM > > To: dev@kafka.apache.org > > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized > administrative operations (Thread 2) > > > > +1 on discussing this on next KIP hangout. I will update KIP-24 before > that. > > > > On Fri, May 29, 2015 at 3:40 AM, Andrii Biletskyi < > > andrii.bilets...@stealth.ly> wrote: > > > > > Guys, > > > > > > I won't be able to attend next meeting. But in the latest patch for > KIP-4 > > > Phase 1 > > > I didn't even evolve TopicMetadataRequest to v1 since we won't be able > > > to change config with AlterTopicRequest, hence with this patch TMR will > > > still > > > return isr. Taking this into account I think yes - it would be good to > fix > > > ISR issue, > > > although I didn't consider it to be a critical one (isr was part of TMR > > > from the very > > > beginning and almost no code relies on this piece of request). > > > > > > Thanks, > > > Andrii Biletskyi > > > > > > On Fri, May 29, 2015 at 8:50 AM, Aditya Auradkar < > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > Thanks. Perhaps we should leave TMR unchanged for now. Should we > discuss > > > > this during the next hangout? > > > > > > > > Aditya > > > > > > > > ________________________________________ > > > > From: Jun Rao [j...@confluent.io] > > > > Sent: Thursday, May 28, 2015 5:32 PM > > > > To: dev@kafka.apache.org > > > > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized > > > administrative > > > > operations (Thread 2) > > > > > > > > There is a reasonable use case of ISR in KAFKA-2225. Basically, for > > > > economical reasons, we may want to let a consumer fetch from a > replica in > > > > ISR that's in the same zone. In order to support that, it will be > > > > convenient to have TMR return the correct ISR for the consumer to > choose. > > > > > > > > So, perhaps it's worth fixing the ISR inconsistency issue in > KAFKA-1367 > > > > (there is some new discussion there on what it takes to fix this). > If we > > > do > > > > that, we can leave TMR unchanged. > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > On Tue, May 26, 2015 at 1:13 PM, Aditya Auradkar < > > > > aaurad...@linkedin.com.invalid> wrote: > > > > > > > > > Andryii, > > > > > > > > > > I made a few edits to this document as discussed in the KIP-21 > thread. > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > > > > > > > > > With these changes. the only difference between > > > TopicMetadataResponse_V1 > > > > > and V0 is the removal of the ISR field. I've altered the KIP with > the > > > > > assumption that this is a good enough reason by itself to evolve > the > > > > > request/response protocol. Any concerns there? > > > > > > > > > > Thanks, > > > > > Aditya > > > > > > > > > > ________________________________________ > > > > > From: Mayuresh Gharat [gharatmayures...@gmail.com] > > > > > Sent: Thursday, May 21, 2015 8:29 PM > > > > > To: dev@kafka.apache.org > > > > > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized > > > > administrative > > > > > operations (Thread 2) > > > > > > > > > > Hi Jun, > > > > > > > > > > Thanks a lot. I get it now. > > > > > Point 4) will actually enable clients to who don't want to create > a > > > > topic > > > > > with default partitions, if it does not exist and then can manually > > > > create > > > > > the topic with their own configs(#partitions). > > > > > > > > > > Thanks, > > > > > > > > > > Mayuresh > > > > > > > > > > On Thu, May 21, 2015 at 6:16 PM, Jun Rao <j...@confluent.io> wrote: > > > > > > > > > > > Mayuresh, > > > > > > > > > > > > The current plan is the following. > > > > > > > > > > > > 1. Add TMR v1, which still triggers auto topic creation. > > > > > > 2. Change the consumer client to TMR v1. Change the producer > client > > > to > > > > > use > > > > > > TMR v1 and on UnknownTopicException, issue TopicCreateRequest to > > > > > explicitly > > > > > > create the topic with the default server side partitions and > > > replicas. > > > > > > 3. At some later time after the new clients are released and > > > deployed, > > > > > > disable auto topic creation in TMR v1. This will make sure > consumers > > > > > never > > > > > > create new topics. > > > > > > 4. If needed, we can add a new config in the producer to control > > > > whether > > > > > > TopicCreateRequest should be issued or not on > UnknownTopicException. > > > If > > > > > > this is disabled and the topic doesn't exist, send will fail and > the > > > > user > > > > > > is expected to create the topic manually. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > On Thu, May 21, 2015 at 5:27 PM, Mayuresh Gharat < > > > > > > gharatmayures...@gmail.com > > > > > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > I had a question about TopicMetadata Request. > > > > > > > Currently the way it works is : > > > > > > > > > > > > > > 1) Suppose a topic T1 does not exist. > > > > > > > 2) Client wants to produce data to T1 using producer P1. > > > > > > > 3) Since T1 does not exist, P1 issues a TopicMetadata request > to > > > > kafka. > > > > > > > This in turn creates the default number of partition. The > number of > > > > > > > partitions is a cluster wide config. > > > > > > > 4) Same goes for a consumer. If the topic does not exist and > new > > > > topic > > > > > > will > > > > > > > be created when the consumer issues TopicMetadata request. > > > > > > > > > > > > > > Here are 2 use cases where it might not be suited : > > > > > > > > > > > > > > The auto creation flag for topics is turned ON. > > > > > > > > > > > > > > a) Some clients might not want to create topic with default > number > > > of > > > > > > > partitions but with lower number of partitions. Currently in a > > > > > > multi-tenant > > > > > > > environment this is not possible without changing the cluster > wide > > > > > > default > > > > > > > config. > > > > > > > > > > > > > > b) Some clients might want to just check if the topic exist or > not > > > > but > > > > > > > currently the topic gets created automatically using default > number > > > > of > > > > > > > partitions. > > > > > > > > > > > > > > Here are some ideas to address this : > > > > > > > > > > > > > > 1) The way this can be addressed is that TopicMetadata request > > > > should > > > > > > have > > > > > > > a way to specify whether it should only check if the topic > exist or > > > > > check > > > > > > > and create a topic with given number of partitions. If the > number > > > of > > > > > > > partitions is not specified use the default cluster wide > config. > > > > > > > > > > > > > > OR > > > > > > > > > > > > > > 2) We should only allow TopicMetadata Request to get the > metadata > > > > > > > explicitly and not allow it to create a new topic. We should > have > > > > > another > > > > > > > Request that takes in config parameters from the user > regarding how > > > > > > he/she > > > > > > > wants the topic to be created. This request can be used if we > get > > > an > > > > > > empty > > > > > > > TopicMetadata Response. > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Mayuresh > > > > > > > > > > > > > > > > > > > > > On Thu, May 14, 2015 at 10:22 AM, Jun Rao <j...@confluent.io> > > > wrote: > > > > > > > > > > > > > > > For ListTopics, we decided not to add a ListTopics request > for > > > now > > > > > and > > > > > > > just > > > > > > > > rely on passing in an empty list to TMR. We can revisit this > in > > > the > > > > > > > future > > > > > > > > if it becomes an issue. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > On Wed, May 13, 2015 at 3:31 PM, Joel Koshy < > jjkosh...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Just had a few minor questions before I join the vote > thread. > > > > > > > > > Apologies if these have been discussed: > > > > > > > > > > > > > > > > > > - Do we need DecreasePartitionsNotAllowed? i.e., can we > just > > > > return > > > > > > > > > InvalidPartitions instead? > > > > > > > > > - AdminClient.listTopics: should we allow listing all > > > partitions? > > > > > Or > > > > > > > > > do you intend for the client to issue listTopics > followed by > > > > > > > > > describeTopics? > > > > > > > > > - On returning future<void> for partition reassignments: > do we > > > > need > > > > > > to > > > > > > > > > return any future especially since you have the > > > > > > > > > verifyReassignPartitions method? For e.g., what happens > if > > > the > > > > > > > > > controller moves? The get should fail right? The client > will > > > > then > > > > > > > > > need to connect to the new controller and reissue the > request > > > > but > > > > > > > > > will then get ReassignPartitionsInProgress. So in that > case > > > the > > > > > > > > > client any way needs to rely in verifyReassignPartitions. > > > > > > > > > - In past hangouts I think either you/Joe were mentioning > the > > > > need > > > > > to > > > > > > > > > locate the controller (and possibly other cluster > metadata). > > > It > > > > > > > > > appears we decided to defer this for a future KIP. > Correct? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Joel > > > > > > > > > > > > > > > > > > On Tue, May 05, 2015 at 04:49:27PM +0300, Andrii Biletskyi > > > wrote: > > > > > > > > > > Guys, > > > > > > > > > > > > > > > > > > > > I've updated the wiki to reflect all previously discussed > > > items > > > > > > > > > > (regarding the schema only - this is included to phase > 1). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > > > > > > > > > > > > > > > > > > > I think we can have the final discussion today (for > phase 1) > > > > and > > > > > > > > > > in case no new remarks I will start the voting thread. > > > > > > > > > > > > > > > > > > > > With regards to AlterTopicRequest semantics. I agree with > > > Jun, > > > > > > > > > > and I think it's my bad I focused on "multiple topics in > one > > > > > > > request". > > > > > > > > > > The same situation is possible in ProduceRequest, Fetch, > > > > > > > TopicMetadata > > > > > > > > > > and we handle it naturally and in the most transparent > way - > > > we > > > > > > > > > > put all separate instructions into a map and thus > silently > > > > ignore > > > > > > > > > > duplicates. > > > > > > > > > > This also makes Response part simple too - it's just a > map > > > > > > > > > Topic->ErrorCode. > > > > > > > > > > I think we need to follow the same approach for Alter > (and > > > > > Create, > > > > > > > > > Delete) > > > > > > > > > > request. With this we add nothing new in terms of batch > > > > requests > > > > > > > > > > semantics. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > -Regards, > > > > > > > Mayuresh R. Gharat > > > > > > > (862) 250-7125 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > -Regards, > > > > > Mayuresh R. Gharat > > > > > (862) 250-7125 > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, > > Ashish > >
