Guys, I won't be able to attend next meeting. But in the latest patch for KIP-4 Phase 1 I didn't even evolve TopicMetadataRequest to v1 since we won't be able to change config with AlterTopicRequest, hence with this patch TMR will still return isr. Taking this into account I think yes - it would be good to fix ISR issue, although I didn't consider it to be a critical one (isr was part of TMR from the very beginning and almost no code relies on this piece of request).
Thanks, Andrii Biletskyi On Fri, May 29, 2015 at 8:50 AM, Aditya Auradkar < aaurad...@linkedin.com.invalid> wrote: > Thanks. Perhaps we should leave TMR unchanged for now. Should we discuss > this during the next hangout? > > Aditya > > ________________________________________ > From: Jun Rao [j...@confluent.io] > Sent: Thursday, May 28, 2015 5:32 PM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative > operations (Thread 2) > > There is a reasonable use case of ISR in KAFKA-2225. Basically, for > economical reasons, we may want to let a consumer fetch from a replica in > ISR that's in the same zone. In order to support that, it will be > convenient to have TMR return the correct ISR for the consumer to choose. > > So, perhaps it's worth fixing the ISR inconsistency issue in KAFKA-1367 > (there is some new discussion there on what it takes to fix this). If we do > that, we can leave TMR unchanged. > > Thanks, > > Jun > > On Tue, May 26, 2015 at 1:13 PM, Aditya Auradkar < > aaurad...@linkedin.com.invalid> wrote: > > > Andryii, > > > > I made a few edits to this document as discussed in the KIP-21 thread. > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > > > With these changes. the only difference between TopicMetadataResponse_V1 > > and V0 is the removal of the ISR field. I've altered the KIP with the > > assumption that this is a good enough reason by itself to evolve the > > request/response protocol. Any concerns there? > > > > Thanks, > > Aditya > > > > ________________________________________ > > From: Mayuresh Gharat [gharatmayures...@gmail.com] > > Sent: Thursday, May 21, 2015 8:29 PM > > To: dev@kafka.apache.org > > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized > administrative > > operations (Thread 2) > > > > Hi Jun, > > > > Thanks a lot. I get it now. > > Point 4) will actually enable clients to who don't want to create a > topic > > with default partitions, if it does not exist and then can manually > create > > the topic with their own configs(#partitions). > > > > Thanks, > > > > Mayuresh > > > > On Thu, May 21, 2015 at 6:16 PM, Jun Rao <j...@confluent.io> wrote: > > > > > Mayuresh, > > > > > > The current plan is the following. > > > > > > 1. Add TMR v1, which still triggers auto topic creation. > > > 2. Change the consumer client to TMR v1. Change the producer client to > > use > > > TMR v1 and on UnknownTopicException, issue TopicCreateRequest to > > explicitly > > > create the topic with the default server side partitions and replicas. > > > 3. At some later time after the new clients are released and deployed, > > > disable auto topic creation in TMR v1. This will make sure consumers > > never > > > create new topics. > > > 4. If needed, we can add a new config in the producer to control > whether > > > TopicCreateRequest should be issued or not on UnknownTopicException. If > > > this is disabled and the topic doesn't exist, send will fail and the > user > > > is expected to create the topic manually. > > > > > > Thanks, > > > > > > Jun > > > > > > > > > On Thu, May 21, 2015 at 5:27 PM, Mayuresh Gharat < > > > gharatmayures...@gmail.com > > > > wrote: > > > > > > > Hi, > > > > I had a question about TopicMetadata Request. > > > > Currently the way it works is : > > > > > > > > 1) Suppose a topic T1 does not exist. > > > > 2) Client wants to produce data to T1 using producer P1. > > > > 3) Since T1 does not exist, P1 issues a TopicMetadata request to > kafka. > > > > This in turn creates the default number of partition. The number of > > > > partitions is a cluster wide config. > > > > 4) Same goes for a consumer. If the topic does not exist and new > topic > > > will > > > > be created when the consumer issues TopicMetadata request. > > > > > > > > Here are 2 use cases where it might not be suited : > > > > > > > > The auto creation flag for topics is turned ON. > > > > > > > > a) Some clients might not want to create topic with default number of > > > > partitions but with lower number of partitions. Currently in a > > > multi-tenant > > > > environment this is not possible without changing the cluster wide > > > default > > > > config. > > > > > > > > b) Some clients might want to just check if the topic exist or not > but > > > > currently the topic gets created automatically using default number > of > > > > partitions. > > > > > > > > Here are some ideas to address this : > > > > > > > > 1) The way this can be addressed is that TopicMetadata request > should > > > have > > > > a way to specify whether it should only check if the topic exist or > > check > > > > and create a topic with given number of partitions. If the number of > > > > partitions is not specified use the default cluster wide config. > > > > > > > > OR > > > > > > > > 2) We should only allow TopicMetadata Request to get the metadata > > > > explicitly and not allow it to create a new topic. We should have > > another > > > > Request that takes in config parameters from the user regarding how > > > he/she > > > > wants the topic to be created. This request can be used if we get an > > > empty > > > > TopicMetadata Response. > > > > > > > > > > > > Thanks, > > > > > > > > Mayuresh > > > > > > > > > > > > On Thu, May 14, 2015 at 10:22 AM, Jun Rao <j...@confluent.io> wrote: > > > > > > > > > For ListTopics, we decided not to add a ListTopics request for now > > and > > > > just > > > > > rely on passing in an empty list to TMR. We can revisit this in the > > > > future > > > > > if it becomes an issue. > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > On Wed, May 13, 2015 at 3:31 PM, Joel Koshy <jjkosh...@gmail.com> > > > wrote: > > > > > > > > > > > Just had a few minor questions before I join the vote thread. > > > > > > Apologies if these have been discussed: > > > > > > > > > > > > - Do we need DecreasePartitionsNotAllowed? i.e., can we just > return > > > > > > InvalidPartitions instead? > > > > > > - AdminClient.listTopics: should we allow listing all partitions? > > Or > > > > > > do you intend for the client to issue listTopics followed by > > > > > > describeTopics? > > > > > > - On returning future<void> for partition reassignments: do we > need > > > to > > > > > > return any future especially since you have the > > > > > > verifyReassignPartitions method? For e.g., what happens if the > > > > > > controller moves? The get should fail right? The client will > then > > > > > > need to connect to the new controller and reissue the request > but > > > > > > will then get ReassignPartitionsInProgress. So in that case the > > > > > > client any way needs to rely in verifyReassignPartitions. > > > > > > - In past hangouts I think either you/Joe were mentioning the > need > > to > > > > > > locate the controller (and possibly other cluster metadata). It > > > > > > appears we decided to defer this for a future KIP. Correct? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Joel > > > > > > > > > > > > On Tue, May 05, 2015 at 04:49:27PM +0300, Andrii Biletskyi wrote: > > > > > > > Guys, > > > > > > > > > > > > > > I've updated the wiki to reflect all previously discussed items > > > > > > > (regarding the schema only - this is included to phase 1). > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > > > > > > > > > > > > > I think we can have the final discussion today (for phase 1) > and > > > > > > > in case no new remarks I will start the voting thread. > > > > > > > > > > > > > > With regards to AlterTopicRequest semantics. I agree with Jun, > > > > > > > and I think it's my bad I focused on "multiple topics in one > > > > request". > > > > > > > The same situation is possible in ProduceRequest, Fetch, > > > > TopicMetadata > > > > > > > and we handle it naturally and in the most transparent way - we > > > > > > > put all separate instructions into a map and thus silently > ignore > > > > > > > duplicates. > > > > > > > This also makes Response part simple too - it's just a map > > > > > > Topic->ErrorCode. > > > > > > > I think we need to follow the same approach for Alter (and > > Create, > > > > > > Delete) > > > > > > > request. With this we add nothing new in terms of batch > requests > > > > > > > semantics. > > > > > > > > > > > > > > Thanks, > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > -Regards, > > > > Mayuresh R. Gharat > > > > (862) 250-7125 > > > > > > > > > > > > > > > -- > > -Regards, > > Mayuresh R. Gharat > > (862) 250-7125 > > >
