Thanks. Perhaps we should leave TMR unchanged for now. Should we discuss this during the next hangout?
Aditya ________________________________________ From: Jun Rao [j...@confluent.io] Sent: Thursday, May 28, 2015 5:32 PM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2) There is a reasonable use case of ISR in KAFKA-2225. Basically, for economical reasons, we may want to let a consumer fetch from a replica in ISR that's in the same zone. In order to support that, it will be convenient to have TMR return the correct ISR for the consumer to choose. So, perhaps it's worth fixing the ISR inconsistency issue in KAFKA-1367 (there is some new discussion there on what it takes to fix this). If we do that, we can leave TMR unchanged. Thanks, Jun On Tue, May 26, 2015 at 1:13 PM, Aditya Auradkar < aaurad...@linkedin.com.invalid> wrote: > Andryii, > > I made a few edits to this document as discussed in the KIP-21 thread. > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > With these changes. the only difference between TopicMetadataResponse_V1 > and V0 is the removal of the ISR field. I've altered the KIP with the > assumption that this is a good enough reason by itself to evolve the > request/response protocol. Any concerns there? > > Thanks, > Aditya > > ________________________________________ > From: Mayuresh Gharat [gharatmayures...@gmail.com] > Sent: Thursday, May 21, 2015 8:29 PM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative > operations (Thread 2) > > Hi Jun, > > Thanks a lot. I get it now. > Point 4) will actually enable clients to who don't want to create a topic > with default partitions, if it does not exist and then can manually create > the topic with their own configs(#partitions). > > Thanks, > > Mayuresh > > On Thu, May 21, 2015 at 6:16 PM, Jun Rao <j...@confluent.io> wrote: > > > Mayuresh, > > > > The current plan is the following. > > > > 1. Add TMR v1, which still triggers auto topic creation. > > 2. Change the consumer client to TMR v1. Change the producer client to > use > > TMR v1 and on UnknownTopicException, issue TopicCreateRequest to > explicitly > > create the topic with the default server side partitions and replicas. > > 3. At some later time after the new clients are released and deployed, > > disable auto topic creation in TMR v1. This will make sure consumers > never > > create new topics. > > 4. If needed, we can add a new config in the producer to control whether > > TopicCreateRequest should be issued or not on UnknownTopicException. If > > this is disabled and the topic doesn't exist, send will fail and the user > > is expected to create the topic manually. > > > > Thanks, > > > > Jun > > > > > > On Thu, May 21, 2015 at 5:27 PM, Mayuresh Gharat < > > gharatmayures...@gmail.com > > > wrote: > > > > > Hi, > > > I had a question about TopicMetadata Request. > > > Currently the way it works is : > > > > > > 1) Suppose a topic T1 does not exist. > > > 2) Client wants to produce data to T1 using producer P1. > > > 3) Since T1 does not exist, P1 issues a TopicMetadata request to kafka. > > > This in turn creates the default number of partition. The number of > > > partitions is a cluster wide config. > > > 4) Same goes for a consumer. If the topic does not exist and new topic > > will > > > be created when the consumer issues TopicMetadata request. > > > > > > Here are 2 use cases where it might not be suited : > > > > > > The auto creation flag for topics is turned ON. > > > > > > a) Some clients might not want to create topic with default number of > > > partitions but with lower number of partitions. Currently in a > > multi-tenant > > > environment this is not possible without changing the cluster wide > > default > > > config. > > > > > > b) Some clients might want to just check if the topic exist or not but > > > currently the topic gets created automatically using default number of > > > partitions. > > > > > > Here are some ideas to address this : > > > > > > 1) The way this can be addressed is that TopicMetadata request should > > have > > > a way to specify whether it should only check if the topic exist or > check > > > and create a topic with given number of partitions. If the number of > > > partitions is not specified use the default cluster wide config. > > > > > > OR > > > > > > 2) We should only allow TopicMetadata Request to get the metadata > > > explicitly and not allow it to create a new topic. We should have > another > > > Request that takes in config parameters from the user regarding how > > he/she > > > wants the topic to be created. This request can be used if we get an > > empty > > > TopicMetadata Response. > > > > > > > > > Thanks, > > > > > > Mayuresh > > > > > > > > > On Thu, May 14, 2015 at 10:22 AM, Jun Rao <j...@confluent.io> wrote: > > > > > > > For ListTopics, we decided not to add a ListTopics request for now > and > > > just > > > > rely on passing in an empty list to TMR. We can revisit this in the > > > future > > > > if it becomes an issue. > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > On Wed, May 13, 2015 at 3:31 PM, Joel Koshy <jjkosh...@gmail.com> > > wrote: > > > > > > > > > Just had a few minor questions before I join the vote thread. > > > > > Apologies if these have been discussed: > > > > > > > > > > - Do we need DecreasePartitionsNotAllowed? i.e., can we just return > > > > > InvalidPartitions instead? > > > > > - AdminClient.listTopics: should we allow listing all partitions? > Or > > > > > do you intend for the client to issue listTopics followed by > > > > > describeTopics? > > > > > - On returning future<void> for partition reassignments: do we need > > to > > > > > return any future especially since you have the > > > > > verifyReassignPartitions method? For e.g., what happens if the > > > > > controller moves? The get should fail right? The client will then > > > > > need to connect to the new controller and reissue the request but > > > > > will then get ReassignPartitionsInProgress. So in that case the > > > > > client any way needs to rely in verifyReassignPartitions. > > > > > - In past hangouts I think either you/Joe were mentioning the need > to > > > > > locate the controller (and possibly other cluster metadata). It > > > > > appears we decided to defer this for a future KIP. Correct? > > > > > > > > > > Thanks, > > > > > > > > > > Joel > > > > > > > > > > On Tue, May 05, 2015 at 04:49:27PM +0300, Andrii Biletskyi wrote: > > > > > > Guys, > > > > > > > > > > > > I've updated the wiki to reflect all previously discussed items > > > > > > (regarding the schema only - this is included to phase 1). > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > > > > > > > > > > > I think we can have the final discussion today (for phase 1) and > > > > > > in case no new remarks I will start the voting thread. > > > > > > > > > > > > With regards to AlterTopicRequest semantics. I agree with Jun, > > > > > > and I think it's my bad I focused on "multiple topics in one > > > request". > > > > > > The same situation is possible in ProduceRequest, Fetch, > > > TopicMetadata > > > > > > and we handle it naturally and in the most transparent way - we > > > > > > put all separate instructions into a map and thus silently ignore > > > > > > duplicates. > > > > > > This also makes Response part simple too - it's just a map > > > > > Topic->ErrorCode. > > > > > > I think we need to follow the same approach for Alter (and > Create, > > > > > Delete) > > > > > > request. With this we add nothing new in terms of batch requests > > > > > > semantics. > > > > > > > > > > > > Thanks, > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > > > > -- > > > -Regards, > > > Mayuresh R. Gharat > > > (862) 250-7125 > > > > > > > > > -- > -Regards, > Mayuresh R. Gharat > (862) 250-7125 >
