Hi Jun, Thanks a lot. I get it now. Point 4) will actually enable clients to who don't want to create a topic with default partitions, if it does not exist and then can manually create the topic with their own configs(#partitions).
Thanks, Mayuresh On Thu, May 21, 2015 at 6:16 PM, Jun Rao <j...@confluent.io> wrote: > Mayuresh, > > The current plan is the following. > > 1. Add TMR v1, which still triggers auto topic creation. > 2. Change the consumer client to TMR v1. Change the producer client to use > TMR v1 and on UnknownTopicException, issue TopicCreateRequest to explicitly > create the topic with the default server side partitions and replicas. > 3. At some later time after the new clients are released and deployed, > disable auto topic creation in TMR v1. This will make sure consumers never > create new topics. > 4. If needed, we can add a new config in the producer to control whether > TopicCreateRequest should be issued or not on UnknownTopicException. If > this is disabled and the topic doesn't exist, send will fail and the user > is expected to create the topic manually. > > Thanks, > > Jun > > > On Thu, May 21, 2015 at 5:27 PM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > Hi, > > I had a question about TopicMetadata Request. > > Currently the way it works is : > > > > 1) Suppose a topic T1 does not exist. > > 2) Client wants to produce data to T1 using producer P1. > > 3) Since T1 does not exist, P1 issues a TopicMetadata request to kafka. > > This in turn creates the default number of partition. The number of > > partitions is a cluster wide config. > > 4) Same goes for a consumer. If the topic does not exist and new topic > will > > be created when the consumer issues TopicMetadata request. > > > > Here are 2 use cases where it might not be suited : > > > > The auto creation flag for topics is turned ON. > > > > a) Some clients might not want to create topic with default number of > > partitions but with lower number of partitions. Currently in a > multi-tenant > > environment this is not possible without changing the cluster wide > default > > config. > > > > b) Some clients might want to just check if the topic exist or not but > > currently the topic gets created automatically using default number of > > partitions. > > > > Here are some ideas to address this : > > > > 1) The way this can be addressed is that TopicMetadata request should > have > > a way to specify whether it should only check if the topic exist or check > > and create a topic with given number of partitions. If the number of > > partitions is not specified use the default cluster wide config. > > > > OR > > > > 2) We should only allow TopicMetadata Request to get the metadata > > explicitly and not allow it to create a new topic. We should have another > > Request that takes in config parameters from the user regarding how > he/she > > wants the topic to be created. This request can be used if we get an > empty > > TopicMetadata Response. > > > > > > Thanks, > > > > Mayuresh > > > > > > On Thu, May 14, 2015 at 10:22 AM, Jun Rao <j...@confluent.io> wrote: > > > > > For ListTopics, we decided not to add a ListTopics request for now and > > just > > > rely on passing in an empty list to TMR. We can revisit this in the > > future > > > if it becomes an issue. > > > > > > Thanks, > > > > > > Jun > > > > > > On Wed, May 13, 2015 at 3:31 PM, Joel Koshy <jjkosh...@gmail.com> > wrote: > > > > > > > Just had a few minor questions before I join the vote thread. > > > > Apologies if these have been discussed: > > > > > > > > - Do we need DecreasePartitionsNotAllowed? i.e., can we just return > > > > InvalidPartitions instead? > > > > - AdminClient.listTopics: should we allow listing all partitions? Or > > > > do you intend for the client to issue listTopics followed by > > > > describeTopics? > > > > - On returning future<void> for partition reassignments: do we need > to > > > > return any future especially since you have the > > > > verifyReassignPartitions method? For e.g., what happens if the > > > > controller moves? The get should fail right? The client will then > > > > need to connect to the new controller and reissue the request but > > > > will then get ReassignPartitionsInProgress. So in that case the > > > > client any way needs to rely in verifyReassignPartitions. > > > > - In past hangouts I think either you/Joe were mentioning the need to > > > > locate the controller (and possibly other cluster metadata). It > > > > appears we decided to defer this for a future KIP. Correct? > > > > > > > > Thanks, > > > > > > > > Joel > > > > > > > > On Tue, May 05, 2015 at 04:49:27PM +0300, Andrii Biletskyi wrote: > > > > > Guys, > > > > > > > > > > I've updated the wiki to reflect all previously discussed items > > > > > (regarding the schema only - this is included to phase 1). > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > > > > > > > > > I think we can have the final discussion today (for phase 1) and > > > > > in case no new remarks I will start the voting thread. > > > > > > > > > > With regards to AlterTopicRequest semantics. I agree with Jun, > > > > > and I think it's my bad I focused on "multiple topics in one > > request". > > > > > The same situation is possible in ProduceRequest, Fetch, > > TopicMetadata > > > > > and we handle it naturally and in the most transparent way - we > > > > > put all separate instructions into a map and thus silently ignore > > > > > duplicates. > > > > > This also makes Response part simple too - it's just a map > > > > Topic->ErrorCode. > > > > > I think we need to follow the same approach for Alter (and Create, > > > > Delete) > > > > > request. With this we add nothing new in terms of batch requests > > > > > semantics. > > > > > > > > > > Thanks, > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > -- > > -Regards, > > Mayuresh R. Gharat > > (862) 250-7125 > > > -- -Regards, Mayuresh R. Gharat (862) 250-7125
