Mayuresh, The current plan is the following.
1. Add TMR v1, which still triggers auto topic creation. 2. Change the consumer client to TMR v1. Change the producer client to use TMR v1 and on UnknownTopicException, issue TopicCreateRequest to explicitly create the topic with the default server side partitions and replicas. 3. At some later time after the new clients are released and deployed, disable auto topic creation in TMR v1. This will make sure consumers never create new topics. 4. If needed, we can add a new config in the producer to control whether TopicCreateRequest should be issued or not on UnknownTopicException. If this is disabled and the topic doesn't exist, send will fail and the user is expected to create the topic manually. Thanks, Jun On Thu, May 21, 2015 at 5:27 PM, Mayuresh Gharat <gharatmayures...@gmail.com > wrote: > Hi, > I had a question about TopicMetadata Request. > Currently the way it works is : > > 1) Suppose a topic T1 does not exist. > 2) Client wants to produce data to T1 using producer P1. > 3) Since T1 does not exist, P1 issues a TopicMetadata request to kafka. > This in turn creates the default number of partition. The number of > partitions is a cluster wide config. > 4) Same goes for a consumer. If the topic does not exist and new topic will > be created when the consumer issues TopicMetadata request. > > Here are 2 use cases where it might not be suited : > > The auto creation flag for topics is turned ON. > > a) Some clients might not want to create topic with default number of > partitions but with lower number of partitions. Currently in a multi-tenant > environment this is not possible without changing the cluster wide default > config. > > b) Some clients might want to just check if the topic exist or not but > currently the topic gets created automatically using default number of > partitions. > > Here are some ideas to address this : > > 1) The way this can be addressed is that TopicMetadata request should have > a way to specify whether it should only check if the topic exist or check > and create a topic with given number of partitions. If the number of > partitions is not specified use the default cluster wide config. > > OR > > 2) We should only allow TopicMetadata Request to get the metadata > explicitly and not allow it to create a new topic. We should have another > Request that takes in config parameters from the user regarding how he/she > wants the topic to be created. This request can be used if we get an empty > TopicMetadata Response. > > > Thanks, > > Mayuresh > > > On Thu, May 14, 2015 at 10:22 AM, Jun Rao <j...@confluent.io> wrote: > > > For ListTopics, we decided not to add a ListTopics request for now and > just > > rely on passing in an empty list to TMR. We can revisit this in the > future > > if it becomes an issue. > > > > Thanks, > > > > Jun > > > > On Wed, May 13, 2015 at 3:31 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > > > Just had a few minor questions before I join the vote thread. > > > Apologies if these have been discussed: > > > > > > - Do we need DecreasePartitionsNotAllowed? i.e., can we just return > > > InvalidPartitions instead? > > > - AdminClient.listTopics: should we allow listing all partitions? Or > > > do you intend for the client to issue listTopics followed by > > > describeTopics? > > > - On returning future<void> for partition reassignments: do we need to > > > return any future especially since you have the > > > verifyReassignPartitions method? For e.g., what happens if the > > > controller moves? The get should fail right? The client will then > > > need to connect to the new controller and reissue the request but > > > will then get ReassignPartitionsInProgress. So in that case the > > > client any way needs to rely in verifyReassignPartitions. > > > - In past hangouts I think either you/Joe were mentioning the need to > > > locate the controller (and possibly other cluster metadata). It > > > appears we decided to defer this for a future KIP. Correct? > > > > > > Thanks, > > > > > > Joel > > > > > > On Tue, May 05, 2015 at 04:49:27PM +0300, Andrii Biletskyi wrote: > > > > Guys, > > > > > > > > I've updated the wiki to reflect all previously discussed items > > > > (regarding the schema only - this is included to phase 1). > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > > > > > > > I think we can have the final discussion today (for phase 1) and > > > > in case no new remarks I will start the voting thread. > > > > > > > > With regards to AlterTopicRequest semantics. I agree with Jun, > > > > and I think it's my bad I focused on "multiple topics in one > request". > > > > The same situation is possible in ProduceRequest, Fetch, > TopicMetadata > > > > and we handle it naturally and in the most transparent way - we > > > > put all separate instructions into a map and thus silently ignore > > > > duplicates. > > > > This also makes Response part simple too - it's just a map > > > Topic->ErrorCode. > > > > I think we need to follow the same approach for Alter (and Create, > > > Delete) > > > > request. With this we add nothing new in terms of batch requests > > > > semantics. > > > > > > > > Thanks, > > > > Andrii Biletskyi > > > > > > > > > -- > -Regards, > Mayuresh R. Gharat > (862) 250-7125 >
