Thanks to all who voted. The KIP-4 Create Topics changes passed with +4 (binding), and +4 (non-binding).
There is a patch available for review here: https://github.com/apache/kafka/pull/1489 On Tue, Jun 21, 2016 at 1:11 AM, Manikumar Reddy <manikumar.re...@gmail.com> wrote: > +1 (non-binding) > > On Tue, Jun 21, 2016 at 9:16 AM, Ewen Cheslack-Postava <e...@confluent.io> > wrote: > > > +1 (binding) and thanks for the work on this Grant! > > > > -Ewen > > > > On Mon, Jun 20, 2016 at 12:18 PM, Gwen Shapira <g...@confluent.io> > wrote: > > > > > +1 (binding) > > > > > > On Mon, Jun 20, 2016 at 12:13 PM, Tom Crayford <tcrayf...@heroku.com> > > > wrote: > > > > +1 (non-binding) > > > > > > > > On Mon, Jun 20, 2016 at 8:07 PM, Harsha <ka...@harsha.io> wrote: > > > > > > > >> +1 (binding) > > > >> -Harsha > > > >> > > > >> On Mon, Jun 20, 2016, at 11:33 AM, Ismael Juma wrote: > > > >> > +1 (binding) > > > >> > > > > >> > On Mon, Jun 20, 2016 at 8:27 PM, Dana Powers < > dana.pow...@gmail.com > > > > > > >> > wrote: > > > >> > > > > >> > > +1 -- thanks for the update > > > >> > > > > > >> > > On Mon, Jun 20, 2016 at 10:49 AM, Grant Henke < > > ghe...@cloudera.com> > > > >> wrote: > > > >> > > > I have update the patch and wiki based on the feedback in the > > > >> discussion > > > >> > > > thread. The only change is that instead of logging and > > > disconnecting > > > >> in > > > >> > > the > > > >> > > > case of invalid messages (duplicate topics or both arguments) > we > > > now > > > >> > > return > > > >> > > > and InvalidRequest error back to the client for that topic. > > > >> > > > > > > >> > > > I would like to restart the vote now including that change. If > > you > > > >> have > > > >> > > > already voted, please revote in this thread. > > > >> > > > > > > >> > > > Thank you, > > > >> > > > Grant > > > >> > > > > > > >> > > > On Sun, Jun 19, 2016 at 8:57 PM, Ewen Cheslack-Postava < > > > >> > > e...@confluent.io> > > > >> > > > wrote: > > > >> > > > > > > >> > > >> Don't necessarily want to add noise here, but I'm -1 based on > > the > > > >> > > >> disconnect part. See discussion in other thread. (I'm +1 > > > otherwise, > > > >> and > > > >> > > >> happy to have my vote applied assuming we clean up that one > > > issue.) > > > >> > > >> > > > >> > > >> -Ewen > > > >> > > >> > > > >> > > >> On Thu, Jun 16, 2016 at 6:05 PM, Harsha <ka...@harsha.io> > > wrote: > > > >> > > >> > > > >> > > >> > +1 (binding) > > > >> > > >> > Thanks, > > > >> > > >> > Harsha > > > >> > > >> > > > > >> > > >> > On Thu, Jun 16, 2016, at 04:15 PM, Guozhang Wang wrote: > > > >> > > >> > > +1. > > > >> > > >> > > > > > >> > > >> > > On Thu, Jun 16, 2016 at 3:47 PM, Ismael Juma < > > > ism...@juma.me.uk > > > >> > > > > >> > > >> wrote: > > > >> > > >> > > > > > >> > > >> > > > +1 (binding) > > > >> > > >> > > > > > > >> > > >> > > > On Thu, Jun 16, 2016 at 11:50 PM, Grant Henke < > > > >> > > ghe...@cloudera.com> > > > >> > > >> > wrote: > > > >> > > >> > > > > > > >> > > >> > > > > I would like to initiate the voting process for the > > > "KIP-4 > > > >> > > Create > > > >> > > >> > Topics > > > >> > > >> > > > > Schema changes". This is not a vote for all of KIP-4, > > but > > > >> > > >> > specifically > > > >> > > >> > > > for > > > >> > > >> > > > > the create topics changes. I have included the exact > > > changes > > > >> > > below > > > >> > > >> > for > > > >> > > >> > > > > clarity: > > > >> > > >> > > > > > > > > >> > > >> > > > > > Create Topics Request (KAFKA-2945 > > > >> > > >> > > > > > <https://issues.apache.org/jira/browse/KAFKA-2945 > >) > > > >> > > >> > > > > > > > > >> > > >> > > > > > CreateTopics Request (Version: 0) => > > > >> [create_topic_requests] > > > >> > > >> > timeout > > > >> > > >> > > > > > create_topic_requests => topic num_partitions > > > >> > > >> replication_factor > > > >> > > >> > > > > [replica_assignment] [configs] > > > >> > > >> > > > > > topic => STRING > > > >> > > >> > > > > > num_partitions => INT32 > > > >> > > >> > > > > > replication_factor => INT16 > > > >> > > >> > > > > > replica_assignment => partition_id [replicas] > > > >> > > >> > > > > > partition_id => INT32 > > > >> > > >> > > > > > replicas => INT32 > > > >> > > >> > > > > > configs => config_key config_value > > > >> > > >> > > > > > config_key => STRING > > > >> > > >> > > > > > config_value => STRING > > > >> > > >> > > > > > timeout => INT32 > > > >> > > >> > > > > > > > > >> > > >> > > > > > CreateTopicsRequest is a batch request to initiate > > > topic > > > >> > > creation > > > >> > > >> > with > > > >> > > >> > > > > > either predefined or automatic replica assignment > and > > > >> > > optionally > > > >> > > >> > topic > > > >> > > >> > > > > > configuration. > > > >> > > >> > > > > > > > > >> > > >> > > > > > Request semantics: > > > >> > > >> > > > > > > > > >> > > >> > > > > > 1. Must be sent to the controller broker > > > >> > > >> > > > > > 2. If there are multiple instructions for the > same > > > >> topic in > > > >> > > >> one > > > >> > > >> > > > > > request an InvalidRequestException will be > logged > > on > > > >> the > > > >> > > >> broker > > > >> > > >> > and > > > >> > > >> > > > > the > > > >> > > >> > > > > > client will be disconnected. > > > >> > > >> > > > > > - This is because the list of topics is > modeled > > > >> server > > > >> > > side > > > >> > > >> > as a > > > >> > > >> > > > > > map with TopicName as the key > > > >> > > >> > > > > > 3. The principal must be authorized to the > > "Create" > > > >> > > Operation > > > >> > > >> > on the > > > >> > > >> > > > > > "Cluster" resource to create topics. > > > >> > > >> > > > > > - Unauthorized requests will receive a > > > >> > > >> > > > > ClusterAuthorizationException > > > >> > > >> > > > > > 4. > > > >> > > >> > > > > > > > > >> > > >> > > > > > Only one from ReplicaAssignment or > > (num_partitions + > > > >> > > >> > > > > replication_factor > > > >> > > >> > > > > > ), can be defined in one instruction. > > > >> > > >> > > > > > - If both parameters are specified an > > > >> > > InvalidRequestException > > > >> > > >> > will > > > >> > > >> > > > be > > > >> > > >> > > > > > logged on the broker and the client will be > > > >> > > disconnected. > > > >> > > >> > > > > > - In the case ReplicaAssignment is defined > > > number of > > > >> > > >> > partitions > > > >> > > >> > > > and > > > >> > > >> > > > > > replicas will be calculated from the supplied > > > >> > > >> > replica_assignment. > > > >> > > >> > > > > > - In the case of defined (num_partitions + > > > >> > > >> > replication_factor) > > > >> > > >> > > > > > replica assignment will be automatically > > > generated > > > >> by > > > >> > > the > > > >> > > >> > server. > > > >> > > >> > > > > > - One or the other must be defined. The > > existing > > > >> broker > > > >> > > >> side > > > >> > > >> > auto > > > >> > > >> > > > > > create defaults will not be used > > > >> > > >> > > > > > (default.replication.factor, num.partitions). > > The > > > >> client > > > >> > > >> > > > > implementation can > > > >> > > >> > > > > > have defaults for these options when > generating > > > the > > > >> > > >> messages. > > > >> > > >> > > > > > - The first replica in [replicas] is assumed > to > > > be > > > >> the > > > >> > > >> > preferred > > > >> > > >> > > > > > leader. This matches current behavior > > elsewhere. > > > >> > > >> > > > > > 5. Setting a timeout > 0 will allow the request > to > > > >> block > > > >> > > until > > > >> > > >> > the > > > >> > > >> > > > > > topic metadata is "complete" on the controller > > node. > > > >> > > >> > > > > > - Complete means the local topic metadata > cache > > > been > > > >> > > >> > completely > > > >> > > >> > > > > > populated and all partitions have leaders > > > >> > > >> > > > > > - The topic metadata is updated when the > > > >> controller > > > >> > > >> sends > > > >> > > >> > out > > > >> > > >> > > > > > update metadata requests to the brokers > > > >> > > >> > > > > > - If a timeout error occurs, the topic could > > > still > > > >> be > > > >> > > >> created > > > >> > > >> > > > > > successfully at a later time. Its up to the > > > client > > > >> to > > > >> > > query > > > >> > > >> > for > > > >> > > >> > > > > the state > > > >> > > >> > > > > > at that point. > > > >> > > >> > > > > > 6. Setting a timeout <= 0 will validate > arguments > > > and > > > >> > > trigger > > > >> > > >> > the > > > >> > > >> > > > > > create topics and return immediately. > > > >> > > >> > > > > > - This is essentially the fully asynchronous > > > mode we > > > >> > > have > > > >> > > >> in > > > >> > > >> > the > > > >> > > >> > > > > > Zookeeper tools today. > > > >> > > >> > > > > > - The error code in the response will either > > > >> contain an > > > >> > > >> > argument > > > >> > > >> > > > > > validation exception or a timeout exception. > If > > > you > > > >> > > >> receive a > > > >> > > >> > > > > timeout > > > >> > > >> > > > > > exception, because you asked for 0 timeout, > you > > > can > > > >> > > assume > > > >> > > >> > the > > > >> > > >> > > > > message was > > > >> > > >> > > > > > valid and the topic creation was triggered. > > > >> > > >> > > > > > 7. The request is not transactional. > > > >> > > >> > > > > > 1. If an error occurs on one topic, the > others > > > could > > > >> > > still > > > >> > > >> be > > > >> > > >> > > > > > created. > > > >> > > >> > > > > > 2. Errors are reported independently. > > > >> > > >> > > > > > > > > >> > > >> > > > > > QA: > > > >> > > >> > > > > > > > > >> > > >> > > > > > - Why is CreateTopicsRequest a batch request? > > > >> > > >> > > > > > - Scenarios where tools or admins want to > > create > > > >> many > > > >> > > >> topics > > > >> > > >> > > > should > > > >> > > >> > > > > > be able to with fewer requests > > > >> > > >> > > > > > - Example: MirrorMaker may want to create the > > > topics > > > >> > > >> > downstream > > > >> > > >> > > > > > - What happens if some topics error immediately? > > > Will > > > >> it > > > >> > > >> > > > > > return immediately? > > > >> > > >> > > > > > - The request will block until all topics > have > > > >> either > > > >> > > been > > > >> > > >> > > > created, > > > >> > > >> > > > > > errors, or the timeout has been hit > > > >> > > >> > > > > > - There is no "short circuiting" where 1 > error > > > >> stops the > > > >> > > >> > other > > > >> > > >> > > > > > topics from being created > > > >> > > >> > > > > > - Why implement "partial blocking" instead of > > fully > > > >> async > > > >> > > or > > > >> > > >> > fully > > > >> > > >> > > > > > consistent? > > > >> > > >> > > > > > - See Cluster Consistent Blocking > > > >> > > >> > > > > > < > > > >> > > >> > > > > > > > >> > > >> > > > > > > >> > > >> > > > > >> > > >> > > > >> > > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-cluster-consistent-blocking > > > >> > > >> > > > > > > > > >> > > >> > > > > > below > > > >> > > >> > > > > > - Why require the request to go to the > controller? > > > >> > > >> > > > > > - The controller is responsible for the > cluster > > > >> metadata > > > >> > > >> and > > > >> > > >> > its > > > >> > > >> > > > > > propagation > > > >> > > >> > > > > > - See Request Forwarding > > > >> > > >> > > > > > < > > > >> > > >> > > > > > > > >> > > >> > > > > > > >> > > >> > > > > >> > > >> > > > >> > > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-request > > > >> > > >> > > > > > > > > >> > > >> > > > > > below > > > >> > > >> > > > > > > > > >> > > >> > > > > > Create Topics Response > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > > >> > > >> > > > > > CreateTopics Response (Version: 0) => > > > [topic_error_codes] > > > >> > > >> > > > > > topic_error_codes => topic error_code > > > >> > > >> > > > > > topic => STRING > > > >> > > >> > > > > > error_code => INT16 > > > >> > > >> > > > > > > > > >> > > >> > > > > > CreateTopicsResponse contains a map between topic > and > > > >> topic > > > >> > > >> > creation > > > >> > > >> > > > > > result error code (see New Protocol Errors > > > >> > > >> > > > > > < > > > >> > > >> > > > > > > > >> > > >> > > > > > > >> > > >> > > > > >> > > >> > > > >> > > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-NewProtocolErrors > > > >> > > >> > > > > > > > > >> > > >> > > > > > ). > > > >> > > >> > > > > > > > > >> > > >> > > > > > > > >> > > >> > > > > The KIP is available here for reference (linked to > the > > > >> Create > > > >> > > >> Topics > > > >> > > >> > > > schema > > > >> > > >> > > > > section): > > > >> > > >> > > > > * > > > >> > > >> > > > > > > > >> > > >> > > > > > > >> > > >> > > > > >> > > >> > > > >> > > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945) > > > >> > > >> > > > > < > > > >> > > >> > > > > > > > >> > > >> > > > > > > >> > > >> > > > > >> > > >> > > > >> > > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945) > > > >> > > >> > > > > >* > > > >> > > >> > > > > > > > >> > > >> > > > > A pull request is available implementing the proposed > > > >> changes > > > >> > > here: > > > >> > > >> > > > > https://github.com/apache/kafka/pull/1489 > > > >> > > >> > > > > > > > >> > > >> > > > > Here is a link to the past discussion on the mailing > > > list: > > > >> > > >> > > > > * > > > >> > > >> > > > > > > > >> > > >> > > > > > > >> > > >> > > > > >> > > >> > > > >> > > > > > >> > > > > > > http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema > > > >> > > >> > > > > < > > > >> > > >> > > > > > > > >> > > >> > > > > > > >> > > >> > > > > >> > > >> > > > >> > > > > > >> > > > > > > http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema > > > >> > > >> > > > > >* > > > >> > > >> > > > > > > > >> > > >> > > > > Thank you, > > > >> > > >> > > > > Grant > > > >> > > >> > > > > -- > > > >> > > >> > > > > Grant Henke > > > >> > > >> > > > > Software Engineer | Cloudera > > > >> > > >> > > > > gr...@cloudera.com | twitter.com/gchenke | > > > >> > > >> > linkedin.com/in/granthenke > > > >> > > >> > > > > > > > >> > > >> > > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > -- > > > >> > > >> > > -- Guozhang > > > >> > > >> > > > > >> > > >> > > > >> > > >> > > > >> > > >> > > > >> > > >> -- > > > >> > > >> Thanks, > > > >> > > >> Ewen > > > >> > > >> > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > -- > > > >> > > > Grant Henke > > > >> > > > Software Engineer | Cloudera > > > >> > > > gr...@cloudera.com | twitter.com/gchenke | > > > >> linkedin.com/in/granthenke > > > >> > > > > > >> > > > > > > > > > > > -- > > Thanks, > > Ewen > > > -- Grant Henke Software Engineer | Cloudera gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
