For 1), 2) and 3), blocking would probably mean that the new metadata is propagated to every broker. To achieve that, the client can keep issuing the describe topic request to every broker until it sees the new metadata in the response.
Thanks, Jun On Fri, Mar 20, 2015 at 12:16 PM, Andrii Biletskyi < andrii.bilets...@stealth.ly> wrote: > Hm, actually the ticket you linked, Guozhang, brings as back > to the problem what should be considered a post-condition for > each of the admin commands. > In my understanding: > > 1) CreateTopic - broker created /brokers/topics/<topic> > (Not the controller picked up changes from zk and broadcasted > LeaderAndIsr and UpdateMetadata) > > 2) AlterTopic - same as 1) - broker changed assignment data > in zookeeper or created admin path for topic config change > > 3) DeleteTopic - admin path /admin/delete_topics is created > > 4) ReassignPartitions and PreferredReplica - corresponding admin > path is created > > Now what can be considered a completed operation from the client's > perspective? > 1) Topic is created once corresponding data is in zk > (I remember there were some thoughts that it'd be good to consider > topic created once all replicas receive information about it and thus > clients can produce/consume from it, but as was discussed this seems > to be a hard thing to do) > > 2) Probably same as 1), so right after AlterTopic is issued > > 3) The topic has been removed from /brokers/topics > > 4) ReassignPartitions and PrefferedReplica were discussed earlier - > in short the former is completed once partition state info in zk matches > reassignment request and admin path is empty, the latter - once data > in zk shows that head of assignned replicas of the partition and leader > is the same replica > > Thoughts? > > Thanks, > Andrii Biletskyi > > On Fri, Mar 20, 2015 at 7:10 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > > I think while loop is fine for supporting blocking, just that we need to > > add back off to avoid bombarding brokers with DescribeTopic requests. > > > > Also I have linked KAFKA-1125 > > <https://issues.apache.org/jira/browse/KAFKA-1125> to your proposal, and > > when KAFKA-1694 is done this ticket can also be closed. > > > > Guozhang > > > > On Fri, Mar 20, 2015 at 9:41 AM, Andrii Biletskyi < > > andrii.bilets...@stealth.ly> wrote: > > > > > Great. > > > I want to elaborate this a bit more, to see we are on the same page > > > concerning the client code. > > > > > > So with all topic commands being async a client (AdminClient in our > > > case or any other other client people would like to implement) to > support > > > a blocking operation (which seems to be a natural use-case e.g. for > topic > > > creation): would have to do: > > > 1. issue CreateTopicRequest > > > 2. if successful, in a "while" loop send DescribeTopicRequest and > > > break the loop once all topics are returned in response (or upon > > timeout). > > > 3. if unsuccessful throw exception > > > Would it be okay? > > > > > > Thanks, > > > Andrii Biletskyi > > > > > > > > > On Fri, Mar 20, 2015 at 6:21 PM, Jun Rao <j...@confluent.io> wrote: > > > > > > > Andrii, > > > > > > > > I think you are right. It seems that only ReassignPartitions needs a > > > > separate verification request. > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > On Thu, Mar 19, 2015 at 9:22 AM, Andrii Biletskyi < > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > Guys, > > > > > I like this idea too. Let's stick with that. I'll update KIP > > > accordingly. > > > > > > > > > > I was also thinking we can avoid adding dedicated status check > > > > > requests for topic commands. - We have everything in DescribeTopic > > > > > for that! E.g.: > > > > > User issued CreateTopic - to check the status client sends > > > DescribeTopic > > > > > and checks whether is something returned for that topic. The same > for > > > > > alteration, deletion. > > > > > Btw, PreferredReplica status can be also checked with > > > > DescribeTopicRequest > > > > > (head of assigned replicas list == leader). > > > > > For ReassignPartitions as discussed we'll need to have a separate > > > > Verify... > > > > > request. > > > > > > > > > > Thanks, > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > On Thu, Mar 19, 2015 at 6:03 PM, Guozhang Wang <wangg...@gmail.com > > > > > > wrote: > > > > > > > > > > > +1 on broker writing to ZK for async handling. I was thinking > that > > in > > > > the > > > > > > end state the admin requests would be eventually sent to > controller > > > > > either > > > > > > through re-routing or clients discovering them, instead of > letting > > > > > > controller listen on ZK admin path. But thinking about it a > second > > > > time, > > > > > I > > > > > > think it is actually simpler to let controller manage > > > > > > incoming queued-up admin requests through ZK. > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 4:16 PM, Joel Koshy <jjkosh...@gmail.com > > > > > > wrote: > > > > > > > > > > > > > +1 as well. I think it helps to keep the rerouting approach > > > > orthogonal > > > > > > > to this KIP. > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 03:40:48PM -0700, Jay Kreps wrote: > > > > > > > > I'm +1 on Jun's suggestion as long as it can work for all the > > > > > requests. > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 3:35 PM, Jun Rao <j...@confluent.io> > > > wrote: > > > > > > > > > > > > > > > > > Andrii, > > > > > > > > > > > > > > > > > > I think we agreed on the following. > > > > > > > > > > > > > > > > > > (a) Admin requests can be sent to and handled by any > broker. > > > > > > > > > (b) Admin requests are processed asynchronously, at least > for > > > > now. > > > > > > > That is, > > > > > > > > > when the client gets a response, it just means that the > > request > > > > is > > > > > > > > > initiated, but not necessarily completed. Then, it's up to > > the > > > > > client > > > > > > > to > > > > > > > > > issue another request to check the status for completion. > > > > > > > > > > > > > > > > > > To support (a), we were thinking of doing request > forwarding > > to > > > > the > > > > > > > > > controller (utilizing KAFKA-1912). I am making an > alternative > > > > > > proposal. > > > > > > > > > Basically, the broker can just write to ZooKeeper to inform > > the > > > > > > > controller > > > > > > > > > about the request. For example, to handle > > > partitionReassignment, > > > > > the > > > > > > > broker > > > > > > > > > will just write the requested partitions to > > > > > > /admin/reassign_partitions > > > > > > > > > (like what AdminUtils currently does) and then send a > > response > > > to > > > > > the > > > > > > > > > client. This shouldn't take long and the implementation > will > > be > > > > > > simpler > > > > > > > > > than forwarding the requests to the controller through RPC. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 3:03 PM, Andrii Biletskyi < > > > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > > > > > > Jun, > > > > > > > > > > > > > > > > > > > > I might be wrong but didn't we agree we will let any > broker > > > > from > > > > > > the > > > > > > > > > > cluster handle *long-running* admin requests (at this > time > > > > > > > > > preferredReplica > > > > > > > > > > and > > > > > > > > > > reassignPartitions), via zk admin path. Thus CreateTopics > > etc > > > > > > should > > > > > > > be > > > > > > > > > > sent > > > > > > > > > > only to the controller. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 11:55 PM, Jun Rao < > > j...@confluent.io> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Joel, Andril, > > > > > > > > > > > > > > > > > > > > > > I think we agreed that those admin requests can be > issued > > > to > > > > > any > > > > > > > > > broker. > > > > > > > > > > > Because of that, there doesn't seem to be a strong need > > to > > > > know > > > > > > the > > > > > > > > > > > controller. So, perhaps we can proceed by not making > any > > > > change > > > > > > to > > > > > > > the > > > > > > > > > > > format of TMR right now. When we start using create > topic > > > > > request > > > > > > > in > > > > > > > > > the > > > > > > > > > > > producer, we will need a new version of TMR that > doesn't > > > > > trigger > > > > > > > auto > > > > > > > > > > topic > > > > > > > > > > > creation. But that can be done later. > > > > > > > > > > > > > > > > > > > > > > As a first cut implementation, I think the broker can > > just > > > > > write > > > > > > > to ZK > > > > > > > > > > > directly for > > > > > > > > > > > > > > > > > > createToipic/alterTopic/reassignPartitions/preferredLeaderElection > > > > > > > > > > > requests, instead of forwarding them to the controller. > > > This > > > > > will > > > > > > > > > > simplify > > > > > > > > > > > the implementation on the broker side. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 11:58 AM, Joel Koshy < > > > > > > jjkosh...@gmail.com> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > (Thanks Andrii for the summary) > > > > > > > > > > > > > > > > > > > > > > > > For (1) yes we will circle back on that shortly after > > > > syncing > > > > > > up > > > > > > > in > > > > > > > > > > > > person. I think it is close to getting committed > > although > > > > > > > development > > > > > > > > > > > > for KAFKA-1927 can probably begin without it. > > > > > > > > > > > > > > > > > > > > > > > > There is one more item we covered at the hangout. > i.e., > > > > > whether > > > > > > > we > > > > > > > > > > > > want to add the coordinator to the topic metadata > > > response > > > > or > > > > > > > provide > > > > > > > > > > > > a clearer ClusterMetadataRequest. > > > > > > > > > > > > > > > > > > > > > > > > There are two reasons I think we should try and avoid > > > > adding > > > > > > the > > > > > > > > > > > > field: > > > > > > > > > > > > - It is irrelevant to topic metadata > > > > > > > > > > > > - If we finally do request rerouting in Kafka then > the > > > > field > > > > > > > would > > > > > > > > > add > > > > > > > > > > > > little to no value. (It still helps to have a > > separate > > > > > > > > > > > > ClusterMetadataRequest to query for cluster-wide > > > > > information > > > > > > > such > > > > > > > > > as > > > > > > > > > > > > 'which broker is the controller?' as Joe > mentioned.) > > > > > > > > > > > > > > > > > > > > > > > > I think it would be cleaner to have an explicit > > > > > > > > > ClusterMetadataRequest > > > > > > > > > > > > that you can send to any broker in order to obtain > the > > > > > > controller > > > > > > > > > (and > > > > > > > > > > > > in the future possibly other cluster-wide > > information). I > > > > > think > > > > > > > the > > > > > > > > > > > > main argument against doing this and instead adding > it > > to > > > > the > > > > > > > topic > > > > > > > > > > > > metadata response was convenience - i.e., you don't > > have > > > to > > > > > > > discover > > > > > > > > > > > > the controller in advance. However, I don't see much > > > actual > > > > > > > > > > > > benefit/convenience in this and in fact think it is a > > > > > > non-issue. > > > > > > > Let > > > > > > > > > > > > me know if I'm overlooking something here. > > > > > > > > > > > > > > > > > > > > > > > > As an example, say we need to initiate partition > > > > reassignment > > > > > > by > > > > > > > > > > > > issuing the new ReassignPartitionsRequest to the > > > controller > > > > > > > (assume > > > > > > > > > we > > > > > > > > > > > > already have the desired manual partition > assignment). > > > If > > > > we > > > > > > > are to > > > > > > > > > > > > augment topic metadata response then the flow be > > > something > > > > > like > > > > > > > this > > > > > > > > > : > > > > > > > > > > > > > > > > > > > > > > > > - Issue topic metadata request to any broker (and > > > discover > > > > > the > > > > > > > > > > > > controller > > > > > > > > > > > > - Connect to controller if required (i.e., if the > > broker > > > > > above > > > > > > != > > > > > > > > > > > > controller) > > > > > > > > > > > > - Issue the partition reassignment request to the > > > > controller. > > > > > > > > > > > > > > > > > > > > > > > > With an explicit cluster metadata request it would > be: > > > > > > > > > > > > - Issue cluster metadata request to any broker > > > > > > > > > > > > - Connect to controller if required (i.e., if the > > broker > > > > > above > > > > > > != > > > > > > > > > > > > controller) > > > > > > > > > > > > - Issue the partition reassignment request > > > > > > > > > > > > > > > > > > > > > > > > So it seems to add little practical value and bloats > > > topic > > > > > > > metadata > > > > > > > > > > > > response with an irrelevant detail. > > > > > > > > > > > > > > > > > > > > > > > > The other angle to this is the following - is it a > > matter > > > > of > > > > > > > naming? > > > > > > > > > > > > Should we just rename topic metadata request/response > > to > > > > just > > > > > > > > > > > > MetadataRequest/Response and add cluster metadata to > > it? > > > By > > > > > > that > > > > > > > same > > > > > > > > > > > > token should we also allow querying for the consumer > > > > > > coordinator > > > > > > > (and > > > > > > > > > > > > in future transaction coordinator) as well? This > leads > > > to a > > > > > > > bloated > > > > > > > > > > > > request which isn't very appealing and altogether > > > > confusing. > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > Joel > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 09:34:12AM -0700, Jun Rao > > wrote: > > > > > > > > > > > > > Andri, > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the summary. > > > > > > > > > > > > > > > > > > > > > > > > > > 1. I just realized that in order to start working > on > > > > > > > KAFKA-1927, we > > > > > > > > > > > will > > > > > > > > > > > > > need to merge the changes to OffsetCommitRequest > > (from > > > > > 0.8.2) > > > > > > > to > > > > > > > > > > trunk. > > > > > > > > > > > > > This is planned to be done as part of KAFKA-1634. > So, > > > we > > > > > will > > > > > > > need > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > and Joel's help to wrap this up. > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Thinking about this a bit more, if the semantic > of > > > > those > > > > > > > "write" > > > > > > > > > > > > > requests is async (i.e., after the client gets a > > > > response, > > > > > it > > > > > > > just > > > > > > > > > > > means > > > > > > > > > > > > > that the operation is initiated, but not > necessarily > > > > > > > completed), we > > > > > > > > > > > don't > > > > > > > > > > > > > really need to forward the requests to the > > controller. > > > > > > > Instead, the > > > > > > > > > > > > > receiving broker can just write the operation to ZK > > as > > > > the > > > > > > > admin > > > > > > > > > > > command > > > > > > > > > > > > > line tool previously does. This will simplify the > > > > > > > implementation. > > > > > > > > > > > > > > > > > > > > > > > > > > 8. There is another implementation detail for > > describe > > > > > topic. > > > > > > > > > > Ideally, > > > > > > > > > > > we > > > > > > > > > > > > > want to read the topic config from the broker > cache, > > > > > instead > > > > > > of > > > > > > > > > > > > ZooKeeper. > > > > > > > > > > > > > Currently, every broker reads the topic-level > config > > > for > > > > > all > > > > > > > > > topics. > > > > > > > > > > > > > However, it ignores those for topics not hosted on > > > > itself. > > > > > > So, > > > > > > > we > > > > > > > > > may > > > > > > > > > > > > need > > > > > > > > > > > > > to change TopicConfigManager a bit so that it > caches > > > the > > > > > > > configs > > > > > > > > > for > > > > > > > > > > > all > > > > > > > > > > > > > topics. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 1:13 PM, Andrii Biletskyi < > > > > > > > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Guys, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for a great discussion! > > > > > > > > > > > > > > Here are the actions points: > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Q: Get rid of all scala requests objects, use > > java > > > > > > > protocol > > > > > > > > > > > > definitions. > > > > > > > > > > > > > > A: Gwen kindly took that (KAFKA-1927). It's > > > > important > > > > > > to > > > > > > > > > speed > > > > > > > > > > up > > > > > > > > > > > > > > review procedure > > > > > > > > > > > > > > there since this ticket blocks other > > > important > > > > > > > changes. > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Q: Generic re-reroute facility vs client > > > maintaining > > > > > > > cluster > > > > > > > > > > > state. > > > > > > > > > > > > > > A: Jay has added pseudo code to KAFKA-1912 - > > need > > > > to > > > > > > > consider > > > > > > > > > > > > whether > > > > > > > > > > > > > > this will be > > > > > > > > > > > > > > easy to implement as a server-side > feature > > > > > > (comments > > > > > > > are > > > > > > > > > > > > > > welcomed!). > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3. Q: Controller field in wire protocol. > > > > > > > > > > > > > > A: This might be useful for clients, add this > > to > > > > > > > > > > > > TopicMetadataResponse > > > > > > > > > > > > > > (already in KIP). > > > > > > > > > > > > > > > > > > > > > > > > > > > > 4. Q: Decoupling topic creation from TMR. > > > > > > > > > > > > > > A: I will add proposed by Jun solution (using > > > > > clientId > > > > > > > for > > > > > > > > > > that) > > > > > > > > > > > > to the > > > > > > > > > > > > > > KIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > 5. Q: Bumping new versions of TMR vs grabbing all > > > > > protocol > > > > > > > > > changes > > > > > > > > > > in > > > > > > > > > > > > one > > > > > > > > > > > > > > version. > > > > > > > > > > > > > > A: It was decided to try to gather all > changes > > to > > > > > > > protocol > > > > > > > > > > > (before > > > > > > > > > > > > > > release). > > > > > > > > > > > > > > In case of TMR it worth checking: > > KAFKA-2020 > > > > and > > > > > > > KIP-13 > > > > > > > > > > > > (quotas) > > > > > > > > > > > > > > > > > > > > > > > > > > > > 6. Q: JSON lib is needed to deserialize user's > > input > > > in > > > > > CLI > > > > > > > tool. > > > > > > > > > > > > > > A: Use jackson for that, /tools project is a > > > > separate > > > > > > > jar so > > > > > > > > > > > > shouldn't > > > > > > > > > > > > > > be a big deal. > > > > > > > > > > > > > > > > > > > > > > > > > > > > 7. Q: VerifyReassingPartitions vs generic status > > > check > > > > > > > command. > > > > > > > > > > > > > > A: For long-running requests like reassign > > > > > partitions > > > > > > > > > > *progress* > > > > > > > > > > > > check > > > > > > > > > > > > > > request is useful, > > > > > > > > > > > > > > it makes sense to introduce it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please add, correct me if I missed something. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 6:20 PM, Andrii > Biletskyi < > > > > > > > > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Joel, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > You are right, I removed ClusterMetadata > because > > we > > > > > have > > > > > > > > > > partially > > > > > > > > > > > > > > > what we need in TopicMetadata. Also, as Jay > > pointed > > > > out > > > > > > > > > earlier, > > > > > > > > > > we > > > > > > > > > > > > > > > would like to have "orthogonal" API, but at the > > > same > > > > > time > > > > > > > we > > > > > > > > > need > > > > > > > > > > > > > > > to be backward compatible. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > But I like your idea and even have some other > > > > arguments > > > > > > for > > > > > > > > > this > > > > > > > > > > > > option: > > > > > > > > > > > > > > > There is also DescribeTopicRequest which was > > > proposed > > > > > in > > > > > > > this > > > > > > > > > > KIP, > > > > > > > > > > > > > > > it returns topic configs, partitions, > replication > > > > > factor > > > > > > > plus > > > > > > > > > > > > partition > > > > > > > > > > > > > > > ISR, ASR, > > > > > > > > > > > > > > > leader replica. The later part is really > already > > > > there > > > > > in > > > > > > > > > > > > > > > TopicMetadataRequest. > > > > > > > > > > > > > > > So again we'll have to add stuff to TMR, not to > > > > > duplicate > > > > > > > some > > > > > > > > > > info > > > > > > > > > > > > in > > > > > > > > > > > > > > > newly added requests. However, this way we'll > end > > > up > > > > > with > > > > > > > > > > "monster" > > > > > > > > > > > > > > > request which returns cluster metadata, topic > > > > > replication > > > > > > > and > > > > > > > > > > > config > > > > > > > > > > > > info > > > > > > > > > > > > > > > plus partition replication data. Seems logical > to > > > > split > > > > > > > TMR to > > > > > > > > > > > > > > > - ClusterMetadata (brokers + controller, maybe > > smth > > > > > else) > > > > > > > > > > > > > > > - TopicMetadata (topic info + partition > details) > > > > > > > > > > > > > > > But since current TMR is involved in lots of > > places > > > > > > > (including > > > > > > > > > > > > network > > > > > > > > > > > > > > > client, > > > > > > > > > > > > > > > as I understand) this might be very serious > > change > > > > and > > > > > it > > > > > > > > > > probably > > > > > > > > > > > > makes > > > > > > > > > > > > > > > sense to stick with current approach. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 5:29 PM, Joel Koshy < > > > > > > > > > jjkosh...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> I may be missing some context but hopefully > this > > > > will > > > > > > > also be > > > > > > > > > > > > covered > > > > > > > > > > > > > > >> today: I thought the earlier proposal where > > there > > > > was > > > > > an > > > > > > > > > > explicit > > > > > > > > > > > > > > >> ClusterMetadata request was clearer and > > explicit. > > > > > During > > > > > > > the > > > > > > > > > > > course > > > > > > > > > > > > of > > > > > > > > > > > > > > >> this thread I think the conclusion was that > the > > > main > > > > > > need > > > > > > > was > > > > > > > > > > for > > > > > > > > > > > > > > >> controller information and that can be rolled > > into > > > > the > > > > > > > topic > > > > > > > > > > > > metadata > > > > > > > > > > > > > > >> response but that seems a bit irrelevant to > > topic > > > > > > > metadata. > > > > > > > > > > FWIW I > > > > > > > > > > > > > > >> think the full broker-list is also irrelevant > to > > > > topic > > > > > > > > > metadata, > > > > > > > > > > > but > > > > > > > > > > > > > > >> it is already there and in use. I think there > is > > > > still > > > > > > > room > > > > > > > > > for > > > > > > > > > > an > > > > > > > > > > > > > > >> explicit ClusterMetadata request since there > may > > > be > > > > > > other > > > > > > > > > > > > > > >> cluster-level information that we may want to > > add > > > > over > > > > > > > time > > > > > > > > > (and > > > > > > > > > > > > that > > > > > > > > > > > > > > >> have nothing to do with topic metadata). > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> On Tue, Mar 17, 2015 at 02:45:30PM +0200, > Andrii > > > > > > Biletskyi > > > > > > > > > > wrote: > > > > > > > > > > > > > > >> > Jun, > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > 101. Okay, if you say that such use case is > > > > > > important. I > > > > > > > > > also > > > > > > > > > > > > think > > > > > > > > > > > > > > >> > using clientId for these purposes is fine - > if > > > we > > > > > > > already > > > > > > > > > have > > > > > > > > > > > > this > > > > > > > > > > > > > > >> field > > > > > > > > > > > > > > >> > as part of all Wire protocol messages, why > not > > > use > > > > > > that. > > > > > > > > > > > > > > >> > I will update KIP-4 page if nobody has other > > > ideas > > > > > > > (which > > > > > > > > > may > > > > > > > > > > > > come up > > > > > > > > > > > > > > >> > during the call today). > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > 102.1 Agree, I'll update the KIP > accordingly. > > I > > > > > think > > > > > > > we can > > > > > > > > > > add > > > > > > > > > > > > new, > > > > > > > > > > > > > > >> > fine-grained error codes if some error code > > > > received > > > > > > in > > > > > > > > > > specific > > > > > > > > > > > > case > > > > > > > > > > > > > > >> > won't give enough context to return a > > > descriptive > > > > > > error > > > > > > > > > > message > > > > > > > > > > > > for > > > > > > > > > > > > > > >> user. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > Look forward to discussing all outstanding > > > issues > > > > in > > > > > > > detail > > > > > > > > > > > today > > > > > > > > > > > > > > during > > > > > > > > > > > > > > >> > the call. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > Thanks, > > > > > > > > > > > > > > >> > Andrii Biletskyi > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > On Mon, Mar 16, 2015 at 10:59 PM, Jun Rao < > > > > > > > j...@confluent.io > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > 101. There may be a use case where you > only > > > want > > > > > the > > > > > > > > > topics > > > > > > > > > > to > > > > > > > > > > > > be > > > > > > > > > > > > > > >> created > > > > > > > > > > > > > > >> > > manually by admins. Currently, you can do > > that > > > > by > > > > > > > > > disabling > > > > > > > > > > > auto > > > > > > > > > > > > > > topic > > > > > > > > > > > > > > >> > > creation and issue topic creation from the > > > > > > > TopicCommand. > > > > > > > > > If > > > > > > > > > > we > > > > > > > > > > > > > > >> disable auto > > > > > > > > > > > > > > >> > > topic creation completely on the broker > and > > > > don't > > > > > > > have a > > > > > > > > > way > > > > > > > > > > > to > > > > > > > > > > > > > > >> distinguish > > > > > > > > > > > > > > >> > > between topic creation requests from the > > > regular > > > > > > > clients > > > > > > > > > and > > > > > > > > > > > the > > > > > > > > > > > > > > >> admin, we > > > > > > > > > > > > > > >> > > can't support manual topic creation any > > more. > > > I > > > > > was > > > > > > > > > thinking > > > > > > > > > > > > that > > > > > > > > > > > > > > >> another > > > > > > > > > > > > > > >> > > way of distinguishing the clients making > the > > > > topic > > > > > > > > > creation > > > > > > > > > > > > requests > > > > > > > > > > > > > > >> is > > > > > > > > > > > > > > >> > > using clientId. For example, the admin > tool > > > can > > > > > set > > > > > > > it to > > > > > > > > > > > > something > > > > > > > > > > > > > > >> like > > > > > > > > > > > > > > >> > > admin and the broker can treat that > clientId > > > > > > > specially. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > Also, there is a related discussion in > > > > KAFKA-2020. > > > > > > > > > > Currently, > > > > > > > > > > > > we do > > > > > > > > > > > > > > >> the > > > > > > > > > > > > > > >> > > following in TopicMetadataResponse: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > 1. If leader is not available, we set the > > > > > partition > > > > > > > level > > > > > > > > > > > error > > > > > > > > > > > > code > > > > > > > > > > > > > > >> to > > > > > > > > > > > > > > >> > > LeaderNotAvailable. > > > > > > > > > > > > > > >> > > 2. If a non-leader replica is not > available, > > > we > > > > > take > > > > > > > that > > > > > > > > > > > > replica > > > > > > > > > > > > > > out > > > > > > > > > > > > > > >> of > > > > > > > > > > > > > > >> > > the assigned replica list and isr in the > > > > response. > > > > > > As > > > > > > > an > > > > > > > > > > > > indication > > > > > > > > > > > > > > >> for > > > > > > > > > > > > > > >> > > doing that, we set the partition level > error > > > > code > > > > > to > > > > > > > > > > > > > > >> ReplicaNotAvailable. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > This has a few problems. First, > > > > > ReplicaNotAvailable > > > > > > > > > probably > > > > > > > > > > > > > > >> shouldn't be > > > > > > > > > > > > > > >> > > an error, at least for the normal > > > > > producer/consumer > > > > > > > > > clients > > > > > > > > > > > that > > > > > > > > > > > > > > just > > > > > > > > > > > > > > >> want > > > > > > > > > > > > > > >> > > to find out the leader. Second, it can > > happen > > > > that > > > > > > > both > > > > > > > > > the > > > > > > > > > > > > leader > > > > > > > > > > > > > > and > > > > > > > > > > > > > > >> > > another replica are not available at the > > same > > > > > time. > > > > > > > There > > > > > > > > > is > > > > > > > > > > > no > > > > > > > > > > > > > > error > > > > > > > > > > > > > > >> code > > > > > > > > > > > > > > >> > > to indicate both. Third, even if a replica > > is > > > > not > > > > > > > > > available, > > > > > > > > > > > > it's > > > > > > > > > > > > > > >> still > > > > > > > > > > > > > > >> > > useful to return its replica id since some > > > > clients > > > > > > > (e.g. > > > > > > > > > > admin > > > > > > > > > > > > tool) > > > > > > > > > > > > > > >> may > > > > > > > > > > > > > > >> > > still make use of it. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > One way to address this issue is to always > > > > return > > > > > > the > > > > > > > > > > replica > > > > > > > > > > > > id for > > > > > > > > > > > > > > >> > > leader, assigned replicas, and isr > > regardless > > > of > > > > > > > whether > > > > > > > > > the > > > > > > > > > > > > > > >> corresponding > > > > > > > > > > > > > > >> > > broker is live or not. Since we also > return > > > the > > > > > list > > > > > > > of > > > > > > > > > live > > > > > > > > > > > > > > brokers, > > > > > > > > > > > > > > >> the > > > > > > > > > > > > > > >> > > client can figure out whether a leader or > a > > > > > replica > > > > > > is > > > > > > > > > live > > > > > > > > > > or > > > > > > > > > > > > not > > > > > > > > > > > > > > >> and act > > > > > > > > > > > > > > >> > > accordingly. This way, we don't need to > set > > > the > > > > > > > partition > > > > > > > > > > > level > > > > > > > > > > > > > > error > > > > > > > > > > > > > > >> code > > > > > > > > > > > > > > >> > > when the leader or a replica is not > > available. > > > > > This > > > > > > > > > doesn't > > > > > > > > > > > > change > > > > > > > > > > > > > > >> the wire > > > > > > > > > > > > > > >> > > protocol, but does change the semantics. > > Since > > > > we > > > > > > are > > > > > > > > > > evolving > > > > > > > > > > > > the > > > > > > > > > > > > > > >> protocol > > > > > > > > > > > > > > >> > > of TopicMetadataRequest here, we can > > > potentially > > > > > > > piggyback > > > > > > > > > > the > > > > > > > > > > > > > > change. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > 102.1 For those types of errors due to > > invalid > > > > > > input, > > > > > > > > > > > shouldn't > > > > > > > > > > > > we > > > > > > > > > > > > > > >> just > > > > > > > > > > > > > > >> > > guard it at parameter validation time and > > > throw > > > > > > > > > > > > > > >> InvalidArgumentException > > > > > > > > > > > > > > >> > > without even sending the request to the > > > broker? > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > Thanks, > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > Jun > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > On Mon, Mar 16, 2015 at 10:37 AM, Andrii > > > > > Biletskyi < > > > > > > > > > > > > > > >> > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > Jun, > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > Answering your questions: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > 101. If I understand you correctly, you > > are > > > > > saying > > > > > > > > > future > > > > > > > > > > > > producer > > > > > > > > > > > > > > >> > > versions > > > > > > > > > > > > > > >> > > > (which > > > > > > > > > > > > > > >> > > > will be ported to TMR_V1) won't be able > to > > > > > > > automatically > > > > > > > > > > > > create > > > > > > > > > > > > > > >> topic (if > > > > > > > > > > > > > > >> > > > we > > > > > > > > > > > > > > >> > > > unconditionally remove topic creation > from > > > > > there). > > > > > > > But > > > > > > > > > we > > > > > > > > > > > > need to > > > > > > > > > > > > > > >> this > > > > > > > > > > > > > > >> > > > preserve logic. > > > > > > > > > > > > > > >> > > > Ok, about your proposal: I'm not a big > fan > > > > too, > > > > > > > when it > > > > > > > > > > > comes > > > > > > > > > > > > to > > > > > > > > > > > > > > >> > > > differentiating > > > > > > > > > > > > > > >> > > > clients directly in protocol schema. And > > > also > > > > > I'm > > > > > > > not > > > > > > > > > > sure I > > > > > > > > > > > > > > >> understand > > > > > > > > > > > > > > >> > > at > > > > > > > > > > > > > > >> > > > all why > > > > > > > > > > > > > > >> > > > auto.create.topics.enable is a server > side > > > > > > > > > configuration. > > > > > > > > > > > Can > > > > > > > > > > > > we > > > > > > > > > > > > > > >> > > deprecate > > > > > > > > > > > > > > >> > > > this setting > > > > > > > > > > > > > > >> > > > in future versions, add this setting to > > > > producer > > > > > > and > > > > > > > > > based > > > > > > > > > > > on > > > > > > > > > > > > that > > > > > > > > > > > > > > >> upon > > > > > > > > > > > > > > >> > > > receiving > > > > > > > > > > > > > > >> > > > UnknownTopic create topic explicitly by > a > > > > > separate > > > > > > > > > > producer > > > > > > > > > > > > call > > > > > > > > > > > > > > via > > > > > > > > > > > > > > >> > > > adminClient? > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > 102.1. Hm, yes. It's because we want to > > > > support > > > > > > > batching > > > > > > > > > > and > > > > > > > > > > > > at > > > > > > > > > > > > > > the > > > > > > > > > > > > > > >> same > > > > > > > > > > > > > > >> > > > time we > > > > > > > > > > > > > > >> > > > want to give descriptive error messages > > for > > > > > > clients. > > > > > > > > > Since > > > > > > > > > > > > > > >> AdminClient > > > > > > > > > > > > > > >> > > > holds the context > > > > > > > > > > > > > > >> > > > to construct such messages (e.g. > > AdminClient > > > > > layer > > > > > > > can > > > > > > > > > > know > > > > > > > > > > > > that > > > > > > > > > > > > > > >> > > > InvalidArgumentsCode > > > > > > > > > > > > > > >> > > > means two cases: either invalid number - > > > e.g. > > > > > -1; > > > > > > or > > > > > > > > > > > > > > >> replication-factor > > > > > > > > > > > > > > >> > > was > > > > > > > > > > > > > > >> > > > provided while > > > > > > > > > > > > > > >> > > > partitions argument wasn't) - I wrapped > > > > > responses > > > > > > in > > > > > > > > > > > > Exceptions. > > > > > > > > > > > > > > >> But I'm > > > > > > > > > > > > > > >> > > > open to any > > > > > > > > > > > > > > >> > > > other ideas, this was just initial > > version. > > > > > > > > > > > > > > >> > > > 102.2. Yes, I agree. I'll change that to > > > > > probably > > > > > > > some > > > > > > > > > > other > > > > > > > > > > > > dto. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > Thanks, > > > > > > > > > > > > > > >> > > > Andrii Biletskyi > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > On Fri, Mar 13, 2015 at 7:16 PM, Jun > Rao < > > > > > > > > > > j...@confluent.io> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > Andrii, > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > 101. That's what I was thinking too, > but > > > it > > > > > may > > > > > > > not be > > > > > > > > > > > that > > > > > > > > > > > > > > >> simple. In > > > > > > > > > > > > > > >> > > > > TopicMetadataRequest_V1, > > > > > > > > > > > > > > >> > > > > we can let it not trigger auto topic > > > > creation. > > > > > > > Then, > > > > > > > > > in > > > > > > > > > > > the > > > > > > > > > > > > > > >> producer > > > > > > > > > > > > > > >> > > > side, > > > > > > > > > > > > > > >> > > > > if it gets an UnknownTopicException, > it > > > can > > > > > > > explicitly > > > > > > > > > > > > issue a > > > > > > > > > > > > > > >> > > > > createTopicRequest for auto topic > > > creation. > > > > On > > > > > > the > > > > > > > > > > > consumer > > > > > > > > > > > > > > side, > > > > > > > > > > > > > > >> it > > > > > > > > > > > > > > >> > > will > > > > > > > > > > > > > > >> > > > > never issue createTopicRequest. This > > works > > > > > when > > > > > > > auto > > > > > > > > > > topic > > > > > > > > > > > > > > >> creation is > > > > > > > > > > > > > > >> > > > > enabled on the broker side. However, I > > am > > > > not > > > > > > > sure how > > > > > > > > > > > > things > > > > > > > > > > > > > > >> will work > > > > > > > > > > > > > > >> > > > > when auto topic creation is disabled > on > > > the > > > > > > broker > > > > > > > > > side. > > > > > > > > > > > In > > > > > > > > > > > > this > > > > > > > > > > > > > > >> case, > > > > > > > > > > > > > > >> > > we > > > > > > > > > > > > > > >> > > > > want to have a way to manually create > a > > > > topic, > > > > > > > > > > potentially > > > > > > > > > > > > > > through > > > > > > > > > > > > > > >> > > admin > > > > > > > > > > > > > > >> > > > > commands. However, then we need a way > to > > > > > > > distinguish > > > > > > > > > > > > > > >> createTopicRequest > > > > > > > > > > > > > > >> > > > > issued from the producer clients and > the > > > > admin > > > > > > > tools. > > > > > > > > > > May > > > > > > > > > > > > be we > > > > > > > > > > > > > > >> can > > > > > > > > > > > > > > >> > > add a > > > > > > > > > > > > > > >> > > > > new field in createTopicRequest and > set > > it > > > > > > > differently > > > > > > > > > > in > > > > > > > > > > > > the > > > > > > > > > > > > > > >> producer > > > > > > > > > > > > > > >> > > > > client and the admin client. However, > I > > am > > > > not > > > > > > > sure if > > > > > > > > > > > > that's > > > > > > > > > > > > > > the > > > > > > > > > > > > > > >> best > > > > > > > > > > > > > > >> > > > > approach. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > 2. Yes, refactoring existing requests > > is a > > > > > > > non-trivial > > > > > > > > > > > > amount of > > > > > > > > > > > > > > >> work. > > > > > > > > > > > > > > >> > > I > > > > > > > > > > > > > > >> > > > > posted some comments in KAFKA-1927. We > > > will > > > > > > > probably > > > > > > > > > > have > > > > > > > > > > > > to fix > > > > > > > > > > > > > > >> > > > KAFKA-1927 > > > > > > > > > > > > > > >> > > > > first, before adding the new logic in > > > > > > KAFKA-1694. > > > > > > > > > > > > Otherwise, the > > > > > > > > > > > > > > >> > > changes > > > > > > > > > > > > > > >> > > > > will be too big. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > 102. About the AdminClient: > > > > > > > > > > > > > > >> > > > > 102.1. It's a bit weird that we return > > > > > exception > > > > > > > in > > > > > > > > > the > > > > > > > > > > > > api. It > > > > > > > > > > > > > > >> seems > > > > > > > > > > > > > > >> > > > that > > > > > > > > > > > > > > >> > > > > we should either return error code or > > > throw > > > > an > > > > > > > > > exception > > > > > > > > > > > > when > > > > > > > > > > > > > > >> getting > > > > > > > > > > > > > > >> > > the > > > > > > > > > > > > > > >> > > > > response state. > > > > > > > > > > > > > > >> > > > > 102.2. We probably shouldn't > explicitly > > > use > > > > > the > > > > > > > > > request > > > > > > > > > > > > object > > > > > > > > > > > > > > in > > > > > > > > > > > > > > >> the > > > > > > > > > > > > > > >> > > > api. > > > > > > > > > > > > > > >> > > > > Not every request evolution requires > an > > > api > > > > > > > change. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > Thanks, > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > Jun > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > On Fri, Mar 13, 2015 at 4:08 AM, > Andrii > > > > > > Biletskyi > > > > > > > < > > > > > > > > > > > > > > >> > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > Jun, > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > Thanks for you comments. Answers > > inline: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > 100. There are a few fields such as > > > > > > > > > ReplicaAssignment, > > > > > > > > > > > > > > >> > > > > > > ReassignPartitionRequest, > > > > > > > > > > > > > > >> > > > > > > and PartitionsSerialized that are > > > > > > represented > > > > > > > as a > > > > > > > > > > > > string, > > > > > > > > > > > > > > but > > > > > > > > > > > > > > >> > > > contain > > > > > > > > > > > > > > >> > > > > > > composite structures in json. > Could > > we > > > > > > flatten > > > > > > > > > them > > > > > > > > > > > out > > > > > > > > > > > > > > >> directly in > > > > > > > > > > > > > > >> > > > the > > > > > > > > > > > > > > >> > > > > > > protocol definition as > > arrays/records? > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > Yes, now with Admin Client this > looks > > a > > > > bit > > > > > > > weird. > > > > > > > > > My > > > > > > > > > > > > initial > > > > > > > > > > > > > > >> > > > motivation > > > > > > > > > > > > > > >> > > > > > was: > > > > > > > > > > > > > > >> > > > > > ReassignPartitionCommand accepts > input > > > in > > > > > > json, > > > > > > > we > > > > > > > > > > want > > > > > > > > > > > to > > > > > > > > > > > > > > >> remain > > > > > > > > > > > > > > >> > > > tools' > > > > > > > > > > > > > > >> > > > > > interfaces unchanged, where > possible. > > > > > > > > > > > > > > >> > > > > > If we port it to deserialized > format, > > in > > > > CLI > > > > > > > (/tools > > > > > > > > > > > > project) > > > > > > > > > > > > > > >> we will > > > > > > > > > > > > > > >> > > > > have > > > > > > > > > > > > > > >> > > > > > to add some > > > > > > > > > > > > > > >> > > > > > json library since /tools is written > > in > > > > java > > > > > > and > > > > > > > > > we'll > > > > > > > > > > > > need to > > > > > > > > > > > > > > >> > > > > deserialize > > > > > > > > > > > > > > >> > > > > > json file > > > > > > > > > > > > > > >> > > > > > provided by a user. Can we quickly > > agree > > > > on > > > > > > what > > > > > > > > > this > > > > > > > > > > > > library > > > > > > > > > > > > > > >> should > > > > > > > > > > > > > > >> > > be > > > > > > > > > > > > > > >> > > > > > (Jackson, GSON, whatever)? > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > 101. Does TopicMetadataRequest v1 > > still > > > > > > trigger > > > > > > > auto > > > > > > > > > > > topic > > > > > > > > > > > > > > >> creation? > > > > > > > > > > > > > > >> > > > This > > > > > > > > > > > > > > >> > > > > > > will be a bit weird now that we > > have a > > > > > > > separate > > > > > > > > > > topic > > > > > > > > > > > > > > >> creation api. > > > > > > > > > > > > > > >> > > > > Have > > > > > > > > > > > > > > >> > > > > > > you thought about how the new > > > > > > > createTopicRequest > > > > > > > > > and > > > > > > > > > > > > > > >> > > > > TopicMetadataRequest > > > > > > > > > > > > > > >> > > > > > > v1 will be used in the > > > producer/consumer > > > > > > > client, > > > > > > > > > in > > > > > > > > > > > > addition > > > > > > > > > > > > > > >> to > > > > > > > > > > > > > > >> > > admin > > > > > > > > > > > > > > >> > > > > > > tools? For example, ideally, we > > don't > > > > want > > > > > > > > > > > > > > >> TopicMetadataRequest > > > > > > > > > > > > > > >> > > from > > > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > > > >> > > > > > > consumer to trigger auto topic > > > creation. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > I agree, this strange logic should > be > > > > fixed. > > > > > > > I'm not > > > > > > > > > > > > confident > > > > > > > > > > > > > > >> in > > > > > > > > > > > > > > >> > > this > > > > > > > > > > > > > > >> > > > > > Kafka part so > > > > > > > > > > > > > > >> > > > > > correct me if I'm wrong, but it > > doesn't > > > > look > > > > > > > like a > > > > > > > > > > hard > > > > > > > > > > > > thing > > > > > > > > > > > > > > >> to > > > > > > > > > > > > > > >> > > do, I > > > > > > > > > > > > > > >> > > > > > think we can > > > > > > > > > > > > > > >> > > > > > leverage AdminClient for that in > > > Producer > > > > > and > > > > > > > > > > > > unconditionally > > > > > > > > > > > > > > >> remove > > > > > > > > > > > > > > >> > > > > topic > > > > > > > > > > > > > > >> > > > > > creation from the > > > TopicMetadataRequest_V1. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > 2. I think Jay meant getting rid of > > > scala > > > > > > > classes > > > > > > > > > > > > > > >> > > > > > > like HeartbeatRequestAndHeader and > > > > > > > > > > > > > > >> HeartbeatResponseAndHeader. We > > > > > > > > > > > > > > >> > > did > > > > > > > > > > > > > > >> > > > > > that > > > > > > > > > > > > > > >> > > > > > > as a stop-gap thing when adding > the > > > new > > > > > > > requests > > > > > > > > > for > > > > > > > > > > > the > > > > > > > > > > > > > > >> consumers. > > > > > > > > > > > > > > >> > > > > > > However, the long term plan is to > > get > > > > rid > > > > > of > > > > > > > all > > > > > > > > > > those > > > > > > > > > > > > and > > > > > > > > > > > > > > >> just > > > > > > > > > > > > > > >> > > reuse > > > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > > > >> > > > > > > java request/response in the > client. > > > > Since > > > > > > > this > > > > > > > > > KIP > > > > > > > > > > > > proposes > > > > > > > > > > > > > > >> to > > > > > > > > > > > > > > >> > > add a > > > > > > > > > > > > > > >> > > > > > > significant number of new > requests, > > > > > perhaps > > > > > > we > > > > > > > > > > should > > > > > > > > > > > > bite > > > > > > > > > > > > > > the > > > > > > > > > > > > > > >> > > bullet > > > > > > > > > > > > > > >> > > > > to > > > > > > > > > > > > > > >> > > > > > > clean up the existing scala > requests > > > > first > > > > > > > before > > > > > > > > > > > > adding new > > > > > > > > > > > > > > >> ones? > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > Yes, looks like I misunderstood the > > > point > > > > of > > > > > > > > > > > > > > >> ...RequestAndHeader. > > > > > > > > > > > > > > >> > > > Okay, I > > > > > > > > > > > > > > >> > > > > > will > > > > > > > > > > > > > > >> > > > > > rework that. The only thing is that > I > > > > don't > > > > > > see > > > > > > > any > > > > > > > > > > > > example > > > > > > > > > > > > > > how > > > > > > > > > > > > > > >> it > > > > > > > > > > > > > > >> > > was > > > > > > > > > > > > > > >> > > > > done > > > > > > > > > > > > > > >> > > > > > for at > > > > > > > > > > > > > > >> > > > > > least one existing protocol message. > > > Thus, > > > > > as > > > > > > I > > > > > > > > > > > > understand, I > > > > > > > > > > > > > > >> have to > > > > > > > > > > > > > > >> > > > > think > > > > > > > > > > > > > > >> > > > > > how we > > > > > > > > > > > > > > >> > > > > > are going to do it. > > > > > > > > > > > > > > >> > > > > > Re porting all existing RQ/RP in > this > > > > patch. > > > > > > > Sounds > > > > > > > > > > > > > > reasonable, > > > > > > > > > > > > > > >> but > > > > > > > > > > > > > > >> > > if > > > > > > > > > > > > > > >> > > > > it's > > > > > > > > > > > > > > >> > > > > > an *obligatory* > > > > > > > > > > > > > > >> > > > > > requirement to have Admin KIP done, > > I'm > > > > > afraid > > > > > > > this > > > > > > > > > > can > > > > > > > > > > > > be a > > > > > > > > > > > > > > >> serious > > > > > > > > > > > > > > >> > > > > > blocker for us. > > > > > > > > > > > > > > >> > > > > > There are 13 protocol messages and > all > > > > that > > > > > > > would > > > > > > > > > > > require > > > > > > > > > > > > not > > > > > > > > > > > > > > >> only > > > > > > > > > > > > > > >> > > unit > > > > > > > > > > > > > > >> > > > > > tests but quite > > > > > > > > > > > > > > >> > > > > > intensive manual testing, no? I'm > > afraid > > > > I'm > > > > > > > not the > > > > > > > > > > > > right guy > > > > > > > > > > > > > > >> to > > > > > > > > > > > > > > >> > > cover > > > > > > > > > > > > > > >> > > > > > pretty much all > > > > > > > > > > > > > > >> > > > > > Kafka core internals :). Let me know > > > your > > > > > > > thoughts > > > > > > > > > on > > > > > > > > > > > this > > > > > > > > > > > > > > >> item. Btw > > > > > > > > > > > > > > >> > > > > there > > > > > > > > > > > > > > >> > > > > > is a ticket to > > > > > > > > > > > > > > >> > > > > > follow-up this issue ( > > > > > > > > > > > > > > >> > > > > > > https://issues.apache.org/jira/browse/KAFKA-2006 > > > > > > > > > > > > > > >> > > > ). > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > Thanks, > > > > > > > > > > > > > > >> > > > > > Andrii Biletskyi > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > On Fri, Mar 13, 2015 at 6:40 AM, Jun > > > Rao < > > > > > > > > > > > > j...@confluent.io> > > > > > > > > > > > > > > >> wrote: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > Andrii, > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > A few more comments. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > 100. There are a few fields such > as > > > > > > > > > > ReplicaAssignment, > > > > > > > > > > > > > > >> > > > > > > ReassignPartitionRequest, > > > > > > > > > > > > > > >> > > > > > > and PartitionsSerialized that are > > > > > > represented > > > > > > > as a > > > > > > > > > > > > string, > > > > > > > > > > > > > > but > > > > > > > > > > > > > > >> > > > contain > > > > > > > > > > > > > > >> > > > > > > composite structures in json. > Could > > we > > > > > > flatten > > > > > > > > > them > > > > > > > > > > > out > > > > > > > > > > > > > > >> directly in > > > > > > > > > > > > > > >> > > > the > > > > > > > > > > > > > > >> > > > > > > protocol definition as > > arrays/records? > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > 101. Does TopicMetadataRequest v1 > > > still > > > > > > > trigger > > > > > > > > > auto > > > > > > > > > > > > topic > > > > > > > > > > > > > > >> > > creation? > > > > > > > > > > > > > > >> > > > > This > > > > > > > > > > > > > > >> > > > > > > will be a bit weird now that we > > have a > > > > > > > separate > > > > > > > > > > topic > > > > > > > > > > > > > > >> creation api. > > > > > > > > > > > > > > >> > > > > Have > > > > > > > > > > > > > > >> > > > > > > you thought about how the new > > > > > > > createTopicRequest > > > > > > > > > and > > > > > > > > > > > > > > >> > > > > TopicMetadataRequest > > > > > > > > > > > > > > >> > > > > > > v1 will be used in the > > > producer/consumer > > > > > > > client, > > > > > > > > > in > > > > > > > > > > > > addition > > > > > > > > > > > > > > >> to > > > > > > > > > > > > > > >> > > admin > > > > > > > > > > > > > > >> > > > > > > tools? For example, ideally, we > > don't > > > > want > > > > > > > > > > > > > > >> TopicMetadataRequest > > > > > > > > > > > > > > >> > > from > > > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > > > >> > > > > > > consumer to trigger auto topic > > > creation. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > 2. I think Jay meant getting rid > of > > > > scala > > > > > > > classes > > > > > > > > > > > > > > >> > > > > > > like HeartbeatRequestAndHeader and > > > > > > > > > > > > > > >> HeartbeatResponseAndHeader. We > > > > > > > > > > > > > > >> > > did > > > > > > > > > > > > > > >> > > > > > that > > > > > > > > > > > > > > >> > > > > > > as a stop-gap thing when adding > the > > > new > > > > > > > requests > > > > > > > > > for > > > > > > > > > > > the > > > > > > > > > > > > > > >> consumers. > > > > > > > > > > > > > > >> > > > > > > However, the long term plan is to > > get > > > > rid > > > > > of > > > > > > > all > > > > > > > > > > those > > > > > > > > > > > > and > > > > > > > > > > > > > > >> just > > > > > > > > > > > > > > >> > > reuse > > > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > > > >> > > > > > > java request/response in the > client. > > > > Since > > > > > > > this > > > > > > > > > KIP > > > > > > > > > > > > proposes > > > > > > > > > > > > > > >> to > > > > > > > > > > > > > > >> > > add a > > > > > > > > > > > > > > >> > > > > > > significant number of new > requests, > > > > > perhaps > > > > > > we > > > > > > > > > > should > > > > > > > > > > > > bite > > > > > > > > > > > > > > the > > > > > > > > > > > > > > >> > > bullet > > > > > > > > > > > > > > >> > > > > to > > > > > > > > > > > > > > >> > > > > > > clean up the existing scala > requests > > > > first > > > > > > > before > > > > > > > > > > > > adding new > > > > > > > > > > > > > > >> ones? > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > Thanks, > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > Jun > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > On Thu, Mar 12, 2015 at 3:37 PM, > > > Andrii > > > > > > > Biletskyi > > > > > > > > > < > > > > > > > > > > > > > > >> > > > > > > andrii.bilets...@stealth.ly> > wrote: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > Hi, > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > As said above - I list again all > > > > > comments > > > > > > > from > > > > > > > > > > this > > > > > > > > > > > > thread > > > > > > > > > > > > > > >> so we > > > > > > > > > > > > > > >> > > > > > > > can see what's left and finalize > > all > > > > > > pending > > > > > > > > > > issues. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > Comments from Jay: > > > > > > > > > > > > > > >> > > > > > > > 1. This is much needed > > > functionality, > > > > > but > > > > > > > there > > > > > > > > > > are > > > > > > > > > > > a > > > > > > > > > > > > lot > > > > > > > > > > > > > > >> of the > > > > > > > > > > > > > > >> > > so > > > > > > > > > > > > > > >> > > > > > let's > > > > > > > > > > > > > > >> > > > > > > > really think these protocols > > > through. > > > > We > > > > > > > really > > > > > > > > > > want > > > > > > > > > > > > to > > > > > > > > > > > > > > end > > > > > > > > > > > > > > >> up > > > > > > > > > > > > > > >> > > > with a > > > > > > > > > > > > > > >> > > > > > set > > > > > > > > > > > > > > >> > > > > > > > of well thought-out, orthoganol > > > apis. > > > > > For > > > > > > > this > > > > > > > > > > > reason > > > > > > > > > > > > I > > > > > > > > > > > > > > >> think it > > > > > > > > > > > > > > >> > > is > > > > > > > > > > > > > > >> > > > > > > really > > > > > > > > > > > > > > >> > > > > > > > important to think through the > end > > > > state > > > > > > > even if > > > > > > > > > > > that > > > > > > > > > > > > > > >> includes > > > > > > > > > > > > > > >> > > APIs > > > > > > > > > > > > > > >> > > > > we > > > > > > > > > > > > > > >> > > > > > > > won't implement in the first > > phase. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Definitely behind this. Would > > > > > > appreciate > > > > > > > if > > > > > > > > > > there > > > > > > > > > > > > are > > > > > > > > > > > > > > >> concrete > > > > > > > > > > > > > > >> > > > > > > comments > > > > > > > > > > > > > > >> > > > > > > > how this can be improved. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 2. Let's please please please > wait > > > > until > > > > > > we > > > > > > > have > > > > > > > > > > > > switched > > > > > > > > > > > > > > >> the > > > > > > > > > > > > > > >> > > > server > > > > > > > > > > > > > > >> > > > > > over > > > > > > > > > > > > > > >> > > > > > > > to the new java protocol > > > definitions. > > > > If > > > > > > we > > > > > > > add > > > > > > > > > > > upteen > > > > > > > > > > > > > > more > > > > > > > > > > > > > > >> ad > > > > > > > > > > > > > > >> > > hoc > > > > > > > > > > > > > > >> > > > > > scala > > > > > > > > > > > > > > >> > > > > > > > objects that is just generating > > more > > > > > work > > > > > > > for > > > > > > > > > the > > > > > > > > > > > > > > >> conversion we > > > > > > > > > > > > > > >> > > > know > > > > > > > > > > > > > > >> > > > > we > > > > > > > > > > > > > > >> > > > > > > > have to do. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - > > > removed > > > > > > scala > > > > > > > > > > > protocol > > > > > > > > > > > > > > >> classes. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 3. This proposal introduces a > new > > > type > > > > > of > > > > > > > > > optional > > > > > > > > > > > > > > >> parameter. > > > > > > > > > > > > > > >> > > This > > > > > > > > > > > > > > >> > > > is > > > > > > > > > > > > > > >> > > > > > > > inconsistent with everything > else > > in > > > > the > > > > > > > > > protocol > > > > > > > > > > > > where we > > > > > > > > > > > > > > >> use -1 > > > > > > > > > > > > > > >> > > > or > > > > > > > > > > > > > > >> > > > > > some > > > > > > > > > > > > > > >> > > > > > > > other marker value. You could > > argue > > > > > either > > > > > > > way > > > > > > > > > but > > > > > > > > > > > > let's > > > > > > > > > > > > > > >> stick > > > > > > > > > > > > > > >> > > with > > > > > > > > > > > > > > >> > > > > > that > > > > > > > > > > > > > > >> > > > > > > > for consistency. For clients > that > > > > > > > implemented > > > > > > > > > the > > > > > > > > > > > > protocol > > > > > > > > > > > > > > >> in a > > > > > > > > > > > > > > >> > > > > better > > > > > > > > > > > > > > >> > > > > > > way > > > > > > > > > > > > > > >> > > > > > > > than our scala code these basic > > > > > primitives > > > > > > > are > > > > > > > > > > hard > > > > > > > > > > > to > > > > > > > > > > > > > > >> change. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - > > > removed > > > > > > > MaybeOf > > > > > > > > > > type > > > > > > > > > > > > and > > > > > > > > > > > > > > >> changed > > > > > > > > > > > > > > >> > > > > > protocol > > > > > > > > > > > > > > >> > > > > > > > accordingly. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 4. ClusterMetadata: This seems > to > > > > > > duplicate > > > > > > > > > > > > > > >> TopicMetadataRequest > > > > > > > > > > > > > > >> > > > > which > > > > > > > > > > > > > > >> > > > > > > has > > > > > > > > > > > > > > >> > > > > > > > brokers, topics, and > partitions. I > > > > think > > > > > > we > > > > > > > > > should > > > > > > > > > > > > rename > > > > > > > > > > > > > > >> that > > > > > > > > > > > > > > >> > > > > request > > > > > > > > > > > > > > >> > > > > > > > ClusterMetadataRequest (or just > > > > > > > MetadataRequest) > > > > > > > > > > and > > > > > > > > > > > > > > >> include the > > > > > > > > > > > > > > >> > > id > > > > > > > > > > > > > > >> > > > > of > > > > > > > > > > > > > > >> > > > > > > the > > > > > > > > > > > > > > >> > > > > > > > controller. Or are there other > > > things > > > > we > > > > > > > could > > > > > > > > > add > > > > > > > > > > > > here? > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: I agree. Updated the KIP. > Let's > > > > > extends > > > > > > > > > > > > TopicMetadata > > > > > > > > > > > > > > to > > > > > > > > > > > > > > >> > > > version 2 > > > > > > > > > > > > > > >> > > > > > and > > > > > > > > > > > > > > >> > > > > > > > include controller. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 5. We have a tendency to try to > > > make a > > > > > lot > > > > > > > of > > > > > > > > > > > requests > > > > > > > > > > > > > > that > > > > > > > > > > > > > > >> can > > > > > > > > > > > > > > >> > > > only > > > > > > > > > > > > > > >> > > > > go > > > > > > > > > > > > > > >> > > > > > > to > > > > > > > > > > > > > > >> > > > > > > > particular nodes. This adds a > lot > > of > > > > > > burden > > > > > > > for > > > > > > > > > > > client > > > > > > > > > > > > > > >> > > > > implementations > > > > > > > > > > > > > > >> > > > > > > (it > > > > > > > > > > > > > > >> > > > > > > > sounds easy but each discovery > can > > > > fail > > > > > in > > > > > > > many > > > > > > > > > > > parts > > > > > > > > > > > > so > > > > > > > > > > > > > > it > > > > > > > > > > > > > > >> ends > > > > > > > > > > > > > > >> > > up > > > > > > > > > > > > > > >> > > > > > > being a > > > > > > > > > > > > > > >> > > > > > > > full state machine to do > right). I > > > > think > > > > > > we > > > > > > > > > should > > > > > > > > > > > > > > consider > > > > > > > > > > > > > > >> > > making > > > > > > > > > > > > > > >> > > > > > admin > > > > > > > > > > > > > > >> > > > > > > > commands and ideally as many of > > the > > > > > other > > > > > > > apis > > > > > > > > > as > > > > > > > > > > > > possible > > > > > > > > > > > > > > >> > > > available > > > > > > > > > > > > > > >> > > > > on > > > > > > > > > > > > > > >> > > > > > > all > > > > > > > > > > > > > > >> > > > > > > > brokers and just redirect to the > > > > > > controller > > > > > > > on > > > > > > > > > the > > > > > > > > > > > > broker > > > > > > > > > > > > > > >> side. > > > > > > > > > > > > > > >> > > > > Perhaps > > > > > > > > > > > > > > >> > > > > > > > there would be a general way to > > > > > > encapsulate > > > > > > > this > > > > > > > > > > > > > > re-routing > > > > > > > > > > > > > > >> > > > behavior. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: It's a very interesting idea, > > but > > > > > seems > > > > > > > there > > > > > > > > > > are > > > > > > > > > > > > some > > > > > > > > > > > > > > >> > > concerns > > > > > > > > > > > > > > >> > > > > > about > > > > > > > > > > > > > > >> > > > > > > > this > > > > > > > > > > > > > > >> > > > > > > > feature (like performance > > > > > considerations, > > > > > > > how > > > > > > > > > this > > > > > > > > > > > > will > > > > > > > > > > > > > > >> > > complicate > > > > > > > > > > > > > > >> > > > > > server > > > > > > > > > > > > > > >> > > > > > > > etc). > > > > > > > > > > > > > > >> > > > > > > > I believe this shouldn't be a > > > blocker. > > > > > If > > > > > > > this > > > > > > > > > > > > feature is > > > > > > > > > > > > > > >> > > > implemented > > > > > > > > > > > > > > >> > > > > > at > > > > > > > > > > > > > > >> > > > > > > > some > > > > > > > > > > > > > > >> > > > > > > > point it won't affect Admin > > changes > > > - > > > > at > > > > > > > least > > > > > > > > > no > > > > > > > > > > > > changes > > > > > > > > > > > > > > to > > > > > > > > > > > > > > >> > > public > > > > > > > > > > > > > > >> > > > > API > > > > > > > > > > > > > > >> > > > > > > > will be required. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 6. We should probably normalize > > the > > > > key > > > > > > > value > > > > > > > > > > pairs > > > > > > > > > > > > used > > > > > > > > > > > > > > for > > > > > > > > > > > > > > >> > > > configs > > > > > > > > > > > > > > >> > > > > > > rather > > > > > > > > > > > > > > >> > > > > > > > than embedding a new formatting. > > So > > > > two > > > > > > > strings > > > > > > > > > > > rather > > > > > > > > > > > > > > than > > > > > > > > > > > > > > >> one > > > > > > > > > > > > > > >> > > > with > > > > > > > > > > > > > > >> > > > > an > > > > > > > > > > > > > > >> > > > > > > > internal equals sign. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - > > > > > normalized > > > > > > > > > configs > > > > > > > > > > > and > > > > > > > > > > > > > > >> changed > > > > > > > > > > > > > > >> > > > > protocol > > > > > > > > > > > > > > >> > > > > > > > accordingly. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 7. Is the postcondition of these > > > APIs > > > > > that > > > > > > > the > > > > > > > > > > > > command has > > > > > > > > > > > > > > >> begun > > > > > > > > > > > > > > >> > > or > > > > > > > > > > > > > > >> > > > > > that > > > > > > > > > > > > > > >> > > > > > > > the command has been completed? > It > > > is > > > > a > > > > > > lot > > > > > > > more > > > > > > > > > > > > usable if > > > > > > > > > > > > > > >> the > > > > > > > > > > > > > > >> > > > > command > > > > > > > > > > > > > > >> > > > > > > has > > > > > > > > > > > > > > >> > > > > > > > been completed so you know that > if > > > you > > > > > > > create a > > > > > > > > > > > topic > > > > > > > > > > > > and > > > > > > > > > > > > > > >> then > > > > > > > > > > > > > > >> > > > > publish > > > > > > > > > > > > > > >> > > > > > to > > > > > > > > > > > > > > >> > > > > > > > it you won't get an exception > > about > > > > > there > > > > > > > being > > > > > > > > > no > > > > > > > > > > > > such > > > > > > > > > > > > > > >> topic. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: For long running requests > (like > > > > > > reassign > > > > > > > > > > > > partitions) - > > > > > > > > > > > > > > >> the > > > > > > > > > > > > > > >> > > post > > > > > > > > > > > > > > >> > > > > > > > condition is > > > > > > > > > > > > > > >> > > > > > > > command has begun - so we don't > > > block > > > > > the > > > > > > > > > client. > > > > > > > > > > In > > > > > > > > > > > > case > > > > > > > > > > > > > > >> of your > > > > > > > > > > > > > > >> > > > > > > example - > > > > > > > > > > > > > > >> > > > > > > > topic commands, this will be > > > > refactored > > > > > > and > > > > > > > > > topic > > > > > > > > > > > > commands > > > > > > > > > > > > > > >> will > > > > > > > > > > > > > > >> > > be > > > > > > > > > > > > > > >> > > > > > > executed > > > > > > > > > > > > > > >> > > > > > > > immediately, since the > Controller > > > will > > > > > > serve > > > > > > > > > Admin > > > > > > > > > > > > > > requests > > > > > > > > > > > > > > >> > > > > > > > (follow-up ticket KAFKA-1777). > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 8. Describe topic and list > topics > > > > > > duplicate > > > > > > > a > > > > > > > > > lot > > > > > > > > > > of > > > > > > > > > > > > stuff > > > > > > > > > > > > > > >> in the > > > > > > > > > > > > > > >> > > > > > > metadata > > > > > > > > > > > > > > >> > > > > > > > request. Is there a reason to > give > > > > back > > > > > > > topics > > > > > > > > > > > marked > > > > > > > > > > > > for > > > > > > > > > > > > > > >> > > > deletion? I > > > > > > > > > > > > > > >> > > > > > > feel > > > > > > > > > > > > > > >> > > > > > > > like if we just make the > > > > post-condition > > > > > of > > > > > > > the > > > > > > > > > > > delete > > > > > > > > > > > > > > >> command be > > > > > > > > > > > > > > >> > > > that > > > > > > > > > > > > > > >> > > > > > the > > > > > > > > > > > > > > >> > > > > > > > topic is deleted that will get > rid > > > of > > > > > the > > > > > > > need > > > > > > > > > for > > > > > > > > > > > > this > > > > > > > > > > > > > > >> right? > > > > > > > > > > > > > > >> > > And > > > > > > > > > > > > > > >> > > > it > > > > > > > > > > > > > > >> > > > > > > will > > > > > > > > > > > > > > >> > > > > > > > be much more intuitive. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - > > > removed > > > > > > > topics > > > > > > > > > > marked > > > > > > > > > > > > for > > > > > > > > > > > > > > >> deletion > > > > > > > > > > > > > > >> > > > in > > > > > > > > > > > > > > >> > > > > > > > ListTopicsRequest. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 9. Should we consider batching > > these > > > > > > > requests? > > > > > > > > > We > > > > > > > > > > > have > > > > > > > > > > > > > > >> generally > > > > > > > > > > > > > > >> > > > > tried > > > > > > > > > > > > > > >> > > > > > to > > > > > > > > > > > > > > >> > > > > > > > allow multiple operations to be > > > > batched. > > > > > > My > > > > > > > > > > > suspicion > > > > > > > > > > > > is > > > > > > > > > > > > > > >> that > > > > > > > > > > > > > > >> > > > without > > > > > > > > > > > > > > >> > > > > > > this > > > > > > > > > > > > > > >> > > > > > > > we will get a lot of code that > > does > > > > > > > something > > > > > > > > > like > > > > > > > > > > > > > > >> > > > > > > > for(topic: > > > > adminClient.listTopics()) > > > > > > > > > > > > > > >> > > > > > > > > > > adminClient.describeTopic(topic) > > > > > > > > > > > > > > >> > > > > > > > this code will work great when > you > > > > test > > > > > > on 5 > > > > > > > > > > topics > > > > > > > > > > > > but > > > > > > > > > > > > > > not > > > > > > > > > > > > > > >> do as > > > > > > > > > > > > > > >> > > > > well > > > > > > > > > > > > > > >> > > > > > if > > > > > > > > > > > > > > >> > > > > > > > you have 50k. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please > check > > > > "Topic > > > > > > > Admin > > > > > > > > > > > Schema" > > > > > > > > > > > > > > >> section. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 10. I think we should also > discuss > > > how > > > > > we > > > > > > > want > > > > > > > > > to > > > > > > > > > > > > expose a > > > > > > > > > > > > > > >> > > > > programmatic > > > > > > > > > > > > > > >> > > > > > > JVM > > > > > > > > > > > > > > >> > > > > > > > client api for these operations. > > > > > Currently > > > > > > > > > people > > > > > > > > > > > > rely on > > > > > > > > > > > > > > >> > > > AdminUtils > > > > > > > > > > > > > > >> > > > > > > which > > > > > > > > > > > > > > >> > > > > > > > is totally sketchy. I think we > > > > probably > > > > > > need > > > > > > > > > > another > > > > > > > > > > > > > > client > > > > > > > > > > > > > > >> under > > > > > > > > > > > > > > >> > > > > > > clients/ > > > > > > > > > > > > > > >> > > > > > > > that exposes administrative > > > > > functionality. > > > > > > > We > > > > > > > > > will > > > > > > > > > > > > need > > > > > > > > > > > > > > >> this just > > > > > > > > > > > > > > >> > > > to > > > > > > > > > > > > > > >> > > > > > > > properly test the new apis, I > > > suspect. > > > > > We > > > > > > > should > > > > > > > > > > > > figure > > > > > > > > > > > > > > out > > > > > > > > > > > > > > >> that > > > > > > > > > > > > > > >> > > > API. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please > check > > > > "Admin > > > > > > > Client" > > > > > > > > > > > > section > > > > > > > > > > > > > > >> with an > > > > > > > > > > > > > > >> > > > > > initial > > > > > > > > > > > > > > >> > > > > > > > API proposal. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 11. The other information that > > would > > > > be > > > > > > > really > > > > > > > > > > > useful > > > > > > > > > > > > to > > > > > > > > > > > > > > get > > > > > > > > > > > > > > >> > > would > > > > > > > > > > > > > > >> > > > be > > > > > > > > > > > > > > >> > > > > > > > information about > partitions--how > > > much > > > > > > data > > > > > > > is > > > > > > > > > in > > > > > > > > > > > the > > > > > > > > > > > > > > >> partition, > > > > > > > > > > > > > > >> > > > what > > > > > > > > > > > > > > >> > > > > > are > > > > > > > > > > > > > > >> > > > > > > > the segment offsets, what is the > > > > log-end > > > > > > > offset > > > > > > > > > > > (i.e. > > > > > > > > > > > > last > > > > > > > > > > > > > > >> > > offset), > > > > > > > > > > > > > > >> > > > > > what > > > > > > > > > > > > > > >> > > > > > > is > > > > > > > > > > > > > > >> > > > > > > > the compaction point, etc. I > think > > > > that > > > > > > done > > > > > > > > > right > > > > > > > > > > > > this > > > > > > > > > > > > > > >> would be > > > > > > > > > > > > > > >> > > > the > > > > > > > > > > > > > > >> > > > > > > > successor to the very awkward > > > > > > OffsetRequest > > > > > > > we > > > > > > > > > > have > > > > > > > > > > > > today. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: I removed > > > > ConsumerGroupOffsetsRequest > > > > > > in > > > > > > > the > > > > > > > > > > > latest > > > > > > > > > > > > > > >> patch. I > > > > > > > > > > > > > > >> > > > > believe > > > > > > > > > > > > > > >> > > > > > > > this should > > > > > > > > > > > > > > >> > > > > > > > be resolved in a separate KIP / > > jira > > > > > > ticket. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 12. Generally we can do good > error > > > > > > handling > > > > > > > > > > without > > > > > > > > > > > > > > needing > > > > > > > > > > > > > > >> > > custom > > > > > > > > > > > > > > >> > > > > > > > server-side > > > > > > > > > > > > > > >> > > > > > > > messages. I.e. generally the > > client > > > > has > > > > > > the > > > > > > > > > > context > > > > > > > > > > > to > > > > > > > > > > > > > > know > > > > > > > > > > > > > > >> that > > > > > > > > > > > > > > >> > > if > > > > > > > > > > > > > > >> > > > > it > > > > > > > > > > > > > > >> > > > > > > got > > > > > > > > > > > > > > >> > > > > > > > an error that the topic doesn't > > > exist > > > > to > > > > > > say > > > > > > > > > > "Topic > > > > > > > > > > > X > > > > > > > > > > > > > > >> doesn't > > > > > > > > > > > > > > >> > > > exist" > > > > > > > > > > > > > > >> > > > > > > rather > > > > > > > > > > > > > > >> > > > > > > > than "error code 14" (or > > whatever). > > > > > Maybe > > > > > > > there > > > > > > > > > > are > > > > > > > > > > > > > > specific > > > > > > > > > > > > > > >> > > cases > > > > > > > > > > > > > > >> > > > > > where > > > > > > > > > > > > > > >> > > > > > > > this is hard? If we want to add > > > > > > server-side > > > > > > > > > error > > > > > > > > > > > > messages > > > > > > > > > > > > > > >> we > > > > > > > > > > > > > > >> > > > really > > > > > > > > > > > > > > >> > > > > do > > > > > > > > > > > > > > >> > > > > > > > need to do this in a consistent > > way > > > > > across > > > > > > > the > > > > > > > > > > > > protocol. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please > check > > > > > > "Protocol > > > > > > > > > > Errors" > > > > > > > > > > > > > > >> section. I > > > > > > > > > > > > > > >> > > > added > > > > > > > > > > > > > > >> > > > > > the > > > > > > > > > > > > > > >> > > > > > > > comprehensive, fine-grained list > > of > > > > > error > > > > > > > codes. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > Comments from Guozhang: > > > > > > > > > > > > > > >> > > > > > > > 13. Describe topic request: it > > would > > > > be > > > > > > > great to > > > > > > > > > > go > > > > > > > > > > > > beyond > > > > > > > > > > > > > > >> just > > > > > > > > > > > > > > >> > > > > > batching > > > > > > > > > > > > > > >> > > > > > > on > > > > > > > > > > > > > > >> > > > > > > > topic name regex for this > request. > > > For > > > > > > > example, > > > > > > > > > a > > > > > > > > > > > very > > > > > > > > > > > > > > >> common use > > > > > > > > > > > > > > >> > > > > case > > > > > > > > > > > > > > >> > > > > > of > > > > > > > > > > > > > > >> > > > > > > > the topic command is to list all > > > > topics > > > > > > > whose > > > > > > > > > > config > > > > > > > > > > > > A's > > > > > > > > > > > > > > >> value is > > > > > > > > > > > > > > >> > > > B. > > > > > > > > > > > > > > >> > > > > > With > > > > > > > > > > > > > > >> > > > > > > > topic name regex then we have to > > > first > > > > > > > retrieve > > > > > > > > > > > > __all__ > > > > > > > > > > > > > > >> topics's > > > > > > > > > > > > > > >> > > > > > > > description info and then filter > > at > > > > the > > > > > > > client > > > > > > > > > > end, > > > > > > > > > > > > which > > > > > > > > > > > > > > >> will > > > > > > > > > > > > > > >> > > be a > > > > > > > > > > > > > > >> > > > > > huge > > > > > > > > > > > > > > >> > > > > > > > burden on ZK. > > > > > > > > > > > > > > >> > > > > > > > AND > > > > > > > > > > > > > > >> > > > > > > > 14. Config K-Vs in create topic: > > > this > > > > is > > > > > > > related > > > > > > > > > > to > > > > > > > > > > > > the > > > > > > > > > > > > > > >> previous > > > > > > > > > > > > > > >> > > > > point; > > > > > > > > > > > > > > >> > > > > > > > maybe we can add another > metadata > > > K-V > > > > or > > > > > > > just a > > > > > > > > > > > > metadata > > > > > > > > > > > > > > >> string > > > > > > > > > > > > > > >> > > > along > > > > > > > > > > > > > > >> > > > > > > side > > > > > > > > > > > > > > >> > > > > > > > with config K-V in create topic > > like > > > > we > > > > > > did > > > > > > > for > > > > > > > > > > > offset > > > > > > > > > > > > > > >> commit > > > > > > > > > > > > > > >> > > > > request. > > > > > > > > > > > > > > >> > > > > > > This > > > > > > > > > > > > > > >> > > > > > > > field can be quite useful in > > storing > > > > > > > information > > > > > > > > > > > like > > > > > > > > > > > > > > >> "owner" of > > > > > > > > > > > > > > >> > > > the > > > > > > > > > > > > > > >> > > > > > > topic > > > > > > > > > > > > > > >> > > > > > > > who issue the create command, > etc, > > > > which > > > > > > is > > > > > > > > > quite > > > > > > > > > > > > > > important > > > > > > > > > > > > > > >> for a > > > > > > > > > > > > > > >> > > > > > > > multi-tenant setting. Then in > the > > > > > describe > > > > > > > topic > > > > > > > > > > > > request > > > > > > > > > > > > > > we > > > > > > > > > > > > > > >> can > > > > > > > > > > > > > > >> > > > also > > > > > > > > > > > > > > >> > > > > > > batch > > > > > > > > > > > > > > >> > > > > > > > on regex of the metadata field. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: As discussed it is very > > > interesting > > > > > but > > > > > > > can > > > > > > > > > be > > > > > > > > > > > > > > >> implemented > > > > > > > > > > > > > > >> > > later > > > > > > > > > > > > > > >> > > > > > after > > > > > > > > > > > > > > >> > > > > > > > we have some basic functionality > > > > there. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 15. Today all the admin > operations > > > are > > > > > > > async in > > > > > > > > > > the > > > > > > > > > > > > sense > > > > > > > > > > > > > > >> that > > > > > > > > > > > > > > >> > > > > command > > > > > > > > > > > > > > >> > > > > > > will > > > > > > > > > > > > > > >> > > > > > > > return once it is written in ZK, > > and > > > > > that > > > > > > > is why > > > > > > > > > > we > > > > > > > > > > > > need > > > > > > > > > > > > > > >> extra > > > > > > > > > > > > > > >> > > > > > > verification > > > > > > > > > > > > > > >> > > > > > > > like > > testUtil.waitForTopicCreated() > > > / > > > > > > verify > > > > > > > > > > > partition > > > > > > > > > > > > > > >> > > reassignment > > > > > > > > > > > > > > >> > > > > > > > request, etc. With admin > requests > > we > > > > > could > > > > > > > add a > > > > > > > > > > > flag > > > > > > > > > > > > to > > > > > > > > > > > > > > >> enable / > > > > > > > > > > > > > > >> > > > > > disable > > > > > > > > > > > > > > >> > > > > > > > synchronous requests; when it is > > > > turned > > > > > > on, > > > > > > > the > > > > > > > > > > > > response > > > > > > > > > > > > > > >> will not > > > > > > > > > > > > > > >> > > > > > return > > > > > > > > > > > > > > >> > > > > > > > until the request has been > > > completed. > > > > > And > > > > > > > for > > > > > > > > > > async > > > > > > > > > > > > > > >> requests we > > > > > > > > > > > > > > >> > > can > > > > > > > > > > > > > > >> > > > > > add a > > > > > > > > > > > > > > >> > > > > > > > "token" field in the response, > and > > > > then > > > > > > only > > > > > > > > > need > > > > > > > > > > a > > > > > > > > > > > > > > general > > > > > > > > > > > > > > >> > > "admin > > > > > > > > > > > > > > >> > > > > > > > verification request" with the > > given > > > > > token > > > > > > > to > > > > > > > > > > check > > > > > > > > > > > > if the > > > > > > > > > > > > > > >> async > > > > > > > > > > > > > > >> > > > > > request > > > > > > > > > > > > > > >> > > > > > > > has been completed. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: I see your point. My idea was > > to > > > > > > provide > > > > > > > > > > specific > > > > > > > > > > > > > > >> > > > Verify...Request > > > > > > > > > > > > > > >> > > > > > per > > > > > > > > > > > > > > >> > > > > > > > each > > > > > > > > > > > > > > >> > > > > > > > long running request, where > > needed. > > > We > > > > > can > > > > > > > do it > > > > > > > > > > the > > > > > > > > > > > > way > > > > > > > > > > > > > > you > > > > > > > > > > > > > > >> > > > suggest. > > > > > > > > > > > > > > >> > > > > > The > > > > > > > > > > > > > > >> > > > > > > > only > > > > > > > > > > > > > > >> > > > > > > > concern is that introducing a > > token > > > we > > > > > > again > > > > > > > > > will > > > > > > > > > > > make > > > > > > > > > > > > > > >> schema > > > > > > > > > > > > > > >> > > > > > "dynamic". > > > > > > > > > > > > > > >> > > > > > > We > > > > > > > > > > > > > > >> > > > > > > > wanted > > > > > > > > > > > > > > >> > > > > > > > to do similar thing introducing > > > single > > > > > > > > > > AdminRequest > > > > > > > > > > > > for > > > > > > > > > > > > > > all > > > > > > > > > > > > > > >> topic > > > > > > > > > > > > > > >> > > > > > > commands > > > > > > > > > > > > > > >> > > > > > > > but rejected > > > > > > > > > > > > > > >> > > > > > > > this idea because we wanted to > > have > > > > > schema > > > > > > > > > > defined. > > > > > > > > > > > So > > > > > > > > > > > > > > this > > > > > > > > > > > > > > >> is > > > > > > > > > > > > > > >> > > > more a > > > > > > > > > > > > > > >> > > > > > > > choice between: > > > > > > > > > > > > > > >> > > > > > > > a) have fixed schema but > introduce > > > > each > > > > > > > time new > > > > > > > > > > > > > > >> Verify...Request > > > > > > > > > > > > > > >> > > > for > > > > > > > > > > > > > > >> > > > > > > > long-running requests > > > > > > > > > > > > > > >> > > > > > > > b) use one request for > > verification > > > > but > > > > > > > > > generalize > > > > > > > > > > > it > > > > > > > > > > > > with > > > > > > > > > > > > > > >> token > > > > > > > > > > > > > > >> > > > > > > > I'm fine with whatever decision > > > > > community > > > > > > > come > > > > > > > > > to. > > > > > > > > > > > > Just > > > > > > > > > > > > > > let > > > > > > > > > > > > > > >> me > > > > > > > > > > > > > > >> > > know > > > > > > > > > > > > > > >> > > > > > your > > > > > > > > > > > > > > >> > > > > > > > thoughts. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > Comment from Gwen: > > > > > > > > > > > > > > >> > > > > > > > 16. Specifically for ownership, > I > > > > think > > > > > > the > > > > > > > plan > > > > > > > > > > is > > > > > > > > > > > > to add > > > > > > > > > > > > > > >> ACL > > > > > > > > > > > > > > >> > > (it > > > > > > > > > > > > > > >> > > > > > sounds > > > > > > > > > > > > > > >> > > > > > > > like you are describing ACL) via > > an > > > > > > external > > > > > > > > > > system > > > > > > > > > > > > > > (Argus, > > > > > > > > > > > > > > >> > > > Sentry). > > > > > > > > > > > > > > >> > > > > > > > I remember KIP-11 described > this, > > > but > > > > I > > > > > > > can't > > > > > > > > > find > > > > > > > > > > > > the KIP > > > > > > > > > > > > > > >> any > > > > > > > > > > > > > > >> > > > > longer. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Okay, no problem. Not sure > > though > > > > how > > > > > > we > > > > > > > are > > > > > > > > > > > going > > > > > > > > > > > > to > > > > > > > > > > > > > > >> handle > > > > > > > > > > > > > > >> > > it. > > > > > > > > > > > > > > >> > > > > > Wait > > > > > > > > > > > > > > >> > > > > > > > which KIP > > > > > > > > > > > > > > >> > > > > > > > will be committed first and > > include > > > > > > changes > > > > > > > to > > > > > > > > > > > > > > >> TopicMetadata from > > > > > > > > > > > > > > >> > > > the > > > > > > > > > > > > > > >> > > > > > > later > > > > > > > > > > > > > > >> > > > > > > > one? > > > > > > > > > > > > > > >> > > > > > > > Anyway, I added this note to > "Open > > > > > > > Questions" > > > > > > > > > > > section > > > > > > > > > > > > so > > > > > > > > > > > > > > we > > > > > > > > > > > > > > >> don't > > > > > > > > > > > > > > >> > > > > miss > > > > > > > > > > > > > > >> > > > > > > this > > > > > > > > > > > > > > >> > > > > > > > piece. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > Thanks, > > > > > > > > > > > > > > >> > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > On Fri, Mar 13, 2015 at 12:34 > AM, > > > > Andrii > > > > > > > > > > Biletskyi < > > > > > > > > > > > > > > >> > > > > > > > andrii.bilets...@stealth.ly> > > wrote: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > Hi all, > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > Today I uploaded the patch > that > > > > covers > > > > > > > some of > > > > > > > > > > the > > > > > > > > > > > > > > >> discussed > > > > > > > > > > > > > > >> > > and > > > > > > > > > > > > > > >> > > > > > agreed > > > > > > > > > > > > > > >> > > > > > > > > items: > > > > > > > > > > > > > > >> > > > > > > > > - removed MaybeOf optional > type > > > > > > > > > > > > > > >> > > > > > > > > - switched to java protocol > > > > > definitions > > > > > > > > > > > > > > >> > > > > > > > > - simplified messages > > (normalized > > > > > > configs, > > > > > > > > > > removed > > > > > > > > > > > > topic > > > > > > > > > > > > > > >> marked > > > > > > > > > > > > > > >> > > > for > > > > > > > > > > > > > > >> > > > > > > > > deletion) > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > I also updated the KIP-4 with > > > > > respective > > > > > > > > > changes > > > > > > > > > > > and > > > > > > > > > > > > > > >> wrote down > > > > > > > > > > > > > > >> > > > my > > > > > > > > > > > > > > >> > > > > > > > > proposal for > > > > > > > > > > > > > > >> > > > > > > > > pending items: > > > > > > > > > > > > > > >> > > > > > > > > - Batch Admin Operations -> > > > updated > > > > > Wire > > > > > > > > > > Protocol > > > > > > > > > > > > schema > > > > > > > > > > > > > > >> > > proposal > > > > > > > > > > > > > > >> > > > > > > > > - Remove ClusterMetadata -> > > > changed > > > > to > > > > > > > extend > > > > > > > > > > > > > > >> > > > TopicMetadataRequest > > > > > > > > > > > > > > >> > > > > > > > > - Admin Client -> updated my > > > initial > > > > > > > proposal > > > > > > > > > to > > > > > > > > > > > > reflect > > > > > > > > > > > > > > >> > > batching > > > > > > > > > > > > > > >> > > > > > > > > - Error codes -> proposed > > > > fine-grained > > > > > > > error > > > > > > > > > > code > > > > > > > > > > > > > > instead > > > > > > > > > > > > > > >> of > > > > > > > > > > > > > > >> > > > > > > > > AdminRequestFailed > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > I will also send a separate > > email > > > to > > > > > > > cover all > > > > > > > > > > > > comments > > > > > > > > > > > > > > >> from > > > > > > > > > > > > > > >> > > this > > > > > > > > > > > > > > >> > > > > > > thread. > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > Thanks, > > > > > > > > > > > > > > >> > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > On Thu, Mar 12, 2015 at 9:26 > PM, > > > > Gwen > > > > > > > Shapira > > > > > > > > > < > > > > > > > > > > > > > > >> > > > > gshap...@cloudera.com > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > wrote: > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> Found KIP-11 ( > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface > > > > > > > > > > > > > > >> > > > > > > > >> ) > > > > > > > > > > > > > > >> > > > > > > > >> It actually specifies changes > > to > > > > the > > > > > > > Metadata > > > > > > > > > > > > protocol, > > > > > > > > > > > > > > >> so > > > > > > > > > > > > > > >> > > > making > > > > > > > > > > > > > > >> > > > > > sure > > > > > > > > > > > > > > >> > > > > > > > >> both KIPs are consistent in > > this > > > > > regard > > > > > > > will > > > > > > > > > be > > > > > > > > > > > > good. > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > >> On Thu, Mar 12, 2015 at 12:21 > > PM, > > > > > Gwen > > > > > > > > > Shapira > > > > > > > > > > < > > > > > > > > > > > > > > >> > > > > > gshap...@cloudera.com > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > > > > > > >> > > > > > > > >> > Specifically for > ownership, I > > > > think > > > > > > the > > > > > > > > > plan > > > > > > > > > > is > > > > > > > > > > > > to > > > > > > > > > > > > > > add > > > > > > > > > > > > > > >> ACL > > > > > > > > > > > > > > >> > > (it > > > > > > > > > > > > > > >> > > > > > > sounds > > > > > > > > > > > > > > >> > > > > > > > >> > like you are describing > ACL) > > > via > > > > an > > > > > > > > > external > > > > > > > > > > > > system > > > > > > > > > > > > > > >> (Argus, > > > > > > > > > > > > > > >> > > > > > Sentry). > > > > > > > > > > > > > > >> > > > > > > > >> > I remember KIP-11 described > > > this, > > > > > > but I > > > > > > > > > can't > > > > > > > > > > > > find > > > > > > > > > > > > > > the > > > > > > > > > > > > > > >> KIP > > > > > > > > > > > > > > >> > > any > > > > > > > > > > > > > > >> > > > > > > longer. > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > >> > Regardless, I think KIP-4 > > > focuses > > > > > on > > > > > > > > > getting > > > > > > > > > > > > > > >> information > > > > > > > > > > > > > > >> > > that > > > > > > > > > > > > > > >> > > > > > > already > > > > > > > > > > > > > > >> > > > > > > > >> > exists from Kafka brokers, > > not > > > on > > > > > > > adding > > > > > > > > > > > > information > > > > > > > > > > > > > > >> that > > > > > > > > > > > > > > >> > > > > perhaps > > > > > > > > > > > > > > >> > > > > > > > >> > should exist but doesn't > yet? > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > >> > Gwen > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > >> > On Thu, Mar 12, 2015 at > 6:37 > > > AM, > > > > > > > Guozhang > > > > > > > > > > Wang > > > > > > > > > > > < > > > > > > > > > > > > > > >> > > > > > wangg...@gmail.com> > > > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > > > > > > >> > > > > > > > >> >> Folks, > > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > > >> > > > > > > > >> >> Just want to elaborate a > bit > > > > more > > > > > on > > > > > > > the > > > > > > > > > > > > > > create-topic > > > > > > > > > > > > > > >> > > > metadata > > > > > > > > > > > > > > >> > > > > > and > > > > > > > > > > > > > > >> > > > > > > > >> batching > > > > > > > > > > > > > > >> > > > > > > > >> >> describe-topic based on > > > config / > > > > > > > metadata > > > > > > > > > in > > > > > > > > > > > my > > > > > > > > > > > > > > >> previous > > > > > > > > > > > > > > >> > > > email > > > > > > > > > > > > > > >> > > > > as > > > > > > > > > > > > > > >> > > > > > > we > > > > > > > > > > > > > > >> > > > > > > > >> work > > > > > > > > > > > > > > >> > > > > > > > >> >> on KAFKA-1694. The main > > > > motivation > > > > > > is > > > > > > > to > > > > > > > > > > have > > > > > > > > > > > > some > > > > > > > > > > > > > > >> sort of > > > > > > > > > > > > > > >> > > > > topic > > > > > > > > > > > > > > >> > > > > > > > >> management > > > > > > > > > > > > > > >> > > > > > > > >> >> mechanisms, which I think > is > > > > quite > > > > > > > > > important > > > > > > > > > > > in > > > > > > > > > > > > a > > > > > > > > > > > > > > >> > > > multi-tenant > > > > > > > > > > > > > > >> > > > > / > > > > > > > > > > > > > > >> > > > > > > > cloud > > > > > > > > > > > > > > >> > > > > > > > >> >> architecture: today anyone > > can > > > > > > create > > > > > > > > > topics > > > > > > > > > > > in > > > > > > > > > > > > a > > > > > > > > > > > > > > >> shared > > > > > > > > > > > > > > >> > > > Kafka > > > > > > > > > > > > > > >> > > > > > > > >> cluster, but > > > > > > > > > > > > > > >> > > > > > > > >> >> there is no concept or > > > > "ownership" > > > > > > of > > > > > > > > > topics > > > > > > > > > > > > that > > > > > > > > > > > > > > are > > > > > > > > > > > > > > >> > > created > > > > > > > > > > > > > > >> > > > > by > > > > > > > > > > > > > > >> > > > > > > > >> different > > > > > > > > > > > > > > >> > > > > > > > >> >> users. For example, at > > > LinkedIn > > > > we > > > > > > > > > basically > > > > > > > > > > > > > > >> distinguish > > > > > > > > > > > > > > >> > > > topic > > > > > > > > > > > > > > >> > > > > > > owners > > > > > > > > > > > > > > >> > > > > > > > >> via > > > > > > > > > > > > > > >> > > > > > > > >> >> some casual topic name > > prefix, > > > > > which > > > > > > > is a > > > > > > > > > > bit > > > > > > > > > > > > > > awkward > > > > > > > > > > > > > > >> and > > > > > > > > > > > > > > >> > > > does > > > > > > > > > > > > > > >> > > > > > not > > > > > > > > > > > > > > >> > > > > > > > fly > > > > > > > > > > > > > > >> > > > > > > > >> as > > > > > > > > > > > > > > >> > > > > > > > >> >> we scale our customers. It > > > would > > > > > be > > > > > > > great > > > > > > > > > to > > > > > > > > > > > use > > > > > > > > > > > > > > >> > > > > describe-topics > > > > > > > > > > > > > > >> > > > > > > such > > > > > > > > > > > > > > >> > > > > > > > >> as: > > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > > >> > > > > > > > >> >> Describe all topics that > is > > > > > created > > > > > > > by me. > > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > > >> > > > > > > > >> >> Describe all topics whose > > > > > retention > > > > > > > time > > > > > > > > > is > > > > > > > > > > > > > > overriden > > > > > > > > > > > > > > >> to X. > > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > > >> > > > > > > > >> >> Describe all topics whose > > > > writable > > > > > > > group > > > > > > > > > > > include > > > > > > > > > > > > > > user > > > > > > > > > > > > > > >> Y > > > > > > > > > > > > > > >> > > (this > > > > > > > > > > > > > > >> > > > > is > > > > > > > > > > > > > > >> > > > > > > > >> related to > > > > > > > > > > > > > > >> > > > > > > > >> >> authorization), etc.. > > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > > >> > > > > > > > >> >> One possible way to > achieve > > > this > > > > > is > > > > > > to > > > > > > > > > add a > > > > > > > > > > > > > > metadata > > > > > > > > > > > > > > >> file > > > > > > > > > > > > > > >> > > in > > > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > > > >> > > > > > > > >> >> create-topic request, > whose > > > > value > > > > > > will > > > > > > > > > also > > > > > > > > > > be > > > > > > > > > > > > > > >> written ZK > > > > > > > > > > > > > > >> > > as > > > > > > > > > > > > > > >> > > > we > > > > > > > > > > > > > > >> > > > > > > > create > > > > > > > > > > > > > > >> > > > > > > > >> the > > > > > > > > > > > > > > >> > > > > > > > >> >> topic; then > describe-topics > > > can > > > > > > > choose to > > > > > > > > > > > batch > > > > > > > > > > > > > > topics > > > > > > > > > > > > > > >> > > based > > > > > > > > > > > > > > >> > > > on > > > > > > > > > > > > > > >> > > > > > 1) > > > > > > > > > > > > > > >> > > > > > > > name > > > > > > > > > > > > > > >> > > > > > > > >> >> regex, 2) config K-V > > matching, > > > > 3) > > > > > > > metadata > > > > > > > > > > > > regex, > > > > > > > > > > > > > > etc. > > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > > >> > > > > > > > >> >> Thoughts? > > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > > >> > > > > > > > >> >> Guozhang > > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > > >> > > > > > > > >> >> On Thu, Mar 5, 2015 at > 4:37 > > > PM, > > > > > > > Guozhang > > > > > > > > > > Wang > > > > > > > > > > > < > > > > > > > > > > > > > > >> > > > > > wangg...@gmail.com> > > > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > > >> > > > > > > > >> >>> Thanks for the updated > > wiki. > > > A > > > > > few > > > > > > > > > comments > > > > > > > > > > > > below: > > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > > >> > > > > > > > >> >>> 1. Error description in > > > > > response: I > > > > > > > think > > > > > > > > > > if > > > > > > > > > > > > some > > > > > > > > > > > > > > >> > > errorCode > > > > > > > > > > > > > > >> > > > > > could > > > > > > > > > > > > > > >> > > > > > > > >> indicate > > > > > > > > > > > > > > >> > > > > > > > >> >>> several different error > > cases > > > > > then > > > > > > we > > > > > > > > > > should > > > > > > > > > > > > really > > > > > > > > > > > > > > >> change > > > > > > > > > > > > > > >> > > > it > > > > > > > > > > > > > > >> > > > > to > > > > > > > > > > > > > > >> > > > > > > > >> multiple > > > > > > > > > > > > > > >> > > > > > > > >> >>> codes. In general the > > > errorCode > > > > > > > itself > > > > > > > > > > would > > > > > > > > > > > be > > > > > > > > > > > > > > >> precise > > > > > > > > > > > > > > >> > > and > > > > > > > > > > > > > > >> > > > > > > > >> sufficient for > > > > > > > > > > > > > > >> > > > > > > > >> >>> describing the server > side > > > > > errors. > > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > > >> > > > > > > > >> >>> 2. Describe topic > request: > > it > > > > > would > > > > > > > be > > > > > > > > > > great > > > > > > > > > > > > to go > > > > > > > > > > > > > > >> beyond > > > > > > > > > > > > > > >> > > > just > > > > > > > > > > > > > > >> > > > > > > > >> batching on > > > > > > > > > > > > > > >> > > > > > > > >> >>> topic name regex for this > > > > > request. > > > > > > > For > > > > > > > > > > > > example, a > > > > > > > > > > > > > > >> very > > > > > > > > > > > > > > >> > > > common > > > > > > > > > > > > > > >> > > > > > use > > > > > > > > > > > > > > >> > > > > > > > >> case of > > > > > > > > > > > > > > >> > > > > > > > >> >>> the topic command is to > > list > > > > all > > > > > > > topics > > > > > > > > > > whose > > > > > > > > > > > > > > config > > > > > > > > > > > > > > >> A's > > > > > > > > > > > > > > >> > > > value > > > > > > > > > > > > > > >> > > > > > is > > > > > > > > > > > > > > >> > > > > > > B. > > > > > > > > > > > > > > >> > > > > > > > >> With > > > > > > > > > > > > > > >> > > > > > > > >> >>> topic name regex then we > > have > > > > to > > > > > > > first > > > > > > > > > > > retrieve > > > > > > > > > > > > > > >> __all__ > > > > > > > > > > > > > > >> > > > > topics's > > > > > > > > > > > > > > >> > > > > > > > >> >>> description info and then > > > > filter > > > > > at > > > > > > > the > > > > > > > > > > > client > > > > > > > > > > > > end, > > > > > > > > > > > > > > >> which > > > > > > > > > > > > > > >> > > > will > > > > > > > > > > > > > > >> > > > > > be > > > > > > > > > > > > > > >> > > > > > > a > > > > > > > > > > > > > > >> > > > > > > > >> huge > > > > > > > > > > > > > > >> > > > > > > > >> >>> burden on ZK. > > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > > >> > > > > > > > >> >>> 3. Config K-Vs in create > > > topic: > > > > > > this > > > > > > > is > > > > > > > > > > > > related to > > > > > > > > > > > > > > >> the > > > > > > > > > > > > > > >> > > > > previous > > > > > > > > > > > > > > >> > > > > > > > point; > > > > > > > > > > > > > > >> > > > > > > > >> >>> maybe we can add another > > > > metadata > > > > > > > K-V or > > > > > > > > > > > just a > > > > > > > > > > > > > > >> metadata > > > > > > > > > > > > > > >> > > > > string > > > > > > > > > > > > > > >> > > > > > > > along > > > > > > > > > > > > > > >> > > > > > > > >> side > > > > > > > > > > > > > > >> > > > > > > > >> >>> with config K-V in create > > > topic > > > > > > like > > > > > > > we > > > > > > > > > did > > > > > > > > > > > for > > > > > > > > > > > > > > >> offset > > > > > > > > > > > > > > >> > > > commit > > > > > > > > > > > > > > >> > > > > > > > >> request. This > > > > > > > > > > > > > > >> > > > > > > > >> >>> field can be quite useful > > in > > > > > > storing > > > > > > > > > > > > information > > > > > > > > > > > > > > like > > > > > > > > > > > > > > >> > > > "owner" > > > > > > > > > > > > > > >> > > > > of > > > > > > > > > > > > > > >> > > > > > > the > > > > > > > > > > > > > > >> > > > > > > > >> topic > > > > > > > > > > > > > > >> > > > > > > > >> >>> who issue the create > > command, > > > > > etc, > > > > > > > which > > > > > > > > > is > > > > > > > > > > > > quite > > > > > > > > > > > > > > >> > > important > > > > > > > > > > > > > > >> > > > > for > > > > > > > > > > > > > > >> > > > > > a > > > > > > > > > > > > > > >> > > > > > > > >> >>> multi-tenant setting. > Then > > in > > > > the > > > > > > > > > describe > > > > > > > > > > > > topic > > > > > > > > > > > > > > >> request > > > > > > > > > > > > > > >> > > we > > > > > > > > > > > > > > >> > > > > can > > > > > > > > > > > > > > >> > > > > > > also > > > > > > > > > > > > > > >> > > > > > > > >> batch > > > > > > > > > > > > > > >> > > > > > > > >> >>> on regex of the metadata > > > field. > > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > > >> > > > > > > > >> >>> 4. Today all the admin > > > > operations > > > > > > are > > > > > > > > > async > > > > > > > > > > > in > > > > > > > > > > > > the > > > > > > > > > > > > > > >> sense > > > > > > > > > > > > > > >> > > > that > > > > > > > > > > > > > > >> > > > > > > > command > > > > > > > > > > > > > > >> > > > > > > > >> will > > > > > > > > > > > > > > >> > > > > > > > >> >>> return once it is written > > in > > > > ZK, > > > > > > and > > > > > > > that > > > > > > > > > > is > > > > > > > > > > > > why we > > > > > > > > > > > > > > >> need > > > > > > > > > > > > > > >> > > > extra > > > > > > > > > > > > > > >> > > > > > > > >> verification > > > > > > > > > > > > > > >> > > > > > > > >> >>> like > > > > > > testUtil.waitForTopicCreated() / > > > > > > > > > > verify > > > > > > > > > > > > > > >> partition > > > > > > > > > > > > > > >> > > > > > > reassignment > > > > > > > > > > > > > > >> > > > > > > > >> >>> request, etc. With admin > > > > requests > > > > > > we > > > > > > > > > could > > > > > > > > > > > add > > > > > > > > > > > > a > > > > > > > > > > > > > > >> flag to > > > > > > > > > > > > > > >> > > > > enable > > > > > > > > > > > > > > >> > > > > > / > > > > > > > > > > > > > > >> > > > > > > > >> disable > > > > > > > > > > > > > > >> > > > > > > > >> >>> synchronous requests; > when > > it > > > > is > > > > > > > turned > > > > > > > > > on, > > > > > > > > > > > the > > > > > > > > > > > > > > >> response > > > > > > > > > > > > > > >> > > > will > > > > > > > > > > > > > > >> > > > > > not > > > > > > > > > > > > > > >> > > > > > > > >> return > > > > > > > > > > > > > > >> > > > > > > > >> >>> until the request has > been > > > > > > > completed. And > > > > > > > > > > for > > > > > > > > > > > > async > > > > > > > > > > > > > > >> > > requests > > > > > > > > > > > > > > >> > > > > we > > > > > > > > > > > > > > >> > > > > > > can > > > > > > > > > > > > > > >> > > > > > > > >> add a > > > > > > > > > > > > > > >> > > > > > > > >> >>> "token" field in the > > > response, > > > > > and > > > > > > > then > > > > > > > > > > only > > > > > > > > > > > > need a > > > > > > > > > > > > > > >> > > general > > > > > > > > > > > > > > >> > > > > > "admin > > > > > > > > > > > > > > >> > > > > > > > >> >>> verification request" > with > > > the > > > > > > given > > > > > > > > > token > > > > > > > > > > to > > > > > > > > > > > > check > > > > > > > > > > > > > > >> if the > > > > > > > > > > > > > > >> > > > > async > > > > > > > > > > > > > > >> > > > > > > > >> request > > > > > > > > > > > > > > >> > > > > > > > >> >>> has been completed. > > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > > >> > > > > > > > >> >>> 5. +1 for extending > > Metadata > > > > > > request > > > > > > > to > > > > > > > > > > > include > > > > > > > > > > > > > > >> > > controller / > > > > > > > > > > > > > > >> > > > > > > > >> coordinator > > > > > > > > > > > > > > >> > > > > > > > >> >>> information, and then we > > can > > > > > remove > > > > > > > the > > > > > > > > > > > > > > >> ConsumerMetadata / > > > > > > > > > > > > > > >> > > > > > > > >> ClusterMetadata > > > > > > > > > > > > > > >> > > > > > > > >> >>> requests. > > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > > >> > > > > > > > >> >>> Guozhang > > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > > >> > > > > > > > >> >>> On Tue, Mar 3, 2015 at > > 10:23 > > > > AM, > > > > > > Joel > > > > > > > > > > Koshy < > > > > > > > > > > > > > > >> > > > > > jjkosh...@gmail.com> > > > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> Thanks for sending that > > out > > > > Joe > > > > > - > > > > > > I > > > > > > > > > don't > > > > > > > > > > > > think I > > > > > > > > > > > > > > >> will be > > > > > > > > > > > > > > >> > > > > able > > > > > > > > > > > > > > >> > > > > > to > > > > > > > > > > > > > > >> > > > > > > > >> make > > > > > > > > > > > > > > >> > > > > > > > >> >>>> it today, so if notes > can > > be > > > > > sent > > > > > > > out > > > > > > > > > > > > afterward > > > > > > > > > > > > > > that > > > > > > > > > > > > > > >> > > would > > > > > > > > > > > > > > >> > > > be > > > > > > > > > > > > > > >> > > > > > > > great. > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> On Mon, Mar 02, 2015 at > > > > > 09:16:13AM > > > > > > > > > -0800, > > > > > > > > > > > Gwen > > > > > > > > > > > > > > >> Shapira > > > > > > > > > > > > > > >> > > > wrote: > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > Thanks for sending > this > > > out > > > > > Joe. > > > > > > > > > Looking > > > > > > > > > > > > forward > > > > > > > > > > > > > > >> to > > > > > > > > > > > > > > >> > > > > chatting > > > > > > > > > > > > > > >> > > > > > > with > > > > > > > > > > > > > > >> > > > > > > > >> >>>> everyone :) > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > On Mon, Mar 2, 2015 at > > > 6:46 > > > > > AM, > > > > > > > Joe > > > > > > > > > > Stein > > > > > > > > > > > < > > > > > > > > > > > > > > >> > > > > > > joe.st...@stealth.ly> > > > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > Hey, I just sent > out a > > > > > google > > > > > > > > > hangout > > > > > > > > > > > > invite > > > > > > > > > > > > > > to > > > > > > > > > > > > > > >> all > > > > > > > > > > > > > > >> > > > pmc, > > > > > > > > > > > > > > >> > > > > > > > >> committers > > > > > > > > > > > > > > >> > > > > > > > >> >>>> and > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > everyone I found > > working > > > > on > > > > > a > > > > > > > KIP. > > > > > > > > > If > > > > > > > > > > I > > > > > > > > > > > > missed > > > > > > > > > > > > > > >> anyone > > > > > > > > > > > > > > >> > > > in > > > > > > > > > > > > > > >> > > > > > the > > > > > > > > > > > > > > >> > > > > > > > >> invite > > > > > > > > > > > > > > >> > > > > > > > >> >>>> please > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > let me know and can > > > update > > > > > it, > > > > > > > np. > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > We should do this > > every > > > > > > Tuesday > > > > > > > @ > > > > > > > > > 2pm > > > > > > > > > > > > Eastern > > > > > > > > > > > > > > >> Time. > > > > > > > > > > > > > > >> > > > Maybe > > > > > > > > > > > > > > >> > > > > > we > > > > > > > > > > > > > > >> > > > > > > > can > > > > > > > > > > > > > > >> > > > > > > > >> get > > > > > > > > > > > > > > >> > > > > > > > >> >>>> INFRA > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > help to make a > google > > > > > account > > > > > > > so we > > > > > > > > > > can > > > > > > > > > > > > manage > > > > > > > > > > > > > > >> > > better? > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > To discuss > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > in progress and > > related > > > > JIRA > > > > > > > that > > > > > > > > > are > > > > > > > > > > > > > > >> interdependent > > > > > > > > > > > > > > >> > > > and > > > > > > > > > > > > > > >> > > > > > > common > > > > > > > > > > > > > > >> > > > > > > > >> work. > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > ~ Joe Stein > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > On Tue, Feb 24, 2015 > > at > > > > 2:59 > > > > > > > PM, Jay > > > > > > > > > > > > Kreps < > > > > > > > > > > > > > > >> > > > > > > > jay.kr...@gmail.com> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> wrote: > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> Let's stay on > Google > > > > > hangouts > > > > > > > that > > > > > > > > > > will > > > > > > > > > > > > also > > > > > > > > > > > > > > >> record > > > > > > > > > > > > > > >> > > > and > > > > > > > > > > > > > > >> > > > > > make > > > > > > > > > > > > > > >> > > > > > > > the > > > > > > > > > > > > > > >> > > > > > > > >> >>>> sessions > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> available on > youtube. > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> -Jay > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> On Tue, Feb 24, > 2015 > > at > > > > > 11:49 > > > > > > > AM, > > > > > > > > > > Jeff > > > > > > > > > > > > > > Holoman > > > > > > > > > > > > > > >> < > > > > > > > > > > > > > > >> > > > > > > > >> >>>> jholo...@cloudera.com> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> wrote: > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > Jay / Joe > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > We're happy to > send > > > > out a > > > > > > > Webex > > > > > > > > > for > > > > > > > > > > > > this > > > > > > > > > > > > > > >> purpose. > > > > > > > > > > > > > > >> > > We > > > > > > > > > > > > > > >> > > > > > could > > > > > > > > > > > > > > >> > > > > > > > >> record > > > > > > > > > > > > > > >> > > > > > > > >> >>>> the > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > sessions if there > > is > > > > > > > interest and > > > > > > > > > > > > publish > > > > > > > > > > > > > > >> them > > > > > > > > > > > > > > >> > > out. > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > Thanks > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > Jeff > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > On Tue, Feb 24, > > 2015 > > > at > > > > > > > 11:28 AM, > > > > > > > > > > Jay > > > > > > > > > > > > > > Kreps < > > > > > > > > > > > > > > >> > > > > > > > >> jay.kr...@gmail.com> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> wrote: > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > Let's try to > get > > > the > > > > > > > technical > > > > > > > > > > > > hang-ups > > > > > > > > > > > > > > >> sorted > > > > > > > > > > > > > > >> > > > out, > > > > > > > > > > > > > > >> > > > > > > > though. > > > > > > > > > > > > > > >> > > > > > > > >> I > > > > > > > > > > > > > > >> > > > > > > > >> >>>> really > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > think > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > there is some > > > benefit > > > > > to > > > > > > > live > > > > > > > > > > > > discussion > > > > > > > > > > > > > > vs > > > > > > > > > > > > > > >> > > > > writing. I > > > > > > > > > > > > > > >> > > > > > > am > > > > > > > > > > > > > > >> > > > > > > > >> >>>> hopeful that > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> if > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > we post > > > instructions > > > > > and > > > > > > > give > > > > > > > > > > > > ourselves a > > > > > > > > > > > > > > >> few > > > > > > > > > > > > > > >> > > > > attempts > > > > > > > > > > > > > > >> > > > > > > we > > > > > > > > > > > > > > >> > > > > > > > >> can > > > > > > > > > > > > > > >> > > > > > > > >> >>>> get it > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > working. > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > Tuesday at that > > > time > > > > > > would > > > > > > > work > > > > > > > > > > for > > > > > > > > > > > > > > >> me...any > > > > > > > > > > > > > > >> > > > > > objections? > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > -Jay > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > On Tue, Feb 24, > > > 2015 > > > > at > > > > > > > 8:18 > > > > > > > > > AM, > > > > > > > > > > > Joe > > > > > > > > > > > > > > Stein > > > > > > > > > > > > > > >> < > > > > > > > > > > > > > > >> > > > > > > > >> joe.st...@stealth.ly > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> wrote: > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > Weekly would > be > > > > great > > > > > > > maybe > > > > > > > > > > like > > > > > > > > > > > > every > > > > > > > > > > > > > > >> > > Tuesday ~ > > > > > > > > > > > > > > >> > > > > 1pm > > > > > > > > > > > > > > >> > > > > > > ET > > > > > > > > > > > > > > >> > > > > > > > / > > > > > > > > > > > > > > >> > > > > > > > >> 10am > > > > > > > > > > > > > > >> > > > > > > > >> >>>> PT > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> ???? > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > I don't mind > > > google > > > > > > > hangout > > > > > > > > > but > > > > > > > > > > > > there > > > > > > > > > > > > > > is > > > > > > > > > > > > > > >> > > always > > > > > > > > > > > > > > >> > > > > some > > > > > > > > > > > > > > >> > > > > > > > >> issue or > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> whatever > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > so > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > we know the > > > apache > > > > > irc > > > > > > > > > channel > > > > > > > > > > > > works. > > > > > > > > > > > > > > We > > > > > > > > > > > > > > >> can > > > > > > > > > > > > > > >> > > > start > > > > > > > > > > > > > > >> > > > > > > there > > > > > > > > > > > > > > >> > > > > > > > >> and > > > > > > > > > > > > > > >> > > > > > > > >> >>>> see how > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> it > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > goes? We can > > pull > > > > > > > transcripts > > > > > > > > > > too > > > > > > > > > > > > and > > > > > > > > > > > > > > >> > > associate > > > > > > > > > > > > > > >> > > > to > > > > > > > > > > > > > > >> > > > > > > > >> tickets if > > > > > > > > > > > > > > >> > > > > > > > >> >>>> need be > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > makes > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > it helpful > for > > > > > things. > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > ~ Joestein > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > On Tue, Feb > 24, > > > > 2015 > > > > > at > > > > > > > 11:10 > > > > > > > > > > AM, > > > > > > > > > > > > Jay > > > > > > > > > > > > > > >> Kreps < > > > > > > > > > > > > > > >> > > > > > > > >> >>>> jay.kr...@gmail.com> > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > wrote: > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > We'd talked > > > about > > > > > > > doing a > > > > > > > > > > > Google > > > > > > > > > > > > > > >> Hangout to > > > > > > > > > > > > > > >> > > > chat > > > > > > > > > > > > > > >> > > > > > > about > > > > > > > > > > > > > > >> > > > > > > > >> this. > > > > > > > > > > > > > > >> > > > > > > > >> >>>> What > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > about > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > generalizing > > > > that a > > > > > > > little > > > > > > > > > > > > > > further...I > > > > > > > > > > > > > > >> > > > actually > > > > > > > > > > > > > > >> > > > > > > think > > > > > > > > > > > > > > >> > > > > > > > it > > > > > > > > > > > > > > >> > > > > > > > >> >>>> would be > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > good > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > for > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > everyone > > > > spending a > > > > > > > > > > reasonable > > > > > > > > > > > > chunk > > > > > > > > > > > > > > of > > > > > > > > > > > > > > >> > > their > > > > > > > > > > > > > > >> > > > > week > > > > > > > > > > > > > > >> > > > > > > on > > > > > > > > > > > > > > >> > > > > > > > >> Kafka > > > > > > > > > > > > > > >> > > > > > > > >> >>>> stuff > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> to > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > maybe > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > sync up > once > > a > > > > > week. > > > > > > I > > > > > > > > > think > > > > > > > > > > we > > > > > > > > > > > > could > > > > > > > > > > > > > > >> use > > > > > > > > > > > > > > >> > > time > > > > > > > > > > > > > > >> > > > > to > > > > > > > > > > > > > > >> > > > > > > talk > > > > > > > > > > > > > > >> > > > > > > > >> >>>> through > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> design > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > stuff, make > > > sure > > > > we > > > > > > > are on > > > > > > > > > > top > > > > > > > > > > > of > > > > > > > > > > > > > > code > > > > > > > > > > > > > > >> > > > reviews, > > > > > > > > > > > > > > >> > > > > > talk > > > > > > > > > > > > > > >> > > > > > > > >> through > > > > > > > > > > > > > > >> > > > > > > > >> >>>> any > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > tricky > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > issues, > etc. > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > We can make > > it > > > > > > publicly > > > > > > > > > > > > available so > > > > > > > > > > > > > > >> that > > > > > > > > > > > > > > >> > > any > > > > > > > > > > > > > > >> > > > > one > > > > > > > > > > > > > > >> > > > > > > can > > > > > > > > > > > > > > >> > > > > > > > >> follow > > > > > > > > > > > > > > >> > > > > > > > >> >>>> along > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > who > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > likes. > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > Any > interest > > in > > > > > doing > > > > > > > this? > > > > > > > > > > If > > > > > > > > > > > so > > > > > > > > > > > > > > I'll > > > > > > > > > > > > > > >> try > > > > > > > > > > > > > > >> > > to > > > > > > > > > > > > > > >> > > > > set > > > > > > > > > > > > > > >> > > > > > it > > > > > > > > > > > > > > >> > > > > > > > up > > > > > > > > > > > > > > >> > > > > > > > >> >>>> starting > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> next > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > week. > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > -Jay > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > On Tue, Feb > > 24, > > > > > 2015 > > > > > > at > > > > > > > > > 3:57 > > > > > > > > > > > AM, > > > > > > > > > > > > > > Andrii > > > > > > > > > > > > > > >> > > > > Biletskyi > > > > > > > > > > > > > > >> > > > > > < > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > andrii.bilets...@stealth.ly> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > Hi all, > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > I've > > updated > > > > KIP > > > > > > > page, > > > > > > > > > > fixed > > > > > > > > > > > / > > > > > > > > > > > > > > >> aligned > > > > > > > > > > > > > > >> > > > > document > > > > > > > > > > > > > > >> > > > > > > > >> structure. > > > > > > > > > > > > > > >> > > > > > > > >> >>>> Also I > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > added > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > some > > > > > > > > > ... > > > > > > > > [Message clipped] > > > > > > > > > > > -- > > -- Guozhang > > >
