Andrii, A few points.
1. Create/Alter can typically complete quickly. So, it's possible to make the request block until it's completed. However, currently, doing this at the broker is a bit involved. To make Create block, we will need to add some callbacks in KafkaController. This is possible. However, the controller logic currently is pretty completed. It would probably be better if we clean it up first before adding more complexity to it. Alter is even trickier. Adding partition is currently handled through KafkaController. So it can be dealt with in a similar way. However, Alter config is done completely differently. It doesn't go through the controller. Instead, each broker listens to ZooKeeper directly. So, it's not clear if there is an easy way on the broker to figure out whether a config is applied on every broker. 2. Delete can potentially take long if a replica to be deleted is offline. PreferredLeader/PartitionReassign can also take long. So, we can't really make those requests block on the broker. As you can see, at this moment it's not easy to make all admin requests block on the broker. So, if we want the blocking feature in the admin utility in the short term, doing the completion check at the admin client is probably an easier route, even though it may not be ideal. Thanks, Jun On Fri, Mar 20, 2015 at 2:38 PM, Andrii Biletskyi < andrii.bilets...@stealth.ly> wrote: > Jun, > > I see your point. But wouldn't that lead to a "fat" client implementations? > Suppose someone would like to implement client for Admin Wire protocol. > Not only people will have to code quite complicated logic like "send > describe > request to each broker" (again state machin?) but it will also mean people > must understand internal kafka logic related to topic storage and how > information is propageted from the controller to brokers. > I see this like a dilemma between having a concise Wire Protocol and > self-sufficient API to make client implementations simple. > I don't have a win-win solution though. > > Thanks, > Andrii Biletskyi > > > On Fri, Mar 20, 2015 at 11:19 PM, Jun Rao <j...@confluent.io> wrote: > > > For 1), 2) and 3), blocking would probably mean that the new metadata is > > propagated to every broker. To achieve that, the client can keep issuing > > the describe topic request to every broker until it sees the new metadata > > in the response. > > > > Thanks, > > > > Jun > > > > On Fri, Mar 20, 2015 at 12:16 PM, Andrii Biletskyi < > > andrii.bilets...@stealth.ly> wrote: > > > > > Hm, actually the ticket you linked, Guozhang, brings as back > > > to the problem what should be considered a post-condition for > > > each of the admin commands. > > > In my understanding: > > > > > > 1) CreateTopic - broker created /brokers/topics/<topic> > > > (Not the controller picked up changes from zk and broadcasted > > > LeaderAndIsr and UpdateMetadata) > > > > > > 2) AlterTopic - same as 1) - broker changed assignment data > > > in zookeeper or created admin path for topic config change > > > > > > 3) DeleteTopic - admin path /admin/delete_topics is created > > > > > > 4) ReassignPartitions and PreferredReplica - corresponding admin > > > path is created > > > > > > Now what can be considered a completed operation from the client's > > > perspective? > > > 1) Topic is created once corresponding data is in zk > > > (I remember there were some thoughts that it'd be good to consider > > > topic created once all replicas receive information about it and thus > > > clients can produce/consume from it, but as was discussed this seems > > > to be a hard thing to do) > > > > > > 2) Probably same as 1), so right after AlterTopic is issued > > > > > > 3) The topic has been removed from /brokers/topics > > > > > > 4) ReassignPartitions and PrefferedReplica were discussed earlier - > > > in short the former is completed once partition state info in zk > matches > > > reassignment request and admin path is empty, the latter - once data > > > in zk shows that head of assignned replicas of the partition and leader > > > is the same replica > > > > > > Thoughts? > > > > > > Thanks, > > > Andrii Biletskyi > > > > > > On Fri, Mar 20, 2015 at 7:10 PM, Guozhang Wang <wangg...@gmail.com> > > wrote: > > > > > > > I think while loop is fine for supporting blocking, just that we need > > to > > > > add back off to avoid bombarding brokers with DescribeTopic requests. > > > > > > > > Also I have linked KAFKA-1125 > > > > <https://issues.apache.org/jira/browse/KAFKA-1125> to your proposal, > > and > > > > when KAFKA-1694 is done this ticket can also be closed. > > > > > > > > Guozhang > > > > > > > > On Fri, Mar 20, 2015 at 9:41 AM, Andrii Biletskyi < > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > Great. > > > > > I want to elaborate this a bit more, to see we are on the same page > > > > > concerning the client code. > > > > > > > > > > So with all topic commands being async a client (AdminClient in our > > > > > case or any other other client people would like to implement) to > > > support > > > > > a blocking operation (which seems to be a natural use-case e.g. for > > > topic > > > > > creation): would have to do: > > > > > 1. issue CreateTopicRequest > > > > > 2. if successful, in a "while" loop send DescribeTopicRequest and > > > > > break the loop once all topics are returned in response (or upon > > > > timeout). > > > > > 3. if unsuccessful throw exception > > > > > Would it be okay? > > > > > > > > > > Thanks, > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > On Fri, Mar 20, 2015 at 6:21 PM, Jun Rao <j...@confluent.io> wrote: > > > > > > > > > > > Andrii, > > > > > > > > > > > > I think you are right. It seems that only ReassignPartitions > needs > > a > > > > > > separate verification request. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > On Thu, Mar 19, 2015 at 9:22 AM, Andrii Biletskyi < > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > Guys, > > > > > > > I like this idea too. Let's stick with that. I'll update KIP > > > > > accordingly. > > > > > > > > > > > > > > I was also thinking we can avoid adding dedicated status check > > > > > > > requests for topic commands. - We have everything in > > DescribeTopic > > > > > > > for that! E.g.: > > > > > > > User issued CreateTopic - to check the status client sends > > > > > DescribeTopic > > > > > > > and checks whether is something returned for that topic. The > same > > > for > > > > > > > alteration, deletion. > > > > > > > Btw, PreferredReplica status can be also checked with > > > > > > DescribeTopicRequest > > > > > > > (head of assigned replicas list == leader). > > > > > > > For ReassignPartitions as discussed we'll need to have a > separate > > > > > > Verify... > > > > > > > request. > > > > > > > > > > > > > > Thanks, > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 19, 2015 at 6:03 PM, Guozhang Wang < > > wangg...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > +1 on broker writing to ZK for async handling. I was thinking > > > that > > > > in > > > > > > the > > > > > > > > end state the admin requests would be eventually sent to > > > controller > > > > > > > either > > > > > > > > through re-routing or clients discovering them, instead of > > > letting > > > > > > > > controller listen on ZK admin path. But thinking about it a > > > second > > > > > > time, > > > > > > > I > > > > > > > > think it is actually simpler to let controller manage > > > > > > > > incoming queued-up admin requests through ZK. > > > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 4:16 PM, Joel Koshy < > > jjkosh...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > +1 as well. I think it helps to keep the rerouting approach > > > > > > orthogonal > > > > > > > > > to this KIP. > > > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 03:40:48PM -0700, Jay Kreps wrote: > > > > > > > > > > I'm +1 on Jun's suggestion as long as it can work for all > > the > > > > > > > requests. > > > > > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 3:35 PM, Jun Rao < > j...@confluent.io > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Andrii, > > > > > > > > > > > > > > > > > > > > > > I think we agreed on the following. > > > > > > > > > > > > > > > > > > > > > > (a) Admin requests can be sent to and handled by any > > > broker. > > > > > > > > > > > (b) Admin requests are processed asynchronously, at > least > > > for > > > > > > now. > > > > > > > > > That is, > > > > > > > > > > > when the client gets a response, it just means that the > > > > request > > > > > > is > > > > > > > > > > > initiated, but not necessarily completed. Then, it's up > > to > > > > the > > > > > > > client > > > > > > > > > to > > > > > > > > > > > issue another request to check the status for > completion. > > > > > > > > > > > > > > > > > > > > > > To support (a), we were thinking of doing request > > > forwarding > > > > to > > > > > > the > > > > > > > > > > > controller (utilizing KAFKA-1912). I am making an > > > alternative > > > > > > > > proposal. > > > > > > > > > > > Basically, the broker can just write to ZooKeeper to > > inform > > > > the > > > > > > > > > controller > > > > > > > > > > > about the request. For example, to handle > > > > > partitionReassignment, > > > > > > > the > > > > > > > > > broker > > > > > > > > > > > will just write the requested partitions to > > > > > > > > /admin/reassign_partitions > > > > > > > > > > > (like what AdminUtils currently does) and then send a > > > > response > > > > > to > > > > > > > the > > > > > > > > > > > client. This shouldn't take long and the implementation > > > will > > > > be > > > > > > > > simpler > > > > > > > > > > > than forwarding the requests to the controller through > > RPC. > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 3:03 PM, Andrii Biletskyi < > > > > > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > > > > > > > > > > Jun, > > > > > > > > > > > > > > > > > > > > > > > > I might be wrong but didn't we agree we will let any > > > broker > > > > > > from > > > > > > > > the > > > > > > > > > > > > cluster handle *long-running* admin requests (at this > > > time > > > > > > > > > > > preferredReplica > > > > > > > > > > > > and > > > > > > > > > > > > reassignPartitions), via zk admin path. Thus > > CreateTopics > > > > etc > > > > > > > > should > > > > > > > > > be > > > > > > > > > > > > sent > > > > > > > > > > > > only to the controller. > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 11:55 PM, Jun Rao < > > > > j...@confluent.io> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Joel, Andril, > > > > > > > > > > > > > > > > > > > > > > > > > > I think we agreed that those admin requests can be > > > issued > > > > > to > > > > > > > any > > > > > > > > > > > broker. > > > > > > > > > > > > > Because of that, there doesn't seem to be a strong > > need > > > > to > > > > > > know > > > > > > > > the > > > > > > > > > > > > > controller. So, perhaps we can proceed by not > making > > > any > > > > > > change > > > > > > > > to > > > > > > > > > the > > > > > > > > > > > > > format of TMR right now. When we start using create > > > topic > > > > > > > request > > > > > > > > > in > > > > > > > > > > > the > > > > > > > > > > > > > producer, we will need a new version of TMR that > > > doesn't > > > > > > > trigger > > > > > > > > > auto > > > > > > > > > > > > topic > > > > > > > > > > > > > creation. But that can be done later. > > > > > > > > > > > > > > > > > > > > > > > > > > As a first cut implementation, I think the broker > can > > > > just > > > > > > > write > > > > > > > > > to ZK > > > > > > > > > > > > > directly for > > > > > > > > > > > > > > > > > > > > > > > > createToipic/alterTopic/reassignPartitions/preferredLeaderElection > > > > > > > > > > > > > requests, instead of forwarding them to the > > controller. > > > > > This > > > > > > > will > > > > > > > > > > > > simplify > > > > > > > > > > > > > the implementation on the broker side. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 11:58 AM, Joel Koshy < > > > > > > > > jjkosh...@gmail.com> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > (Thanks Andrii for the summary) > > > > > > > > > > > > > > > > > > > > > > > > > > > > For (1) yes we will circle back on that shortly > > after > > > > > > syncing > > > > > > > > up > > > > > > > > > in > > > > > > > > > > > > > > person. I think it is close to getting committed > > > > although > > > > > > > > > development > > > > > > > > > > > > > > for KAFKA-1927 can probably begin without it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > There is one more item we covered at the hangout. > > > i.e., > > > > > > > whether > > > > > > > > > we > > > > > > > > > > > > > > want to add the coordinator to the topic metadata > > > > > response > > > > > > or > > > > > > > > > provide > > > > > > > > > > > > > > a clearer ClusterMetadataRequest. > > > > > > > > > > > > > > > > > > > > > > > > > > > > There are two reasons I think we should try and > > avoid > > > > > > adding > > > > > > > > the > > > > > > > > > > > > > > field: > > > > > > > > > > > > > > - It is irrelevant to topic metadata > > > > > > > > > > > > > > - If we finally do request rerouting in Kafka > then > > > the > > > > > > field > > > > > > > > > would > > > > > > > > > > > add > > > > > > > > > > > > > > little to no value. (It still helps to have a > > > > separate > > > > > > > > > > > > > > ClusterMetadataRequest to query for > cluster-wide > > > > > > > information > > > > > > > > > such > > > > > > > > > > > as > > > > > > > > > > > > > > 'which broker is the controller?' as Joe > > > mentioned.) > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think it would be cleaner to have an explicit > > > > > > > > > > > ClusterMetadataRequest > > > > > > > > > > > > > > that you can send to any broker in order to > obtain > > > the > > > > > > > > controller > > > > > > > > > > > (and > > > > > > > > > > > > > > in the future possibly other cluster-wide > > > > information). I > > > > > > > think > > > > > > > > > the > > > > > > > > > > > > > > main argument against doing this and instead > adding > > > it > > > > to > > > > > > the > > > > > > > > > topic > > > > > > > > > > > > > > metadata response was convenience - i.e., you > don't > > > > have > > > > > to > > > > > > > > > discover > > > > > > > > > > > > > > the controller in advance. However, I don't see > > much > > > > > actual > > > > > > > > > > > > > > benefit/convenience in this and in fact think it > > is a > > > > > > > > non-issue. > > > > > > > > > Let > > > > > > > > > > > > > > me know if I'm overlooking something here. > > > > > > > > > > > > > > > > > > > > > > > > > > > > As an example, say we need to initiate partition > > > > > > reassignment > > > > > > > > by > > > > > > > > > > > > > > issuing the new ReassignPartitionsRequest to the > > > > > controller > > > > > > > > > (assume > > > > > > > > > > > we > > > > > > > > > > > > > > already have the desired manual partition > > > assignment). > > > > > If > > > > > > we > > > > > > > > > are to > > > > > > > > > > > > > > augment topic metadata response then the flow be > > > > > something > > > > > > > like > > > > > > > > > this > > > > > > > > > > > : > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Issue topic metadata request to any broker (and > > > > > discover > > > > > > > the > > > > > > > > > > > > > > controller > > > > > > > > > > > > > > - Connect to controller if required (i.e., if the > > > > broker > > > > > > > above > > > > > > > > != > > > > > > > > > > > > > > controller) > > > > > > > > > > > > > > - Issue the partition reassignment request to the > > > > > > controller. > > > > > > > > > > > > > > > > > > > > > > > > > > > > With an explicit cluster metadata request it > would > > > be: > > > > > > > > > > > > > > - Issue cluster metadata request to any broker > > > > > > > > > > > > > > - Connect to controller if required (i.e., if the > > > > broker > > > > > > > above > > > > > > > > != > > > > > > > > > > > > > > controller) > > > > > > > > > > > > > > - Issue the partition reassignment request > > > > > > > > > > > > > > > > > > > > > > > > > > > > So it seems to add little practical value and > > bloats > > > > > topic > > > > > > > > > metadata > > > > > > > > > > > > > > response with an irrelevant detail. > > > > > > > > > > > > > > > > > > > > > > > > > > > > The other angle to this is the following - is it > a > > > > matter > > > > > > of > > > > > > > > > naming? > > > > > > > > > > > > > > Should we just rename topic metadata > > request/response > > > > to > > > > > > just > > > > > > > > > > > > > > MetadataRequest/Response and add cluster metadata > > to > > > > it? > > > > > By > > > > > > > > that > > > > > > > > > same > > > > > > > > > > > > > > token should we also allow querying for the > > consumer > > > > > > > > coordinator > > > > > > > > > (and > > > > > > > > > > > > > > in future transaction coordinator) as well? This > > > leads > > > > > to a > > > > > > > > > bloated > > > > > > > > > > > > > > request which isn't very appealing and altogether > > > > > > confusing. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > Joel > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 18, 2015 at 09:34:12AM -0700, Jun Rao > > > > wrote: > > > > > > > > > > > > > > > Andri, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the summary. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. I just realized that in order to start > working > > > on > > > > > > > > > KAFKA-1927, we > > > > > > > > > > > > > will > > > > > > > > > > > > > > > need to merge the changes to > OffsetCommitRequest > > > > (from > > > > > > > 0.8.2) > > > > > > > > > to > > > > > > > > > > > > trunk. > > > > > > > > > > > > > > > This is planned to be done as part of > KAFKA-1634. > > > So, > > > > > we > > > > > > > will > > > > > > > > > need > > > > > > > > > > > > > > Guozhang > > > > > > > > > > > > > > > and Joel's help to wrap this up. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Thinking about this a bit more, if the > > semantic > > > of > > > > > > those > > > > > > > > > "write" > > > > > > > > > > > > > > > requests is async (i.e., after the client gets > a > > > > > > response, > > > > > > > it > > > > > > > > > just > > > > > > > > > > > > > means > > > > > > > > > > > > > > > that the operation is initiated, but not > > > necessarily > > > > > > > > > completed), we > > > > > > > > > > > > > don't > > > > > > > > > > > > > > > really need to forward the requests to the > > > > controller. > > > > > > > > > Instead, the > > > > > > > > > > > > > > > receiving broker can just write the operation > to > > ZK > > > > as > > > > > > the > > > > > > > > > admin > > > > > > > > > > > > > command > > > > > > > > > > > > > > > line tool previously does. This will simplify > the > > > > > > > > > implementation. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 8. There is another implementation detail for > > > > describe > > > > > > > topic. > > > > > > > > > > > > Ideally, > > > > > > > > > > > > > we > > > > > > > > > > > > > > > want to read the topic config from the broker > > > cache, > > > > > > > instead > > > > > > > > of > > > > > > > > > > > > > > ZooKeeper. > > > > > > > > > > > > > > > Currently, every broker reads the topic-level > > > config > > > > > for > > > > > > > all > > > > > > > > > > > topics. > > > > > > > > > > > > > > > However, it ignores those for topics not hosted > > on > > > > > > itself. > > > > > > > > So, > > > > > > > > > we > > > > > > > > > > > may > > > > > > > > > > > > > > need > > > > > > > > > > > > > > > to change TopicConfigManager a bit so that it > > > caches > > > > > the > > > > > > > > > configs > > > > > > > > > > > for > > > > > > > > > > > > > all > > > > > > > > > > > > > > > topics. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 1:13 PM, Andrii > > Biletskyi < > > > > > > > > > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Guys, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for a great discussion! > > > > > > > > > > > > > > > > Here are the actions points: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Q: Get rid of all scala requests objects, > > use > > > > java > > > > > > > > > protocol > > > > > > > > > > > > > > definitions. > > > > > > > > > > > > > > > > A: Gwen kindly took that (KAFKA-1927). > It's > > > > > > important > > > > > > > > to > > > > > > > > > > > speed > > > > > > > > > > > > up > > > > > > > > > > > > > > > > review procedure > > > > > > > > > > > > > > > > there since this ticket blocks other > > > > > important > > > > > > > > > changes. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2. Q: Generic re-reroute facility vs client > > > > > maintaining > > > > > > > > > cluster > > > > > > > > > > > > > state. > > > > > > > > > > > > > > > > A: Jay has added pseudo code to > KAFKA-1912 > > - > > > > need > > > > > > to > > > > > > > > > consider > > > > > > > > > > > > > > whether > > > > > > > > > > > > > > > > this will be > > > > > > > > > > > > > > > > easy to implement as a server-side > > > feature > > > > > > > > (comments > > > > > > > > > are > > > > > > > > > > > > > > > > welcomed!). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 3. Q: Controller field in wire protocol. > > > > > > > > > > > > > > > > A: This might be useful for clients, add > > this > > > > to > > > > > > > > > > > > > > TopicMetadataResponse > > > > > > > > > > > > > > > > (already in KIP). > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 4. Q: Decoupling topic creation from TMR. > > > > > > > > > > > > > > > > A: I will add proposed by Jun solution > > (using > > > > > > > clientId > > > > > > > > > for > > > > > > > > > > > > that) > > > > > > > > > > > > > > to the > > > > > > > > > > > > > > > > KIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 5. Q: Bumping new versions of TMR vs grabbing > > all > > > > > > > protocol > > > > > > > > > > > changes > > > > > > > > > > > > in > > > > > > > > > > > > > > one > > > > > > > > > > > > > > > > version. > > > > > > > > > > > > > > > > A: It was decided to try to gather all > > > changes > > > > to > > > > > > > > > protocol > > > > > > > > > > > > > (before > > > > > > > > > > > > > > > > release). > > > > > > > > > > > > > > > > In case of TMR it worth checking: > > > > KAFKA-2020 > > > > > > and > > > > > > > > > KIP-13 > > > > > > > > > > > > > > (quotas) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 6. Q: JSON lib is needed to deserialize > user's > > > > input > > > > > in > > > > > > > CLI > > > > > > > > > tool. > > > > > > > > > > > > > > > > A: Use jackson for that, /tools project > is > > a > > > > > > separate > > > > > > > > > jar so > > > > > > > > > > > > > > shouldn't > > > > > > > > > > > > > > > > be a big deal. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 7. Q: VerifyReassingPartitions vs generic > > status > > > > > check > > > > > > > > > command. > > > > > > > > > > > > > > > > A: For long-running requests like > reassign > > > > > > > partitions > > > > > > > > > > > > *progress* > > > > > > > > > > > > > > check > > > > > > > > > > > > > > > > request is useful, > > > > > > > > > > > > > > > > it makes sense to introduce it. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please add, correct me if I missed > something. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 6:20 PM, Andrii > > > Biletskyi < > > > > > > > > > > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Joel, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > You are right, I removed ClusterMetadata > > > because > > > > we > > > > > > > have > > > > > > > > > > > > partially > > > > > > > > > > > > > > > > > what we need in TopicMetadata. Also, as Jay > > > > pointed > > > > > > out > > > > > > > > > > > earlier, > > > > > > > > > > > > we > > > > > > > > > > > > > > > > > would like to have "orthogonal" API, but at > > the > > > > > same > > > > > > > time > > > > > > > > > we > > > > > > > > > > > need > > > > > > > > > > > > > > > > > to be backward compatible. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > But I like your idea and even have some > other > > > > > > arguments > > > > > > > > for > > > > > > > > > > > this > > > > > > > > > > > > > > option: > > > > > > > > > > > > > > > > > There is also DescribeTopicRequest which > was > > > > > proposed > > > > > > > in > > > > > > > > > this > > > > > > > > > > > > KIP, > > > > > > > > > > > > > > > > > it returns topic configs, partitions, > > > replication > > > > > > > factor > > > > > > > > > plus > > > > > > > > > > > > > > partition > > > > > > > > > > > > > > > > > ISR, ASR, > > > > > > > > > > > > > > > > > leader replica. The later part is really > > > already > > > > > > there > > > > > > > in > > > > > > > > > > > > > > > > > TopicMetadataRequest. > > > > > > > > > > > > > > > > > So again we'll have to add stuff to TMR, > not > > to > > > > > > > duplicate > > > > > > > > > some > > > > > > > > > > > > info > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > > newly added requests. However, this way > we'll > > > end > > > > > up > > > > > > > with > > > > > > > > > > > > "monster" > > > > > > > > > > > > > > > > > request which returns cluster metadata, > topic > > > > > > > replication > > > > > > > > > and > > > > > > > > > > > > > config > > > > > > > > > > > > > > info > > > > > > > > > > > > > > > > > plus partition replication data. Seems > > logical > > > to > > > > > > split > > > > > > > > > TMR to > > > > > > > > > > > > > > > > > - ClusterMetadata (brokers + controller, > > maybe > > > > smth > > > > > > > else) > > > > > > > > > > > > > > > > > - TopicMetadata (topic info + partition > > > details) > > > > > > > > > > > > > > > > > But since current TMR is involved in lots > of > > > > places > > > > > > > > > (including > > > > > > > > > > > > > > network > > > > > > > > > > > > > > > > > client, > > > > > > > > > > > > > > > > > as I understand) this might be very serious > > > > change > > > > > > and > > > > > > > it > > > > > > > > > > > > probably > > > > > > > > > > > > > > makes > > > > > > > > > > > > > > > > > sense to stick with current approach. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 5:29 PM, Joel > Koshy < > > > > > > > > > > > jjkosh...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> I may be missing some context but > hopefully > > > this > > > > > > will > > > > > > > > > also be > > > > > > > > > > > > > > covered > > > > > > > > > > > > > > > > >> today: I thought the earlier proposal > where > > > > there > > > > > > was > > > > > > > an > > > > > > > > > > > > explicit > > > > > > > > > > > > > > > > >> ClusterMetadata request was clearer and > > > > explicit. > > > > > > > During > > > > > > > > > the > > > > > > > > > > > > > course > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > >> this thread I think the conclusion was > that > > > the > > > > > main > > > > > > > > need > > > > > > > > > was > > > > > > > > > > > > for > > > > > > > > > > > > > > > > >> controller information and that can be > > rolled > > > > into > > > > > > the > > > > > > > > > topic > > > > > > > > > > > > > > metadata > > > > > > > > > > > > > > > > >> response but that seems a bit irrelevant > to > > > > topic > > > > > > > > > metadata. > > > > > > > > > > > > FWIW I > > > > > > > > > > > > > > > > >> think the full broker-list is also > > irrelevant > > > to > > > > > > topic > > > > > > > > > > > metadata, > > > > > > > > > > > > > but > > > > > > > > > > > > > > > > >> it is already there and in use. I think > > there > > > is > > > > > > still > > > > > > > > > room > > > > > > > > > > > for > > > > > > > > > > > > an > > > > > > > > > > > > > > > > >> explicit ClusterMetadata request since > there > > > may > > > > > be > > > > > > > > other > > > > > > > > > > > > > > > > >> cluster-level information that we may want > > to > > > > add > > > > > > over > > > > > > > > > time > > > > > > > > > > > (and > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > >> have nothing to do with topic metadata). > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> On Tue, Mar 17, 2015 at 02:45:30PM +0200, > > > Andrii > > > > > > > > Biletskyi > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > >> > Jun, > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > 101. Okay, if you say that such use case > > is > > > > > > > > important. I > > > > > > > > > > > also > > > > > > > > > > > > > > think > > > > > > > > > > > > > > > > >> > using clientId for these purposes is > fine > > - > > > if > > > > > we > > > > > > > > > already > > > > > > > > > > > have > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > >> field > > > > > > > > > > > > > > > > >> > as part of all Wire protocol messages, > why > > > not > > > > > use > > > > > > > > that. > > > > > > > > > > > > > > > > >> > I will update KIP-4 page if nobody has > > other > > > > > ideas > > > > > > > > > (which > > > > > > > > > > > may > > > > > > > > > > > > > > come up > > > > > > > > > > > > > > > > >> > during the call today). > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > 102.1 Agree, I'll update the KIP > > > accordingly. > > > > I > > > > > > > think > > > > > > > > > we can > > > > > > > > > > > > add > > > > > > > > > > > > > > new, > > > > > > > > > > > > > > > > >> > fine-grained error codes if some error > > code > > > > > > received > > > > > > > > in > > > > > > > > > > > > specific > > > > > > > > > > > > > > case > > > > > > > > > > > > > > > > >> > won't give enough context to return a > > > > > descriptive > > > > > > > > error > > > > > > > > > > > > message > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > >> user. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > Look forward to discussing all > outstanding > > > > > issues > > > > > > in > > > > > > > > > detail > > > > > > > > > > > > > today > > > > > > > > > > > > > > > > during > > > > > > > > > > > > > > > > >> > the call. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > Thanks, > > > > > > > > > > > > > > > > >> > Andrii Biletskyi > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > On Mon, Mar 16, 2015 at 10:59 PM, Jun > Rao > > < > > > > > > > > > j...@confluent.io > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > 101. There may be a use case where you > > > only > > > > > want > > > > > > > the > > > > > > > > > > > topics > > > > > > > > > > > > to > > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > >> created > > > > > > > > > > > > > > > > >> > > manually by admins. Currently, you can > > do > > > > that > > > > > > by > > > > > > > > > > > disabling > > > > > > > > > > > > > auto > > > > > > > > > > > > > > > > topic > > > > > > > > > > > > > > > > >> > > creation and issue topic creation from > > the > > > > > > > > > TopicCommand. > > > > > > > > > > > If > > > > > > > > > > > > we > > > > > > > > > > > > > > > > >> disable auto > > > > > > > > > > > > > > > > >> > > topic creation completely on the > broker > > > and > > > > > > don't > > > > > > > > > have a > > > > > > > > > > > way > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > >> distinguish > > > > > > > > > > > > > > > > >> > > between topic creation requests from > the > > > > > regular > > > > > > > > > clients > > > > > > > > > > > and > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > >> admin, we > > > > > > > > > > > > > > > > >> > > can't support manual topic creation > any > > > > more. > > > > > I > > > > > > > was > > > > > > > > > > > thinking > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > >> another > > > > > > > > > > > > > > > > >> > > way of distinguishing the clients > making > > > the > > > > > > topic > > > > > > > > > > > creation > > > > > > > > > > > > > > requests > > > > > > > > > > > > > > > > >> is > > > > > > > > > > > > > > > > >> > > using clientId. For example, the admin > > > tool > > > > > can > > > > > > > set > > > > > > > > > it to > > > > > > > > > > > > > > something > > > > > > > > > > > > > > > > >> like > > > > > > > > > > > > > > > > >> > > admin and the broker can treat that > > > clientId > > > > > > > > > specially. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > Also, there is a related discussion in > > > > > > KAFKA-2020. > > > > > > > > > > > > Currently, > > > > > > > > > > > > > > we do > > > > > > > > > > > > > > > > >> the > > > > > > > > > > > > > > > > >> > > following in TopicMetadataResponse: > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > 1. If leader is not available, we set > > the > > > > > > > partition > > > > > > > > > level > > > > > > > > > > > > > error > > > > > > > > > > > > > > code > > > > > > > > > > > > > > > > >> to > > > > > > > > > > > > > > > > >> > > LeaderNotAvailable. > > > > > > > > > > > > > > > > >> > > 2. If a non-leader replica is not > > > available, > > > > > we > > > > > > > take > > > > > > > > > that > > > > > > > > > > > > > > replica > > > > > > > > > > > > > > > > out > > > > > > > > > > > > > > > > >> of > > > > > > > > > > > > > > > > >> > > the assigned replica list and isr in > the > > > > > > response. > > > > > > > > As > > > > > > > > > an > > > > > > > > > > > > > > indication > > > > > > > > > > > > > > > > >> for > > > > > > > > > > > > > > > > >> > > doing that, we set the partition level > > > error > > > > > > code > > > > > > > to > > > > > > > > > > > > > > > > >> ReplicaNotAvailable. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > This has a few problems. First, > > > > > > > ReplicaNotAvailable > > > > > > > > > > > probably > > > > > > > > > > > > > > > > >> shouldn't be > > > > > > > > > > > > > > > > >> > > an error, at least for the normal > > > > > > > producer/consumer > > > > > > > > > > > clients > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > just > > > > > > > > > > > > > > > > >> want > > > > > > > > > > > > > > > > >> > > to find out the leader. Second, it can > > > > happen > > > > > > that > > > > > > > > > both > > > > > > > > > > > the > > > > > > > > > > > > > > leader > > > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > >> > > another replica are not available at > the > > > > same > > > > > > > time. > > > > > > > > > There > > > > > > > > > > > is > > > > > > > > > > > > > no > > > > > > > > > > > > > > > > error > > > > > > > > > > > > > > > > >> code > > > > > > > > > > > > > > > > >> > > to indicate both. Third, even if a > > replica > > > > is > > > > > > not > > > > > > > > > > > available, > > > > > > > > > > > > > > it's > > > > > > > > > > > > > > > > >> still > > > > > > > > > > > > > > > > >> > > useful to return its replica id since > > some > > > > > > clients > > > > > > > > > (e.g. > > > > > > > > > > > > admin > > > > > > > > > > > > > > tool) > > > > > > > > > > > > > > > > >> may > > > > > > > > > > > > > > > > >> > > still make use of it. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > One way to address this issue is to > > always > > > > > > return > > > > > > > > the > > > > > > > > > > > > replica > > > > > > > > > > > > > > id for > > > > > > > > > > > > > > > > >> > > leader, assigned replicas, and isr > > > > regardless > > > > > of > > > > > > > > > whether > > > > > > > > > > > the > > > > > > > > > > > > > > > > >> corresponding > > > > > > > > > > > > > > > > >> > > broker is live or not. Since we also > > > return > > > > > the > > > > > > > list > > > > > > > > > of > > > > > > > > > > > live > > > > > > > > > > > > > > > > brokers, > > > > > > > > > > > > > > > > >> the > > > > > > > > > > > > > > > > >> > > client can figure out whether a leader > > or > > > a > > > > > > > replica > > > > > > > > is > > > > > > > > > > > live > > > > > > > > > > > > or > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > >> and act > > > > > > > > > > > > > > > > >> > > accordingly. This way, we don't need > to > > > set > > > > > the > > > > > > > > > partition > > > > > > > > > > > > > level > > > > > > > > > > > > > > > > error > > > > > > > > > > > > > > > > >> code > > > > > > > > > > > > > > > > >> > > when the leader or a replica is not > > > > available. > > > > > > > This > > > > > > > > > > > doesn't > > > > > > > > > > > > > > change > > > > > > > > > > > > > > > > >> the wire > > > > > > > > > > > > > > > > >> > > protocol, but does change the > semantics. > > > > Since > > > > > > we > > > > > > > > are > > > > > > > > > > > > evolving > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > >> protocol > > > > > > > > > > > > > > > > >> > > of TopicMetadataRequest here, we can > > > > > potentially > > > > > > > > > piggyback > > > > > > > > > > > > the > > > > > > > > > > > > > > > > change. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > 102.1 For those types of errors due to > > > > invalid > > > > > > > > input, > > > > > > > > > > > > > shouldn't > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > >> just > > > > > > > > > > > > > > > > >> > > guard it at parameter validation time > > and > > > > > throw > > > > > > > > > > > > > > > > >> InvalidArgumentException > > > > > > > > > > > > > > > > >> > > without even sending the request to > the > > > > > broker? > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > Thanks, > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > Jun > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > On Mon, Mar 16, 2015 at 10:37 AM, > Andrii > > > > > > > Biletskyi < > > > > > > > > > > > > > > > > >> > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > Jun, > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > Answering your questions: > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > 101. If I understand you correctly, > > you > > > > are > > > > > > > saying > > > > > > > > > > > future > > > > > > > > > > > > > > producer > > > > > > > > > > > > > > > > >> > > versions > > > > > > > > > > > > > > > > >> > > > (which > > > > > > > > > > > > > > > > >> > > > will be ported to TMR_V1) won't be > > able > > > to > > > > > > > > > automatically > > > > > > > > > > > > > > create > > > > > > > > > > > > > > > > >> topic (if > > > > > > > > > > > > > > > > >> > > > we > > > > > > > > > > > > > > > > >> > > > unconditionally remove topic > creation > > > from > > > > > > > there). > > > > > > > > > But > > > > > > > > > > > we > > > > > > > > > > > > > > need to > > > > > > > > > > > > > > > > >> this > > > > > > > > > > > > > > > > >> > > > preserve logic. > > > > > > > > > > > > > > > > >> > > > Ok, about your proposal: I'm not a > big > > > fan > > > > > > too, > > > > > > > > > when it > > > > > > > > > > > > > comes > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > >> > > > differentiating > > > > > > > > > > > > > > > > >> > > > clients directly in protocol schema. > > And > > > > > also > > > > > > > I'm > > > > > > > > > not > > > > > > > > > > > > sure I > > > > > > > > > > > > > > > > >> understand > > > > > > > > > > > > > > > > >> > > at > > > > > > > > > > > > > > > > >> > > > all why > > > > > > > > > > > > > > > > >> > > > auto.create.topics.enable is a > server > > > side > > > > > > > > > > > configuration. > > > > > > > > > > > > > Can > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > >> > > deprecate > > > > > > > > > > > > > > > > >> > > > this setting > > > > > > > > > > > > > > > > >> > > > in future versions, add this setting > > to > > > > > > producer > > > > > > > > and > > > > > > > > > > > based > > > > > > > > > > > > > on > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > >> upon > > > > > > > > > > > > > > > > >> > > > receiving > > > > > > > > > > > > > > > > >> > > > UnknownTopic create topic explicitly > > by > > > a > > > > > > > separate > > > > > > > > > > > > producer > > > > > > > > > > > > > > call > > > > > > > > > > > > > > > > via > > > > > > > > > > > > > > > > >> > > > adminClient? > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > 102.1. Hm, yes. It's because we want > > to > > > > > > support > > > > > > > > > batching > > > > > > > > > > > > and > > > > > > > > > > > > > > at > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > >> same > > > > > > > > > > > > > > > > >> > > > time we > > > > > > > > > > > > > > > > >> > > > want to give descriptive error > > messages > > > > for > > > > > > > > clients. > > > > > > > > > > > Since > > > > > > > > > > > > > > > > >> AdminClient > > > > > > > > > > > > > > > > >> > > > holds the context > > > > > > > > > > > > > > > > >> > > > to construct such messages (e.g. > > > > AdminClient > > > > > > > layer > > > > > > > > > can > > > > > > > > > > > > know > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > >> > > > InvalidArgumentsCode > > > > > > > > > > > > > > > > >> > > > means two cases: either invalid > > number - > > > > > e.g. > > > > > > > -1; > > > > > > > > or > > > > > > > > > > > > > > > > >> replication-factor > > > > > > > > > > > > > > > > >> > > was > > > > > > > > > > > > > > > > >> > > > provided while > > > > > > > > > > > > > > > > >> > > > partitions argument wasn't) - I > > wrapped > > > > > > > responses > > > > > > > > in > > > > > > > > > > > > > > Exceptions. > > > > > > > > > > > > > > > > >> But I'm > > > > > > > > > > > > > > > > >> > > > open to any > > > > > > > > > > > > > > > > >> > > > other ideas, this was just initial > > > > version. > > > > > > > > > > > > > > > > >> > > > 102.2. Yes, I agree. I'll change > that > > to > > > > > > > probably > > > > > > > > > some > > > > > > > > > > > > other > > > > > > > > > > > > > > dto. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > Thanks, > > > > > > > > > > > > > > > > >> > > > Andrii Biletskyi > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > On Fri, Mar 13, 2015 at 7:16 PM, Jun > > > Rao < > > > > > > > > > > > > j...@confluent.io> > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > Andrii, > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > 101. That's what I was thinking > too, > > > but > > > > > it > > > > > > > may > > > > > > > > > not be > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > >> simple. In > > > > > > > > > > > > > > > > >> > > > > TopicMetadataRequest_V1, > > > > > > > > > > > > > > > > >> > > > > we can let it not trigger auto > topic > > > > > > creation. > > > > > > > > > Then, > > > > > > > > > > > in > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > >> producer > > > > > > > > > > > > > > > > >> > > > side, > > > > > > > > > > > > > > > > >> > > > > if it gets an > UnknownTopicException, > > > it > > > > > can > > > > > > > > > explicitly > > > > > > > > > > > > > > issue a > > > > > > > > > > > > > > > > >> > > > > createTopicRequest for auto topic > > > > > creation. > > > > > > On > > > > > > > > the > > > > > > > > > > > > > consumer > > > > > > > > > > > > > > > > side, > > > > > > > > > > > > > > > > >> it > > > > > > > > > > > > > > > > >> > > will > > > > > > > > > > > > > > > > >> > > > > never issue createTopicRequest. > This > > > > works > > > > > > > when > > > > > > > > > auto > > > > > > > > > > > > topic > > > > > > > > > > > > > > > > >> creation is > > > > > > > > > > > > > > > > >> > > > > enabled on the broker side. > > However, I > > > > am > > > > > > not > > > > > > > > > sure how > > > > > > > > > > > > > > things > > > > > > > > > > > > > > > > >> will work > > > > > > > > > > > > > > > > >> > > > > when auto topic creation is > disabled > > > on > > > > > the > > > > > > > > broker > > > > > > > > > > > side. > > > > > > > > > > > > > In > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > >> case, > > > > > > > > > > > > > > > > >> > > we > > > > > > > > > > > > > > > > >> > > > > want to have a way to manually > > create > > > a > > > > > > topic, > > > > > > > > > > > > potentially > > > > > > > > > > > > > > > > through > > > > > > > > > > > > > > > > >> > > admin > > > > > > > > > > > > > > > > >> > > > > commands. However, then we need a > > way > > > to > > > > > > > > > distinguish > > > > > > > > > > > > > > > > >> createTopicRequest > > > > > > > > > > > > > > > > >> > > > > issued from the producer clients > and > > > the > > > > > > admin > > > > > > > > > tools. > > > > > > > > > > > > May > > > > > > > > > > > > > > be we > > > > > > > > > > > > > > > > >> can > > > > > > > > > > > > > > > > >> > > add a > > > > > > > > > > > > > > > > >> > > > > new field in createTopicRequest > and > > > set > > > > it > > > > > > > > > differently > > > > > > > > > > > > in > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > >> producer > > > > > > > > > > > > > > > > >> > > > > client and the admin client. > > However, > > > I > > > > am > > > > > > not > > > > > > > > > sure if > > > > > > > > > > > > > > that's > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > >> best > > > > > > > > > > > > > > > > >> > > > > approach. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > 2. Yes, refactoring existing > > requests > > > > is a > > > > > > > > > non-trivial > > > > > > > > > > > > > > amount of > > > > > > > > > > > > > > > > >> work. > > > > > > > > > > > > > > > > >> > > I > > > > > > > > > > > > > > > > >> > > > > posted some comments in > KAFKA-1927. > > We > > > > > will > > > > > > > > > probably > > > > > > > > > > > > have > > > > > > > > > > > > > > to fix > > > > > > > > > > > > > > > > >> > > > KAFKA-1927 > > > > > > > > > > > > > > > > >> > > > > first, before adding the new logic > > in > > > > > > > > KAFKA-1694. > > > > > > > > > > > > > > Otherwise, the > > > > > > > > > > > > > > > > >> > > changes > > > > > > > > > > > > > > > > >> > > > > will be too big. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > 102. About the AdminClient: > > > > > > > > > > > > > > > > >> > > > > 102.1. It's a bit weird that we > > return > > > > > > > exception > > > > > > > > > in > > > > > > > > > > > the > > > > > > > > > > > > > > api. It > > > > > > > > > > > > > > > > >> seems > > > > > > > > > > > > > > > > >> > > > that > > > > > > > > > > > > > > > > >> > > > > we should either return error code > > or > > > > > throw > > > > > > an > > > > > > > > > > > exception > > > > > > > > > > > > > > when > > > > > > > > > > > > > > > > >> getting > > > > > > > > > > > > > > > > >> > > the > > > > > > > > > > > > > > > > >> > > > > response state. > > > > > > > > > > > > > > > > >> > > > > 102.2. We probably shouldn't > > > explicitly > > > > > use > > > > > > > the > > > > > > > > > > > request > > > > > > > > > > > > > > object > > > > > > > > > > > > > > > > in > > > > > > > > > > > > > > > > >> the > > > > > > > > > > > > > > > > >> > > > api. > > > > > > > > > > > > > > > > >> > > > > Not every request evolution > requires > > > an > > > > > api > > > > > > > > > change. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > Thanks, > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > Jun > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > On Fri, Mar 13, 2015 at 4:08 AM, > > > Andrii > > > > > > > > Biletskyi > > > > > > > > > < > > > > > > > > > > > > > > > > >> > > > > andrii.bilets...@stealth.ly> > wrote: > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > Jun, > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > Thanks for you comments. Answers > > > > inline: > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > 100. There are a few fields such > > as > > > > > > > > > > > ReplicaAssignment, > > > > > > > > > > > > > > > > >> > > > > > > ReassignPartitionRequest, > > > > > > > > > > > > > > > > >> > > > > > > and PartitionsSerialized that > > are > > > > > > > > represented > > > > > > > > > as a > > > > > > > > > > > > > > string, > > > > > > > > > > > > > > > > but > > > > > > > > > > > > > > > > >> > > > contain > > > > > > > > > > > > > > > > >> > > > > > > composite structures in json. > > > Could > > > > we > > > > > > > > flatten > > > > > > > > > > > them > > > > > > > > > > > > > out > > > > > > > > > > > > > > > > >> directly in > > > > > > > > > > > > > > > > >> > > > the > > > > > > > > > > > > > > > > >> > > > > > > protocol definition as > > > > arrays/records? > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > Yes, now with Admin Client this > > > looks > > > > a > > > > > > bit > > > > > > > > > weird. > > > > > > > > > > > My > > > > > > > > > > > > > > initial > > > > > > > > > > > > > > > > >> > > > motivation > > > > > > > > > > > > > > > > >> > > > > > was: > > > > > > > > > > > > > > > > >> > > > > > ReassignPartitionCommand accepts > > > input > > > > > in > > > > > > > > json, > > > > > > > > > we > > > > > > > > > > > > want > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > >> remain > > > > > > > > > > > > > > > > >> > > > tools' > > > > > > > > > > > > > > > > >> > > > > > interfaces unchanged, where > > > possible. > > > > > > > > > > > > > > > > >> > > > > > If we port it to deserialized > > > format, > > > > in > > > > > > CLI > > > > > > > > > (/tools > > > > > > > > > > > > > > project) > > > > > > > > > > > > > > > > >> we will > > > > > > > > > > > > > > > > >> > > > > have > > > > > > > > > > > > > > > > >> > > > > > to add some > > > > > > > > > > > > > > > > >> > > > > > json library since /tools is > > written > > > > in > > > > > > java > > > > > > > > and > > > > > > > > > > > we'll > > > > > > > > > > > > > > need to > > > > > > > > > > > > > > > > >> > > > > deserialize > > > > > > > > > > > > > > > > >> > > > > > json file > > > > > > > > > > > > > > > > >> > > > > > provided by a user. Can we > quickly > > > > agree > > > > > > on > > > > > > > > what > > > > > > > > > > > this > > > > > > > > > > > > > > library > > > > > > > > > > > > > > > > >> should > > > > > > > > > > > > > > > > >> > > be > > > > > > > > > > > > > > > > >> > > > > > (Jackson, GSON, whatever)? > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > 101. Does TopicMetadataRequest > v1 > > > > still > > > > > > > > trigger > > > > > > > > > auto > > > > > > > > > > > > > topic > > > > > > > > > > > > > > > > >> creation? > > > > > > > > > > > > > > > > >> > > > This > > > > > > > > > > > > > > > > >> > > > > > > will be a bit weird now that > we > > > > have a > > > > > > > > > separate > > > > > > > > > > > > topic > > > > > > > > > > > > > > > > >> creation api. > > > > > > > > > > > > > > > > >> > > > > Have > > > > > > > > > > > > > > > > >> > > > > > > you thought about how the new > > > > > > > > > createTopicRequest > > > > > > > > > > > and > > > > > > > > > > > > > > > > >> > > > > TopicMetadataRequest > > > > > > > > > > > > > > > > >> > > > > > > v1 will be used in the > > > > > producer/consumer > > > > > > > > > client, > > > > > > > > > > > in > > > > > > > > > > > > > > addition > > > > > > > > > > > > > > > > >> to > > > > > > > > > > > > > > > > >> > > admin > > > > > > > > > > > > > > > > >> > > > > > > tools? For example, ideally, > we > > > > don't > > > > > > want > > > > > > > > > > > > > > > > >> TopicMetadataRequest > > > > > > > > > > > > > > > > >> > > from > > > > > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > > > > > >> > > > > > > consumer to trigger auto topic > > > > > creation. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > I agree, this strange logic > should > > > be > > > > > > fixed. > > > > > > > > > I'm not > > > > > > > > > > > > > > confident > > > > > > > > > > > > > > > > >> in > > > > > > > > > > > > > > > > >> > > this > > > > > > > > > > > > > > > > >> > > > > > Kafka part so > > > > > > > > > > > > > > > > >> > > > > > correct me if I'm wrong, but it > > > > doesn't > > > > > > look > > > > > > > > > like a > > > > > > > > > > > > hard > > > > > > > > > > > > > > thing > > > > > > > > > > > > > > > > >> to > > > > > > > > > > > > > > > > >> > > do, I > > > > > > > > > > > > > > > > >> > > > > > think we can > > > > > > > > > > > > > > > > >> > > > > > leverage AdminClient for that in > > > > > Producer > > > > > > > and > > > > > > > > > > > > > > unconditionally > > > > > > > > > > > > > > > > >> remove > > > > > > > > > > > > > > > > >> > > > > topic > > > > > > > > > > > > > > > > >> > > > > > creation from the > > > > > TopicMetadataRequest_V1. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > 2. I think Jay meant getting rid > > of > > > > > scala > > > > > > > > > classes > > > > > > > > > > > > > > > > >> > > > > > > like HeartbeatRequestAndHeader > > and > > > > > > > > > > > > > > > > >> HeartbeatResponseAndHeader. We > > > > > > > > > > > > > > > > >> > > did > > > > > > > > > > > > > > > > >> > > > > > that > > > > > > > > > > > > > > > > >> > > > > > > as a stop-gap thing when > adding > > > the > > > > > new > > > > > > > > > requests > > > > > > > > > > > for > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > >> consumers. > > > > > > > > > > > > > > > > >> > > > > > > However, the long term plan is > > to > > > > get > > > > > > rid > > > > > > > of > > > > > > > > > all > > > > > > > > > > > > those > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > >> just > > > > > > > > > > > > > > > > >> > > reuse > > > > > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > > > > > >> > > > > > > java request/response in the > > > client. > > > > > > Since > > > > > > > > > this > > > > > > > > > > > KIP > > > > > > > > > > > > > > proposes > > > > > > > > > > > > > > > > >> to > > > > > > > > > > > > > > > > >> > > add a > > > > > > > > > > > > > > > > >> > > > > > > significant number of new > > > requests, > > > > > > > perhaps > > > > > > > > we > > > > > > > > > > > > should > > > > > > > > > > > > > > bite > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > >> > > bullet > > > > > > > > > > > > > > > > >> > > > > to > > > > > > > > > > > > > > > > >> > > > > > > clean up the existing scala > > > requests > > > > > > first > > > > > > > > > before > > > > > > > > > > > > > > adding new > > > > > > > > > > > > > > > > >> ones? > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > Yes, looks like I misunderstood > > the > > > > > point > > > > > > of > > > > > > > > > > > > > > > > >> ...RequestAndHeader. > > > > > > > > > > > > > > > > >> > > > Okay, I > > > > > > > > > > > > > > > > >> > > > > > will > > > > > > > > > > > > > > > > >> > > > > > rework that. The only thing is > > that > > > I > > > > > > don't > > > > > > > > see > > > > > > > > > any > > > > > > > > > > > > > > example > > > > > > > > > > > > > > > > how > > > > > > > > > > > > > > > > >> it > > > > > > > > > > > > > > > > >> > > was > > > > > > > > > > > > > > > > >> > > > > done > > > > > > > > > > > > > > > > >> > > > > > for at > > > > > > > > > > > > > > > > >> > > > > > least one existing protocol > > message. > > > > > Thus, > > > > > > > as > > > > > > > > I > > > > > > > > > > > > > > understand, I > > > > > > > > > > > > > > > > >> have to > > > > > > > > > > > > > > > > >> > > > > think > > > > > > > > > > > > > > > > >> > > > > > how we > > > > > > > > > > > > > > > > >> > > > > > are going to do it. > > > > > > > > > > > > > > > > >> > > > > > Re porting all existing RQ/RP in > > > this > > > > > > patch. > > > > > > > > > Sounds > > > > > > > > > > > > > > > > reasonable, > > > > > > > > > > > > > > > > >> but > > > > > > > > > > > > > > > > >> > > if > > > > > > > > > > > > > > > > >> > > > > it's > > > > > > > > > > > > > > > > >> > > > > > an *obligatory* > > > > > > > > > > > > > > > > >> > > > > > requirement to have Admin KIP > > done, > > > > I'm > > > > > > > afraid > > > > > > > > > this > > > > > > > > > > > > can > > > > > > > > > > > > > > be a > > > > > > > > > > > > > > > > >> serious > > > > > > > > > > > > > > > > >> > > > > > blocker for us. > > > > > > > > > > > > > > > > >> > > > > > There are 13 protocol messages > and > > > all > > > > > > that > > > > > > > > > would > > > > > > > > > > > > > require > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > >> only > > > > > > > > > > > > > > > > >> > > unit > > > > > > > > > > > > > > > > >> > > > > > tests but quite > > > > > > > > > > > > > > > > >> > > > > > intensive manual testing, no? > I'm > > > > afraid > > > > > > I'm > > > > > > > > > not the > > > > > > > > > > > > > > right guy > > > > > > > > > > > > > > > > >> to > > > > > > > > > > > > > > > > >> > > cover > > > > > > > > > > > > > > > > >> > > > > > pretty much all > > > > > > > > > > > > > > > > >> > > > > > Kafka core internals :). Let me > > know > > > > > your > > > > > > > > > thoughts > > > > > > > > > > > on > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > >> item. Btw > > > > > > > > > > > > > > > > >> > > > > there > > > > > > > > > > > > > > > > >> > > > > > is a ticket to > > > > > > > > > > > > > > > > >> > > > > > follow-up this issue ( > > > > > > > > > > > > > > > > >> > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-2006 > > > > > > > > > > > > > > > > >> > > > ). > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > Thanks, > > > > > > > > > > > > > > > > >> > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > On Fri, Mar 13, 2015 at 6:40 AM, > > Jun > > > > > Rao < > > > > > > > > > > > > > > j...@confluent.io> > > > > > > > > > > > > > > > > >> wrote: > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > Andrii, > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > A few more comments. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > 100. There are a few fields > such > > > as > > > > > > > > > > > > ReplicaAssignment, > > > > > > > > > > > > > > > > >> > > > > > > ReassignPartitionRequest, > > > > > > > > > > > > > > > > >> > > > > > > and PartitionsSerialized that > > are > > > > > > > > represented > > > > > > > > > as a > > > > > > > > > > > > > > string, > > > > > > > > > > > > > > > > but > > > > > > > > > > > > > > > > >> > > > contain > > > > > > > > > > > > > > > > >> > > > > > > composite structures in json. > > > Could > > > > we > > > > > > > > flatten > > > > > > > > > > > them > > > > > > > > > > > > > out > > > > > > > > > > > > > > > > >> directly in > > > > > > > > > > > > > > > > >> > > > the > > > > > > > > > > > > > > > > >> > > > > > > protocol definition as > > > > arrays/records? > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > 101. Does TopicMetadataRequest > > v1 > > > > > still > > > > > > > > > trigger > > > > > > > > > > > auto > > > > > > > > > > > > > > topic > > > > > > > > > > > > > > > > >> > > creation? > > > > > > > > > > > > > > > > >> > > > > This > > > > > > > > > > > > > > > > >> > > > > > > will be a bit weird now that > we > > > > have a > > > > > > > > > separate > > > > > > > > > > > > topic > > > > > > > > > > > > > > > > >> creation api. > > > > > > > > > > > > > > > > >> > > > > Have > > > > > > > > > > > > > > > > >> > > > > > > you thought about how the new > > > > > > > > > createTopicRequest > > > > > > > > > > > and > > > > > > > > > > > > > > > > >> > > > > TopicMetadataRequest > > > > > > > > > > > > > > > > >> > > > > > > v1 will be used in the > > > > > producer/consumer > > > > > > > > > client, > > > > > > > > > > > in > > > > > > > > > > > > > > addition > > > > > > > > > > > > > > > > >> to > > > > > > > > > > > > > > > > >> > > admin > > > > > > > > > > > > > > > > >> > > > > > > tools? For example, ideally, > we > > > > don't > > > > > > want > > > > > > > > > > > > > > > > >> TopicMetadataRequest > > > > > > > > > > > > > > > > >> > > from > > > > > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > > > > > >> > > > > > > consumer to trigger auto topic > > > > > creation. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > 2. I think Jay meant getting > rid > > > of > > > > > > scala > > > > > > > > > classes > > > > > > > > > > > > > > > > >> > > > > > > like HeartbeatRequestAndHeader > > and > > > > > > > > > > > > > > > > >> HeartbeatResponseAndHeader. We > > > > > > > > > > > > > > > > >> > > did > > > > > > > > > > > > > > > > >> > > > > > that > > > > > > > > > > > > > > > > >> > > > > > > as a stop-gap thing when > adding > > > the > > > > > new > > > > > > > > > requests > > > > > > > > > > > for > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > >> consumers. > > > > > > > > > > > > > > > > >> > > > > > > However, the long term plan is > > to > > > > get > > > > > > rid > > > > > > > of > > > > > > > > > all > > > > > > > > > > > > those > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > >> just > > > > > > > > > > > > > > > > >> > > reuse > > > > > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > > > > > >> > > > > > > java request/response in the > > > client. > > > > > > Since > > > > > > > > > this > > > > > > > > > > > KIP > > > > > > > > > > > > > > proposes > > > > > > > > > > > > > > > > >> to > > > > > > > > > > > > > > > > >> > > add a > > > > > > > > > > > > > > > > >> > > > > > > significant number of new > > > requests, > > > > > > > perhaps > > > > > > > > we > > > > > > > > > > > > should > > > > > > > > > > > > > > bite > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > >> > > bullet > > > > > > > > > > > > > > > > >> > > > > to > > > > > > > > > > > > > > > > >> > > > > > > clean up the existing scala > > > requests > > > > > > first > > > > > > > > > before > > > > > > > > > > > > > > adding new > > > > > > > > > > > > > > > > >> ones? > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > Thanks, > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > Jun > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > On Thu, Mar 12, 2015 at 3:37 > PM, > > > > > Andrii > > > > > > > > > Biletskyi > > > > > > > > > > > < > > > > > > > > > > > > > > > > >> > > > > > > andrii.bilets...@stealth.ly> > > > wrote: > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > Hi, > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > As said above - I list again > > all > > > > > > > comments > > > > > > > > > from > > > > > > > > > > > > this > > > > > > > > > > > > > > thread > > > > > > > > > > > > > > > > >> so we > > > > > > > > > > > > > > > > >> > > > > > > > can see what's left and > > finalize > > > > all > > > > > > > > pending > > > > > > > > > > > > issues. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > Comments from Jay: > > > > > > > > > > > > > > > > >> > > > > > > > 1. This is much needed > > > > > functionality, > > > > > > > but > > > > > > > > > there > > > > > > > > > > > > are > > > > > > > > > > > > > a > > > > > > > > > > > > > > lot > > > > > > > > > > > > > > > > >> of the > > > > > > > > > > > > > > > > >> > > so > > > > > > > > > > > > > > > > >> > > > > > let's > > > > > > > > > > > > > > > > >> > > > > > > > really think these protocols > > > > > through. > > > > > > We > > > > > > > > > really > > > > > > > > > > > > want > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > end > > > > > > > > > > > > > > > > >> up > > > > > > > > > > > > > > > > >> > > > with a > > > > > > > > > > > > > > > > >> > > > > > set > > > > > > > > > > > > > > > > >> > > > > > > > of well thought-out, > > orthoganol > > > > > apis. > > > > > > > For > > > > > > > > > this > > > > > > > > > > > > > reason > > > > > > > > > > > > > > I > > > > > > > > > > > > > > > > >> think it > > > > > > > > > > > > > > > > >> > > is > > > > > > > > > > > > > > > > >> > > > > > > really > > > > > > > > > > > > > > > > >> > > > > > > > important to think through > the > > > end > > > > > > state > > > > > > > > > even if > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > >> includes > > > > > > > > > > > > > > > > >> > > APIs > > > > > > > > > > > > > > > > >> > > > > we > > > > > > > > > > > > > > > > >> > > > > > > > won't implement in the first > > > > phase. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Definitely behind this. > > Would > > > > > > > > appreciate > > > > > > > > > if > > > > > > > > > > > > there > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > >> concrete > > > > > > > > > > > > > > > > >> > > > > > > comments > > > > > > > > > > > > > > > > >> > > > > > > > how this can be improved. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 2. Let's please please > please > > > wait > > > > > > until > > > > > > > > we > > > > > > > > > have > > > > > > > > > > > > > > switched > > > > > > > > > > > > > > > > >> the > > > > > > > > > > > > > > > > >> > > > server > > > > > > > > > > > > > > > > >> > > > > > over > > > > > > > > > > > > > > > > >> > > > > > > > to the new java protocol > > > > > definitions. > > > > > > If > > > > > > > > we > > > > > > > > > add > > > > > > > > > > > > > upteen > > > > > > > > > > > > > > > > more > > > > > > > > > > > > > > > > >> ad > > > > > > > > > > > > > > > > >> > > hoc > > > > > > > > > > > > > > > > >> > > > > > scala > > > > > > > > > > > > > > > > >> > > > > > > > objects that is just > > generating > > > > more > > > > > > > work > > > > > > > > > for > > > > > > > > > > > the > > > > > > > > > > > > > > > > >> conversion we > > > > > > > > > > > > > > > > >> > > > know > > > > > > > > > > > > > > > > >> > > > > we > > > > > > > > > > > > > > > > >> > > > > > > > have to do. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest > patch - > > > > > removed > > > > > > > > scala > > > > > > > > > > > > > protocol > > > > > > > > > > > > > > > > >> classes. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 3. This proposal introduces > a > > > new > > > > > type > > > > > > > of > > > > > > > > > > > optional > > > > > > > > > > > > > > > > >> parameter. > > > > > > > > > > > > > > > > >> > > This > > > > > > > > > > > > > > > > >> > > > is > > > > > > > > > > > > > > > > >> > > > > > > > inconsistent with everything > > > else > > > > in > > > > > > the > > > > > > > > > > > protocol > > > > > > > > > > > > > > where we > > > > > > > > > > > > > > > > >> use -1 > > > > > > > > > > > > > > > > >> > > > or > > > > > > > > > > > > > > > > >> > > > > > some > > > > > > > > > > > > > > > > >> > > > > > > > other marker value. You > could > > > > argue > > > > > > > either > > > > > > > > > way > > > > > > > > > > > but > > > > > > > > > > > > > > let's > > > > > > > > > > > > > > > > >> stick > > > > > > > > > > > > > > > > >> > > with > > > > > > > > > > > > > > > > >> > > > > > that > > > > > > > > > > > > > > > > >> > > > > > > > for consistency. For clients > > > that > > > > > > > > > implemented > > > > > > > > > > > the > > > > > > > > > > > > > > protocol > > > > > > > > > > > > > > > > >> in a > > > > > > > > > > > > > > > > >> > > > > better > > > > > > > > > > > > > > > > >> > > > > > > way > > > > > > > > > > > > > > > > >> > > > > > > > than our scala code these > > basic > > > > > > > primitives > > > > > > > > > are > > > > > > > > > > > > hard > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > >> change. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest > patch - > > > > > removed > > > > > > > > > MaybeOf > > > > > > > > > > > > type > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > >> changed > > > > > > > > > > > > > > > > >> > > > > > protocol > > > > > > > > > > > > > > > > >> > > > > > > > accordingly. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 4. ClusterMetadata: This > seems > > > to > > > > > > > > duplicate > > > > > > > > > > > > > > > > >> TopicMetadataRequest > > > > > > > > > > > > > > > > >> > > > > which > > > > > > > > > > > > > > > > >> > > > > > > has > > > > > > > > > > > > > > > > >> > > > > > > > brokers, topics, and > > > partitions. I > > > > > > think > > > > > > > > we > > > > > > > > > > > should > > > > > > > > > > > > > > rename > > > > > > > > > > > > > > > > >> that > > > > > > > > > > > > > > > > >> > > > > request > > > > > > > > > > > > > > > > >> > > > > > > > ClusterMetadataRequest (or > > just > > > > > > > > > MetadataRequest) > > > > > > > > > > > > and > > > > > > > > > > > > > > > > >> include the > > > > > > > > > > > > > > > > >> > > id > > > > > > > > > > > > > > > > >> > > > > of > > > > > > > > > > > > > > > > >> > > > > > > the > > > > > > > > > > > > > > > > >> > > > > > > > controller. Or are there > other > > > > > things > > > > > > we > > > > > > > > > could > > > > > > > > > > > add > > > > > > > > > > > > > > here? > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: I agree. Updated the KIP. > > > Let's > > > > > > > extends > > > > > > > > > > > > > > TopicMetadata > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > >> > > > version 2 > > > > > > > > > > > > > > > > >> > > > > > and > > > > > > > > > > > > > > > > >> > > > > > > > include controller. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 5. We have a tendency to try > > to > > > > > make a > > > > > > > lot > > > > > > > > > of > > > > > > > > > > > > > requests > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > >> can > > > > > > > > > > > > > > > > >> > > > only > > > > > > > > > > > > > > > > >> > > > > go > > > > > > > > > > > > > > > > >> > > > > > > to > > > > > > > > > > > > > > > > >> > > > > > > > particular nodes. This adds > a > > > lot > > > > of > > > > > > > > burden > > > > > > > > > for > > > > > > > > > > > > > client > > > > > > > > > > > > > > > > >> > > > > implementations > > > > > > > > > > > > > > > > >> > > > > > > (it > > > > > > > > > > > > > > > > >> > > > > > > > sounds easy but each > discovery > > > can > > > > > > fail > > > > > > > in > > > > > > > > > many > > > > > > > > > > > > > parts > > > > > > > > > > > > > > so > > > > > > > > > > > > > > > > it > > > > > > > > > > > > > > > > >> ends > > > > > > > > > > > > > > > > >> > > up > > > > > > > > > > > > > > > > >> > > > > > > being a > > > > > > > > > > > > > > > > >> > > > > > > > full state machine to do > > > right). I > > > > > > think > > > > > > > > we > > > > > > > > > > > should > > > > > > > > > > > > > > > > consider > > > > > > > > > > > > > > > > >> > > making > > > > > > > > > > > > > > > > >> > > > > > admin > > > > > > > > > > > > > > > > >> > > > > > > > commands and ideally as many > > of > > > > the > > > > > > > other > > > > > > > > > apis > > > > > > > > > > > as > > > > > > > > > > > > > > possible > > > > > > > > > > > > > > > > >> > > > available > > > > > > > > > > > > > > > > >> > > > > on > > > > > > > > > > > > > > > > >> > > > > > > all > > > > > > > > > > > > > > > > >> > > > > > > > brokers and just redirect to > > the > > > > > > > > controller > > > > > > > > > on > > > > > > > > > > > the > > > > > > > > > > > > > > broker > > > > > > > > > > > > > > > > >> side. > > > > > > > > > > > > > > > > >> > > > > Perhaps > > > > > > > > > > > > > > > > >> > > > > > > > there would be a general way > > to > > > > > > > > encapsulate > > > > > > > > > this > > > > > > > > > > > > > > > > re-routing > > > > > > > > > > > > > > > > >> > > > behavior. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: It's a very interesting > > idea, > > > > but > > > > > > > seems > > > > > > > > > there > > > > > > > > > > > > are > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > >> > > concerns > > > > > > > > > > > > > > > > >> > > > > > about > > > > > > > > > > > > > > > > >> > > > > > > > this > > > > > > > > > > > > > > > > >> > > > > > > > feature (like performance > > > > > > > considerations, > > > > > > > > > how > > > > > > > > > > > this > > > > > > > > > > > > > > will > > > > > > > > > > > > > > > > >> > > complicate > > > > > > > > > > > > > > > > >> > > > > > server > > > > > > > > > > > > > > > > >> > > > > > > > etc). > > > > > > > > > > > > > > > > >> > > > > > > > I believe this shouldn't be > a > > > > > blocker. > > > > > > > If > > > > > > > > > this > > > > > > > > > > > > > > feature is > > > > > > > > > > > > > > > > >> > > > implemented > > > > > > > > > > > > > > > > >> > > > > > at > > > > > > > > > > > > > > > > >> > > > > > > > some > > > > > > > > > > > > > > > > >> > > > > > > > point it won't affect Admin > > > > changes > > > > > - > > > > > > at > > > > > > > > > least > > > > > > > > > > > no > > > > > > > > > > > > > > changes > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > >> > > public > > > > > > > > > > > > > > > > >> > > > > API > > > > > > > > > > > > > > > > >> > > > > > > > will be required. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 6. We should probably > > normalize > > > > the > > > > > > key > > > > > > > > > value > > > > > > > > > > > > pairs > > > > > > > > > > > > > > used > > > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > >> > > > configs > > > > > > > > > > > > > > > > >> > > > > > > rather > > > > > > > > > > > > > > > > >> > > > > > > > than embedding a new > > formatting. > > > > So > > > > > > two > > > > > > > > > strings > > > > > > > > > > > > > rather > > > > > > > > > > > > > > > > than > > > > > > > > > > > > > > > > >> one > > > > > > > > > > > > > > > > >> > > > with > > > > > > > > > > > > > > > > >> > > > > an > > > > > > > > > > > > > > > > >> > > > > > > > internal equals sign. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest > patch - > > > > > > > normalized > > > > > > > > > > > configs > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > >> changed > > > > > > > > > > > > > > > > >> > > > > protocol > > > > > > > > > > > > > > > > >> > > > > > > > accordingly. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 7. Is the postcondition of > > these > > > > > APIs > > > > > > > that > > > > > > > > > the > > > > > > > > > > > > > > command has > > > > > > > > > > > > > > > > >> begun > > > > > > > > > > > > > > > > >> > > or > > > > > > > > > > > > > > > > >> > > > > > that > > > > > > > > > > > > > > > > >> > > > > > > > the command has been > > completed? > > > It > > > > > is > > > > > > a > > > > > > > > lot > > > > > > > > > more > > > > > > > > > > > > > > usable if > > > > > > > > > > > > > > > > >> the > > > > > > > > > > > > > > > > >> > > > > command > > > > > > > > > > > > > > > > >> > > > > > > has > > > > > > > > > > > > > > > > >> > > > > > > > been completed so you know > > that > > > if > > > > > you > > > > > > > > > create a > > > > > > > > > > > > > topic > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > >> then > > > > > > > > > > > > > > > > >> > > > > publish > > > > > > > > > > > > > > > > >> > > > > > to > > > > > > > > > > > > > > > > >> > > > > > > > it you won't get an > exception > > > > about > > > > > > > there > > > > > > > > > being > > > > > > > > > > > no > > > > > > > > > > > > > > such > > > > > > > > > > > > > > > > >> topic. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: For long running requests > > > (like > > > > > > > > reassign > > > > > > > > > > > > > > partitions) - > > > > > > > > > > > > > > > > >> the > > > > > > > > > > > > > > > > >> > > post > > > > > > > > > > > > > > > > >> > > > > > > > condition is > > > > > > > > > > > > > > > > >> > > > > > > > command has begun - so we > > don't > > > > > block > > > > > > > the > > > > > > > > > > > client. > > > > > > > > > > > > In > > > > > > > > > > > > > > case > > > > > > > > > > > > > > > > >> of your > > > > > > > > > > > > > > > > >> > > > > > > example - > > > > > > > > > > > > > > > > >> > > > > > > > topic commands, this will be > > > > > > refactored > > > > > > > > and > > > > > > > > > > > topic > > > > > > > > > > > > > > commands > > > > > > > > > > > > > > > > >> will > > > > > > > > > > > > > > > > >> > > be > > > > > > > > > > > > > > > > >> > > > > > > executed > > > > > > > > > > > > > > > > >> > > > > > > > immediately, since the > > > Controller > > > > > will > > > > > > > > serve > > > > > > > > > > > Admin > > > > > > > > > > > > > > > > requests > > > > > > > > > > > > > > > > >> > > > > > > > (follow-up ticket > KAFKA-1777). > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 8. Describe topic and list > > > topics > > > > > > > > duplicate > > > > > > > > > a > > > > > > > > > > > lot > > > > > > > > > > > > of > > > > > > > > > > > > > > stuff > > > > > > > > > > > > > > > > >> in the > > > > > > > > > > > > > > > > >> > > > > > > metadata > > > > > > > > > > > > > > > > >> > > > > > > > request. Is there a reason > to > > > give > > > > > > back > > > > > > > > > topics > > > > > > > > > > > > > marked > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > >> > > > deletion? I > > > > > > > > > > > > > > > > >> > > > > > > feel > > > > > > > > > > > > > > > > >> > > > > > > > like if we just make the > > > > > > post-condition > > > > > > > of > > > > > > > > > the > > > > > > > > > > > > > delete > > > > > > > > > > > > > > > > >> command be > > > > > > > > > > > > > > > > >> > > > that > > > > > > > > > > > > > > > > >> > > > > > the > > > > > > > > > > > > > > > > >> > > > > > > > topic is deleted that will > get > > > rid > > > > > of > > > > > > > the > > > > > > > > > need > > > > > > > > > > > for > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > >> right? > > > > > > > > > > > > > > > > >> > > And > > > > > > > > > > > > > > > > >> > > > it > > > > > > > > > > > > > > > > >> > > > > > > will > > > > > > > > > > > > > > > > >> > > > > > > > be much more intuitive. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest > patch - > > > > > removed > > > > > > > > > topics > > > > > > > > > > > > marked > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > >> deletion > > > > > > > > > > > > > > > > >> > > > in > > > > > > > > > > > > > > > > >> > > > > > > > ListTopicsRequest. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 9. Should we consider > batching > > > > these > > > > > > > > > requests? > > > > > > > > > > > We > > > > > > > > > > > > > have > > > > > > > > > > > > > > > > >> generally > > > > > > > > > > > > > > > > >> > > > > tried > > > > > > > > > > > > > > > > >> > > > > > to > > > > > > > > > > > > > > > > >> > > > > > > > allow multiple operations to > > be > > > > > > batched. > > > > > > > > My > > > > > > > > > > > > > suspicion > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > >> that > > > > > > > > > > > > > > > > >> > > > without > > > > > > > > > > > > > > > > >> > > > > > > this > > > > > > > > > > > > > > > > >> > > > > > > > we will get a lot of code > that > > > > does > > > > > > > > > something > > > > > > > > > > > like > > > > > > > > > > > > > > > > >> > > > > > > > for(topic: > > > > > > adminClient.listTopics()) > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > adminClient.describeTopic(topic) > > > > > > > > > > > > > > > > >> > > > > > > > this code will work great > when > > > you > > > > > > test > > > > > > > > on 5 > > > > > > > > > > > > topics > > > > > > > > > > > > > > but > > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > > >> do as > > > > > > > > > > > > > > > > >> > > > > well > > > > > > > > > > > > > > > > >> > > > > > if > > > > > > > > > > > > > > > > >> > > > > > > > you have 50k. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please > > > check > > > > > > "Topic > > > > > > > > > Admin > > > > > > > > > > > > > Schema" > > > > > > > > > > > > > > > > >> section. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 10. I think we should also > > > discuss > > > > > how > > > > > > > we > > > > > > > > > want > > > > > > > > > > > to > > > > > > > > > > > > > > expose a > > > > > > > > > > > > > > > > >> > > > > programmatic > > > > > > > > > > > > > > > > >> > > > > > > JVM > > > > > > > > > > > > > > > > >> > > > > > > > client api for these > > operations. > > > > > > > Currently > > > > > > > > > > > people > > > > > > > > > > > > > > rely on > > > > > > > > > > > > > > > > >> > > > AdminUtils > > > > > > > > > > > > > > > > >> > > > > > > which > > > > > > > > > > > > > > > > >> > > > > > > > is totally sketchy. I think > we > > > > > > probably > > > > > > > > need > > > > > > > > > > > > another > > > > > > > > > > > > > > > > client > > > > > > > > > > > > > > > > >> under > > > > > > > > > > > > > > > > >> > > > > > > clients/ > > > > > > > > > > > > > > > > >> > > > > > > > that exposes administrative > > > > > > > functionality. > > > > > > > > > We > > > > > > > > > > > will > > > > > > > > > > > > > > need > > > > > > > > > > > > > > > > >> this just > > > > > > > > > > > > > > > > >> > > > to > > > > > > > > > > > > > > > > >> > > > > > > > properly test the new apis, > I > > > > > suspect. > > > > > > > We > > > > > > > > > should > > > > > > > > > > > > > > figure > > > > > > > > > > > > > > > > out > > > > > > > > > > > > > > > > >> that > > > > > > > > > > > > > > > > >> > > > API. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please > > > check > > > > > > "Admin > > > > > > > > > Client" > > > > > > > > > > > > > > section > > > > > > > > > > > > > > > > >> with an > > > > > > > > > > > > > > > > >> > > > > > initial > > > > > > > > > > > > > > > > >> > > > > > > > API proposal. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 11. The other information > that > > > > would > > > > > > be > > > > > > > > > really > > > > > > > > > > > > > useful > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > get > > > > > > > > > > > > > > > > >> > > would > > > > > > > > > > > > > > > > >> > > > be > > > > > > > > > > > > > > > > >> > > > > > > > information about > > > partitions--how > > > > > much > > > > > > > > data > > > > > > > > > is > > > > > > > > > > > in > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > >> partition, > > > > > > > > > > > > > > > > >> > > > what > > > > > > > > > > > > > > > > >> > > > > > are > > > > > > > > > > > > > > > > >> > > > > > > > the segment offsets, what is > > the > > > > > > log-end > > > > > > > > > offset > > > > > > > > > > > > > (i.e. > > > > > > > > > > > > > > last > > > > > > > > > > > > > > > > >> > > offset), > > > > > > > > > > > > > > > > >> > > > > > what > > > > > > > > > > > > > > > > >> > > > > > > is > > > > > > > > > > > > > > > > >> > > > > > > > the compaction point, etc. I > > > think > > > > > > that > > > > > > > > done > > > > > > > > > > > right > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > >> would be > > > > > > > > > > > > > > > > >> > > > the > > > > > > > > > > > > > > > > >> > > > > > > > successor to the very > awkward > > > > > > > > OffsetRequest > > > > > > > > > we > > > > > > > > > > > > have > > > > > > > > > > > > > > today. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: I removed > > > > > > ConsumerGroupOffsetsRequest > > > > > > > > in > > > > > > > > > the > > > > > > > > > > > > > latest > > > > > > > > > > > > > > > > >> patch. I > > > > > > > > > > > > > > > > >> > > > > believe > > > > > > > > > > > > > > > > >> > > > > > > > this should > > > > > > > > > > > > > > > > >> > > > > > > > be resolved in a separate > KIP > > / > > > > jira > > > > > > > > ticket. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 12. Generally we can do good > > > error > > > > > > > > handling > > > > > > > > > > > > without > > > > > > > > > > > > > > > > needing > > > > > > > > > > > > > > > > >> > > custom > > > > > > > > > > > > > > > > >> > > > > > > > server-side > > > > > > > > > > > > > > > > >> > > > > > > > messages. I.e. generally the > > > > client > > > > > > has > > > > > > > > the > > > > > > > > > > > > context > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > know > > > > > > > > > > > > > > > > >> that > > > > > > > > > > > > > > > > >> > > if > > > > > > > > > > > > > > > > >> > > > > it > > > > > > > > > > > > > > > > >> > > > > > > got > > > > > > > > > > > > > > > > >> > > > > > > > an error that the topic > > doesn't > > > > > exist > > > > > > to > > > > > > > > say > > > > > > > > > > > > "Topic > > > > > > > > > > > > > X > > > > > > > > > > > > > > > > >> doesn't > > > > > > > > > > > > > > > > >> > > > exist" > > > > > > > > > > > > > > > > >> > > > > > > rather > > > > > > > > > > > > > > > > >> > > > > > > > than "error code 14" (or > > > > whatever). > > > > > > > Maybe > > > > > > > > > there > > > > > > > > > > > > are > > > > > > > > > > > > > > > > specific > > > > > > > > > > > > > > > > >> > > cases > > > > > > > > > > > > > > > > >> > > > > > where > > > > > > > > > > > > > > > > >> > > > > > > > this is hard? If we want to > > add > > > > > > > > server-side > > > > > > > > > > > error > > > > > > > > > > > > > > messages > > > > > > > > > > > > > > > > >> we > > > > > > > > > > > > > > > > >> > > > really > > > > > > > > > > > > > > > > >> > > > > do > > > > > > > > > > > > > > > > >> > > > > > > > need to do this in a > > consistent > > > > way > > > > > > > across > > > > > > > > > the > > > > > > > > > > > > > > protocol. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please > > > check > > > > > > > > "Protocol > > > > > > > > > > > > Errors" > > > > > > > > > > > > > > > > >> section. I > > > > > > > > > > > > > > > > >> > > > added > > > > > > > > > > > > > > > > >> > > > > > the > > > > > > > > > > > > > > > > >> > > > > > > > comprehensive, fine-grained > > list > > > > of > > > > > > > error > > > > > > > > > codes. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > Comments from Guozhang: > > > > > > > > > > > > > > > > >> > > > > > > > 13. Describe topic request: > it > > > > would > > > > > > be > > > > > > > > > great to > > > > > > > > > > > > go > > > > > > > > > > > > > > beyond > > > > > > > > > > > > > > > > >> just > > > > > > > > > > > > > > > > >> > > > > > batching > > > > > > > > > > > > > > > > >> > > > > > > on > > > > > > > > > > > > > > > > >> > > > > > > > topic name regex for this > > > request. > > > > > For > > > > > > > > > example, > > > > > > > > > > > a > > > > > > > > > > > > > very > > > > > > > > > > > > > > > > >> common use > > > > > > > > > > > > > > > > >> > > > > case > > > > > > > > > > > > > > > > >> > > > > > of > > > > > > > > > > > > > > > > >> > > > > > > > the topic command is to list > > all > > > > > > topics > > > > > > > > > whose > > > > > > > > > > > > config > > > > > > > > > > > > > > A's > > > > > > > > > > > > > > > > >> value is > > > > > > > > > > > > > > > > >> > > > B. > > > > > > > > > > > > > > > > >> > > > > > With > > > > > > > > > > > > > > > > >> > > > > > > > topic name regex then we > have > > to > > > > > first > > > > > > > > > retrieve > > > > > > > > > > > > > > __all__ > > > > > > > > > > > > > > > > >> topics's > > > > > > > > > > > > > > > > >> > > > > > > > description info and then > > filter > > > > at > > > > > > the > > > > > > > > > client > > > > > > > > > > > > end, > > > > > > > > > > > > > > which > > > > > > > > > > > > > > > > >> will > > > > > > > > > > > > > > > > >> > > be a > > > > > > > > > > > > > > > > >> > > > > > huge > > > > > > > > > > > > > > > > >> > > > > > > > burden on ZK. > > > > > > > > > > > > > > > > >> > > > > > > > AND > > > > > > > > > > > > > > > > >> > > > > > > > 14. Config K-Vs in create > > topic: > > > > > this > > > > > > is > > > > > > > > > related > > > > > > > > > > > > to > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > >> previous > > > > > > > > > > > > > > > > >> > > > > point; > > > > > > > > > > > > > > > > >> > > > > > > > maybe we can add another > > > metadata > > > > > K-V > > > > > > or > > > > > > > > > just a > > > > > > > > > > > > > > metadata > > > > > > > > > > > > > > > > >> string > > > > > > > > > > > > > > > > >> > > > along > > > > > > > > > > > > > > > > >> > > > > > > side > > > > > > > > > > > > > > > > >> > > > > > > > with config K-V in create > > topic > > > > like > > > > > > we > > > > > > > > did > > > > > > > > > for > > > > > > > > > > > > > offset > > > > > > > > > > > > > > > > >> commit > > > > > > > > > > > > > > > > >> > > > > request. > > > > > > > > > > > > > > > > >> > > > > > > This > > > > > > > > > > > > > > > > >> > > > > > > > field can be quite useful in > > > > storing > > > > > > > > > information > > > > > > > > > > > > > like > > > > > > > > > > > > > > > > >> "owner" of > > > > > > > > > > > > > > > > >> > > > the > > > > > > > > > > > > > > > > >> > > > > > > topic > > > > > > > > > > > > > > > > >> > > > > > > > who issue the create > command, > > > etc, > > > > > > which > > > > > > > > is > > > > > > > > > > > quite > > > > > > > > > > > > > > > > important > > > > > > > > > > > > > > > > >> for a > > > > > > > > > > > > > > > > >> > > > > > > > multi-tenant setting. Then > in > > > the > > > > > > > describe > > > > > > > > > topic > > > > > > > > > > > > > > request > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > >> can > > > > > > > > > > > > > > > > >> > > > also > > > > > > > > > > > > > > > > >> > > > > > > batch > > > > > > > > > > > > > > > > >> > > > > > > > on regex of the metadata > > field. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: As discussed it is very > > > > > interesting > > > > > > > but > > > > > > > > > can > > > > > > > > > > > be > > > > > > > > > > > > > > > > >> implemented > > > > > > > > > > > > > > > > >> > > later > > > > > > > > > > > > > > > > >> > > > > > after > > > > > > > > > > > > > > > > >> > > > > > > > we have some basic > > functionality > > > > > > there. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > 15. Today all the admin > > > operations > > > > > are > > > > > > > > > async in > > > > > > > > > > > > the > > > > > > > > > > > > > > sense > > > > > > > > > > > > > > > > >> that > > > > > > > > > > > > > > > > >> > > > > command > > > > > > > > > > > > > > > > >> > > > > > > will > > > > > > > > > > > > > > > > >> > > > > > > > return once it is written in > > ZK, > > > > and > > > > > > > that > > > > > > > > > is why > > > > > > > > > > > > we > > > > > > > > > > > > > > need > > > > > > > > > > > > > > > > >> extra > > > > > > > > > > > > > > > > >> > > > > > > verification > > > > > > > > > > > > > > > > >> > > > > > > > like > > > > testUtil.waitForTopicCreated() > > > > > / > > > > > > > > verify > > > > > > > > > > > > > partition > > > > > > > > > > > > > > > > >> > > reassignment > > > > > > > > > > > > > > > > >> > > > > > > > request, etc. With admin > > > requests > > > > we > > > > > > > could > > > > > > > > > add a > > > > > > > > > > > > > flag > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > >> enable / > > > > > > > > > > > > > > > > >> > > > > > disable > > > > > > > > > > > > > > > > >> > > > > > > > synchronous requests; when > it > > is > > > > > > turned > > > > > > > > on, > > > > > > > > > the > > > > > > > > > > > > > > response > > > > > > > > > > > > > > > > >> will not > > > > > > > > > > > > > > > > >> > > > > > return > > > > > > > > > > > > > > > > >> > > > > > > > until the request has been > > > > > completed. > > > > > > > And > > > > > > > > > for > > > > > > > > > > > > async > > > > > > > > > > > > > > > > >> requests we > > > > > > > > > > > > > > > > >> > > can > > > > > > > > > > > > > > > > >> > > > > > add a > > > > > > > > > > > > > > > > >> > > > > > > > "token" field in the > response, > > > and > > > > > > then > > > > > > > > only > > > > > > > > > > > need > > > > > > > > > > > > a > > > > > > > > > > > > > > > > general > > > > > > > > > > > > > > > > >> > > "admin > > > > > > > > > > > > > > > > >> > > > > > > > verification request" with > the > > > > given > > > > > > > token > > > > > > > > > to > > > > > > > > > > > > check > > > > > > > > > > > > > > if the > > > > > > > > > > > > > > > > >> async > > > > > > > > > > > > > > > > >> > > > > > request > > > > > > > > > > > > > > > > >> > > > > > > > has been completed. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: I see your point. My idea > > was > > > > to > > > > > > > > provide > > > > > > > > > > > > specific > > > > > > > > > > > > > > > > >> > > > Verify...Request > > > > > > > > > > > > > > > > >> > > > > > per > > > > > > > > > > > > > > > > >> > > > > > > > each > > > > > > > > > > > > > > > > >> > > > > > > > long running request, where > > > > needed. > > > > > We > > > > > > > can > > > > > > > > > do it > > > > > > > > > > > > the > > > > > > > > > > > > > > way > > > > > > > > > > > > > > > > you > > > > > > > > > > > > > > > > >> > > > suggest. > > > > > > > > > > > > > > > > >> > > > > > The > > > > > > > > > > > > > > > > >> > > > > > > > only > > > > > > > > > > > > > > > > >> > > > > > > > concern is that introducing > a > > > > token > > > > > we > > > > > > > > again > > > > > > > > > > > will > > > > > > > > > > > > > make > > > > > > > > > > > > > > > > >> schema > > > > > > > > > > > > > > > > >> > > > > > "dynamic". > > > > > > > > > > > > > > > > >> > > > > > > We > > > > > > > > > > > > > > > > >> > > > > > > > wanted > > > > > > > > > > > > > > > > >> > > > > > > > to do similar thing > > introducing > > > > > single > > > > > > > > > > > > AdminRequest > > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > all > > > > > > > > > > > > > > > > >> topic > > > > > > > > > > > > > > > > >> > > > > > > commands > > > > > > > > > > > > > > > > >> > > > > > > > but rejected > > > > > > > > > > > > > > > > >> > > > > > > > this idea because we wanted > to > > > > have > > > > > > > schema > > > > > > > > > > > > defined. > > > > > > > > > > > > > So > > > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > >> is > > > > > > > > > > > > > > > > >> > > > more a > > > > > > > > > > > > > > > > >> > > > > > > > choice between: > > > > > > > > > > > > > > > > >> > > > > > > > a) have fixed schema but > > > introduce > > > > > > each > > > > > > > > > time new > > > > > > > > > > > > > > > > >> Verify...Request > > > > > > > > > > > > > > > > >> > > > for > > > > > > > > > > > > > > > > >> > > > > > > > long-running requests > > > > > > > > > > > > > > > > >> > > > > > > > b) use one request for > > > > verification > > > > > > but > > > > > > > > > > > generalize > > > > > > > > > > > > > it > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > >> token > > > > > > > > > > > > > > > > >> > > > > > > > I'm fine with whatever > > decision > > > > > > > community > > > > > > > > > come > > > > > > > > > > > to. > > > > > > > > > > > > > > Just > > > > > > > > > > > > > > > > let > > > > > > > > > > > > > > > > >> me > > > > > > > > > > > > > > > > >> > > know > > > > > > > > > > > > > > > > >> > > > > > your > > > > > > > > > > > > > > > > >> > > > > > > > thoughts. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > Comment from Gwen: > > > > > > > > > > > > > > > > >> > > > > > > > 16. Specifically for > > ownership, > > > I > > > > > > think > > > > > > > > the > > > > > > > > > plan > > > > > > > > > > > > is > > > > > > > > > > > > > > to add > > > > > > > > > > > > > > > > >> ACL > > > > > > > > > > > > > > > > >> > > (it > > > > > > > > > > > > > > > > >> > > > > > sounds > > > > > > > > > > > > > > > > >> > > > > > > > like you are describing ACL) > > via > > > > an > > > > > > > > external > > > > > > > > > > > > system > > > > > > > > > > > > > > > > (Argus, > > > > > > > > > > > > > > > > >> > > > Sentry). > > > > > > > > > > > > > > > > >> > > > > > > > I remember KIP-11 described > > > this, > > > > > but > > > > > > I > > > > > > > > > can't > > > > > > > > > > > find > > > > > > > > > > > > > > the KIP > > > > > > > > > > > > > > > > >> any > > > > > > > > > > > > > > > > >> > > > > longer. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > A: Okay, no problem. Not > sure > > > > though > > > > > > how > > > > > > > > we > > > > > > > > > are > > > > > > > > > > > > > going > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > >> handle > > > > > > > > > > > > > > > > >> > > it. > > > > > > > > > > > > > > > > >> > > > > > Wait > > > > > > > > > > > > > > > > >> > > > > > > > which KIP > > > > > > > > > > > > > > > > >> > > > > > > > will be committed first and > > > > include > > > > > > > > changes > > > > > > > > > to > > > > > > > > > > > > > > > > >> TopicMetadata from > > > > > > > > > > > > > > > > >> > > > the > > > > > > > > > > > > > > > > >> > > > > > > later > > > > > > > > > > > > > > > > >> > > > > > > > one? > > > > > > > > > > > > > > > > >> > > > > > > > Anyway, I added this note to > > > "Open > > > > > > > > > Questions" > > > > > > > > > > > > > section > > > > > > > > > > > > > > so > > > > > > > > > > > > > > > > we > > > > > > > > > > > > > > > > >> don't > > > > > > > > > > > > > > > > >> > > > > miss > > > > > > > > > > > > > > > > >> > > > > > > this > > > > > > > > > > > > > > > > >> > > > > > > > piece. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > Thanks, > > > > > > > > > > > > > > > > >> > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > On Fri, Mar 13, 2015 at > 12:34 > > > AM, > > > > > > Andrii > > > > > > > > > > > > Biletskyi < > > > > > > > > > > > > > > > > >> > > > > > > > andrii.bilets...@stealth.ly > > > > > > wrote: > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > Hi all, > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > Today I uploaded the patch > > > that > > > > > > covers > > > > > > > > > some of > > > > > > > > > > > > the > > > > > > > > > > > > > > > > >> discussed > > > > > > > > > > > > > > > > >> > > and > > > > > > > > > > > > > > > > >> > > > > > agreed > > > > > > > > > > > > > > > > >> > > > > > > > > items: > > > > > > > > > > > > > > > > >> > > > > > > > > - removed MaybeOf optional > > > type > > > > > > > > > > > > > > > > >> > > > > > > > > - switched to java > protocol > > > > > > > definitions > > > > > > > > > > > > > > > > >> > > > > > > > > - simplified messages > > > > (normalized > > > > > > > > configs, > > > > > > > > > > > > removed > > > > > > > > > > > > > > topic > > > > > > > > > > > > > > > > >> marked > > > > > > > > > > > > > > > > >> > > > for > > > > > > > > > > > > > > > > >> > > > > > > > > deletion) > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > I also updated the KIP-4 > > with > > > > > > > respective > > > > > > > > > > > changes > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > >> wrote down > > > > > > > > > > > > > > > > >> > > > my > > > > > > > > > > > > > > > > >> > > > > > > > > proposal for > > > > > > > > > > > > > > > > >> > > > > > > > > pending items: > > > > > > > > > > > > > > > > >> > > > > > > > > - Batch Admin Operations > -> > > > > > updated > > > > > > > Wire > > > > > > > > > > > > Protocol > > > > > > > > > > > > > > schema > > > > > > > > > > > > > > > > >> > > proposal > > > > > > > > > > > > > > > > >> > > > > > > > > - Remove ClusterMetadata > -> > > > > > changed > > > > > > to > > > > > > > > > extend > > > > > > > > > > > > > > > > >> > > > TopicMetadataRequest > > > > > > > > > > > > > > > > >> > > > > > > > > - Admin Client -> updated > my > > > > > initial > > > > > > > > > proposal > > > > > > > > > > > to > > > > > > > > > > > > > > reflect > > > > > > > > > > > > > > > > >> > > batching > > > > > > > > > > > > > > > > >> > > > > > > > > - Error codes -> proposed > > > > > > fine-grained > > > > > > > > > error > > > > > > > > > > > > code > > > > > > > > > > > > > > > > instead > > > > > > > > > > > > > > > > >> of > > > > > > > > > > > > > > > > >> > > > > > > > > AdminRequestFailed > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > I will also send a > separate > > > > email > > > > > to > > > > > > > > > cover all > > > > > > > > > > > > > > comments > > > > > > > > > > > > > > > > >> from > > > > > > > > > > > > > > > > >> > > this > > > > > > > > > > > > > > > > >> > > > > > > thread. > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > Thanks, > > > > > > > > > > > > > > > > >> > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > On Thu, Mar 12, 2015 at > 9:26 > > > PM, > > > > > > Gwen > > > > > > > > > Shapira > > > > > > > > > > > < > > > > > > > > > > > > > > > > >> > > > > gshap...@cloudera.com > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > wrote: > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> Found KIP-11 ( > > > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface > > > > > > > > > > > > > > > > >> > > > > > > > >> ) > > > > > > > > > > > > > > > > >> > > > > > > > >> It actually specifies > > changes > > > > to > > > > > > the > > > > > > > > > Metadata > > > > > > > > > > > > > > protocol, > > > > > > > > > > > > > > > > >> so > > > > > > > > > > > > > > > > >> > > > making > > > > > > > > > > > > > > > > >> > > > > > sure > > > > > > > > > > > > > > > > >> > > > > > > > >> both KIPs are consistent > in > > > > this > > > > > > > regard > > > > > > > > > will > > > > > > > > > > > be > > > > > > > > > > > > > > good. > > > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> On Thu, Mar 12, 2015 at > > 12:21 > > > > PM, > > > > > > > Gwen > > > > > > > > > > > Shapira > > > > > > > > > > > > < > > > > > > > > > > > > > > > > >> > > > > > gshap...@cloudera.com > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > > > > > > > > >> > > > > > > > >> > Specifically for > > > ownership, I > > > > > > think > > > > > > > > the > > > > > > > > > > > plan > > > > > > > > > > > > is > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > add > > > > > > > > > > > > > > > > >> ACL > > > > > > > > > > > > > > > > >> > > (it > > > > > > > > > > > > > > > > >> > > > > > > sounds > > > > > > > > > > > > > > > > >> > > > > > > > >> > like you are describing > > > ACL) > > > > > via > > > > > > an > > > > > > > > > > > external > > > > > > > > > > > > > > system > > > > > > > > > > > > > > > > >> (Argus, > > > > > > > > > > > > > > > > >> > > > > > Sentry). > > > > > > > > > > > > > > > > >> > > > > > > > >> > I remember KIP-11 > > described > > > > > this, > > > > > > > > but I > > > > > > > > > > > can't > > > > > > > > > > > > > > find > > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > >> KIP > > > > > > > > > > > > > > > > >> > > any > > > > > > > > > > > > > > > > >> > > > > > > longer. > > > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> > Regardless, I think > KIP-4 > > > > > focuses > > > > > > > on > > > > > > > > > > > getting > > > > > > > > > > > > > > > > >> information > > > > > > > > > > > > > > > > >> > > that > > > > > > > > > > > > > > > > >> > > > > > > already > > > > > > > > > > > > > > > > >> > > > > > > > >> > exists from Kafka > > brokers, > > > > not > > > > > on > > > > > > > > > adding > > > > > > > > > > > > > > information > > > > > > > > > > > > > > > > >> that > > > > > > > > > > > > > > > > >> > > > > perhaps > > > > > > > > > > > > > > > > >> > > > > > > > >> > should exist but > doesn't > > > yet? > > > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> > Gwen > > > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> > On Thu, Mar 12, 2015 at > > > 6:37 > > > > > AM, > > > > > > > > > Guozhang > > > > > > > > > > > > Wang > > > > > > > > > > > > > < > > > > > > > > > > > > > > > > >> > > > > > wangg...@gmail.com> > > > > > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > > > > > > > > >> > > > > > > > >> >> Folks, > > > > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >> Just want to > elaborate a > > > bit > > > > > > more > > > > > > > on > > > > > > > > > the > > > > > > > > > > > > > > > > create-topic > > > > > > > > > > > > > > > > >> > > > metadata > > > > > > > > > > > > > > > > >> > > > > > and > > > > > > > > > > > > > > > > >> > > > > > > > >> batching > > > > > > > > > > > > > > > > >> > > > > > > > >> >> describe-topic based > on > > > > > config / > > > > > > > > > metadata > > > > > > > > > > > in > > > > > > > > > > > > > my > > > > > > > > > > > > > > > > >> previous > > > > > > > > > > > > > > > > >> > > > email > > > > > > > > > > > > > > > > >> > > > > as > > > > > > > > > > > > > > > > >> > > > > > > we > > > > > > > > > > > > > > > > >> > > > > > > > >> work > > > > > > > > > > > > > > > > >> > > > > > > > >> >> on KAFKA-1694. The > main > > > > > > motivation > > > > > > > > is > > > > > > > > > to > > > > > > > > > > > > have > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > >> sort of > > > > > > > > > > > > > > > > >> > > > > topic > > > > > > > > > > > > > > > > >> > > > > > > > >> management > > > > > > > > > > > > > > > > >> > > > > > > > >> >> mechanisms, which I > > think > > > is > > > > > > quite > > > > > > > > > > > important > > > > > > > > > > > > > in > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > >> > > > multi-tenant > > > > > > > > > > > > > > > > >> > > > > / > > > > > > > > > > > > > > > > >> > > > > > > > cloud > > > > > > > > > > > > > > > > >> > > > > > > > >> >> architecture: today > > anyone > > > > can > > > > > > > > create > > > > > > > > > > > topics > > > > > > > > > > > > > in > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > >> shared > > > > > > > > > > > > > > > > >> > > > Kafka > > > > > > > > > > > > > > > > >> > > > > > > > >> cluster, but > > > > > > > > > > > > > > > > >> > > > > > > > >> >> there is no concept or > > > > > > "ownership" > > > > > > > > of > > > > > > > > > > > topics > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > are > > > > > > > > > > > > > > > > >> > > created > > > > > > > > > > > > > > > > >> > > > > by > > > > > > > > > > > > > > > > >> > > > > > > > >> different > > > > > > > > > > > > > > > > >> > > > > > > > >> >> users. For example, at > > > > > LinkedIn > > > > > > we > > > > > > > > > > > basically > > > > > > > > > > > > > > > > >> distinguish > > > > > > > > > > > > > > > > >> > > > topic > > > > > > > > > > > > > > > > >> > > > > > > owners > > > > > > > > > > > > > > > > >> > > > > > > > >> via > > > > > > > > > > > > > > > > >> > > > > > > > >> >> some casual topic name > > > > prefix, > > > > > > > which > > > > > > > > > is a > > > > > > > > > > > > bit > > > > > > > > > > > > > > > > awkward > > > > > > > > > > > > > > > > >> and > > > > > > > > > > > > > > > > >> > > > does > > > > > > > > > > > > > > > > >> > > > > > not > > > > > > > > > > > > > > > > >> > > > > > > > fly > > > > > > > > > > > > > > > > >> > > > > > > > >> as > > > > > > > > > > > > > > > > >> > > > > > > > >> >> we scale our > customers. > > It > > > > > would > > > > > > > be > > > > > > > > > great > > > > > > > > > > > to > > > > > > > > > > > > > use > > > > > > > > > > > > > > > > >> > > > > describe-topics > > > > > > > > > > > > > > > > >> > > > > > > such > > > > > > > > > > > > > > > > >> > > > > > > > >> as: > > > > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >> Describe all topics > that > > > is > > > > > > > created > > > > > > > > > by me. > > > > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >> Describe all topics > > whose > > > > > > > retention > > > > > > > > > time > > > > > > > > > > > is > > > > > > > > > > > > > > > > overriden > > > > > > > > > > > > > > > > >> to X. > > > > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >> Describe all topics > > whose > > > > > > writable > > > > > > > > > group > > > > > > > > > > > > > include > > > > > > > > > > > > > > > > user > > > > > > > > > > > > > > > > >> Y > > > > > > > > > > > > > > > > >> > > (this > > > > > > > > > > > > > > > > >> > > > > is > > > > > > > > > > > > > > > > >> > > > > > > > >> related to > > > > > > > > > > > > > > > > >> > > > > > > > >> >> authorization), etc.. > > > > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >> One possible way to > > > achieve > > > > > this > > > > > > > is > > > > > > > > to > > > > > > > > > > > add a > > > > > > > > > > > > > > > > metadata > > > > > > > > > > > > > > > > >> file > > > > > > > > > > > > > > > > >> > > in > > > > > > > > > > > > > > > > >> > > > > the > > > > > > > > > > > > > > > > >> > > > > > > > >> >> create-topic request, > > > whose > > > > > > value > > > > > > > > will > > > > > > > > > > > also > > > > > > > > > > > > be > > > > > > > > > > > > > > > > >> written ZK > > > > > > > > > > > > > > > > >> > > as > > > > > > > > > > > > > > > > >> > > > we > > > > > > > > > > > > > > > > >> > > > > > > > create > > > > > > > > > > > > > > > > >> > > > > > > > >> the > > > > > > > > > > > > > > > > >> > > > > > > > >> >> topic; then > > > describe-topics > > > > > can > > > > > > > > > choose to > > > > > > > > > > > > > batch > > > > > > > > > > > > > > > > topics > > > > > > > > > > > > > > > > >> > > based > > > > > > > > > > > > > > > > >> > > > on > > > > > > > > > > > > > > > > >> > > > > > 1) > > > > > > > > > > > > > > > > >> > > > > > > > name > > > > > > > > > > > > > > > > >> > > > > > > > >> >> regex, 2) config K-V > > > > matching, > > > > > > 3) > > > > > > > > > metadata > > > > > > > > > > > > > > regex, > > > > > > > > > > > > > > > > etc. > > > > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >> Thoughts? > > > > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >> Guozhang > > > > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >> On Thu, Mar 5, 2015 at > > > 4:37 > > > > > PM, > > > > > > > > > Guozhang > > > > > > > > > > > > Wang > > > > > > > > > > > > > < > > > > > > > > > > > > > > > > >> > > > > > wangg...@gmail.com> > > > > > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > > > > > > > > >> > > > > > > > >> >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> Thanks for the > updated > > > > wiki. > > > > > A > > > > > > > few > > > > > > > > > > > comments > > > > > > > > > > > > > > below: > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> 1. Error description > in > > > > > > > response: I > > > > > > > > > think > > > > > > > > > > > > if > > > > > > > > > > > > > > some > > > > > > > > > > > > > > > > >> > > errorCode > > > > > > > > > > > > > > > > >> > > > > > could > > > > > > > > > > > > > > > > >> > > > > > > > >> indicate > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> several different > error > > > > cases > > > > > > > then > > > > > > > > we > > > > > > > > > > > > should > > > > > > > > > > > > > > really > > > > > > > > > > > > > > > > >> change > > > > > > > > > > > > > > > > >> > > > it > > > > > > > > > > > > > > > > >> > > > > to > > > > > > > > > > > > > > > > >> > > > > > > > >> multiple > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> codes. In general the > > > > > errorCode > > > > > > > > > itself > > > > > > > > > > > > would > > > > > > > > > > > > > be > > > > > > > > > > > > > > > > >> precise > > > > > > > > > > > > > > > > >> > > and > > > > > > > > > > > > > > > > >> > > > > > > > >> sufficient for > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> describing the server > > > side > > > > > > > errors. > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> 2. Describe topic > > > request: > > > > it > > > > > > > would > > > > > > > > > be > > > > > > > > > > > > great > > > > > > > > > > > > > > to go > > > > > > > > > > > > > > > > >> beyond > > > > > > > > > > > > > > > > >> > > > just > > > > > > > > > > > > > > > > >> > > > > > > > >> batching on > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> topic name regex for > > this > > > > > > > request. > > > > > > > > > For > > > > > > > > > > > > > > example, a > > > > > > > > > > > > > > > > >> very > > > > > > > > > > > > > > > > >> > > > common > > > > > > > > > > > > > > > > >> > > > > > use > > > > > > > > > > > > > > > > >> > > > > > > > >> case of > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> the topic command is > to > > > > list > > > > > > all > > > > > > > > > topics > > > > > > > > > > > > whose > > > > > > > > > > > > > > > > config > > > > > > > > > > > > > > > > >> A's > > > > > > > > > > > > > > > > >> > > > value > > > > > > > > > > > > > > > > >> > > > > > is > > > > > > > > > > > > > > > > >> > > > > > > B. > > > > > > > > > > > > > > > > >> > > > > > > > >> With > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> topic name regex then > > we > > > > have > > > > > > to > > > > > > > > > first > > > > > > > > > > > > > retrieve > > > > > > > > > > > > > > > > >> __all__ > > > > > > > > > > > > > > > > >> > > > > topics's > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> description info and > > then > > > > > > filter > > > > > > > at > > > > > > > > > the > > > > > > > > > > > > > client > > > > > > > > > > > > > > end, > > > > > > > > > > > > > > > > >> which > > > > > > > > > > > > > > > > >> > > > will > > > > > > > > > > > > > > > > >> > > > > > be > > > > > > > > > > > > > > > > >> > > > > > > a > > > > > > > > > > > > > > > > >> > > > > > > > >> huge > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> burden on ZK. > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> 3. Config K-Vs in > > create > > > > > topic: > > > > > > > > this > > > > > > > > > is > > > > > > > > > > > > > > related to > > > > > > > > > > > > > > > > >> the > > > > > > > > > > > > > > > > >> > > > > previous > > > > > > > > > > > > > > > > >> > > > > > > > point; > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> maybe we can add > > another > > > > > > metadata > > > > > > > > > K-V or > > > > > > > > > > > > > just a > > > > > > > > > > > > > > > > >> metadata > > > > > > > > > > > > > > > > >> > > > > string > > > > > > > > > > > > > > > > >> > > > > > > > along > > > > > > > > > > > > > > > > >> > > > > > > > >> side > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> with config K-V in > > create > > > > > topic > > > > > > > > like > > > > > > > > > we > > > > > > > > > > > did > > > > > > > > > > > > > for > > > > > > > > > > > > > > > > >> offset > > > > > > > > > > > > > > > > >> > > > commit > > > > > > > > > > > > > > > > >> > > > > > > > >> request. This > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> field can be quite > > useful > > > > in > > > > > > > > storing > > > > > > > > > > > > > > information > > > > > > > > > > > > > > > > like > > > > > > > > > > > > > > > > >> > > > "owner" > > > > > > > > > > > > > > > > >> > > > > of > > > > > > > > > > > > > > > > >> > > > > > > the > > > > > > > > > > > > > > > > >> > > > > > > > >> topic > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> who issue the create > > > > command, > > > > > > > etc, > > > > > > > > > which > > > > > > > > > > > is > > > > > > > > > > > > > > quite > > > > > > > > > > > > > > > > >> > > important > > > > > > > > > > > > > > > > >> > > > > for > > > > > > > > > > > > > > > > >> > > > > > a > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> multi-tenant setting. > > > Then > > > > in > > > > > > the > > > > > > > > > > > describe > > > > > > > > > > > > > > topic > > > > > > > > > > > > > > > > >> request > > > > > > > > > > > > > > > > >> > > we > > > > > > > > > > > > > > > > >> > > > > can > > > > > > > > > > > > > > > > >> > > > > > > also > > > > > > > > > > > > > > > > >> > > > > > > > >> batch > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> on regex of the > > metadata > > > > > field. > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> 4. Today all the > admin > > > > > > operations > > > > > > > > are > > > > > > > > > > > async > > > > > > > > > > > > > in > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > >> sense > > > > > > > > > > > > > > > > >> > > > that > > > > > > > > > > > > > > > > >> > > > > > > > command > > > > > > > > > > > > > > > > >> > > > > > > > >> will > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> return once it is > > written > > > > in > > > > > > ZK, > > > > > > > > and > > > > > > > > > that > > > > > > > > > > > > is > > > > > > > > > > > > > > why we > > > > > > > > > > > > > > > > >> need > > > > > > > > > > > > > > > > >> > > > extra > > > > > > > > > > > > > > > > >> > > > > > > > >> verification > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> like > > > > > > > > testUtil.waitForTopicCreated() / > > > > > > > > > > > > verify > > > > > > > > > > > > > > > > >> partition > > > > > > > > > > > > > > > > >> > > > > > > reassignment > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> request, etc. With > > admin > > > > > > requests > > > > > > > > we > > > > > > > > > > > could > > > > > > > > > > > > > add > > > > > > > > > > > > > > a > > > > > > > > > > > > > > > > >> flag to > > > > > > > > > > > > > > > > >> > > > > enable > > > > > > > > > > > > > > > > >> > > > > > / > > > > > > > > > > > > > > > > >> > > > > > > > >> disable > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> synchronous requests; > > > when > > > > it > > > > > > is > > > > > > > > > turned > > > > > > > > > > > on, > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > >> response > > > > > > > > > > > > > > > > >> > > > will > > > > > > > > > > > > > > > > >> > > > > > not > > > > > > > > > > > > > > > > >> > > > > > > > >> return > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> until the request has > > > been > > > > > > > > > completed. And > > > > > > > > > > > > for > > > > > > > > > > > > > > async > > > > > > > > > > > > > > > > >> > > requests > > > > > > > > > > > > > > > > >> > > > > we > > > > > > > > > > > > > > > > >> > > > > > > can > > > > > > > > > > > > > > > > >> > > > > > > > >> add a > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> "token" field in the > > > > > response, > > > > > > > and > > > > > > > > > then > > > > > > > > > > > > only > > > > > > > > > > > > > > need a > > > > > > > > > > > > > > > > >> > > general > > > > > > > > > > > > > > > > >> > > > > > "admin > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> verification request" > > > with > > > > > the > > > > > > > > given > > > > > > > > > > > token > > > > > > > > > > > > to > > > > > > > > > > > > > > check > > > > > > > > > > > > > > > > >> if the > > > > > > > > > > > > > > > > >> > > > > async > > > > > > > > > > > > > > > > >> > > > > > > > >> request > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> has been completed. > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> 5. +1 for extending > > > > Metadata > > > > > > > > request > > > > > > > > > to > > > > > > > > > > > > > include > > > > > > > > > > > > > > > > >> > > controller / > > > > > > > > > > > > > > > > >> > > > > > > > >> coordinator > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> information, and then > > we > > > > can > > > > > > > remove > > > > > > > > > the > > > > > > > > > > > > > > > > >> ConsumerMetadata / > > > > > > > > > > > > > > > > >> > > > > > > > >> ClusterMetadata > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> requests. > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> Guozhang > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> On Tue, Mar 3, 2015 > at > > > > 10:23 > > > > > > AM, > > > > > > > > Joel > > > > > > > > > > > > Koshy < > > > > > > > > > > > > > > > > >> > > > > > jjkosh...@gmail.com> > > > > > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > > > > > > > > >> > > > > > > > >> >>> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> Thanks for sending > > that > > > > out > > > > > > Joe > > > > > > > - > > > > > > > > I > > > > > > > > > > > don't > > > > > > > > > > > > > > think I > > > > > > > > > > > > > > > > >> will be > > > > > > > > > > > > > > > > >> > > > > able > > > > > > > > > > > > > > > > >> > > > > > to > > > > > > > > > > > > > > > > >> > > > > > > > >> make > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> it today, so if > notes > > > can > > > > be > > > > > > > sent > > > > > > > > > out > > > > > > > > > > > > > > afterward > > > > > > > > > > > > > > > > that > > > > > > > > > > > > > > > > >> > > would > > > > > > > > > > > > > > > > >> > > > be > > > > > > > > > > > > > > > > >> > > > > > > > great. > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> On Mon, Mar 02, 2015 > > at > > > > > > > 09:16:13AM > > > > > > > > > > > -0800, > > > > > > > > > > > > > Gwen > > > > > > > > > > > > > > > > >> Shapira > > > > > > > > > > > > > > > > >> > > > wrote: > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > Thanks for sending > > > this > > > > > out > > > > > > > Joe. > > > > > > > > > > > Looking > > > > > > > > > > > > > > forward > > > > > > > > > > > > > > > > >> to > > > > > > > > > > > > > > > > >> > > > > chatting > > > > > > > > > > > > > > > > >> > > > > > > with > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> everyone :) > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > On Mon, Mar 2, > 2015 > > at > > > > > 6:46 > > > > > > > AM, > > > > > > > > > Joe > > > > > > > > > > > > Stein > > > > > > > > > > > > > < > > > > > > > > > > > > > > > > >> > > > > > > joe.st...@stealth.ly> > > > > > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > Hey, I just sent > > > out a > > > > > > > google > > > > > > > > > > > hangout > > > > > > > > > > > > > > invite > > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > >> all > > > > > > > > > > > > > > > > >> > > > pmc, > > > > > > > > > > > > > > > > >> > > > > > > > >> committers > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> and > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > everyone I found > > > > working > > > > > > on > > > > > > > a > > > > > > > > > KIP. > > > > > > > > > > > If > > > > > > > > > > > > I > > > > > > > > > > > > > > missed > > > > > > > > > > > > > > > > >> anyone > > > > > > > > > > > > > > > > >> > > > in > > > > > > > > > > > > > > > > >> > > > > > the > > > > > > > > > > > > > > > > >> > > > > > > > >> invite > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> please > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > let me know and > > can > > > > > update > > > > > > > it, > > > > > > > > > np. > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > We should do > this > > > > every > > > > > > > > Tuesday > > > > > > > > > @ > > > > > > > > > > > 2pm > > > > > > > > > > > > > > Eastern > > > > > > > > > > > > > > > > >> Time. > > > > > > > > > > > > > > > > >> > > > Maybe > > > > > > > > > > > > > > > > >> > > > > > we > > > > > > > > > > > > > > > > >> > > > > > > > can > > > > > > > > > > > > > > > > >> > > > > > > > >> get > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> INFRA > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > help to make a > > > google > > > > > > > account > > > > > > > > > so we > > > > > > > > > > > > can > > > > > > > > > > > > > > manage > > > > > > > > > > > > > > > > >> > > better? > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > To discuss > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > in progress and > > > > related > > > > > > JIRA > > > > > > > > > that > > > > > > > > > > > are > > > > > > > > > > > > > > > > >> interdependent > > > > > > > > > > > > > > > > >> > > > and > > > > > > > > > > > > > > > > >> > > > > > > common > > > > > > > > > > > > > > > > >> > > > > > > > >> work. > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > ~ Joe Stein > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > On Tue, Feb 24, > > 2015 > > > > at > > > > > > 2:59 > > > > > > > > > PM, Jay > > > > > > > > > > > > > > Kreps < > > > > > > > > > > > > > > > > >> > > > > > > > jay.kr...@gmail.com> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> wrote: > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> Let's stay on > > > Google > > > > > > > hangouts > > > > > > > > > that > > > > > > > > > > > > will > > > > > > > > > > > > > > also > > > > > > > > > > > > > > > > >> record > > > > > > > > > > > > > > > > >> > > > and > > > > > > > > > > > > > > > > >> > > > > > make > > > > > > > > > > > > > > > > >> > > > > > > > the > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> sessions > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> available on > > > youtube. > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> -Jay > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> On Tue, Feb 24, > > > 2015 > > > > at > > > > > > > 11:49 > > > > > > > > > AM, > > > > > > > > > > > > Jeff > > > > > > > > > > > > > > > > Holoman > > > > > > > > > > > > > > > > >> < > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > jholo...@cloudera.com > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> wrote: > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > Jay / Joe > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > We're happy > to > > > send > > > > > > out a > > > > > > > > > Webex > > > > > > > > > > > for > > > > > > > > > > > > > > this > > > > > > > > > > > > > > > > >> purpose. > > > > > > > > > > > > > > > > >> > > We > > > > > > > > > > > > > > > > >> > > > > > could > > > > > > > > > > > > > > > > >> > > > > > > > >> record > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> the > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > sessions if > > there > > > > is > > > > > > > > > interest and > > > > > > > > > > > > > > publish > > > > > > > > > > > > > > > > >> them > > > > > > > > > > > > > > > > >> > > out. > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > Thanks > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > Jeff > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > On Tue, Feb > 24, > > > > 2015 > > > > > at > > > > > > > > > 11:28 AM, > > > > > > > > > > > > Jay > > > > > > > > > > > > > > > > Kreps < > > > > > > > > > > > > > > > > >> > > > > > > > >> jay.kr...@gmail.com> > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> wrote: > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > Let's try > to > > > get > > > > > the > > > > > > > > > technical > > > > > > > > > > > > > > hang-ups > > > > > > > > > > > > > > > > >> sorted > > > > > > > > > > > > > > > > >> > > > out, > > > > > > > > > > > > > > > > >> > > > > > > > though. > > > > > > > > > > > > > > > > >> > > > > > > > >> I > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> really > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > think > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > there is > some > > > > > benefit > > > > > > > to > > > > > > > > > live > > > > > > > > > > > > > > discussion > > > > > > > > > > > > > > > > vs > > > > > > > > > > > > > > > > >> > > > > writing. I > > > > > > > > > > > > > > > > >> > > > > > > am > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> hopeful that > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> if > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > we post > > > > > instructions > > > > > > > and > > > > > > > > > give > > > > > > > > > > > > > > ourselves a > > > > > > > > > > > > > > > > >> few > > > > > > > > > > > > > > > > >> > > > > attempts > > > > > > > > > > > > > > > > >> > > > > > > we > > > > > > > > > > > > > > > > >> > > > > > > > >> can > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> get it > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > working. > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > Tuesday at > > that > > > > > time > > > > > > > > would > > > > > > > > > work > > > > > > > > > > > > for > > > > > > > > > > > > > > > > >> me...any > > > > > > > > > > > > > > > > >> > > > > > objections? > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > -Jay > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > On Tue, Feb > > 24, > > > > > 2015 > > > > > > at > > > > > > > > > 8:18 > > > > > > > > > > > AM, > > > > > > > > > > > > > Joe > > > > > > > > > > > > > > > > Stein > > > > > > > > > > > > > > > > >> < > > > > > > > > > > > > > > > > >> > > > > > > > >> joe.st...@stealth.ly > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> wrote: > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > Weekly > > would > > > be > > > > > > great > > > > > > > > > maybe > > > > > > > > > > > > like > > > > > > > > > > > > > > every > > > > > > > > > > > > > > > > >> > > Tuesday ~ > > > > > > > > > > > > > > > > >> > > > > 1pm > > > > > > > > > > > > > > > > >> > > > > > > ET > > > > > > > > > > > > > > > > >> > > > > > > > / > > > > > > > > > > > > > > > > >> > > > > > > > >> 10am > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> PT > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> ???? > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > I don't > > mind > > > > > google > > > > > > > > > hangout > > > > > > > > > > > but > > > > > > > > > > > > > > there > > > > > > > > > > > > > > > > is > > > > > > > > > > > > > > > > >> > > always > > > > > > > > > > > > > > > > >> > > > > some > > > > > > > > > > > > > > > > >> > > > > > > > >> issue or > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> whatever > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > so > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > we know > the > > > > > apache > > > > > > > irc > > > > > > > > > > > channel > > > > > > > > > > > > > > works. > > > > > > > > > > > > > > > > We > > > > > > > > > > > > > > > > >> can > > > > > > > > > > > > > > > > >> > > > start > > > > > > > > > > > > > > > > >> > > > > > > there > > > > > > > > > > > > > > > > >> > > > > > > > >> and > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> see how > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> it > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > goes? We > > can > > > > pull > > > > > > > > > transcripts > > > > > > > > > > > > too > > > > > > > > > > > > > > and > > > > > > > > > > > > > > > > >> > > associate > > > > > > > > > > > > > > > > >> > > > to > > > > > > > > > > > > > > > > >> > > > > > > > >> tickets if > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> need be > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > makes > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > it > helpful > > > for > > > > > > > things. > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > ~ > Joestein > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > On Tue, > Feb > > > 24, > > > > > > 2015 > > > > > > > at > > > > > > > > > 11:10 > > > > > > > > > > > > AM, > > > > > > > > > > > > > > Jay > > > > > > > > > > > > > > > > >> Kreps < > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> jay.kr...@gmail.com > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > wrote: > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > We'd > > talked > > > > > about > > > > > > > > > doing a > > > > > > > > > > > > > Google > > > > > > > > > > > > > > > > >> Hangout to > > > > > > > > > > > > > > > > >> > > > chat > > > > > > > > > > > > > > > > >> > > > > > > about > > > > > > > > > > > > > > > > >> > > > > > > > >> this. > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> What > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > about > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > generalizing > > > > > > that a > > > > > > > > > little > > > > > > > > > > > > > > > > further...I > > > > > > > > > > > > > > > > >> > > > actually > > > > > > > > > > > > > > > > >> > > > > > > think > > > > > > > > > > > > > > > > >> > > > > > > > it > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> would be > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > good > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > for > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > everyone > > > > > > spending a > > > > > > > > > > > > reasonable > > > > > > > > > > > > > > chunk > > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > >> > > their > > > > > > > > > > > > > > > > >> > > > > week > > > > > > > > > > > > > > > > >> > > > > > > on > > > > > > > > > > > > > > > > >> > > > > > > > >> Kafka > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> stuff > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> to > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > maybe > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > sync up > > > once > > > > a > > > > > > > week. > > > > > > > > I > > > > > > > > > > > think > > > > > > > > > > > > we > > > > > > > > > > > > > > could > > > > > > > > > > > > > > > > >> use > > > > > > > > > > > > > > > > >> > > time > > > > > > > > > > > > > > > > >> > > > > to > > > > > > > > > > > > > > > > >> > > > > > > talk > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> through > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> design > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > stuff, > > make > > > > > sure > > > > > > we > > > > > > > > > are on > > > > > > > > > > > > top > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > code > > > > > > > > > > > > > > > > >> > > > reviews, > > > > > > > > > > > > > > > > >> > > > > > talk > > > > > > > > > > > > > > > > >> > > > > > > > >> through > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> any > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > tricky > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > issues, > > > etc. > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > We can > > make > > > > it > > > > > > > > publicly > > > > > > > > > > > > > > available so > > > > > > > > > > > > > > > > >> that > > > > > > > > > > > > > > > > >> > > any > > > > > > > > > > > > > > > > >> > > > > one > > > > > > > > > > > > > > > > >> > > > > > > can > > > > > > > > > > > > > > > > >> > > > > > > > >> follow > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> along > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > who > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > likes. > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > Any > > > interest > > > > in > > > > > > > doing > > > > > > > > > this? > > > > > > > > > > > > If > > > > > > > > > > > > > so > > > > > > > > > > > > > > > > I'll > > > > > > > > > > > > > > > > >> try > > > > > > > > > > > > > > > > >> > > to > > > > > > > > > > > > > > > > >> > > > > set > > > > > > > > > > > > > > > > >> > > > > > it > > > > > > > > > > > > > > > > >> > > > > > > > up > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> starting > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> next > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > week. > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > -Jay > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > On Tue, > > Feb > > > > 24, > > > > > > > 2015 > > > > > > > > at > > > > > > > > > > > 3:57 > > > > > > > > > > > > > AM, > > > > > > > > > > > > > > > > Andrii > > > > > > > > > > > > > > > > >> > > > > Biletskyi > > > > > > > > > > > > > > > > >> > > > > > < > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > andrii.bilets...@stealth.ly> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > Hi > all, > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > I've > > > > updated > > > > > > KIP > > > > > > > > > page, > > > > > > > > > > > > fixed > > > > > > > > > > > > > / > > > > > > > > > > > > > > > > >> aligned > > > > > > > > > > > > > > > > >> > > > > document > > > > > > > > > > > > > > > > >> > > > > > > > >> structure. > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> Also I > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > added > > > > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > some > > > > > > > > > > > > > ... > > > > > > > > > > > > [Message clipped] > > > > > > > > > > > > > > > > > > > > > -- > > > > -- Guozhang > > > > > > > > > >
