Gwen, Yes, looks like KAFKA-1927 will leave TopicMetadataRequest/Response. But I believe, KIP is still tightly related with KAFKA-1927 since we are not only going to update TopicMetadataRequest there but we will introduce a bunch of new requests too. And it probably makes sense to do those correctly from scratch - without introducing scala request objects. As I understand you'll have this common infrastructure code done in KAFKA-1927.
Thanks, Andrii Biletskyi On Wed, Mar 18, 2015 at 8:38 PM, Gwen Shapira <gshap...@cloudera.com> wrote: > On Wed, Mar 18, 2015 at 9:34 AM, Jun Rao <j...@confluent.io> wrote: > > Andri, > > > > Thanks for the summary. > > > > 1. I just realized that in order to start working on KAFKA-1927, we will > > need to merge the changes to OffsetCommitRequest (from 0.8.2) to trunk. > > This is planned to be done as part of KAFKA-1634. So, we will need > Guozhang > > and Joel's help to wrap this up. > > I mentioned this in a separate thread, but it may be more relevant here: > It looks like the SimpleConsumer API exposes TopicMetadataRequest and > TopicMetadataResponse. > This means that KAFKA-1927 doesn't remove this duplication. > > So I'm not sure we actually need KAFKA-1927 before implementing this KIP. > This doesn't mean I'm stopping work on KAFKA-1927, but perhaps it > means we can proceed in parallel? > > > 2. Thinking about this a bit more, if the semantic of those "write" > > requests is async (i.e., after the client gets a response, it just means > > that the operation is initiated, but not necessarily completed), we don't > > really need to forward the requests to the controller. Instead, the > > receiving broker can just write the operation to ZK as the admin command > > line tool previously does. This will simplify the implementation. > > > > 8. There is another implementation detail for describe topic. Ideally, we > > want to read the topic config from the broker cache, instead of > ZooKeeper. > > Currently, every broker reads the topic-level config for all topics. > > However, it ignores those for topics not hosted on itself. So, we may > need > > to change TopicConfigManager a bit so that it caches the configs for all > > topics. > > > > Thanks, > > > > Jun > > > > > > On Tue, Mar 17, 2015 at 1:13 PM, Andrii Biletskyi < > > andrii.bilets...@stealth.ly> wrote: > > > >> Guys, > >> > >> Thanks for a great discussion! > >> Here are the actions points: > >> > >> 1. Q: Get rid of all scala requests objects, use java protocol > definitions. > >> A: Gwen kindly took that (KAFKA-1927). It's important to speed up > >> review procedure > >> there since this ticket blocks other important changes. > >> > >> 2. Q: Generic re-reroute facility vs client maintaining cluster state. > >> A: Jay has added pseudo code to KAFKA-1912 - need to consider > whether > >> this will be > >> easy to implement as a server-side feature (comments are > >> welcomed!). > >> > >> 3. Q: Controller field in wire protocol. > >> A: This might be useful for clients, add this to > TopicMetadataResponse > >> (already in KIP). > >> > >> 4. Q: Decoupling topic creation from TMR. > >> A: I will add proposed by Jun solution (using clientId for that) to > the > >> KIP. > >> > >> 5. Q: Bumping new versions of TMR vs grabbing all protocol changes in > one > >> version. > >> A: It was decided to try to gather all changes to protocol (before > >> release). > >> In case of TMR it worth checking: KAFKA-2020 and KIP-13 (quotas) > >> > >> 6. Q: JSON lib is needed to deserialize user's input in CLI tool. > >> A: Use jackson for that, /tools project is a separate jar so > shouldn't > >> be a big deal. > >> > >> 7. Q: VerifyReassingPartitions vs generic status check command. > >> A: For long-running requests like reassign partitions *progress* > check > >> request is useful, > >> it makes sense to introduce it. > >> > >> Please add, correct me if I missed something. > >> > >> Thanks, > >> Andrii Biletskyi > >> > >> On Tue, Mar 17, 2015 at 6:20 PM, Andrii Biletskyi < > >> andrii.bilets...@stealth.ly> wrote: > >> > >> > Joel, > >> > > >> > You are right, I removed ClusterMetadata because we have partially > >> > what we need in TopicMetadata. Also, as Jay pointed out earlier, we > >> > would like to have "orthogonal" API, but at the same time we need > >> > to be backward compatible. > >> > > >> > But I like your idea and even have some other arguments for this > option: > >> > There is also DescribeTopicRequest which was proposed in this KIP, > >> > it returns topic configs, partitions, replication factor plus > partition > >> > ISR, ASR, > >> > leader replica. The later part is really already there in > >> > TopicMetadataRequest. > >> > So again we'll have to add stuff to TMR, not to duplicate some info in > >> > newly added requests. However, this way we'll end up with "monster" > >> > request which returns cluster metadata, topic replication and config > info > >> > plus partition replication data. Seems logical to split TMR to > >> > - ClusterMetadata (brokers + controller, maybe smth else) > >> > - TopicMetadata (topic info + partition details) > >> > But since current TMR is involved in lots of places (including network > >> > client, > >> > as I understand) this might be very serious change and it probably > makes > >> > sense to stick with current approach. > >> > > >> > Thanks, > >> > Andrii Biletskyi > >> > > >> > > >> > On Tue, Mar 17, 2015 at 5:29 PM, Joel Koshy <jjkosh...@gmail.com> > wrote: > >> > > >> >> I may be missing some context but hopefully this will also be covered > >> >> today: I thought the earlier proposal where there was an explicit > >> >> ClusterMetadata request was clearer and explicit. During the course > of > >> >> this thread I think the conclusion was that the main need was for > >> >> controller information and that can be rolled into the topic metadata > >> >> response but that seems a bit irrelevant to topic metadata. FWIW I > >> >> think the full broker-list is also irrelevant to topic metadata, but > >> >> it is already there and in use. I think there is still room for an > >> >> explicit ClusterMetadata request since there may be other > >> >> cluster-level information that we may want to add over time (and that > >> >> have nothing to do with topic metadata). > >> >> > >> >> On Tue, Mar 17, 2015 at 02:45:30PM +0200, Andrii Biletskyi wrote: > >> >> > Jun, > >> >> > > >> >> > 101. Okay, if you say that such use case is important. I also think > >> >> > using clientId for these purposes is fine - if we already have this > >> >> field > >> >> > as part of all Wire protocol messages, why not use that. > >> >> > I will update KIP-4 page if nobody has other ideas (which may come > up > >> >> > during the call today). > >> >> > > >> >> > 102.1 Agree, I'll update the KIP accordingly. I think we can add > new, > >> >> > fine-grained error codes if some error code received in specific > case > >> >> > won't give enough context to return a descriptive error message for > >> >> user. > >> >> > > >> >> > Look forward to discussing all outstanding issues in detail today > >> during > >> >> > the call. > >> >> > > >> >> > Thanks, > >> >> > Andrii Biletskyi > >> >> > > >> >> > > >> >> > > >> >> > On Mon, Mar 16, 2015 at 10:59 PM, Jun Rao <j...@confluent.io> > wrote: > >> >> > > >> >> > > 101. There may be a use case where you only want the topics to be > >> >> created > >> >> > > manually by admins. Currently, you can do that by disabling auto > >> topic > >> >> > > creation and issue topic creation from the TopicCommand. If we > >> >> disable auto > >> >> > > topic creation completely on the broker and don't have a way to > >> >> distinguish > >> >> > > between topic creation requests from the regular clients and the > >> >> admin, we > >> >> > > can't support manual topic creation any more. I was thinking that > >> >> another > >> >> > > way of distinguishing the clients making the topic creation > requests > >> >> is > >> >> > > using clientId. For example, the admin tool can set it to > something > >> >> like > >> >> > > admin and the broker can treat that clientId specially. > >> >> > > > >> >> > > Also, there is a related discussion in KAFKA-2020. Currently, we > do > >> >> the > >> >> > > following in TopicMetadataResponse: > >> >> > > > >> >> > > 1. If leader is not available, we set the partition level error > code > >> >> to > >> >> > > LeaderNotAvailable. > >> >> > > 2. If a non-leader replica is not available, we take that replica > >> out > >> >> of > >> >> > > the assigned replica list and isr in the response. As an > indication > >> >> for > >> >> > > doing that, we set the partition level error code to > >> >> ReplicaNotAvailable. > >> >> > > > >> >> > > This has a few problems. First, ReplicaNotAvailable probably > >> >> shouldn't be > >> >> > > an error, at least for the normal producer/consumer clients that > >> just > >> >> want > >> >> > > to find out the leader. Second, it can happen that both the > leader > >> and > >> >> > > another replica are not available at the same time. There is no > >> error > >> >> code > >> >> > > to indicate both. Third, even if a replica is not available, it's > >> >> still > >> >> > > useful to return its replica id since some clients (e.g. admin > tool) > >> >> may > >> >> > > still make use of it. > >> >> > > > >> >> > > One way to address this issue is to always return the replica id > for > >> >> > > leader, assigned replicas, and isr regardless of whether the > >> >> corresponding > >> >> > > broker is live or not. Since we also return the list of live > >> brokers, > >> >> the > >> >> > > client can figure out whether a leader or a replica is live or > not > >> >> and act > >> >> > > accordingly. This way, we don't need to set the partition level > >> error > >> >> code > >> >> > > when the leader or a replica is not available. This doesn't > change > >> >> the wire > >> >> > > protocol, but does change the semantics. Since we are evolving > the > >> >> protocol > >> >> > > of TopicMetadataRequest here, we can potentially piggyback the > >> change. > >> >> > > > >> >> > > 102.1 For those types of errors due to invalid input, shouldn't > we > >> >> just > >> >> > > guard it at parameter validation time and throw > >> >> InvalidArgumentException > >> >> > > without even sending the request to the broker? > >> >> > > > >> >> > > Thanks, > >> >> > > > >> >> > > Jun > >> >> > > > >> >> > > > >> >> > > On Mon, Mar 16, 2015 at 10:37 AM, Andrii Biletskyi < > >> >> > > andrii.bilets...@stealth.ly> wrote: > >> >> > > > >> >> > > > Jun, > >> >> > > > > >> >> > > > Answering your questions: > >> >> > > > > >> >> > > > 101. If I understand you correctly, you are saying future > producer > >> >> > > versions > >> >> > > > (which > >> >> > > > will be ported to TMR_V1) won't be able to automatically create > >> >> topic (if > >> >> > > > we > >> >> > > > unconditionally remove topic creation from there). But we need > to > >> >> this > >> >> > > > preserve logic. > >> >> > > > Ok, about your proposal: I'm not a big fan too, when it comes > to > >> >> > > > differentiating > >> >> > > > clients directly in protocol schema. And also I'm not sure I > >> >> understand > >> >> > > at > >> >> > > > all why > >> >> > > > auto.create.topics.enable is a server side configuration. Can > we > >> >> > > deprecate > >> >> > > > this setting > >> >> > > > in future versions, add this setting to producer and based on > that > >> >> upon > >> >> > > > receiving > >> >> > > > UnknownTopic create topic explicitly by a separate producer > call > >> via > >> >> > > > adminClient? > >> >> > > > > >> >> > > > 102.1. Hm, yes. It's because we want to support batching and at > >> the > >> >> same > >> >> > > > time we > >> >> > > > want to give descriptive error messages for clients. Since > >> >> AdminClient > >> >> > > > holds the context > >> >> > > > to construct such messages (e.g. AdminClient layer can know > that > >> >> > > > InvalidArgumentsCode > >> >> > > > means two cases: either invalid number - e.g. -1; or > >> >> replication-factor > >> >> > > was > >> >> > > > provided while > >> >> > > > partitions argument wasn't) - I wrapped responses in > Exceptions. > >> >> But I'm > >> >> > > > open to any > >> >> > > > other ideas, this was just initial version. > >> >> > > > 102.2. Yes, I agree. I'll change that to probably some other > dto. > >> >> > > > > >> >> > > > Thanks, > >> >> > > > Andrii Biletskyi > >> >> > > > > >> >> > > > On Fri, Mar 13, 2015 at 7:16 PM, Jun Rao <j...@confluent.io> > >> wrote: > >> >> > > > > >> >> > > > > Andrii, > >> >> > > > > > >> >> > > > > 101. That's what I was thinking too, but it may not be that > >> >> simple. In > >> >> > > > > TopicMetadataRequest_V1, > >> >> > > > > we can let it not trigger auto topic creation. Then, in the > >> >> producer > >> >> > > > side, > >> >> > > > > if it gets an UnknownTopicException, it can explicitly issue > a > >> >> > > > > createTopicRequest for auto topic creation. On the consumer > >> side, > >> >> it > >> >> > > will > >> >> > > > > never issue createTopicRequest. This works when auto topic > >> >> creation is > >> >> > > > > enabled on the broker side. However, I am not sure how things > >> >> will work > >> >> > > > > when auto topic creation is disabled on the broker side. In > this > >> >> case, > >> >> > > we > >> >> > > > > want to have a way to manually create a topic, potentially > >> through > >> >> > > admin > >> >> > > > > commands. However, then we need a way to distinguish > >> >> createTopicRequest > >> >> > > > > issued from the producer clients and the admin tools. May be > we > >> >> can > >> >> > > add a > >> >> > > > > new field in createTopicRequest and set it differently in the > >> >> producer > >> >> > > > > client and the admin client. However, I am not sure if that's > >> the > >> >> best > >> >> > > > > approach. > >> >> > > > > > >> >> > > > > 2. Yes, refactoring existing requests is a non-trivial > amount of > >> >> work. > >> >> > > I > >> >> > > > > posted some comments in KAFKA-1927. We will probably have to > fix > >> >> > > > KAFKA-1927 > >> >> > > > > first, before adding the new logic in KAFKA-1694. Otherwise, > the > >> >> > > changes > >> >> > > > > will be too big. > >> >> > > > > > >> >> > > > > 102. About the AdminClient: > >> >> > > > > 102.1. It's a bit weird that we return exception in the api. > It > >> >> seems > >> >> > > > that > >> >> > > > > we should either return error code or throw an exception when > >> >> getting > >> >> > > the > >> >> > > > > response state. > >> >> > > > > 102.2. We probably shouldn't explicitly use the request > object > >> in > >> >> the > >> >> > > > api. > >> >> > > > > Not every request evolution requires an api change. > >> >> > > > > > >> >> > > > > Thanks, > >> >> > > > > > >> >> > > > > Jun > >> >> > > > > > >> >> > > > > > >> >> > > > > On Fri, Mar 13, 2015 at 4:08 AM, Andrii Biletskyi < > >> >> > > > > andrii.bilets...@stealth.ly> wrote: > >> >> > > > > > >> >> > > > > > Jun, > >> >> > > > > > > >> >> > > > > > Thanks for you comments. Answers inline: > >> >> > > > > > > >> >> > > > > > 100. There are a few fields such as ReplicaAssignment, > >> >> > > > > > > ReassignPartitionRequest, > >> >> > > > > > > and PartitionsSerialized that are represented as a > string, > >> but > >> >> > > > contain > >> >> > > > > > > composite structures in json. Could we flatten them out > >> >> directly in > >> >> > > > the > >> >> > > > > > > protocol definition as arrays/records? > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > Yes, now with Admin Client this looks a bit weird. My > initial > >> >> > > > motivation > >> >> > > > > > was: > >> >> > > > > > ReassignPartitionCommand accepts input in json, we want to > >> >> remain > >> >> > > > tools' > >> >> > > > > > interfaces unchanged, where possible. > >> >> > > > > > If we port it to deserialized format, in CLI (/tools > project) > >> >> we will > >> >> > > > > have > >> >> > > > > > to add some > >> >> > > > > > json library since /tools is written in java and we'll > need to > >> >> > > > > deserialize > >> >> > > > > > json file > >> >> > > > > > provided by a user. Can we quickly agree on what this > library > >> >> should > >> >> > > be > >> >> > > > > > (Jackson, GSON, whatever)? > >> >> > > > > > > >> >> > > > > > 101. Does TopicMetadataRequest v1 still trigger auto topic > >> >> creation? > >> >> > > > This > >> >> > > > > > > will be a bit weird now that we have a separate topic > >> >> creation api. > >> >> > > > > Have > >> >> > > > > > > you thought about how the new createTopicRequest and > >> >> > > > > TopicMetadataRequest > >> >> > > > > > > v1 will be used in the producer/consumer client, in > addition > >> >> to > >> >> > > admin > >> >> > > > > > > tools? For example, ideally, we don't want > >> >> TopicMetadataRequest > >> >> > > from > >> >> > > > > the > >> >> > > > > > > consumer to trigger auto topic creation. > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > I agree, this strange logic should be fixed. I'm not > confident > >> >> in > >> >> > > this > >> >> > > > > > Kafka part so > >> >> > > > > > correct me if I'm wrong, but it doesn't look like a hard > thing > >> >> to > >> >> > > do, I > >> >> > > > > > think we can > >> >> > > > > > leverage AdminClient for that in Producer and > unconditionally > >> >> remove > >> >> > > > > topic > >> >> > > > > > creation from the TopicMetadataRequest_V1. > >> >> > > > > > > >> >> > > > > > 2. I think Jay meant getting rid of scala classes > >> >> > > > > > > like HeartbeatRequestAndHeader and > >> >> HeartbeatResponseAndHeader. We > >> >> > > did > >> >> > > > > > that > >> >> > > > > > > as a stop-gap thing when adding the new requests for the > >> >> consumers. > >> >> > > > > > > However, the long term plan is to get rid of all those > and > >> >> just > >> >> > > reuse > >> >> > > > > the > >> >> > > > > > > java request/response in the client. Since this KIP > proposes > >> >> to > >> >> > > add a > >> >> > > > > > > significant number of new requests, perhaps we should > bite > >> the > >> >> > > bullet > >> >> > > > > to > >> >> > > > > > > clean up the existing scala requests first before adding > new > >> >> ones? > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > Yes, looks like I misunderstood the point of > >> >> ...RequestAndHeader. > >> >> > > > Okay, I > >> >> > > > > > will > >> >> > > > > > rework that. The only thing is that I don't see any example > >> how > >> >> it > >> >> > > was > >> >> > > > > done > >> >> > > > > > for at > >> >> > > > > > least one existing protocol message. Thus, as I > understand, I > >> >> have to > >> >> > > > > think > >> >> > > > > > how we > >> >> > > > > > are going to do it. > >> >> > > > > > Re porting all existing RQ/RP in this patch. Sounds > >> reasonable, > >> >> but > >> >> > > if > >> >> > > > > it's > >> >> > > > > > an *obligatory* > >> >> > > > > > requirement to have Admin KIP done, I'm afraid this can be > a > >> >> serious > >> >> > > > > > blocker for us. > >> >> > > > > > There are 13 protocol messages and all that would require > not > >> >> only > >> >> > > unit > >> >> > > > > > tests but quite > >> >> > > > > > intensive manual testing, no? I'm afraid I'm not the right > guy > >> >> to > >> >> > > cover > >> >> > > > > > pretty much all > >> >> > > > > > Kafka core internals :). Let me know your thoughts on this > >> >> item. Btw > >> >> > > > > there > >> >> > > > > > is a ticket to > >> >> > > > > > follow-up this issue ( > >> >> > > https://issues.apache.org/jira/browse/KAFKA-2006 > >> >> > > > ). > >> >> > > > > > > >> >> > > > > > Thanks, > >> >> > > > > > Andrii Biletskyi > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > On Fri, Mar 13, 2015 at 6:40 AM, Jun Rao <j...@confluent.io > > > >> >> wrote: > >> >> > > > > > > >> >> > > > > > > Andrii, > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > A few more comments. > >> >> > > > > > > > >> >> > > > > > > 100. There are a few fields such as ReplicaAssignment, > >> >> > > > > > > ReassignPartitionRequest, > >> >> > > > > > > and PartitionsSerialized that are represented as a > string, > >> but > >> >> > > > contain > >> >> > > > > > > composite structures in json. Could we flatten them out > >> >> directly in > >> >> > > > the > >> >> > > > > > > protocol definition as arrays/records? > >> >> > > > > > > > >> >> > > > > > > 101. Does TopicMetadataRequest v1 still trigger auto > topic > >> >> > > creation? > >> >> > > > > This > >> >> > > > > > > will be a bit weird now that we have a separate topic > >> >> creation api. > >> >> > > > > Have > >> >> > > > > > > you thought about how the new createTopicRequest and > >> >> > > > > TopicMetadataRequest > >> >> > > > > > > v1 will be used in the producer/consumer client, in > addition > >> >> to > >> >> > > admin > >> >> > > > > > > tools? For example, ideally, we don't want > >> >> TopicMetadataRequest > >> >> > > from > >> >> > > > > the > >> >> > > > > > > consumer to trigger auto topic creation. > >> >> > > > > > > > >> >> > > > > > > 2. I think Jay meant getting rid of scala classes > >> >> > > > > > > like HeartbeatRequestAndHeader and > >> >> HeartbeatResponseAndHeader. We > >> >> > > did > >> >> > > > > > that > >> >> > > > > > > as a stop-gap thing when adding the new requests for the > >> >> consumers. > >> >> > > > > > > However, the long term plan is to get rid of all those > and > >> >> just > >> >> > > reuse > >> >> > > > > the > >> >> > > > > > > java request/response in the client. Since this KIP > proposes > >> >> to > >> >> > > add a > >> >> > > > > > > significant number of new requests, perhaps we should > bite > >> the > >> >> > > bullet > >> >> > > > > to > >> >> > > > > > > clean up the existing scala requests first before adding > new > >> >> ones? > >> >> > > > > > > > >> >> > > > > > > Thanks, > >> >> > > > > > > > >> >> > > > > > > Jun > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > On Thu, Mar 12, 2015 at 3:37 PM, Andrii Biletskyi < > >> >> > > > > > > andrii.bilets...@stealth.ly> wrote: > >> >> > > > > > > > >> >> > > > > > > > Hi, > >> >> > > > > > > > > >> >> > > > > > > > As said above - I list again all comments from this > thread > >> >> so we > >> >> > > > > > > > can see what's left and finalize all pending issues. > >> >> > > > > > > > > >> >> > > > > > > > Comments from Jay: > >> >> > > > > > > > 1. This is much needed functionality, but there are a > lot > >> >> of the > >> >> > > so > >> >> > > > > > let's > >> >> > > > > > > > really think these protocols through. We really want to > >> end > >> >> up > >> >> > > > with a > >> >> > > > > > set > >> >> > > > > > > > of well thought-out, orthoganol apis. For this reason I > >> >> think it > >> >> > > is > >> >> > > > > > > really > >> >> > > > > > > > important to think through the end state even if that > >> >> includes > >> >> > > APIs > >> >> > > > > we > >> >> > > > > > > > won't implement in the first phase. > >> >> > > > > > > > > >> >> > > > > > > > A: Definitely behind this. Would appreciate if there > are > >> >> concrete > >> >> > > > > > > comments > >> >> > > > > > > > how this can be improved. > >> >> > > > > > > > > >> >> > > > > > > > 2. Let's please please please wait until we have > switched > >> >> the > >> >> > > > server > >> >> > > > > > over > >> >> > > > > > > > to the new java protocol definitions. If we add upteen > >> more > >> >> ad > >> >> > > hoc > >> >> > > > > > scala > >> >> > > > > > > > objects that is just generating more work for the > >> >> conversion we > >> >> > > > know > >> >> > > > > we > >> >> > > > > > > > have to do. > >> >> > > > > > > > > >> >> > > > > > > > A: Fixed in the latest patch - removed scala protocol > >> >> classes. > >> >> > > > > > > > > >> >> > > > > > > > 3. This proposal introduces a new type of optional > >> >> parameter. > >> >> > > This > >> >> > > > is > >> >> > > > > > > > inconsistent with everything else in the protocol > where we > >> >> use -1 > >> >> > > > or > >> >> > > > > > some > >> >> > > > > > > > other marker value. You could argue either way but > let's > >> >> stick > >> >> > > with > >> >> > > > > > that > >> >> > > > > > > > for consistency. For clients that implemented the > protocol > >> >> in a > >> >> > > > > better > >> >> > > > > > > way > >> >> > > > > > > > than our scala code these basic primitives are hard to > >> >> change. > >> >> > > > > > > > > >> >> > > > > > > > A: Fixed in the latest patch - removed MaybeOf type and > >> >> changed > >> >> > > > > > protocol > >> >> > > > > > > > accordingly. > >> >> > > > > > > > > >> >> > > > > > > > 4. ClusterMetadata: This seems to duplicate > >> >> TopicMetadataRequest > >> >> > > > > which > >> >> > > > > > > has > >> >> > > > > > > > brokers, topics, and partitions. I think we should > rename > >> >> that > >> >> > > > > request > >> >> > > > > > > > ClusterMetadataRequest (or just MetadataRequest) and > >> >> include the > >> >> > > id > >> >> > > > > of > >> >> > > > > > > the > >> >> > > > > > > > controller. Or are there other things we could add > here? > >> >> > > > > > > > > >> >> > > > > > > > A: I agree. Updated the KIP. Let's extends > TopicMetadata > >> to > >> >> > > > version 2 > >> >> > > > > > and > >> >> > > > > > > > include controller. > >> >> > > > > > > > > >> >> > > > > > > > 5. We have a tendency to try to make a lot of requests > >> that > >> >> can > >> >> > > > only > >> >> > > > > go > >> >> > > > > > > to > >> >> > > > > > > > particular nodes. This adds a lot of burden for client > >> >> > > > > implementations > >> >> > > > > > > (it > >> >> > > > > > > > sounds easy but each discovery can fail in many parts > so > >> it > >> >> ends > >> >> > > up > >> >> > > > > > > being a > >> >> > > > > > > > full state machine to do right). I think we should > >> consider > >> >> > > making > >> >> > > > > > admin > >> >> > > > > > > > commands and ideally as many of the other apis as > possible > >> >> > > > available > >> >> > > > > on > >> >> > > > > > > all > >> >> > > > > > > > brokers and just redirect to the controller on the > broker > >> >> side. > >> >> > > > > Perhaps > >> >> > > > > > > > there would be a general way to encapsulate this > >> re-routing > >> >> > > > behavior. > >> >> > > > > > > > > >> >> > > > > > > > A: It's a very interesting idea, but seems there are > some > >> >> > > concerns > >> >> > > > > > about > >> >> > > > > > > > this > >> >> > > > > > > > feature (like performance considerations, how this will > >> >> > > complicate > >> >> > > > > > server > >> >> > > > > > > > etc). > >> >> > > > > > > > I believe this shouldn't be a blocker. If this feature > is > >> >> > > > implemented > >> >> > > > > > at > >> >> > > > > > > > some > >> >> > > > > > > > point it won't affect Admin changes - at least no > changes > >> to > >> >> > > public > >> >> > > > > API > >> >> > > > > > > > will be required. > >> >> > > > > > > > > >> >> > > > > > > > 6. We should probably normalize the key value pairs > used > >> for > >> >> > > > configs > >> >> > > > > > > rather > >> >> > > > > > > > than embedding a new formatting. So two strings rather > >> than > >> >> one > >> >> > > > with > >> >> > > > > an > >> >> > > > > > > > internal equals sign. > >> >> > > > > > > > > >> >> > > > > > > > A: Fixed in the latest patch - normalized configs and > >> >> changed > >> >> > > > > protocol > >> >> > > > > > > > accordingly. > >> >> > > > > > > > > >> >> > > > > > > > 7. Is the postcondition of these APIs that the command > has > >> >> begun > >> >> > > or > >> >> > > > > > that > >> >> > > > > > > > the command has been completed? It is a lot more > usable if > >> >> the > >> >> > > > > command > >> >> > > > > > > has > >> >> > > > > > > > been completed so you know that if you create a topic > and > >> >> then > >> >> > > > > publish > >> >> > > > > > to > >> >> > > > > > > > it you won't get an exception about there being no such > >> >> topic. > >> >> > > > > > > > > >> >> > > > > > > > A: For long running requests (like reassign > partitions) - > >> >> the > >> >> > > post > >> >> > > > > > > > condition is > >> >> > > > > > > > command has begun - so we don't block the client. In > case > >> >> of your > >> >> > > > > > > example - > >> >> > > > > > > > topic commands, this will be refactored and topic > commands > >> >> will > >> >> > > be > >> >> > > > > > > executed > >> >> > > > > > > > immediately, since the Controller will serve Admin > >> requests > >> >> > > > > > > > (follow-up ticket KAFKA-1777). > >> >> > > > > > > > > >> >> > > > > > > > 8. Describe topic and list topics duplicate a lot of > stuff > >> >> in the > >> >> > > > > > > metadata > >> >> > > > > > > > request. Is there a reason to give back topics marked > for > >> >> > > > deletion? I > >> >> > > > > > > feel > >> >> > > > > > > > like if we just make the post-condition of the delete > >> >> command be > >> >> > > > that > >> >> > > > > > the > >> >> > > > > > > > topic is deleted that will get rid of the need for this > >> >> right? > >> >> > > And > >> >> > > > it > >> >> > > > > > > will > >> >> > > > > > > > be much more intuitive. > >> >> > > > > > > > > >> >> > > > > > > > A: Fixed in the latest patch - removed topics marked > for > >> >> deletion > >> >> > > > in > >> >> > > > > > > > ListTopicsRequest. > >> >> > > > > > > > > >> >> > > > > > > > 9. Should we consider batching these requests? We have > >> >> generally > >> >> > > > > tried > >> >> > > > > > to > >> >> > > > > > > > allow multiple operations to be batched. My suspicion > is > >> >> that > >> >> > > > without > >> >> > > > > > > this > >> >> > > > > > > > we will get a lot of code that does something like > >> >> > > > > > > > for(topic: adminClient.listTopics()) > >> >> > > > > > > > adminClient.describeTopic(topic) > >> >> > > > > > > > this code will work great when you test on 5 topics but > >> not > >> >> do as > >> >> > > > > well > >> >> > > > > > if > >> >> > > > > > > > you have 50k. > >> >> > > > > > > > > >> >> > > > > > > > A: Updated the KIP - please check "Topic Admin Schema" > >> >> section. > >> >> > > > > > > > > >> >> > > > > > > > 10. I think we should also discuss how we want to > expose a > >> >> > > > > programmatic > >> >> > > > > > > JVM > >> >> > > > > > > > client api for these operations. Currently people rely > on > >> >> > > > AdminUtils > >> >> > > > > > > which > >> >> > > > > > > > is totally sketchy. I think we probably need another > >> client > >> >> under > >> >> > > > > > > clients/ > >> >> > > > > > > > that exposes administrative functionality. We will need > >> >> this just > >> >> > > > to > >> >> > > > > > > > properly test the new apis, I suspect. We should figure > >> out > >> >> that > >> >> > > > API. > >> >> > > > > > > > > >> >> > > > > > > > A: Updated the KIP - please check "Admin Client" > section > >> >> with an > >> >> > > > > > initial > >> >> > > > > > > > API proposal. > >> >> > > > > > > > > >> >> > > > > > > > 11. The other information that would be really useful > to > >> get > >> >> > > would > >> >> > > > be > >> >> > > > > > > > information about partitions--how much data is in the > >> >> partition, > >> >> > > > what > >> >> > > > > > are > >> >> > > > > > > > the segment offsets, what is the log-end offset (i.e. > last > >> >> > > offset), > >> >> > > > > > what > >> >> > > > > > > is > >> >> > > > > > > > the compaction point, etc. I think that done right this > >> >> would be > >> >> > > > the > >> >> > > > > > > > successor to the very awkward OffsetRequest we have > today. > >> >> > > > > > > > > >> >> > > > > > > > A: I removed ConsumerGroupOffsetsRequest in the latest > >> >> patch. I > >> >> > > > > believe > >> >> > > > > > > > this should > >> >> > > > > > > > be resolved in a separate KIP / jira ticket. > >> >> > > > > > > > > >> >> > > > > > > > 12. Generally we can do good error handling without > >> needing > >> >> > > custom > >> >> > > > > > > > server-side > >> >> > > > > > > > messages. I.e. generally the client has the context to > >> know > >> >> that > >> >> > > if > >> >> > > > > it > >> >> > > > > > > got > >> >> > > > > > > > an error that the topic doesn't exist to say "Topic X > >> >> doesn't > >> >> > > > exist" > >> >> > > > > > > rather > >> >> > > > > > > > than "error code 14" (or whatever). Maybe there are > >> specific > >> >> > > cases > >> >> > > > > > where > >> >> > > > > > > > this is hard? If we want to add server-side error > messages > >> >> we > >> >> > > > really > >> >> > > > > do > >> >> > > > > > > > need to do this in a consistent way across the > protocol. > >> >> > > > > > > > > >> >> > > > > > > > A: Updated the KIP - please check "Protocol Errors" > >> >> section. I > >> >> > > > added > >> >> > > > > > the > >> >> > > > > > > > comprehensive, fine-grained list of error codes. > >> >> > > > > > > > > >> >> > > > > > > > Comments from Guozhang: > >> >> > > > > > > > 13. Describe topic request: it would be great to go > beyond > >> >> just > >> >> > > > > > batching > >> >> > > > > > > on > >> >> > > > > > > > topic name regex for this request. For example, a very > >> >> common use > >> >> > > > > case > >> >> > > > > > of > >> >> > > > > > > > the topic command is to list all topics whose config > A's > >> >> value is > >> >> > > > B. > >> >> > > > > > With > >> >> > > > > > > > topic name regex then we have to first retrieve __all__ > >> >> topics's > >> >> > > > > > > > description info and then filter at the client end, > which > >> >> will > >> >> > > be a > >> >> > > > > > huge > >> >> > > > > > > > burden on ZK. > >> >> > > > > > > > AND > >> >> > > > > > > > 14. Config K-Vs in create topic: this is related to the > >> >> previous > >> >> > > > > point; > >> >> > > > > > > > maybe we can add another metadata K-V or just a > metadata > >> >> string > >> >> > > > along > >> >> > > > > > > side > >> >> > > > > > > > with config K-V in create topic like we did for offset > >> >> commit > >> >> > > > > request. > >> >> > > > > > > This > >> >> > > > > > > > field can be quite useful in storing information like > >> >> "owner" of > >> >> > > > the > >> >> > > > > > > topic > >> >> > > > > > > > who issue the create command, etc, which is quite > >> important > >> >> for a > >> >> > > > > > > > multi-tenant setting. Then in the describe topic > request > >> we > >> >> can > >> >> > > > also > >> >> > > > > > > batch > >> >> > > > > > > > on regex of the metadata field. > >> >> > > > > > > > > >> >> > > > > > > > A: As discussed it is very interesting but can be > >> >> implemented > >> >> > > later > >> >> > > > > > after > >> >> > > > > > > > we have some basic functionality there. > >> >> > > > > > > > > >> >> > > > > > > > 15. Today all the admin operations are async in the > sense > >> >> that > >> >> > > > > command > >> >> > > > > > > will > >> >> > > > > > > > return once it is written in ZK, and that is why we > need > >> >> extra > >> >> > > > > > > verification > >> >> > > > > > > > like testUtil.waitForTopicCreated() / verify partition > >> >> > > reassignment > >> >> > > > > > > > request, etc. With admin requests we could add a flag > to > >> >> enable / > >> >> > > > > > disable > >> >> > > > > > > > synchronous requests; when it is turned on, the > response > >> >> will not > >> >> > > > > > return > >> >> > > > > > > > until the request has been completed. And for async > >> >> requests we > >> >> > > can > >> >> > > > > > add a > >> >> > > > > > > > "token" field in the response, and then only need a > >> general > >> >> > > "admin > >> >> > > > > > > > verification request" with the given token to check if > the > >> >> async > >> >> > > > > > request > >> >> > > > > > > > has been completed. > >> >> > > > > > > > > >> >> > > > > > > > A: I see your point. My idea was to provide specific > >> >> > > > Verify...Request > >> >> > > > > > per > >> >> > > > > > > > each > >> >> > > > > > > > long running request, where needed. We can do it the > way > >> you > >> >> > > > suggest. > >> >> > > > > > The > >> >> > > > > > > > only > >> >> > > > > > > > concern is that introducing a token we again will make > >> >> schema > >> >> > > > > > "dynamic". > >> >> > > > > > > We > >> >> > > > > > > > wanted > >> >> > > > > > > > to do similar thing introducing single AdminRequest for > >> all > >> >> topic > >> >> > > > > > > commands > >> >> > > > > > > > but rejected > >> >> > > > > > > > this idea because we wanted to have schema defined. So > >> this > >> >> is > >> >> > > > more a > >> >> > > > > > > > choice between: > >> >> > > > > > > > a) have fixed schema but introduce each time new > >> >> Verify...Request > >> >> > > > for > >> >> > > > > > > > long-running requests > >> >> > > > > > > > b) use one request for verification but generalize it > with > >> >> token > >> >> > > > > > > > I'm fine with whatever decision community come to. Just > >> let > >> >> me > >> >> > > know > >> >> > > > > > your > >> >> > > > > > > > thoughts. > >> >> > > > > > > > > >> >> > > > > > > > Comment from Gwen: > >> >> > > > > > > > 16. Specifically for ownership, I think the plan is to > add > >> >> ACL > >> >> > > (it > >> >> > > > > > sounds > >> >> > > > > > > > like you are describing ACL) via an external system > >> (Argus, > >> >> > > > Sentry). > >> >> > > > > > > > I remember KIP-11 described this, but I can't find the > KIP > >> >> any > >> >> > > > > longer. > >> >> > > > > > > > > >> >> > > > > > > > A: Okay, no problem. Not sure though how we are going > to > >> >> handle > >> >> > > it. > >> >> > > > > > Wait > >> >> > > > > > > > which KIP > >> >> > > > > > > > will be committed first and include changes to > >> >> TopicMetadata from > >> >> > > > the > >> >> > > > > > > later > >> >> > > > > > > > one? > >> >> > > > > > > > Anyway, I added this note to "Open Questions" section > so > >> we > >> >> don't > >> >> > > > > miss > >> >> > > > > > > this > >> >> > > > > > > > piece. > >> >> > > > > > > > > >> >> > > > > > > > Thanks, > >> >> > > > > > > > Andrii Biletskyi > >> >> > > > > > > > > >> >> > > > > > > > On Fri, Mar 13, 2015 at 12:34 AM, Andrii Biletskyi < > >> >> > > > > > > > andrii.bilets...@stealth.ly> wrote: > >> >> > > > > > > > > >> >> > > > > > > > > Hi all, > >> >> > > > > > > > > > >> >> > > > > > > > > Today I uploaded the patch that covers some of the > >> >> discussed > >> >> > > and > >> >> > > > > > agreed > >> >> > > > > > > > > items: > >> >> > > > > > > > > - removed MaybeOf optional type > >> >> > > > > > > > > - switched to java protocol definitions > >> >> > > > > > > > > - simplified messages (normalized configs, removed > topic > >> >> marked > >> >> > > > for > >> >> > > > > > > > > deletion) > >> >> > > > > > > > > > >> >> > > > > > > > > I also updated the KIP-4 with respective changes and > >> >> wrote down > >> >> > > > my > >> >> > > > > > > > > proposal for > >> >> > > > > > > > > pending items: > >> >> > > > > > > > > - Batch Admin Operations -> updated Wire Protocol > schema > >> >> > > proposal > >> >> > > > > > > > > - Remove ClusterMetadata -> changed to extend > >> >> > > > TopicMetadataRequest > >> >> > > > > > > > > - Admin Client -> updated my initial proposal to > reflect > >> >> > > batching > >> >> > > > > > > > > - Error codes -> proposed fine-grained error code > >> instead > >> >> of > >> >> > > > > > > > > AdminRequestFailed > >> >> > > > > > > > > > >> >> > > > > > > > > I will also send a separate email to cover all > comments > >> >> from > >> >> > > this > >> >> > > > > > > thread. > >> >> > > > > > > > > > >> >> > > > > > > > > Thanks, > >> >> > > > > > > > > Andrii Biletskyi > >> >> > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > On Thu, Mar 12, 2015 at 9:26 PM, Gwen Shapira < > >> >> > > > > gshap...@cloudera.com > >> >> > > > > > > > >> >> > > > > > > > > wrote: > >> >> > > > > > > > > > >> >> > > > > > > > >> Found KIP-11 ( > >> >> > > > > > > > >> > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface > >> >> > > > > > > > >> ) > >> >> > > > > > > > >> It actually specifies changes to the Metadata > protocol, > >> >> so > >> >> > > > making > >> >> > > > > > sure > >> >> > > > > > > > >> both KIPs are consistent in this regard will be > good. > >> >> > > > > > > > >> > >> >> > > > > > > > >> On Thu, Mar 12, 2015 at 12:21 PM, Gwen Shapira < > >> >> > > > > > gshap...@cloudera.com > >> >> > > > > > > > > >> >> > > > > > > > >> wrote: > >> >> > > > > > > > >> > Specifically for ownership, I think the plan is to > >> add > >> >> ACL > >> >> > > (it > >> >> > > > > > > sounds > >> >> > > > > > > > >> > like you are describing ACL) via an external > system > >> >> (Argus, > >> >> > > > > > Sentry). > >> >> > > > > > > > >> > I remember KIP-11 described this, but I can't find > >> the > >> >> KIP > >> >> > > any > >> >> > > > > > > longer. > >> >> > > > > > > > >> > > >> >> > > > > > > > >> > Regardless, I think KIP-4 focuses on getting > >> >> information > >> >> > > that > >> >> > > > > > > already > >> >> > > > > > > > >> > exists from Kafka brokers, not on adding > information > >> >> that > >> >> > > > > perhaps > >> >> > > > > > > > >> > should exist but doesn't yet? > >> >> > > > > > > > >> > > >> >> > > > > > > > >> > Gwen > >> >> > > > > > > > >> > > >> >> > > > > > > > >> > > >> >> > > > > > > > >> > > >> >> > > > > > > > >> > > >> >> > > > > > > > >> > > >> >> > > > > > > > >> > On Thu, Mar 12, 2015 at 6:37 AM, Guozhang Wang < > >> >> > > > > > wangg...@gmail.com> > >> >> > > > > > > > >> wrote: > >> >> > > > > > > > >> >> Folks, > >> >> > > > > > > > >> >> > >> >> > > > > > > > >> >> Just want to elaborate a bit more on the > >> create-topic > >> >> > > > metadata > >> >> > > > > > and > >> >> > > > > > > > >> batching > >> >> > > > > > > > >> >> describe-topic based on config / metadata in my > >> >> previous > >> >> > > > email > >> >> > > > > as > >> >> > > > > > > we > >> >> > > > > > > > >> work > >> >> > > > > > > > >> >> on KAFKA-1694. The main motivation is to have > some > >> >> sort of > >> >> > > > > topic > >> >> > > > > > > > >> management > >> >> > > > > > > > >> >> mechanisms, which I think is quite important in a > >> >> > > > multi-tenant > >> >> > > > > / > >> >> > > > > > > > cloud > >> >> > > > > > > > >> >> architecture: today anyone can create topics in a > >> >> shared > >> >> > > > Kafka > >> >> > > > > > > > >> cluster, but > >> >> > > > > > > > >> >> there is no concept or "ownership" of topics that > >> are > >> >> > > created > >> >> > > > > by > >> >> > > > > > > > >> different > >> >> > > > > > > > >> >> users. For example, at LinkedIn we basically > >> >> distinguish > >> >> > > > topic > >> >> > > > > > > owners > >> >> > > > > > > > >> via > >> >> > > > > > > > >> >> some casual topic name prefix, which is a bit > >> awkward > >> >> and > >> >> > > > does > >> >> > > > > > not > >> >> > > > > > > > fly > >> >> > > > > > > > >> as > >> >> > > > > > > > >> >> we scale our customers. It would be great to use > >> >> > > > > describe-topics > >> >> > > > > > > such > >> >> > > > > > > > >> as: > >> >> > > > > > > > >> >> > >> >> > > > > > > > >> >> Describe all topics that is created by me. > >> >> > > > > > > > >> >> > >> >> > > > > > > > >> >> Describe all topics whose retention time is > >> overriden > >> >> to X. > >> >> > > > > > > > >> >> > >> >> > > > > > > > >> >> Describe all topics whose writable group include > >> user > >> >> Y > >> >> > > (this > >> >> > > > > is > >> >> > > > > > > > >> related to > >> >> > > > > > > > >> >> authorization), etc.. > >> >> > > > > > > > >> >> > >> >> > > > > > > > >> >> One possible way to achieve this is to add a > >> metadata > >> >> file > >> >> > > in > >> >> > > > > the > >> >> > > > > > > > >> >> create-topic request, whose value will also be > >> >> written ZK > >> >> > > as > >> >> > > > we > >> >> > > > > > > > create > >> >> > > > > > > > >> the > >> >> > > > > > > > >> >> topic; then describe-topics can choose to batch > >> topics > >> >> > > based > >> >> > > > on > >> >> > > > > > 1) > >> >> > > > > > > > name > >> >> > > > > > > > >> >> regex, 2) config K-V matching, 3) metadata regex, > >> etc. > >> >> > > > > > > > >> >> > >> >> > > > > > > > >> >> Thoughts? > >> >> > > > > > > > >> >> > >> >> > > > > > > > >> >> Guozhang > >> >> > > > > > > > >> >> > >> >> > > > > > > > >> >> On Thu, Mar 5, 2015 at 4:37 PM, Guozhang Wang < > >> >> > > > > > wangg...@gmail.com> > >> >> > > > > > > > >> wrote: > >> >> > > > > > > > >> >> > >> >> > > > > > > > >> >>> Thanks for the updated wiki. A few comments > below: > >> >> > > > > > > > >> >>> > >> >> > > > > > > > >> >>> 1. Error description in response: I think if > some > >> >> > > errorCode > >> >> > > > > > could > >> >> > > > > > > > >> indicate > >> >> > > > > > > > >> >>> several different error cases then we should > really > >> >> change > >> >> > > > it > >> >> > > > > to > >> >> > > > > > > > >> multiple > >> >> > > > > > > > >> >>> codes. In general the errorCode itself would be > >> >> precise > >> >> > > and > >> >> > > > > > > > >> sufficient for > >> >> > > > > > > > >> >>> describing the server side errors. > >> >> > > > > > > > >> >>> > >> >> > > > > > > > >> >>> 2. Describe topic request: it would be great to > go > >> >> beyond > >> >> > > > just > >> >> > > > > > > > >> batching on > >> >> > > > > > > > >> >>> topic name regex for this request. For example, > a > >> >> very > >> >> > > > common > >> >> > > > > > use > >> >> > > > > > > > >> case of > >> >> > > > > > > > >> >>> the topic command is to list all topics whose > >> config > >> >> A's > >> >> > > > value > >> >> > > > > > is > >> >> > > > > > > B. > >> >> > > > > > > > >> With > >> >> > > > > > > > >> >>> topic name regex then we have to first retrieve > >> >> __all__ > >> >> > > > > topics's > >> >> > > > > > > > >> >>> description info and then filter at the client > end, > >> >> which > >> >> > > > will > >> >> > > > > > be > >> >> > > > > > > a > >> >> > > > > > > > >> huge > >> >> > > > > > > > >> >>> burden on ZK. > >> >> > > > > > > > >> >>> > >> >> > > > > > > > >> >>> 3. Config K-Vs in create topic: this is related > to > >> >> the > >> >> > > > > previous > >> >> > > > > > > > point; > >> >> > > > > > > > >> >>> maybe we can add another metadata K-V or just a > >> >> metadata > >> >> > > > > string > >> >> > > > > > > > along > >> >> > > > > > > > >> side > >> >> > > > > > > > >> >>> with config K-V in create topic like we did for > >> >> offset > >> >> > > > commit > >> >> > > > > > > > >> request. This > >> >> > > > > > > > >> >>> field can be quite useful in storing information > >> like > >> >> > > > "owner" > >> >> > > > > of > >> >> > > > > > > the > >> >> > > > > > > > >> topic > >> >> > > > > > > > >> >>> who issue the create command, etc, which is > quite > >> >> > > important > >> >> > > > > for > >> >> > > > > > a > >> >> > > > > > > > >> >>> multi-tenant setting. Then in the describe topic > >> >> request > >> >> > > we > >> >> > > > > can > >> >> > > > > > > also > >> >> > > > > > > > >> batch > >> >> > > > > > > > >> >>> on regex of the metadata field. > >> >> > > > > > > > >> >>> > >> >> > > > > > > > >> >>> 4. Today all the admin operations are async in > the > >> >> sense > >> >> > > > that > >> >> > > > > > > > command > >> >> > > > > > > > >> will > >> >> > > > > > > > >> >>> return once it is written in ZK, and that is > why we > >> >> need > >> >> > > > extra > >> >> > > > > > > > >> verification > >> >> > > > > > > > >> >>> like testUtil.waitForTopicCreated() / verify > >> >> partition > >> >> > > > > > > reassignment > >> >> > > > > > > > >> >>> request, etc. With admin requests we could add a > >> >> flag to > >> >> > > > > enable > >> >> > > > > > / > >> >> > > > > > > > >> disable > >> >> > > > > > > > >> >>> synchronous requests; when it is turned on, the > >> >> response > >> >> > > > will > >> >> > > > > > not > >> >> > > > > > > > >> return > >> >> > > > > > > > >> >>> until the request has been completed. And for > async > >> >> > > requests > >> >> > > > > we > >> >> > > > > > > can > >> >> > > > > > > > >> add a > >> >> > > > > > > > >> >>> "token" field in the response, and then only > need a > >> >> > > general > >> >> > > > > > "admin > >> >> > > > > > > > >> >>> verification request" with the given token to > check > >> >> if the > >> >> > > > > async > >> >> > > > > > > > >> request > >> >> > > > > > > > >> >>> has been completed. > >> >> > > > > > > > >> >>> > >> >> > > > > > > > >> >>> 5. +1 for extending Metadata request to include > >> >> > > controller / > >> >> > > > > > > > >> coordinator > >> >> > > > > > > > >> >>> information, and then we can remove the > >> >> ConsumerMetadata / > >> >> > > > > > > > >> ClusterMetadata > >> >> > > > > > > > >> >>> requests. > >> >> > > > > > > > >> >>> > >> >> > > > > > > > >> >>> Guozhang > >> >> > > > > > > > >> >>> > >> >> > > > > > > > >> >>> On Tue, Mar 3, 2015 at 10:23 AM, Joel Koshy < > >> >> > > > > > jjkosh...@gmail.com> > >> >> > > > > > > > >> wrote: > >> >> > > > > > > > >> >>> > >> >> > > > > > > > >> >>>> Thanks for sending that out Joe - I don't > think I > >> >> will be > >> >> > > > > able > >> >> > > > > > to > >> >> > > > > > > > >> make > >> >> > > > > > > > >> >>>> it today, so if notes can be sent out afterward > >> that > >> >> > > would > >> >> > > > be > >> >> > > > > > > > great. > >> >> > > > > > > > >> >>>> > >> >> > > > > > > > >> >>>> On Mon, Mar 02, 2015 at 09:16:13AM -0800, Gwen > >> >> Shapira > >> >> > > > wrote: > >> >> > > > > > > > >> >>>> > Thanks for sending this out Joe. Looking > forward > >> >> to > >> >> > > > > chatting > >> >> > > > > > > with > >> >> > > > > > > > >> >>>> everyone :) > >> >> > > > > > > > >> >>>> > > >> >> > > > > > > > >> >>>> > On Mon, Mar 2, 2015 at 6:46 AM, Joe Stein < > >> >> > > > > > > joe.st...@stealth.ly> > >> >> > > > > > > > >> wrote: > >> >> > > > > > > > >> >>>> > > Hey, I just sent out a google hangout > invite > >> to > >> >> all > >> >> > > > pmc, > >> >> > > > > > > > >> committers > >> >> > > > > > > > >> >>>> and > >> >> > > > > > > > >> >>>> > > everyone I found working on a KIP. If I > missed > >> >> anyone > >> >> > > > in > >> >> > > > > > the > >> >> > > > > > > > >> invite > >> >> > > > > > > > >> >>>> please > >> >> > > > > > > > >> >>>> > > let me know and can update it, np. > >> >> > > > > > > > >> >>>> > > > >> >> > > > > > > > >> >>>> > > We should do this every Tuesday @ 2pm > Eastern > >> >> Time. > >> >> > > > Maybe > >> >> > > > > > we > >> >> > > > > > > > can > >> >> > > > > > > > >> get > >> >> > > > > > > > >> >>>> INFRA > >> >> > > > > > > > >> >>>> > > help to make a google account so we can > manage > >> >> > > better? > >> >> > > > > > > > >> >>>> > > > >> >> > > > > > > > >> >>>> > > To discuss > >> >> > > > > > > > >> >>>> > > > >> >> > > > > > > > >> >>>> > >> >> > > > > > > > >> > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > >> >> > > > > > > > >> >>>> > > in progress and related JIRA that are > >> >> interdependent > >> >> > > > and > >> >> > > > > > > common > >> >> > > > > > > > >> work. > >> >> > > > > > > > >> >>>> > > > >> >> > > > > > > > >> >>>> > > ~ Joe Stein > >> >> > > > > > > > >> >>>> > > > >> >> > > > > > > > >> >>>> > > On Tue, Feb 24, 2015 at 2:59 PM, Jay Kreps > < > >> >> > > > > > > > jay.kr...@gmail.com> > >> >> > > > > > > > >> >>>> wrote: > >> >> > > > > > > > >> >>>> > > > >> >> > > > > > > > >> >>>> > >> Let's stay on Google hangouts that will > also > >> >> record > >> >> > > > and > >> >> > > > > > make > >> >> > > > > > > > the > >> >> > > > > > > > >> >>>> sessions > >> >> > > > > > > > >> >>>> > >> available on youtube. > >> >> > > > > > > > >> >>>> > >> > >> >> > > > > > > > >> >>>> > >> -Jay > >> >> > > > > > > > >> >>>> > >> > >> >> > > > > > > > >> >>>> > >> On Tue, Feb 24, 2015 at 11:49 AM, Jeff > >> Holoman > >> >> < > >> >> > > > > > > > >> >>>> jholo...@cloudera.com> > >> >> > > > > > > > >> >>>> > >> wrote: > >> >> > > > > > > > >> >>>> > >> > >> >> > > > > > > > >> >>>> > >> > Jay / Joe > >> >> > > > > > > > >> >>>> > >> > > >> >> > > > > > > > >> >>>> > >> > We're happy to send out a Webex for this > >> >> purpose. > >> >> > > We > >> >> > > > > > could > >> >> > > > > > > > >> record > >> >> > > > > > > > >> >>>> the > >> >> > > > > > > > >> >>>> > >> > sessions if there is interest and > publish > >> >> them > >> >> > > out. > >> >> > > > > > > > >> >>>> > >> > > >> >> > > > > > > > >> >>>> > >> > Thanks > >> >> > > > > > > > >> >>>> > >> > > >> >> > > > > > > > >> >>>> > >> > Jeff > >> >> > > > > > > > >> >>>> > >> > > >> >> > > > > > > > >> >>>> > >> > On Tue, Feb 24, 2015 at 11:28 AM, Jay > >> Kreps < > >> >> > > > > > > > >> jay.kr...@gmail.com> > >> >> > > > > > > > >> >>>> wrote: > >> >> > > > > > > > >> >>>> > >> > > >> >> > > > > > > > >> >>>> > >> > > Let's try to get the technical > hang-ups > >> >> sorted > >> >> > > > out, > >> >> > > > > > > > though. > >> >> > > > > > > > >> I > >> >> > > > > > > > >> >>>> really > >> >> > > > > > > > >> >>>> > >> > think > >> >> > > > > > > > >> >>>> > >> > > there is some benefit to live > discussion > >> vs > >> >> > > > > writing. I > >> >> > > > > > > am > >> >> > > > > > > > >> >>>> hopeful that > >> >> > > > > > > > >> >>>> > >> if > >> >> > > > > > > > >> >>>> > >> > > we post instructions and give > ourselves a > >> >> few > >> >> > > > > attempts > >> >> > > > > > > we > >> >> > > > > > > > >> can > >> >> > > > > > > > >> >>>> get it > >> >> > > > > > > > >> >>>> > >> > > working. > >> >> > > > > > > > >> >>>> > >> > > > >> >> > > > > > > > >> >>>> > >> > > Tuesday at that time would work for > >> >> me...any > >> >> > > > > > objections? > >> >> > > > > > > > >> >>>> > >> > > > >> >> > > > > > > > >> >>>> > >> > > -Jay > >> >> > > > > > > > >> >>>> > >> > > > >> >> > > > > > > > >> >>>> > >> > > On Tue, Feb 24, 2015 at 8:18 AM, Joe > >> Stein > >> >> < > >> >> > > > > > > > >> joe.st...@stealth.ly > >> >> > > > > > > > >> >>>> > > >> >> > > > > > > > >> >>>> > >> wrote: > >> >> > > > > > > > >> >>>> > >> > > > >> >> > > > > > > > >> >>>> > >> > > > Weekly would be great maybe like > every > >> >> > > Tuesday ~ > >> >> > > > > 1pm > >> >> > > > > > > ET > >> >> > > > > > > > / > >> >> > > > > > > > >> 10am > >> >> > > > > > > > >> >>>> PT > >> >> > > > > > > > >> >>>> > >> ???? > >> >> > > > > > > > >> >>>> > >> > > > > >> >> > > > > > > > >> >>>> > >> > > > I don't mind google hangout but > there > >> is > >> >> > > always > >> >> > > > > some > >> >> > > > > > > > >> issue or > >> >> > > > > > > > >> >>>> > >> whatever > >> >> > > > > > > > >> >>>> > >> > so > >> >> > > > > > > > >> >>>> > >> > > > we know the apache irc channel > works. > >> We > >> >> can > >> >> > > > start > >> >> > > > > > > there > >> >> > > > > > > > >> and > >> >> > > > > > > > >> >>>> see how > >> >> > > > > > > > >> >>>> > >> it > >> >> > > > > > > > >> >>>> > >> > > > goes? We can pull transcripts too > and > >> >> > > associate > >> >> > > > to > >> >> > > > > > > > >> tickets if > >> >> > > > > > > > >> >>>> need be > >> >> > > > > > > > >> >>>> > >> > > makes > >> >> > > > > > > > >> >>>> > >> > > > it helpful for things. > >> >> > > > > > > > >> >>>> > >> > > > > >> >> > > > > > > > >> >>>> > >> > > > ~ Joestein > >> >> > > > > > > > >> >>>> > >> > > > > >> >> > > > > > > > >> >>>> > >> > > > On Tue, Feb 24, 2015 at 11:10 AM, > Jay > >> >> Kreps < > >> >> > > > > > > > >> >>>> jay.kr...@gmail.com> > >> >> > > > > > > > >> >>>> > >> > wrote: > >> >> > > > > > > > >> >>>> > >> > > > > >> >> > > > > > > > >> >>>> > >> > > > > We'd talked about doing a Google > >> >> Hangout to > >> >> > > > chat > >> >> > > > > > > about > >> >> > > > > > > > >> this. > >> >> > > > > > > > >> >>>> What > >> >> > > > > > > > >> >>>> > >> > about > >> >> > > > > > > > >> >>>> > >> > > > > generalizing that a little > >> further...I > >> >> > > > actually > >> >> > > > > > > think > >> >> > > > > > > > it > >> >> > > > > > > > >> >>>> would be > >> >> > > > > > > > >> >>>> > >> > good > >> >> > > > > > > > >> >>>> > >> > > > for > >> >> > > > > > > > >> >>>> > >> > > > > everyone spending a reasonable > chunk > >> of > >> >> > > their > >> >> > > > > week > >> >> > > > > > > on > >> >> > > > > > > > >> Kafka > >> >> > > > > > > > >> >>>> stuff > >> >> > > > > > > > >> >>>> > >> to > >> >> > > > > > > > >> >>>> > >> > > > maybe > >> >> > > > > > > > >> >>>> > >> > > > > sync up once a week. I think we > could > >> >> use > >> >> > > time > >> >> > > > > to > >> >> > > > > > > talk > >> >> > > > > > > > >> >>>> through > >> >> > > > > > > > >> >>>> > >> design > >> >> > > > > > > > >> >>>> > >> > > > > stuff, make sure we are on top of > >> code > >> >> > > > reviews, > >> >> > > > > > talk > >> >> > > > > > > > >> through > >> >> > > > > > > > >> >>>> any > >> >> > > > > > > > >> >>>> > >> > tricky > >> >> > > > > > > > >> >>>> > >> > > > > issues, etc. > >> >> > > > > > > > >> >>>> > >> > > > > > >> >> > > > > > > > >> >>>> > >> > > > > We can make it publicly available > so > >> >> that > >> >> > > any > >> >> > > > > one > >> >> > > > > > > can > >> >> > > > > > > > >> follow > >> >> > > > > > > > >> >>>> along > >> >> > > > > > > > >> >>>> > >> > who > >> >> > > > > > > > >> >>>> > >> > > > > likes. > >> >> > > > > > > > >> >>>> > >> > > > > > >> >> > > > > > > > >> >>>> > >> > > > > Any interest in doing this? If so > >> I'll > >> >> try > >> >> > > to > >> >> > > > > set > >> >> > > > > > it > >> >> > > > > > > > up > >> >> > > > > > > > >> >>>> starting > >> >> > > > > > > > >> >>>> > >> next > >> >> > > > > > > > >> >>>> > >> > > > week. > >> >> > > > > > > > >> >>>> > >> > > > > > >> >> > > > > > > > >> >>>> > >> > > > > -Jay > >> >> > > > > > > > >> >>>> > >> > > > > > >> >> > > > > > > > >> >>>> > >> > > > > On Tue, Feb 24, 2015 at 3:57 AM, > >> Andrii > >> >> > > > > Biletskyi > >> >> > > > > > < > >> >> > > > > > > > >> >>>> > >> > > > > andrii.bilets...@stealth.ly> > wrote: > >> >> > > > > > > > >> >>>> > >> > > > > > >> >> > > > > > > > >> >>>> > >> > > > > > Hi all, > >> >> > > > > > > > >> >>>> > >> > > > > > > >> >> > > > > > > > >> >>>> > >> > > > > > I've updated KIP page, fixed / > >> >> aligned > >> >> > > > > document > >> >> > > > > > > > >> structure. > >> >> > > > > > > > >> >>>> Also I > >> >> > > > > > > > >> >>>> > >> > > added > >> >> > > > > > > > >> >>>> > >> > > > > > some > >> >> > > > > > > > >> >>>> > >> > > > > > very initial proposal for > >> >> AdminClient so > >> >> > > we > >> >> > > > > have > >> >> > > > > > > > >> something > >> >> > > > > > > > >> >>>> to > >> >> > > > > > > > >> >>>> > >> start > >> >> > > > > > > > >> >>>> > >> > > > from > >> >> > > > > > > > >> >>>> > >> > > > > > while > >> >> > > > > > > > >> >>>> > >> > > > > > discussing the KIP. > >> >> > > > > > > > >> >>>> > >> > > > > > > >> >> > > > > > > > >> >>>> > >> > > > > > > >> >> > > > > > > > >> >>>> > >> > > > > > > >> >> > > > > > > > >> >>>> > >> > > > > > >> >> > > > > > > > >> >>>> > >> > > > > >> >> > > > > > > > >> >>>> > >> > > > >> >> > > > > > > > >> >>>> > >> > > >> >> > > > > > > > >> >>>> > >> > >> >> > > > > > > > >> >>>> > >> >> > > > > > > > >> > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > >> >> > > > > > > > >> >>>> > >> > > > > > > >> >> > > > > > > > >> >>>> > >> > > > > > Thanks, > >> >> > > > > > > > >> >>>> > >> > > > > > Andrii Biletskyi > >> >> > > > > > > > >> >>>> > >> > > > > > > >> >> > > > > > > > >> >>>> > >> > > > > > On Wed, Feb 18, 2015 at 9:01 PM, > >> >> Andrii > >> >> > > > > > Biletskyi > >> >> > > > > > > < > >> >> > > > > > > > >> >>>> > >> > > > > > andrii.bilets...@stealth.ly> > >> wrote: > >> >> > > > > > > > >> >>>> > >> > > > > > > >> >> > > > > > > > >> >>>> > >> > > > > > > Jay, > >> >> > > > > > > > >> >>>> > >> > > > > > > > >> >> > > > > > > > >> >>>> > >> > > > > > > Re error messages: you are > right, > >> >> in > >> >> > > most > >> >> > > > > > cases > >> >> > > > > > > > >> client > >> >> > > > > > > > >> >>>> will > >> >> > > > > > > > >> >>>> > >> have > >> >> > > > > > > > >> >>>> > >> > > > enough > >> >> > > > > > > > >> >>>> > >> > > > > > > context to show descriptive > error > >> >> > > message. > >> >> > > > > My > >> >> > > > > > > > >> concern is > >> >> > > > > > > > >> >>>> that > >> >> > > > > > > > >> >>>> > >> we > >> >> > > > > > > > >> >>>> > >> > > will > >> >> > > > > > > > >> >>>> > >> > > > > > have > >> >> > > > > > > > >> >>>> > >> > > > > > > to > >> >> > > > > > > > >> >>>> > >> > > > > > > add lots of new error codes > for > >> >> each > >> >> > > > > possible > >> >> > > > > > > > >> error. Of > >> >> > > > > > > > >> >>>> course, > >> >> > > > > > > > >> >>>> > >> > we > >> >> > > > > > > > >> >>>> > >> > > > > could > >> >> > > > > > > > >> >>>> > >> > > > > > > reuse > >> >> > > > > > > > >> >>>> > >> > > > > > > some of existing like > >> >> > > > > > > UknownTopicOrPartitionCode, > >> >> > > > > > > > >> but we > >> >> > > > > > > > >> >>>> will > >> >> > > > > > > > >> >>>> > >> > also > >> >> > > > > > > > >> >>>> > >> > > > need > >> >> > > > > > > > >> >>>> > >> > > > > > to > >> >> > > > > > > > >> >>>> > >> > > > > > > add smth like: > >> >> TopicAlreadyExistsCode, > >> >> > > > > > > > >> >>>> TopicConfigInvalid (both > >> >> > > > > > > > >> >>>> > >> > for > >> >> > > > > > > > >> >>>> > >> > > > > topic > >> >> > > > > > > > >> >>>> > >> > > > > > > name and config, and probably > >> user > >> >> would > >> >> > > > > like > >> >> > > > > > to > >> >> > > > > > > > >> know > >> >> > > > > > > > >> >>>> what > >> >> > > > > > > > >> >>>> > >> > exactly > >> >> > > > > > > > >> >>>> > >> > > > > > > is wrong in his config), > >> >> > > > > > > InvalidReplicaAssignment, > >> >> > > > > > > > >> >>>> > >> InternalError > >> >> > > > > > > > >> >>>> > >> > > > (e.g. > >> >> > > > > > > > >> >>>> > >> > > > > > > zookeeper failure) etc. > >> >> > > > > > > > >> >>>> > >> > > > > > > And this is only for > >> TopicCommand, > >> >> we > >> >> > > will > >> >> > > > > > also > >> >> > > > > > > > >> need to > >> >> > > > > > > > >> >>>> add > >> >> > > > > > > > >> >>>> > >> > similar > >> >> > > > > > > > >> >>>> > >> > > > > stuff > >> >> > > > > > > > >> >>>> > >> > > > > > > for > >> >> > > > > > > > >> >>>> > >> > > > > > > ReassignPartitions, > >> >> PreferredReplica. So > >> >> > > > > we'll > >> >> > > > > > > end > >> >> > > > > > > > >> up > >> >> > > > > > > > >> >>>> with a > >> >> > > > > > > > >> >>>> > >> > large > >> >> > > > > > > > >> >>>> > >> > > > list > >> >> > > > > > > > >> >>>> > >> > > > > > of > >> >> > > > > > > > >> >>>> > >> > > > > > > error codes, used only in > Admin > >> >> > > protocol. > >> >> > > > > > > > >> >>>> > >> > > > > > > Having said that, I agree my > >> >> proposal is > >> >> > > > not > >> >> > > > > > > > >> consistent > >> >> > > > > > > > >> >>>> with > >> >> > > > > > > > >> >>>> > >> > other > >> >> > > > > > > > >> >>>> > >> > > > > cases. > >> >> > > > > > > > >> >>>> > >> > > > > > > Maybe we can find better > solution > >> >> or > >> >> > > > > something > >> >> > > > > > > > >> >>>> in-between. > >> >> > > > > > > > >> >>>> > >> > > > > > > > >> >> > > > > > > > >> >>>> > >> > > > > > > Re Hangout chat: I think it > is a > >> >> great > >> >> > > > idea. > >> >> > > > > > > This > >> >> > > > > > > > >> way we > >> >> > > > > > > > >> >>>> can > >> >> > > > > > > > >> >>>> > >> move > >> >> > > > > > > > >> >>>> > >> > > on > >> >> > > > > > > > >> >>>> > >> > > > > > > faster. > >> >> > > > > > > > >> >>>> > >> > > > > > > Let's agree somehow on > date/time > >> so > >> >> > > people > >> >> > > > > can > >> >> > > > > > > > join. > >> >> > > > > > > > >> >>>> Will work > >> >> > > > > > > > >> >>>> > >> > for > >> >> > > > > > > > >> >>>> > >> > > me > >> >> > > > > > > > >> >>>> > >> > > > > > this > >> >> > > > > > > > >> >>>> > >> > > > > > > and > >> >> > > > > > > > >> >>>> > >> > > > > > > next week almost anytime if > >> agreed > >> >> in > >> >> > > > > advance. > >> >> > > > > > > > >> >>>> > >> > > > > > > > >> >> > > > > > > > >> >>>> > >> > > > > > > Thanks, > >> >> > > > > > > > >> >>>> > >> > > > > > > Andrii > >> >> > > > > > > > >> >>>> > >> > > > > > > > >> >> > > > > > > > >> >>>> > >> > > > > > > On Wed, Feb 18, 2015 at 7:09 > PM, > >> >> Jay > >> >> > > > Kreps < > >> >> > > > > > > > >> >>>> > >> jay.kr...@gmail.com> > >> >> > > > > > > > >> >>>> > >> > > > > wrote: > >> >> > > > > > > > >> >>>> > >> > > > > > > > >> >> > > > > > > > >> >>>> > >> > > > > > >> Hey Andrii, > >> >> > > > > > > > >> >>>> > >> > > > > > >> > >> >> > > > > > > > >> >>>> > >> > > > > > >> Generally we can do good > error > >> >> handling > >> >> > > > > > without > >> >> > > > > > > > >> needing > >> >> > > > > > > > >> >>>> custom > >> >> > > > > > > > >> >>>> > >> > > > > > server-side > >> >> > > > > > > > >> >>>> > >> > > > > > >> messages. I.e. generally the > >> >> client has > >> >> > > > the > >> >> > > > > > > > >> context to > >> >> > > > > > > > >> >>>> know > >> >> > > > > > > > >> >>>> > >> that > >> >> > > > > > > > >> >>>> > >> > > if > >> >> > > > > > > > >> >>>> > >> > > > it > >> >> > > > > > > > >> >>>> > >> > > > > > got > >> >> > > > > > > > >> >>>> > >> > > > > > >> an error that the topic > doesn't > >> >> exist > >> >> > > to > >> >> > > > > say > >> >> > > > > > > > >> "Topic X > >> >> > > > > > > > >> >>>> doesn't > >> >> > > > > > > > >> >>>> > >> > > exist" > >> >> > > > > > > > >> >>>> > >> > > > > > >> rather > >> >> > > > > > > > >> >>>> > >> > > > > > >> than "error code 14" (or > >> >> whatever). > >> >> > > Maybe > >> >> > > > > > there > >> >> > > > > > > > are > >> >> > > > > > > > >> >>>> specific > >> >> > > > > > > > >> >>>> > >> > cases > >> >> > > > > > > > >> >>>> > >> > > > > where > >> >> > > > > > > > >> >>>> > >> > > > > > >> this is hard? If we want to > add > >> >> > > > server-side > >> >> > > > > > > error > >> >> > > > > > > > >> >>>> messages we > >> >> > > > > > > > >> >>>> > >> > > really > >> >> > > > > > > > >> >>>> > >> > > > > do > >> >> > > > > > > > >> >>>> > >> > > > > > >> need to do this in a > consistent > >> >> way > >> >> > > > across > >> >> > > > > > the > >> >> > > > > > > > >> protocol. > >> >> > > > > > > > >> >>>> > >> > > > > > >> > >> >> > > > > > > > >> >>>> > >> > > > > > >> I still have a bunch of open > >> >> questions > >> >> > > > here > >> >> > > > > > > from > >> >> > > > > > > > my > >> >> > > > > > > > >> >>>> previous > >> >> > > > > > > > >> >>>> > >> > > list. I > >> >> > > > > > > > >> >>>> > >> > > > > > will > >> >> > > > > > > > >> >>>> > >> > > > > > >> be out for the next few days > for > >> >> Strata > >> >> > > > > > though. > >> >> > > > > > > > >> Maybe > >> >> > > > > > > > >> >>>> we could > >> >> > > > > > > > >> >>>> > >> > do > >> >> > > > > > > > >> >>>> > >> > > a > >> >> > > > > > > > >> >>>> > >> > > > > > Google > >> >> > > > > > > > >> >>>> > >> > > > > > >> Hangout chat on any open > issues > >> >> some > >> >> > > time > >> >> > > > > > > towards > >> >> > > > > > > > >> the > >> >> > > > > > > > >> >>>> end of > >> >> > > > > > > > >> >>>> > >> > next > >> >> > > > > > > > >> >>>> > >> > > > week > >> >> > > > > > > > >> >>>> > >> > > > > > for > >> >> > > > > > > > >> >>>> > >> > > > > > >> anyone interested in this > >> ticket? > >> >> I > >> >> > > have > >> >> > > > a > >> >> > > > > > > > feeling > >> >> > > > > > > > >> that > >> >> > > > > > > > >> >>>> might > >> >> > > > > > > > >> >>>> > >> > > > progress > >> >> > > > > > > > >> >>>> > >> > > > > > >> things a little faster than > >> >> email--I > >> >> > > > think > >> >> > > > > we > >> >> > > > > > > > >> could talk > >> >> > > > > > > > >> >>>> > >> through > >> >> > > > > > > > >> >>>> > >> > > > those > >> >> > > > > > > > >> >>>> > >> > > > > > >> issues I brought up fairly > >> >> quickly... > >> >> > > > > > > > >> >>>> > >> > > > > > >> > >> >> > > > > > > > >> >>>> > >> > > > > > >> -Jay > >> >> > > > > > > > >> >>>> > >> > > > > > >> > >> >> > > > > > > > >> >>>> > >> > > > > > >> On Wed, Feb 18, 2015 at 7:27 > AM, > >> >> Andrii > >> >> > > > > > > > Biletskyi < > >> >> > > > > > > > >> >>>> > >> > > > > > >> andrii.bilets...@stealth.ly> > >> >> wrote: > >> >> > > > > > > > >> >>>> > >> > > > > > >> > >> >> > > > > > > > >> >>>> > >> > > > > > >> > Hi all, > >> >> > > > > > > > >> >>>> > >> > > > > > >> > > >> >> > > > > > > > >> >>>> > >> > > > > > >> > I'm trying to address some > of > >> >> the > >> >> > > > issues > >> >> > > > > > > which > >> >> > > > > > > > >> were > >> >> > > > > > > > >> >>>> > >> mentioned > >> >> > > > > > > > >> >>>> > >> > > > > earlier > >> >> > > > > > > > >> >>>> > >> > > > > > >> about > >> >> > > > > > > > >> >>>> > >> > > > > > >> > Admin RQ/RP format. One of > >> >> those was > >> >> > > > > about > >> >> > > > > > > > >> batching > >> >> > > > > > > > >> >>>> > >> > operations. > >> >> > > > > > > > >> >>>> > >> > > > What > >> >> > > > > > > > >> >>>> > >> > > > > > if > >> >> > > > > > > > >> >>>> > >> > > > > > >> we > >> >> > > > > > > > >> >>>> > >> > > > > > >> > follow TopicCommand > approach > >> >> and let > >> >> > > > > people > >> >> > > > > > > > >> specify > >> >> > > > > > > > >> >>>> > >> topic-name > >> >> > > > > > > > >> >>>> > >> > > by > >> >> > > > > > > > >> >>>> > >> > > > > > >> regexp - > >> >> > > > > > > > >> >>>> > >> > > > > > >> > would that cover most of > the > >> use > >> >> > > cases? > >> >> > > > > > > > >> >>>> > >> > > > > > >> > > >> >> > > > > > > > >> >>>> > >> > > > > > >> > Secondly, is what > information > >> >> should > >> >> > > we > >> >> > > > > > > > generally > >> >> > > > > > > > >> >>>> provide in > >> >> > > > > > > > >> >>>> > >> > > Admin > >> >> > > > > > > > >> >>>> > >> > > > > > >> > responses. > >> >> > > > > > > > >> >>>> > >> > > > > > >> > I realize that Admin > commands > >> >> don't > >> >> > > > imply > >> >> > > > > > > they > >> >> > > > > > > > >> will > >> >> > > > > > > > >> >>>> be used > >> >> > > > > > > > >> >>>> > >> > only > >> >> > > > > > > > >> >>>> > >> > > > in > >> >> > > > > > > > >> >>>> > >> > > > > > CLI > >> >> > > > > > > > >> >>>> > >> > > > > > >> > but, > >> >> > > > > > > > >> >>>> > >> > > > > > >> > it seems to me, CLI is a > very > >> >> > > important > >> >> > > > > > > client > >> >> > > > > > > > >> of this > >> >> > > > > > > > >> >>>> > >> > feature. > >> >> > > > > > > > >> >>>> > >> > > In > >> >> > > > > > > > >> >>>> > >> > > > > > this > >> >> > > > > > > > >> >>>> > >> > > > > > >> > case, > >> >> > > > > > > > >> >>>> > >> > > > > > >> > seems logical, we would > like > >> to > >> >> > > provide > >> >> > > > > > users > >> >> > > > > > > > >> with > >> >> > > > > > > > >> >>>> rich > >> >> > > > > > > > >> >>>> > >> > > experience > >> >> > > > > > > > >> >>>> > >> > > > > in > >> >> > > > > > > > >> >>>> > >> > > > > > >> terms > >> >> > > > > > > > >> >>>> > >> > > > > > >> > of > >> >> > > > > > > > >> >>>> > >> > > > > > >> > getting results / errors of > >> the > >> >> > > > executed > >> >> > > > > > > > >> commands. > >> >> > > > > > > > >> >>>> Usually > >> >> > > > > > > > >> >>>> > >> we > >> >> > > > > > > > >> >>>> > >> > > > supply > >> >> > > > > > > > >> >>>> > >> > > > > > >> with > >> >> > > > > > > > >> >>>> > >> > > > > > >> > responses only errorCode, > >> which > >> >> looks > >> >> > > > > very > >> >> > > > > > > > >> limiting, > >> >> > > > > > > > >> >>>> in case > >> >> > > > > > > > >> >>>> > >> > of > >> >> > > > > > > > >> >>>> > >> > > > CLI > >> >> > > > > > > > >> >>>> > >> > > > > we > >> >> > > > > > > > >> >>>> > >> > > > > > >> may > >> >> > > > > > > > >> >>>> > >> > > > > > >> > want to print human > readable > >> >> error > >> >> > > > > > > description. > >> >> > > > > > > > >> >>>> > >> > > > > > >> > > >> >> > > > > > > > >> >>>> > >> > > > > > >> > So, taking into account > >> >> previous item > >> >> > > > > about > >> >> > > > > > > > >> batching, > >> >> > > > > > > > >> >>>> what > >> >> > > > > > > > >> >>>> > >> do > >> >> > > > > > > > >> >>>> > >> > > you > >> >> > > > > > > > >> >>>> > >> > > > > > think > >> >> > > > > > > > >> >>>> > >> > > > > > >> > about > >> >> > > > > > > > >> >>>> > >> > > > > > >> > having smth like: > >> >> > > > > > > > >> >>>> > >> > > > > > >> > > >> >> > > > > > > > >> >>>> > >> > > > > > >> > ('create' doesn't support > >> >> regexp) > >> >> > > > > > > > >> >>>> > >> > > > > > >> > CreateTopicRequest => > >> TopicName > >> >> > > > > Partitions > >> >> > > > > > > > >> Replicas > >> >> > > > > > > > >> >>>> > >> > > > > ReplicaAssignment > >> >> > > > > > > > >> >>>> > >> > > > > > >> > [Config] > >> >> > > > > > > > >> >>>> > >> > > > > > >> > CreateTopicResponse => > >> ErrorCode > >> >> > > > > > > > ErrorDescription > >> >> > > > > > > > >> >>>> > >> > > > > > >> > ErrorCode => int16 > >> >> > > > > > > > >> >>>> > >> > > > > > >> > ErrorDescription => > string > >> >> (empty > >> >> > > if > >> >> > > > > > > > >> successful) > >> >> > > > > > > > >> >>>> > >> > > > > > >> > > >> >> > > > > > > > >> >>>> > >> > > > > > >> > AlterTopicRequest -> > >> >> TopicNameRegexp > >> >> > > > > >> > > >> > ... > >> > > >> > [Message clipped] > >> >
