Andrii, 101. That's what I was thinking too, but it may not be that simple. In TopicMetadataRequest_V1, we can let it not trigger auto topic creation. Then, in the producer side, if it gets an UnknownTopicException, it can explicitly issue a createTopicRequest for auto topic creation. On the consumer side, it will never issue createTopicRequest. This works when auto topic creation is enabled on the broker side. However, I am not sure how things will work when auto topic creation is disabled on the broker side. In this case, we want to have a way to manually create a topic, potentially through admin commands. However, then we need a way to distinguish createTopicRequest issued from the producer clients and the admin tools. May be we can add a new field in createTopicRequest and set it differently in the producer client and the admin client. However, I am not sure if that's the best approach.
2. Yes, refactoring existing requests is a non-trivial amount of work. I posted some comments in KAFKA-1927. We will probably have to fix KAFKA-1927 first, before adding the new logic in KAFKA-1694. Otherwise, the changes will be too big. 102. About the AdminClient: 102.1. It's a bit weird that we return exception in the api. It seems that we should either return error code or throw an exception when getting the response state. 102.2. We probably shouldn't explicitly use the request object in the api. Not every request evolution requires an api change. Thanks, Jun On Fri, Mar 13, 2015 at 4:08 AM, Andrii Biletskyi < andrii.bilets...@stealth.ly> wrote: > Jun, > > Thanks for you comments. Answers inline: > > 100. There are a few fields such as ReplicaAssignment, > > ReassignPartitionRequest, > > and PartitionsSerialized that are represented as a string, but contain > > composite structures in json. Could we flatten them out directly in the > > protocol definition as arrays/records? > > > Yes, now with Admin Client this looks a bit weird. My initial motivation > was: > ReassignPartitionCommand accepts input in json, we want to remain tools' > interfaces unchanged, where possible. > If we port it to deserialized format, in CLI (/tools project) we will have > to add some > json library since /tools is written in java and we'll need to deserialize > json file > provided by a user. Can we quickly agree on what this library should be > (Jackson, GSON, whatever)? > > 101. Does TopicMetadataRequest v1 still trigger auto topic creation? This > > will be a bit weird now that we have a separate topic creation api. Have > > you thought about how the new createTopicRequest and TopicMetadataRequest > > v1 will be used in the producer/consumer client, in addition to admin > > tools? For example, ideally, we don't want TopicMetadataRequest from the > > consumer to trigger auto topic creation. > > > I agree, this strange logic should be fixed. I'm not confident in this > Kafka part so > correct me if I'm wrong, but it doesn't look like a hard thing to do, I > think we can > leverage AdminClient for that in Producer and unconditionally remove topic > creation from the TopicMetadataRequest_V1. > > 2. I think Jay meant getting rid of scala classes > > like HeartbeatRequestAndHeader and HeartbeatResponseAndHeader. We did > that > > as a stop-gap thing when adding the new requests for the consumers. > > However, the long term plan is to get rid of all those and just reuse the > > java request/response in the client. Since this KIP proposes to add a > > significant number of new requests, perhaps we should bite the bullet to > > clean up the existing scala requests first before adding new ones? > > > > Yes, looks like I misunderstood the point of ...RequestAndHeader. Okay, I > will > rework that. The only thing is that I don't see any example how it was done > for at > least one existing protocol message. Thus, as I understand, I have to think > how we > are going to do it. > Re porting all existing RQ/RP in this patch. Sounds reasonable, but if it's > an *obligatory* > requirement to have Admin KIP done, I'm afraid this can be a serious > blocker for us. > There are 13 protocol messages and all that would require not only unit > tests but quite > intensive manual testing, no? I'm afraid I'm not the right guy to cover > pretty much all > Kafka core internals :). Let me know your thoughts on this item. Btw there > is a ticket to > follow-up this issue (https://issues.apache.org/jira/browse/KAFKA-2006). > > Thanks, > Andrii Biletskyi > > > On Fri, Mar 13, 2015 at 6:40 AM, Jun Rao <j...@confluent.io> wrote: > > > Andrii, > > > > > > A few more comments. > > > > 100. There are a few fields such as ReplicaAssignment, > > ReassignPartitionRequest, > > and PartitionsSerialized that are represented as a string, but contain > > composite structures in json. Could we flatten them out directly in the > > protocol definition as arrays/records? > > > > 101. Does TopicMetadataRequest v1 still trigger auto topic creation? This > > will be a bit weird now that we have a separate topic creation api. Have > > you thought about how the new createTopicRequest and TopicMetadataRequest > > v1 will be used in the producer/consumer client, in addition to admin > > tools? For example, ideally, we don't want TopicMetadataRequest from the > > consumer to trigger auto topic creation. > > > > 2. I think Jay meant getting rid of scala classes > > like HeartbeatRequestAndHeader and HeartbeatResponseAndHeader. We did > that > > as a stop-gap thing when adding the new requests for the consumers. > > However, the long term plan is to get rid of all those and just reuse the > > java request/response in the client. Since this KIP proposes to add a > > significant number of new requests, perhaps we should bite the bullet to > > clean up the existing scala requests first before adding new ones? > > > > Thanks, > > > > Jun > > > > > > > > On Thu, Mar 12, 2015 at 3:37 PM, Andrii Biletskyi < > > andrii.bilets...@stealth.ly> wrote: > > > > > Hi, > > > > > > As said above - I list again all comments from this thread so we > > > can see what's left and finalize all pending issues. > > > > > > Comments from Jay: > > > 1. This is much needed functionality, but there are a lot of the so > let's > > > really think these protocols through. We really want to end up with a > set > > > of well thought-out, orthoganol apis. For this reason I think it is > > really > > > important to think through the end state even if that includes APIs we > > > won't implement in the first phase. > > > > > > A: Definitely behind this. Would appreciate if there are concrete > > comments > > > how this can be improved. > > > > > > 2. Let's please please please wait until we have switched the server > over > > > to the new java protocol definitions. If we add upteen more ad hoc > scala > > > objects that is just generating more work for the conversion we know we > > > have to do. > > > > > > A: Fixed in the latest patch - removed scala protocol classes. > > > > > > 3. This proposal introduces a new type of optional parameter. This is > > > inconsistent with everything else in the protocol where we use -1 or > some > > > other marker value. You could argue either way but let's stick with > that > > > for consistency. For clients that implemented the protocol in a better > > way > > > than our scala code these basic primitives are hard to change. > > > > > > A: Fixed in the latest patch - removed MaybeOf type and changed > protocol > > > accordingly. > > > > > > 4. ClusterMetadata: This seems to duplicate TopicMetadataRequest which > > has > > > brokers, topics, and partitions. I think we should rename that request > > > ClusterMetadataRequest (or just MetadataRequest) and include the id of > > the > > > controller. Or are there other things we could add here? > > > > > > A: I agree. Updated the KIP. Let's extends TopicMetadata to version 2 > and > > > include controller. > > > > > > 5. We have a tendency to try to make a lot of requests that can only go > > to > > > particular nodes. This adds a lot of burden for client implementations > > (it > > > sounds easy but each discovery can fail in many parts so it ends up > > being a > > > full state machine to do right). I think we should consider making > admin > > > commands and ideally as many of the other apis as possible available on > > all > > > brokers and just redirect to the controller on the broker side. Perhaps > > > there would be a general way to encapsulate this re-routing behavior. > > > > > > A: It's a very interesting idea, but seems there are some concerns > about > > > this > > > feature (like performance considerations, how this will complicate > server > > > etc). > > > I believe this shouldn't be a blocker. If this feature is implemented > at > > > some > > > point it won't affect Admin changes - at least no changes to public API > > > will be required. > > > > > > 6. We should probably normalize the key value pairs used for configs > > rather > > > than embedding a new formatting. So two strings rather than one with an > > > internal equals sign. > > > > > > A: Fixed in the latest patch - normalized configs and changed protocol > > > accordingly. > > > > > > 7. Is the postcondition of these APIs that the command has begun or > that > > > the command has been completed? It is a lot more usable if the command > > has > > > been completed so you know that if you create a topic and then publish > to > > > it you won't get an exception about there being no such topic. > > > > > > A: For long running requests (like reassign partitions) - the post > > > condition is > > > command has begun - so we don't block the client. In case of your > > example - > > > topic commands, this will be refactored and topic commands will be > > executed > > > immediately, since the Controller will serve Admin requests > > > (follow-up ticket KAFKA-1777). > > > > > > 8. Describe topic and list topics duplicate a lot of stuff in the > > metadata > > > request. Is there a reason to give back topics marked for deletion? I > > feel > > > like if we just make the post-condition of the delete command be that > the > > > topic is deleted that will get rid of the need for this right? And it > > will > > > be much more intuitive. > > > > > > A: Fixed in the latest patch - removed topics marked for deletion in > > > ListTopicsRequest. > > > > > > 9. Should we consider batching these requests? We have generally tried > to > > > allow multiple operations to be batched. My suspicion is that without > > this > > > we will get a lot of code that does something like > > > for(topic: adminClient.listTopics()) > > > adminClient.describeTopic(topic) > > > this code will work great when you test on 5 topics but not do as well > if > > > you have 50k. > > > > > > A: Updated the KIP - please check "Topic Admin Schema" section. > > > > > > 10. I think we should also discuss how we want to expose a programmatic > > JVM > > > client api for these operations. Currently people rely on AdminUtils > > which > > > is totally sketchy. I think we probably need another client under > > clients/ > > > that exposes administrative functionality. We will need this just to > > > properly test the new apis, I suspect. We should figure out that API. > > > > > > A: Updated the KIP - please check "Admin Client" section with an > initial > > > API proposal. > > > > > > 11. The other information that would be really useful to get would be > > > information about partitions--how much data is in the partition, what > are > > > the segment offsets, what is the log-end offset (i.e. last offset), > what > > is > > > the compaction point, etc. I think that done right this would be the > > > successor to the very awkward OffsetRequest we have today. > > > > > > A: I removed ConsumerGroupOffsetsRequest in the latest patch. I believe > > > this should > > > be resolved in a separate KIP / jira ticket. > > > > > > 12. Generally we can do good error handling without needing custom > > > server-side > > > messages. I.e. generally the client has the context to know that if it > > got > > > an error that the topic doesn't exist to say "Topic X doesn't exist" > > rather > > > than "error code 14" (or whatever). Maybe there are specific cases > where > > > this is hard? If we want to add server-side error messages we really do > > > need to do this in a consistent way across the protocol. > > > > > > A: Updated the KIP - please check "Protocol Errors" section. I added > the > > > comprehensive, fine-grained list of error codes. > > > > > > Comments from Guozhang: > > > 13. Describe topic request: it would be great to go beyond just > batching > > on > > > topic name regex for this request. For example, a very common use case > of > > > the topic command is to list all topics whose config A's value is B. > With > > > topic name regex then we have to first retrieve __all__ topics's > > > description info and then filter at the client end, which will be a > huge > > > burden on ZK. > > > AND > > > 14. Config K-Vs in create topic: this is related to the previous point; > > > maybe we can add another metadata K-V or just a metadata string along > > side > > > with config K-V in create topic like we did for offset commit request. > > This > > > field can be quite useful in storing information like "owner" of the > > topic > > > who issue the create command, etc, which is quite important for a > > > multi-tenant setting. Then in the describe topic request we can also > > batch > > > on regex of the metadata field. > > > > > > A: As discussed it is very interesting but can be implemented later > after > > > we have some basic functionality there. > > > > > > 15. Today all the admin operations are async in the sense that command > > will > > > return once it is written in ZK, and that is why we need extra > > verification > > > like testUtil.waitForTopicCreated() / verify partition reassignment > > > request, etc. With admin requests we could add a flag to enable / > disable > > > synchronous requests; when it is turned on, the response will not > return > > > until the request has been completed. And for async requests we can > add a > > > "token" field in the response, and then only need a general "admin > > > verification request" with the given token to check if the async > request > > > has been completed. > > > > > > A: I see your point. My idea was to provide specific Verify...Request > per > > > each > > > long running request, where needed. We can do it the way you suggest. > The > > > only > > > concern is that introducing a token we again will make schema > "dynamic". > > We > > > wanted > > > to do similar thing introducing single AdminRequest for all topic > > commands > > > but rejected > > > this idea because we wanted to have schema defined. So this is more a > > > choice between: > > > a) have fixed schema but introduce each time new Verify...Request for > > > long-running requests > > > b) use one request for verification but generalize it with token > > > I'm fine with whatever decision community come to. Just let me know > your > > > thoughts. > > > > > > Comment from Gwen: > > > 16. Specifically for ownership, I think the plan is to add ACL (it > sounds > > > like you are describing ACL) via an external system (Argus, Sentry). > > > I remember KIP-11 described this, but I can't find the KIP any longer. > > > > > > A: Okay, no problem. Not sure though how we are going to handle it. > Wait > > > which KIP > > > will be committed first and include changes to TopicMetadata from the > > later > > > one? > > > Anyway, I added this note to "Open Questions" section so we don't miss > > this > > > piece. > > > > > > Thanks, > > > Andrii Biletskyi > > > > > > On Fri, Mar 13, 2015 at 12:34 AM, Andrii Biletskyi < > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > Hi all, > > > > > > > > Today I uploaded the patch that covers some of the discussed and > agreed > > > > items: > > > > - removed MaybeOf optional type > > > > - switched to java protocol definitions > > > > - simplified messages (normalized configs, removed topic marked for > > > > deletion) > > > > > > > > I also updated the KIP-4 with respective changes and wrote down my > > > > proposal for > > > > pending items: > > > > - Batch Admin Operations -> updated Wire Protocol schema proposal > > > > - Remove ClusterMetadata -> changed to extend TopicMetadataRequest > > > > - Admin Client -> updated my initial proposal to reflect batching > > > > - Error codes -> proposed fine-grained error code instead of > > > > AdminRequestFailed > > > > > > > > I will also send a separate email to cover all comments from this > > thread. > > > > > > > > Thanks, > > > > Andrii Biletskyi > > > > > > > > > > > > On Thu, Mar 12, 2015 at 9:26 PM, Gwen Shapira <gshap...@cloudera.com > > > > > > wrote: > > > > > > > >> Found KIP-11 ( > > > >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface > > > >> ) > > > >> It actually specifies changes to the Metadata protocol, so making > sure > > > >> both KIPs are consistent in this regard will be good. > > > >> > > > >> On Thu, Mar 12, 2015 at 12:21 PM, Gwen Shapira < > gshap...@cloudera.com > > > > > > >> wrote: > > > >> > Specifically for ownership, I think the plan is to add ACL (it > > sounds > > > >> > like you are describing ACL) via an external system (Argus, > Sentry). > > > >> > I remember KIP-11 described this, but I can't find the KIP any > > longer. > > > >> > > > > >> > Regardless, I think KIP-4 focuses on getting information that > > already > > > >> > exists from Kafka brokers, not on adding information that perhaps > > > >> > should exist but doesn't yet? > > > >> > > > > >> > Gwen > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > On Thu, Mar 12, 2015 at 6:37 AM, Guozhang Wang < > wangg...@gmail.com> > > > >> wrote: > > > >> >> Folks, > > > >> >> > > > >> >> Just want to elaborate a bit more on the create-topic metadata > and > > > >> batching > > > >> >> describe-topic based on config / metadata in my previous email as > > we > > > >> work > > > >> >> on KAFKA-1694. The main motivation is to have some sort of topic > > > >> management > > > >> >> mechanisms, which I think is quite important in a multi-tenant / > > > cloud > > > >> >> architecture: today anyone can create topics in a shared Kafka > > > >> cluster, but > > > >> >> there is no concept or "ownership" of topics that are created by > > > >> different > > > >> >> users. For example, at LinkedIn we basically distinguish topic > > owners > > > >> via > > > >> >> some casual topic name prefix, which is a bit awkward and does > not > > > fly > > > >> as > > > >> >> we scale our customers. It would be great to use describe-topics > > such > > > >> as: > > > >> >> > > > >> >> Describe all topics that is created by me. > > > >> >> > > > >> >> Describe all topics whose retention time is overriden to X. > > > >> >> > > > >> >> Describe all topics whose writable group include user Y (this is > > > >> related to > > > >> >> authorization), etc.. > > > >> >> > > > >> >> One possible way to achieve this is to add a metadata file in the > > > >> >> create-topic request, whose value will also be written ZK as we > > > create > > > >> the > > > >> >> topic; then describe-topics can choose to batch topics based on > 1) > > > name > > > >> >> regex, 2) config K-V matching, 3) metadata regex, etc. > > > >> >> > > > >> >> Thoughts? > > > >> >> > > > >> >> Guozhang > > > >> >> > > > >> >> On Thu, Mar 5, 2015 at 4:37 PM, Guozhang Wang < > wangg...@gmail.com> > > > >> wrote: > > > >> >> > > > >> >>> Thanks for the updated wiki. A few comments below: > > > >> >>> > > > >> >>> 1. Error description in response: I think if some errorCode > could > > > >> indicate > > > >> >>> several different error cases then we should really change it to > > > >> multiple > > > >> >>> codes. In general the errorCode itself would be precise and > > > >> sufficient for > > > >> >>> describing the server side errors. > > > >> >>> > > > >> >>> 2. Describe topic request: it would be great to go beyond just > > > >> batching on > > > >> >>> topic name regex for this request. For example, a very common > use > > > >> case of > > > >> >>> the topic command is to list all topics whose config A's value > is > > B. > > > >> With > > > >> >>> topic name regex then we have to first retrieve __all__ topics's > > > >> >>> description info and then filter at the client end, which will > be > > a > > > >> huge > > > >> >>> burden on ZK. > > > >> >>> > > > >> >>> 3. Config K-Vs in create topic: this is related to the previous > > > point; > > > >> >>> maybe we can add another metadata K-V or just a metadata string > > > along > > > >> side > > > >> >>> with config K-V in create topic like we did for offset commit > > > >> request. This > > > >> >>> field can be quite useful in storing information like "owner" of > > the > > > >> topic > > > >> >>> who issue the create command, etc, which is quite important for > a > > > >> >>> multi-tenant setting. Then in the describe topic request we can > > also > > > >> batch > > > >> >>> on regex of the metadata field. > > > >> >>> > > > >> >>> 4. Today all the admin operations are async in the sense that > > > command > > > >> will > > > >> >>> return once it is written in ZK, and that is why we need extra > > > >> verification > > > >> >>> like testUtil.waitForTopicCreated() / verify partition > > reassignment > > > >> >>> request, etc. With admin requests we could add a flag to enable > / > > > >> disable > > > >> >>> synchronous requests; when it is turned on, the response will > not > > > >> return > > > >> >>> until the request has been completed. And for async requests we > > can > > > >> add a > > > >> >>> "token" field in the response, and then only need a general > "admin > > > >> >>> verification request" with the given token to check if the async > > > >> request > > > >> >>> has been completed. > > > >> >>> > > > >> >>> 5. +1 for extending Metadata request to include controller / > > > >> coordinator > > > >> >>> information, and then we can remove the ConsumerMetadata / > > > >> ClusterMetadata > > > >> >>> requests. > > > >> >>> > > > >> >>> Guozhang > > > >> >>> > > > >> >>> On Tue, Mar 3, 2015 at 10:23 AM, Joel Koshy < > jjkosh...@gmail.com> > > > >> wrote: > > > >> >>> > > > >> >>>> Thanks for sending that out Joe - I don't think I will be able > to > > > >> make > > > >> >>>> it today, so if notes can be sent out afterward that would be > > > great. > > > >> >>>> > > > >> >>>> On Mon, Mar 02, 2015 at 09:16:13AM -0800, Gwen Shapira wrote: > > > >> >>>> > Thanks for sending this out Joe. Looking forward to chatting > > with > > > >> >>>> everyone :) > > > >> >>>> > > > > >> >>>> > On Mon, Mar 2, 2015 at 6:46 AM, Joe Stein < > > joe.st...@stealth.ly> > > > >> wrote: > > > >> >>>> > > Hey, I just sent out a google hangout invite to all pmc, > > > >> committers > > > >> >>>> and > > > >> >>>> > > everyone I found working on a KIP. If I missed anyone in > the > > > >> invite > > > >> >>>> please > > > >> >>>> > > let me know and can update it, np. > > > >> >>>> > > > > > >> >>>> > > We should do this every Tuesday @ 2pm Eastern Time. Maybe > we > > > can > > > >> get > > > >> >>>> INFRA > > > >> >>>> > > help to make a google account so we can manage better? > > > >> >>>> > > > > > >> >>>> > > To discuss > > > >> >>>> > > > > > >> >>>> > > > >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > >> >>>> > > in progress and related JIRA that are interdependent and > > common > > > >> work. > > > >> >>>> > > > > > >> >>>> > > ~ Joe Stein > > > >> >>>> > > > > > >> >>>> > > On Tue, Feb 24, 2015 at 2:59 PM, Jay Kreps < > > > jay.kr...@gmail.com> > > > >> >>>> wrote: > > > >> >>>> > > > > > >> >>>> > >> Let's stay on Google hangouts that will also record and > make > > > the > > > >> >>>> sessions > > > >> >>>> > >> available on youtube. > > > >> >>>> > >> > > > >> >>>> > >> -Jay > > > >> >>>> > >> > > > >> >>>> > >> On Tue, Feb 24, 2015 at 11:49 AM, Jeff Holoman < > > > >> >>>> jholo...@cloudera.com> > > > >> >>>> > >> wrote: > > > >> >>>> > >> > > > >> >>>> > >> > Jay / Joe > > > >> >>>> > >> > > > > >> >>>> > >> > We're happy to send out a Webex for this purpose. We > could > > > >> record > > > >> >>>> the > > > >> >>>> > >> > sessions if there is interest and publish them out. > > > >> >>>> > >> > > > > >> >>>> > >> > Thanks > > > >> >>>> > >> > > > > >> >>>> > >> > Jeff > > > >> >>>> > >> > > > > >> >>>> > >> > On Tue, Feb 24, 2015 at 11:28 AM, Jay Kreps < > > > >> jay.kr...@gmail.com> > > > >> >>>> wrote: > > > >> >>>> > >> > > > > >> >>>> > >> > > Let's try to get the technical hang-ups sorted out, > > > though. > > > >> I > > > >> >>>> really > > > >> >>>> > >> > think > > > >> >>>> > >> > > there is some benefit to live discussion vs writing. I > > am > > > >> >>>> hopeful that > > > >> >>>> > >> if > > > >> >>>> > >> > > we post instructions and give ourselves a few attempts > > we > > > >> can > > > >> >>>> get it > > > >> >>>> > >> > > working. > > > >> >>>> > >> > > > > > >> >>>> > >> > > Tuesday at that time would work for me...any > objections? > > > >> >>>> > >> > > > > > >> >>>> > >> > > -Jay > > > >> >>>> > >> > > > > > >> >>>> > >> > > On Tue, Feb 24, 2015 at 8:18 AM, Joe Stein < > > > >> joe.st...@stealth.ly > > > >> >>>> > > > > >> >>>> > >> wrote: > > > >> >>>> > >> > > > > > >> >>>> > >> > > > Weekly would be great maybe like every Tuesday ~ 1pm > > ET > > > / > > > >> 10am > > > >> >>>> PT > > > >> >>>> > >> ???? > > > >> >>>> > >> > > > > > > >> >>>> > >> > > > I don't mind google hangout but there is always some > > > >> issue or > > > >> >>>> > >> whatever > > > >> >>>> > >> > so > > > >> >>>> > >> > > > we know the apache irc channel works. We can start > > there > > > >> and > > > >> >>>> see how > > > >> >>>> > >> it > > > >> >>>> > >> > > > goes? We can pull transcripts too and associate to > > > >> tickets if > > > >> >>>> need be > > > >> >>>> > >> > > makes > > > >> >>>> > >> > > > it helpful for things. > > > >> >>>> > >> > > > > > > >> >>>> > >> > > > ~ Joestein > > > >> >>>> > >> > > > > > > >> >>>> > >> > > > On Tue, Feb 24, 2015 at 11:10 AM, Jay Kreps < > > > >> >>>> jay.kr...@gmail.com> > > > >> >>>> > >> > wrote: > > > >> >>>> > >> > > > > > > >> >>>> > >> > > > > We'd talked about doing a Google Hangout to chat > > about > > > >> this. > > > >> >>>> What > > > >> >>>> > >> > about > > > >> >>>> > >> > > > > generalizing that a little further...I actually > > think > > > it > > > >> >>>> would be > > > >> >>>> > >> > good > > > >> >>>> > >> > > > for > > > >> >>>> > >> > > > > everyone spending a reasonable chunk of their week > > on > > > >> Kafka > > > >> >>>> stuff > > > >> >>>> > >> to > > > >> >>>> > >> > > > maybe > > > >> >>>> > >> > > > > sync up once a week. I think we could use time to > > talk > > > >> >>>> through > > > >> >>>> > >> design > > > >> >>>> > >> > > > > stuff, make sure we are on top of code reviews, > talk > > > >> through > > > >> >>>> any > > > >> >>>> > >> > tricky > > > >> >>>> > >> > > > > issues, etc. > > > >> >>>> > >> > > > > > > > >> >>>> > >> > > > > We can make it publicly available so that any one > > can > > > >> follow > > > >> >>>> along > > > >> >>>> > >> > who > > > >> >>>> > >> > > > > likes. > > > >> >>>> > >> > > > > > > > >> >>>> > >> > > > > Any interest in doing this? If so I'll try to set > it > > > up > > > >> >>>> starting > > > >> >>>> > >> next > > > >> >>>> > >> > > > week. > > > >> >>>> > >> > > > > > > > >> >>>> > >> > > > > -Jay > > > >> >>>> > >> > > > > > > > >> >>>> > >> > > > > On Tue, Feb 24, 2015 at 3:57 AM, Andrii Biletskyi > < > > > >> >>>> > >> > > > > andrii.bilets...@stealth.ly> wrote: > > > >> >>>> > >> > > > > > > > >> >>>> > >> > > > > > Hi all, > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > > > > > I've updated KIP page, fixed / aligned document > > > >> structure. > > > >> >>>> Also I > > > >> >>>> > >> > > added > > > >> >>>> > >> > > > > > some > > > >> >>>> > >> > > > > > very initial proposal for AdminClient so we have > > > >> something > > > >> >>>> to > > > >> >>>> > >> start > > > >> >>>> > >> > > > from > > > >> >>>> > >> > > > > > while > > > >> >>>> > >> > > > > > discussing the KIP. > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > > > > > > > >> >>>> > >> > > > > > > >> >>>> > >> > > > > > >> >>>> > >> > > > > >> >>>> > >> > > > >> >>>> > > > >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > > > > > Thanks, > > > >> >>>> > >> > > > > > Andrii Biletskyi > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > > > > > On Wed, Feb 18, 2015 at 9:01 PM, Andrii > Biletskyi > > < > > > >> >>>> > >> > > > > > andrii.bilets...@stealth.ly> wrote: > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > > > > > > Jay, > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > > > > > Re error messages: you are right, in most > cases > > > >> client > > > >> >>>> will > > > >> >>>> > >> have > > > >> >>>> > >> > > > enough > > > >> >>>> > >> > > > > > > context to show descriptive error message. My > > > >> concern is > > > >> >>>> that > > > >> >>>> > >> we > > > >> >>>> > >> > > will > > > >> >>>> > >> > > > > > have > > > >> >>>> > >> > > > > > > to > > > >> >>>> > >> > > > > > > add lots of new error codes for each possible > > > >> error. Of > > > >> >>>> course, > > > >> >>>> > >> > we > > > >> >>>> > >> > > > > could > > > >> >>>> > >> > > > > > > reuse > > > >> >>>> > >> > > > > > > some of existing like > > UknownTopicOrPartitionCode, > > > >> but we > > > >> >>>> will > > > >> >>>> > >> > also > > > >> >>>> > >> > > > need > > > >> >>>> > >> > > > > > to > > > >> >>>> > >> > > > > > > add smth like: TopicAlreadyExistsCode, > > > >> >>>> TopicConfigInvalid (both > > > >> >>>> > >> > for > > > >> >>>> > >> > > > > topic > > > >> >>>> > >> > > > > > > name and config, and probably user would like > to > > > >> know > > > >> >>>> what > > > >> >>>> > >> > exactly > > > >> >>>> > >> > > > > > > is wrong in his config), > > InvalidReplicaAssignment, > > > >> >>>> > >> InternalError > > > >> >>>> > >> > > > (e.g. > > > >> >>>> > >> > > > > > > zookeeper failure) etc. > > > >> >>>> > >> > > > > > > And this is only for TopicCommand, we will > also > > > >> need to > > > >> >>>> add > > > >> >>>> > >> > similar > > > >> >>>> > >> > > > > stuff > > > >> >>>> > >> > > > > > > for > > > >> >>>> > >> > > > > > > ReassignPartitions, PreferredReplica. So we'll > > end > > > >> up > > > >> >>>> with a > > > >> >>>> > >> > large > > > >> >>>> > >> > > > list > > > >> >>>> > >> > > > > > of > > > >> >>>> > >> > > > > > > error codes, used only in Admin protocol. > > > >> >>>> > >> > > > > > > Having said that, I agree my proposal is not > > > >> consistent > > > >> >>>> with > > > >> >>>> > >> > other > > > >> >>>> > >> > > > > cases. > > > >> >>>> > >> > > > > > > Maybe we can find better solution or something > > > >> >>>> in-between. > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > > > > > Re Hangout chat: I think it is a great idea. > > This > > > >> way we > > > >> >>>> can > > > >> >>>> > >> move > > > >> >>>> > >> > > on > > > >> >>>> > >> > > > > > > faster. > > > >> >>>> > >> > > > > > > Let's agree somehow on date/time so people can > > > join. > > > >> >>>> Will work > > > >> >>>> > >> > for > > > >> >>>> > >> > > me > > > >> >>>> > >> > > > > > this > > > >> >>>> > >> > > > > > > and > > > >> >>>> > >> > > > > > > next week almost anytime if agreed in advance. > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > > > > > Thanks, > > > >> >>>> > >> > > > > > > Andrii > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > > > > > On Wed, Feb 18, 2015 at 7:09 PM, Jay Kreps < > > > >> >>>> > >> jay.kr...@gmail.com> > > > >> >>>> > >> > > > > wrote: > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > > > > >> Hey Andrii, > > > >> >>>> > >> > > > > > >> > > > >> >>>> > >> > > > > > >> Generally we can do good error handling > without > > > >> needing > > > >> >>>> custom > > > >> >>>> > >> > > > > > server-side > > > >> >>>> > >> > > > > > >> messages. I.e. generally the client has the > > > >> context to > > > >> >>>> know > > > >> >>>> > >> that > > > >> >>>> > >> > > if > > > >> >>>> > >> > > > it > > > >> >>>> > >> > > > > > got > > > >> >>>> > >> > > > > > >> an error that the topic doesn't exist to say > > > >> "Topic X > > > >> >>>> doesn't > > > >> >>>> > >> > > exist" > > > >> >>>> > >> > > > > > >> rather > > > >> >>>> > >> > > > > > >> than "error code 14" (or whatever). Maybe > there > > > are > > > >> >>>> specific > > > >> >>>> > >> > cases > > > >> >>>> > >> > > > > where > > > >> >>>> > >> > > > > > >> this is hard? If we want to add server-side > > error > > > >> >>>> messages we > > > >> >>>> > >> > > really > > > >> >>>> > >> > > > > do > > > >> >>>> > >> > > > > > >> need to do this in a consistent way across > the > > > >> protocol. > > > >> >>>> > >> > > > > > >> > > > >> >>>> > >> > > > > > >> I still have a bunch of open questions here > > from > > > my > > > >> >>>> previous > > > >> >>>> > >> > > list. I > > > >> >>>> > >> > > > > > will > > > >> >>>> > >> > > > > > >> be out for the next few days for Strata > though. > > > >> Maybe > > > >> >>>> we could > > > >> >>>> > >> > do > > > >> >>>> > >> > > a > > > >> >>>> > >> > > > > > Google > > > >> >>>> > >> > > > > > >> Hangout chat on any open issues some time > > towards > > > >> the > > > >> >>>> end of > > > >> >>>> > >> > next > > > >> >>>> > >> > > > week > > > >> >>>> > >> > > > > > for > > > >> >>>> > >> > > > > > >> anyone interested in this ticket? I have a > > > feeling > > > >> that > > > >> >>>> might > > > >> >>>> > >> > > > progress > > > >> >>>> > >> > > > > > >> things a little faster than email--I think we > > > >> could talk > > > >> >>>> > >> through > > > >> >>>> > >> > > > those > > > >> >>>> > >> > > > > > >> issues I brought up fairly quickly... > > > >> >>>> > >> > > > > > >> > > > >> >>>> > >> > > > > > >> -Jay > > > >> >>>> > >> > > > > > >> > > > >> >>>> > >> > > > > > >> On Wed, Feb 18, 2015 at 7:27 AM, Andrii > > > Biletskyi < > > > >> >>>> > >> > > > > > >> andrii.bilets...@stealth.ly> wrote: > > > >> >>>> > >> > > > > > >> > > > >> >>>> > >> > > > > > >> > Hi all, > > > >> >>>> > >> > > > > > >> > > > > >> >>>> > >> > > > > > >> > I'm trying to address some of the issues > > which > > > >> were > > > >> >>>> > >> mentioned > > > >> >>>> > >> > > > > earlier > > > >> >>>> > >> > > > > > >> about > > > >> >>>> > >> > > > > > >> > Admin RQ/RP format. One of those was about > > > >> batching > > > >> >>>> > >> > operations. > > > >> >>>> > >> > > > What > > > >> >>>> > >> > > > > > if > > > >> >>>> > >> > > > > > >> we > > > >> >>>> > >> > > > > > >> > follow TopicCommand approach and let people > > > >> specify > > > >> >>>> > >> topic-name > > > >> >>>> > >> > > by > > > >> >>>> > >> > > > > > >> regexp - > > > >> >>>> > >> > > > > > >> > would that cover most of the use cases? > > > >> >>>> > >> > > > > > >> > > > > >> >>>> > >> > > > > > >> > Secondly, is what information should we > > > generally > > > >> >>>> provide in > > > >> >>>> > >> > > Admin > > > >> >>>> > >> > > > > > >> > responses. > > > >> >>>> > >> > > > > > >> > I realize that Admin commands don't imply > > they > > > >> will > > > >> >>>> be used > > > >> >>>> > >> > only > > > >> >>>> > >> > > > in > > > >> >>>> > >> > > > > > CLI > > > >> >>>> > >> > > > > > >> > but, > > > >> >>>> > >> > > > > > >> > it seems to me, CLI is a very important > > client > > > >> of this > > > >> >>>> > >> > feature. > > > >> >>>> > >> > > In > > > >> >>>> > >> > > > > > this > > > >> >>>> > >> > > > > > >> > case, > > > >> >>>> > >> > > > > > >> > seems logical, we would like to provide > users > > > >> with > > > >> >>>> rich > > > >> >>>> > >> > > experience > > > >> >>>> > >> > > > > in > > > >> >>>> > >> > > > > > >> terms > > > >> >>>> > >> > > > > > >> > of > > > >> >>>> > >> > > > > > >> > getting results / errors of the executed > > > >> commands. > > > >> >>>> Usually > > > >> >>>> > >> we > > > >> >>>> > >> > > > supply > > > >> >>>> > >> > > > > > >> with > > > >> >>>> > >> > > > > > >> > responses only errorCode, which looks very > > > >> limiting, > > > >> >>>> in case > > > >> >>>> > >> > of > > > >> >>>> > >> > > > CLI > > > >> >>>> > >> > > > > we > > > >> >>>> > >> > > > > > >> may > > > >> >>>> > >> > > > > > >> > want to print human readable error > > description. > > > >> >>>> > >> > > > > > >> > > > > >> >>>> > >> > > > > > >> > So, taking into account previous item about > > > >> batching, > > > >> >>>> what > > > >> >>>> > >> do > > > >> >>>> > >> > > you > > > >> >>>> > >> > > > > > think > > > >> >>>> > >> > > > > > >> > about > > > >> >>>> > >> > > > > > >> > having smth like: > > > >> >>>> > >> > > > > > >> > > > > >> >>>> > >> > > > > > >> > ('create' doesn't support regexp) > > > >> >>>> > >> > > > > > >> > CreateTopicRequest => TopicName Partitions > > > >> Replicas > > > >> >>>> > >> > > > > ReplicaAssignment > > > >> >>>> > >> > > > > > >> > [Config] > > > >> >>>> > >> > > > > > >> > CreateTopicResponse => ErrorCode > > > ErrorDescription > > > >> >>>> > >> > > > > > >> > ErrorCode => int16 > > > >> >>>> > >> > > > > > >> > ErrorDescription => string (empty if > > > >> successful) > > > >> >>>> > >> > > > > > >> > > > > >> >>>> > >> > > > > > >> > AlterTopicRequest -> TopicNameRegexp > > Partitions > > > >> >>>> > >> > > ReplicaAssignment > > > >> >>>> > >> > > > > > >> > [AddedConfig] [DeletedConfig] > > > >> >>>> > >> > > > > > >> > AlterTopicResponse -> [TopicName ErrorCode > > > >> >>>> ErrorDescription] > > > >> >>>> > >> > > > > > >> > CommandErrorCode CommandErrorDescription > > > >> >>>> > >> > > > > > >> > CommandErrorCode => int16 > > > >> >>>> > >> > > > > > >> > CommandErrorDescription => string > (nonempty > > > in > > > >> case > > > >> >>>> of > > > >> >>>> > >> fatal > > > >> >>>> > >> > > > > error, > > > >> >>>> > >> > > > > > >> e.g. > > > >> >>>> > >> > > > > > >> > we couldn't get topics by regexp) > > > >> >>>> > >> > > > > > >> > > > > >> >>>> > >> > > > > > >> > DescribeTopicRequest -> TopicNameRegexp > > > >> >>>> > >> > > > > > >> > DescribeTopicResponse -> [TopicName > > > >> TopicDescription > > > >> >>>> > >> ErrorCode > > > >> >>>> > >> > > > > > >> > ErrorDescription] CommandErrorCode > > > >> >>>> CommandErrorDescription > > > >> >>>> > >> > > > > > >> > > > > >> >>>> > >> > > > > > >> > Also, any thoughts about our discussion > > > regarding > > > >> >>>> re-routing > > > >> >>>> > >> > > > > facility? > > > >> >>>> > >> > > > > > >> In > > > >> >>>> > >> > > > > > >> > my > > > >> >>>> > >> > > > > > >> > understanding, it is like between > augmenting > > > >> >>>> > >> > > TopicMetadataRequest > > > >> >>>> > >> > > > > > >> > (to include at least controllerId) and > > > >> implementing > > > >> >>>> new > > > >> >>>> > >> > generic > > > >> >>>> > >> > > > > > >> re-routing > > > >> >>>> > >> > > > > > >> > facility so sending messages to controller > > will > > > >> be > > > >> >>>> handled > > > >> >>>> > >> by > > > >> >>>> > >> > > it. > > > >> >>>> > >> > > > > > >> > > > > >> >>>> > >> > > > > > >> > Thanks, > > > >> >>>> > >> > > > > > >> > Andrii Biletskyi > > > >> >>>> > >> > > > > > >> > > > > >> >>>> > >> > > > > > >> > On Mon, Feb 16, 2015 at 5:26 PM, Andrii > > > >> Biletskyi < > > > >> >>>> > >> > > > > > >> > andrii.bilets...@stealth.ly> wrote: > > > >> >>>> > >> > > > > > >> > > > > >> >>>> > >> > > > > > >> > > @Guozhang: > > > >> >>>> > >> > > > > > >> > > Thanks for your comments, I've answered > > some > > > of > > > >> >>>> those. The > > > >> >>>> > >> > > main > > > >> >>>> > >> > > > > > thing > > > >> >>>> > >> > > > > > >> is > > > >> >>>> > >> > > > > > >> > > having merged request for > > > >> >>>> create-alter-delete-describe - I > > > >> >>>> > >> > > have > > > >> >>>> > >> > > > > some > > > >> >>>> > >> > > > > > >> > > concerns about this approach. > > > >> >>>> > >> > > > > > >> > > > > > >> >>>> > >> > > > > > >> > > @*Jay*: > > > >> >>>> > >> > > > > > >> > > I see that introduced > ClusterMetadaRequest > > is > > > >> also > > > >> >>>> one of > > > >> >>>> > >> > the > > > >> >>>> > >> > > > > > >> concerns. > > > >> >>>> > >> > > > > > >> > We > > > >> >>>> > >> > > > > > >> > > can solve it if we implement re-routing > > > >> facility. > > > >> >>>> But I > > > >> >>>> > >> > agree > > > >> >>>> > >> > > > with > > > >> >>>> > >> > > > > > >> > > Guozhang - it will make clients' > internals > > a > > > >> little > > > >> >>>> bit > > > >> >>>> > >> > easier > > > >> >>>> > >> > > > but > > > >> >>>> > >> > > > > > >> this > > > >> >>>> > >> > > > > > >> > > seems to be a complex logic to implement > > and > > > >> >>>> support then. > > > >> >>>> > >> > > > > > Especially > > > >> >>>> > >> > > > > > >> for > > > >> >>>> > >> > > > > > >> > > Fetch and Produce (even if we add > > re-routing > > > >> later > > > >> >>>> for > > > >> >>>> > >> these > > > >> >>>> > >> > > > > > >> requests). > > > >> >>>> > >> > > > > > >> > > Also people will tend to avoid this > > > re-routing > > > >> >>>> facility > > > >> >>>> > >> and > > > >> >>>> > >> > > hold > > > >> >>>> > >> > > > > > local > > > >> >>>> > >> > > > > > >> > > cluster cache to ensure their > high-priority > > > >> requests > > > >> >>>> > >> (which > > > >> >>>> > >> > > some > > > >> >>>> > >> > > > > of > > > >> >>>> > >> > > > > > >> the > > > >> >>>> > >> > > > > > >> > > admin requests are) not sent to some busy > > > >> broker > > > >> >>>> where > > > >> >>>> > >> they > > > >> >>>> > >> > > wait > > > >> >>>> > >> > > > > to > > > >> >>>> > >> > > > > > be > > > >> >>>> > >> > > > > > >> > > routed to the correct one. > > > >> >>>> > >> > > > > > >> > > As pointed out by Jun here ( > > > >> >>>> > >> > > > > > >> > > > > > >> >>>> > >> > > > > > >> > > > > >> >>>> > >> > > > > > >> > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > > > > > > > >> >>>> > >> > > > > > > >> >>>> > >> > > > > > >> >>>> > >> > > > > >> >>>> > >> > > > >> >>>> > > > >> > > > > > > https://issues.apache.org/jira/browse/KAFKA-1772?focusedCommentId=14234530&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14234530 > > > >> >>>> > >> > > > > > >> > ) > > > >> >>>> > >> > > > > > >> > > to solve the issue we might introduce a > > > message > > > >> >>>> type to > > > >> >>>> > >> get > > > >> >>>> > >> > > > > cluster > > > >> >>>> > >> > > > > > >> > state. > > > >> >>>> > >> > > > > > >> > > But I agree we can just update > > > >> >>>> TopicMetadataResponse to > > > >> >>>> > >> > > include > > > >> >>>> > >> > > > > > >> > > controllerId (and probably smth else). > > > >> >>>> > >> > > > > > >> > > What are you thougths? > > > >> >>>> > >> > > > > > >> > > > > > >> >>>> > >> > > > > > >> > > Thanks, > > > >> >>>> > >> > > > > > >> > > Andrii > > > >> >>>> > >> > > > > > >> > > > > > >> >>>> > >> > > > > > >> > > On Thu, Feb 12, 2015 at 8:31 AM, Guozhang > > > Wang > > > >> < > > > >> >>>> > >> > > > > wangg...@gmail.com> > > > >> >>>> > >> > > > > > >> > wrote: > > > >> >>>> > >> > > > > > >> > > > > > >> >>>> > >> > > > > > >> > >> I think for the topics commands we can > > > >> actually > > > >> >>>> merge > > > >> >>>> > >> > > > > > >> > >> create/alter/delete/describe as one > > request > > > >> type > > > >> >>>> since > > > >> >>>> > >> > their > > > >> >>>> > >> > > > > > formats > > > >> >>>> > >> > > > > > >> are > > > >> >>>> > >> > > > > > >> > >> very much similar, and keep list-topics > > and > > > >> others > > > >> >>>> like > > > >> >>>> > >> > > > > > >> > >> partition-reassignment / > > > >> preferred-leader-election > > > >> >>>> as > > > >> >>>> > >> > > separate > > > >> >>>> > >> > > > > > >> request > > > >> >>>> > >> > > > > > >> > >> types, I also left some other comments > on > > > the > > > >> RB ( > > > >> >>>> > >> > > > > > >> > >> https://reviews.apache.org/r/29301/). > > > >> >>>> > >> > > > > > >> > >> > > > >> >>>> > >> > > > > > >> > >> On Wed, Feb 11, 2015 at 2:04 PM, Jay > > Kreps < > > > >> >>>> > >> > > > jay.kr...@gmail.com> > > > >> >>>> > >> > > > > > >> wrote: > > > >> >>>> > >> > > > > > >> > >> > > > >> >>>> > >> > > > > > >> > >> > Yeah I totally agree that we don't > want > > to > > > >> just > > > >> >>>> have > > > >> >>>> > >> one > > > >> >>>> > >> > > "do > > > >> >>>> > >> > > > > > admin > > > >> >>>> > >> > > > > > >> > >> stuff" > > > >> >>>> > >> > > > > > >> > >> > command that has the union of all > > > >> parameters. > > > >> >>>> > >> > > > > > >> > >> > > > > >> >>>> > >> > > > > > >> > >> > What I am saying is that command line > > > tools > > > >> are > > > >> >>>> one > > > >> >>>> > >> > client > > > >> >>>> > >> > > of > > > >> >>>> > >> > > > > the > > > >> >>>> > >> > > > > > >> > >> > administrative apis, but these will be > > > used > > > >> in a > > > >> >>>> number > > > >> >>>> > >> > of > > > >> >>>> > >> > > > > > >> scenarios > > > >> >>>> > >> > > > > > >> > so > > > >> >>>> > >> > > > > > >> > >> > they should make logical sense even in > > the > > > >> >>>> absence of > > > >> >>>> > >> the > > > >> >>>> > >> > > > > command > > > >> >>>> > >> > > > > > >> line > > > >> >>>> > >> > > > > > >> > >> > tool. Hence comments like trying to > > > clarify > > > >> the > > > >> >>>> > >> > > relationship > > > >> >>>> > >> > > > > > >> between > > > >> >>>> > >> > > > > > >> > >> > ClusterMetadata and > > TopicMetadata...these > > > >> kinds > > > >> >>>> of > > > >> >>>> > >> things > > > >> >>>> > >> > > > > really > > > >> >>>> > >> > > > > > >> need > > > >> >>>> > >> > > > > > >> > >> to be > > > >> >>>> > >> > > > > > >> > >> > thought through. > > > >> >>>> > >> > > > > > >> > >> > > > > >> >>>> > >> > > > > > >> > >> > Hope that makes sense. > > > >> >>>> > >> > > > > > >> > >> > > > > >> >>>> > >> > > > > > >> > >> > -Jay > > > >> >>>> > >> > > > > > >> > >> > > > > >> >>>> > >> > > > > > >> > >> > On Wed, Feb 11, 2015 at 1:41 PM, > Andrii > > > >> >>>> Biletskyi < > > > >> >>>> > >> > > > > > >> > >> > andrii.bilets...@stealth.ly> wrote: > > > >> >>>> > >> > > > > > >> > >> > > > > >> >>>> > >> > > > > > >> > >> > > Jay, > > > >> >>>> > >> > > > > > >> > >> > > > > > >> >>>> > >> > > > > > >> > >> > > Thanks for answering. You understood > > > >> >>>> correctly, most > > > >> >>>> > >> of > > > >> >>>> > >> > > my > > > >> >>>> > >> > > > > > >> comments > > > >> >>>> > >> > > > > > >> > >> were > > > >> >>>> > >> > > > > > >> > >> > > related to your point 1) - about > "well > > > >> >>>> thought-out" > > > >> >>>> > >> > apis. > > > >> >>>> > >> > > > > Also, > > > >> >>>> > >> > > > > > >> yes, > > > >> >>>> > >> > > > > > >> > >> as I > > > >> >>>> > >> > > > > > >> > >> > > understood we would like to > introduce > > a > > > >> single > > > >> >>>> > >> unified > > > >> >>>> > >> > > CLI > > > >> >>>> > >> > > > > tool > > > >> >>>> > >> > > > > > >> with > > > >> >>>> > >> > > > > > >> > >> > > centralized server-side request > > handling > > > >> for > > > >> >>>> lots of > > > >> >>>> > >> > > > existing > > > >> >>>> > >> > > > > > >> ones > > > >> >>>> > >> > > > > > >> > >> (incl. > > > >> >>>> > >> > > > > > >> > >> > > TopicCommand, CommitOffsetChecker, > > > >> >>>> > >> ReassignPartitions, > > > >> >>>> > >> > > smth > > > >> >>>> > >> > > > > > else > > > >> >>>> > >> > > > > > >> if > > > >> >>>> > >> > > > > > >> > >> added > > > >> >>>> > >> > > > > > >> > >> > > in future). In our previous > > discussion ( > > > >> >>>> > >> > > > > > >> > >> > > > > > >> >>>> https://issues.apache.org/jira/browse/KAFKA-1694) > > > >> >>>> > >> > people > > > >> >>>> > >> > > > > said > > > >> >>>> > >> > > > > > >> > they'd > > > >> >>>> > >> > > > > > >> > >> > > rather > > > >> >>>> > >> > > > > > >> > >> > > have a separate message for each > > > command, > > > >> so, > > > >> >>>> yes, > > > >> >>>> > >> this > > > >> >>>> > >> > > > way I > > > >> >>>> > >> > > > > > >> came > > > >> >>>> > >> > > > > > >> > to > > > >> >>>> > >> > > > > > >> > >> 1-1 > > > >> >>>> > >> > > > > > >> > >> > > mapping between commands in the tool > > and > > > >> >>>> protocol > > > >> >>>> > >> > > > additions. > > > >> >>>> > >> > > > > > But > > > >> >>>> > >> > > > > > >> I > > > >> >>>> > >> > > > > > >> > >> might > > > >> >>>> > >> > > > > > >> > >> > be > > > >> >>>> > >> > > > > > >> > >> > > wrong. > > > >> >>>> > >> > > > > > >> > >> > > At the end I just try to start > > > discussion > > > >> how > > > >> >>>> at > > > >> >>>> > >> least > > > >> >>>> > >> > > > > > generally > > > >> >>>> > >> > > > > > >> > this > > > >> >>>> > >> > > > > > >> > >> > > protocol should look like. > > > >> >>>> > >> > > > > > >> > >> > > > > > >> >>>> > >> > > > > > >> > >> > > Thanks, > > > >> >>>> > >> > > > > > >> > >> > > Andrii > > > >> >>>> > >> > > > > > >> > >> > > > > > >> >>>> > >> > > > > > >> > >> > > On Wed, Feb 11, 2015 at 11:10 PM, > Jay > > > >> Kreps < > > > >> >>>> > >> > > > > > jay.kr...@gmail.com > > > >> >>>> > >> > > > > > >> > > > > >> >>>> > >> > > > > > >> > >> wrote: > > > >> >>>> > >> > > > > > >> > >> > > > > > >> >>>> > >> > > > > > >> > >> > > > Hey Andrii, > > > >> >>>> > >> > > > > > >> > >> > > > > > > >> >>>> > >> > > > > > >> > >> > > > To answer your earlier question we > > > just > > > >> >>>> really > > > >> >>>> > >> can't > > > >> >>>> > >> > be > > > >> >>>> > >> > > > > > adding > > > >> >>>> > >> > > > > > >> any > > > >> >>>> > >> > > > > > >> > >> more > > > >> >>>> > >> > > > > > >> > >> > > > scala protocol objects. These > things > > > are > > > >> >>>> super hard > > > >> >>>> > >> > to > > > >> >>>> > >> > > > > > maintain > > > >> >>>> > >> > > > > > >> > >> because > > > >> >>>> > >> > > > > > >> > >> > > > they hand code the byte parsing > and > > > >> don't > > > >> >>>> have good > > > >> >>>> > >> > > > > > versioning > > > >> >>>> > >> > > > > > >> > >> support. > > > >> >>>> > >> > > > > > >> > >> > > > Since we are already planning on > > > >> converting > > > >> >>>> we > > > >> >>>> > >> > > definitely > > > >> >>>> > >> > > > > > don't > > > >> >>>> > >> > > > > > >> > >> want to > > > >> >>>> > >> > > > > > >> > >> > > add > > > >> >>>> > >> > > > > > >> > >> > > > a ton more of these--they are > total > > > tech > > > >> >>>> debt. > > > >> >>>> > >> > > > > > >> > >> > > > > > > >> >>>> > >> > > > > > >> > >> > > > What does it mean that the changes > > are > > > >> >>>> isolated > > > >> >>>> > >> from > > > >> >>>> > >> > > the > > > >> >>>> > >> > > > > > >> current > > > >> >>>> > >> > > > > > >> > >> code > > > >> >>>> > >> > > > > > >> > >> > > base? > > > >> >>>> > >> > > > > > >> > >> > > > > > > >> >>>> > >> > > > > > >> > >> > > > I actually didn't understand the > > > >> remaining > > > >> >>>> > >> comments, > > > >> >>>> > >> > > > which > > > >> >>>> > >> > > > > of > > > >> >>>> > >> > > > > > >> the > > > >> >>>> > >> > > > > > >> > >> > points > > > >> >>>> > >> > > > > > >> > >> > > > are you responding to? > > > >> >>>> > >> > > > > > >> > >> > > > > > > >> >>>> > >> > > > > > >> > >> > > > Maybe one sticking point here is > > that > > > it > > > >> >>>> seems like > > > >> >>>> > >> > you > > > >> >>>> > >> > > > > want > > > >> >>>> > >> > > > > > to > > > >> >>>> > >> > > > > > >> > make > > > >> >>>> > >> > > > > > >> > >> > some > > > >> >>>> > >> > > > > > >> > >> > > > kind of tool, and you have made a > > 1-1 > > > >> mapping > > > >> >>>> > >> between > > > >> >>>> > >> > > > > > commands > > > >> >>>> > >> > > > > > >> you > > > >> >>>> > >> > > > > > >> > >> > > imagine > > > >> >>>> > >> > > > > > >> > >> > > > in the tool and protocol > additions. > > I > > > >> want > > > >> >>>> to make > > > >> >>>> > >> > sure > > > >> >>>> > >> > > > we > > > >> >>>> > >> > > > > > >> don't > > > >> >>>> > >> > > > > > >> > do > > > >> >>>> > >> > > > > > >> > >> > that. > > > >> >>>> > >> > > > > > >> > >> > > > The protocol needs to be really > > really > > > >> well > > > >> >>>> thought > > > >> >>>> > >> > out > > > >> >>>> > >> > > > > > against > > > >> >>>> > >> > > > > > >> > many > > > >> >>>> > >> > > > > > >> > >> > use > > > >> >>>> > >> > > > > > >> > >> > > > cases so it should make perfect > > > logical > > > >> >>>> sense in > > > >> >>>> > >> the > > > >> >>>> > >> > > > > absence > > > >> >>>> > >> > > > > > of > > > >> >>>> > >> > > > > > >> > >> knowing > > > >> >>>> > >> > > > > > >> > >> > > the > > > >> >>>> > >> > > > > > >> > >> > > > command line tool, right? > > > >> >>>> > >> > > > > > >> > >> > > > > > > >> >>>> > >> > > > > > >> > >> > > > -Jay > > > >> >>>> > >> > > > > > >> > >> > > > > > > >> >>>> > >> > > > > > >> > >> > > > On Wed, Feb 11, 2015 at 11:57 AM, > > > Andrii > > > >> >>>> Biletskyi > > > >> >>>> > >> < > > > >> >>>> > >> > > > > > >> > >> > > > andrii.bilets...@stealth.ly> > wrote: > > > >> >>>> > >> > > > > > >> > >> > > > > > > >> >>>> > >> > > > > > >> > >> > > > > Hey Jay, > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > I would like to continue this > > > >> discussion > > > >> >>>> as it > > > >> >>>> > >> seem > > > >> >>>> > >> > > > there > > > >> >>>> > >> > > > > > is > > > >> >>>> > >> > > > > > >> no > > > >> >>>> > >> > > > > > >> > >> > > progress > > > >> >>>> > >> > > > > > >> > >> > > > > here. > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > First of all, could you please > > > explain > > > >> >>>> what did > > > >> >>>> > >> you > > > >> >>>> > >> > > > mean > > > >> >>>> > >> > > > > in > > > >> >>>> > >> > > > > > >> 2? > > > >> >>>> > >> > > > > > >> > How > > > >> >>>> > >> > > > > > >> > >> > > > exactly > > > >> >>>> > >> > > > > > >> > >> > > > > are we going to migrate to the > new > > > >> java > > > >> >>>> protocol > > > >> >>>> > >> > > > > > definitions. > > > >> >>>> > >> > > > > > >> > And > > > >> >>>> > >> > > > > > >> > >> why > > > >> >>>> > >> > > > > > >> > >> > > > it's > > > >> >>>> > >> > > > > > >> > >> > > > > a blocker for centralized CLI? > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > I agree with you, this feature > > > >> includes > > > >> >>>> lots of > > > >> >>>> > >> > > stuff, > > > >> >>>> > >> > > > > but > > > >> >>>> > >> > > > > > >> > >> thankfully > > > >> >>>> > >> > > > > > >> > >> > > > > almost all changes are isolated > > from > > > >> the > > > >> >>>> current > > > >> >>>> > >> > code > > > >> >>>> > >> > > > > base, > > > >> >>>> > >> > > > > > >> > >> > > > > so the main thing, I think, we > > need > > > to > > > >> >>>> agree is > > > >> >>>> > >> > RQ/RP > > > >> >>>> > >> > > > > > format. > > > >> >>>> > >> > > > > > >> > >> > > > > So how can we start discussion > > about > > > >> the > > > >> >>>> concrete > > > >> >>>> > >> > > > > messages > > > >> >>>> > >> > > > > > >> > format? > > > >> >>>> > >> > > > > > >> > >> > > > > Can we take ( > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > >> >>>> > >> > > > > > >> > >> > > > > >> >>>> > >> > > > > > >> > >> > > > >> >>>> > >> > > > > > >> > > > > >> >>>> > >> > > > > > >> > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > > > > > > > >> >>>> > >> > > > > > > >> >>>> > >> > > > > > >> >>>> > >> > > > > >> >>>> > >> > > > >> >>>> > > > >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-ProposedRQ/RPFormat > > > >> >>>> > >> > > > > > >> > >> > > > > ) > > > >> >>>> > >> > > > > > >> > >> > > > > as starting point? > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > We had some doubts earlier > whether > > > it > > > >> worth > > > >> >>>> > >> > > introducing > > > >> >>>> > >> > > > > one > > > >> >>>> > >> > > > > > >> > >> generic > > > >> >>>> > >> > > > > > >> > >> > > Admin > > > >> >>>> > >> > > > > > >> > >> > > > > Request for all commands ( > > > >> >>>> > >> > > > > > >> > >> > > > > > > >> >>>> https://issues.apache.org/jira/browse/KAFKA-1694 > > > >> >>>> > >> > > > > > >> > >> > > > > ) > > > >> >>>> > >> > > > > > >> > >> > > > > but then everybody agreed it > would > > > be > > > >> >>>> better to > > > >> >>>> > >> > have > > > >> >>>> > >> > > > > > separate > > > >> >>>> > >> > > > > > >> > >> message > > > >> >>>> > >> > > > > > >> > >> > > for > > > >> >>>> > >> > > > > > >> > >> > > > > each admin command. The Request > > part > > > >> is > > > >> >>>> really > > > >> >>>> > >> > > dictated > > > >> >>>> > >> > > > > > from > > > >> >>>> > >> > > > > > >> the > > > >> >>>> > >> > > > > > >> > >> > > command > > > >> >>>> > >> > > > > > >> > >> > > > > (e.g. TopicCommand) arguments > > > itself, > > > >> so > > > >> >>>> the > > > >> >>>> > >> > proposed > > > >> >>>> > >> > > > > > version > > > >> >>>> > >> > > > > > >> > >> should > > > >> >>>> > >> > > > > > >> > >> > be > > > >> >>>> > >> > > > > > >> > >> > > > > fine (let's put aside for now > > > remarks > > > >> about > > > >> >>>> > >> > Optional > > > >> >>>> > >> > > > > type, > > > >> >>>> > >> > > > > > >> > >> batching, > > > >> >>>> > >> > > > > > >> > >> > > > > configs normalization - I agree > > with > > > >> all of > > > >> >>>> > >> them). > > > >> >>>> > >> > > > > > >> > >> > > > > So the second part is Response. > I > > > see > > > >> >>>> there are > > > >> >>>> > >> two > > > >> >>>> > >> > > > cases > > > >> >>>> > >> > > > > > >> here. > > > >> >>>> > >> > > > > > >> > >> > > > > a) "Mutate" requests - > > > >> Create/Alter/... ; > > > >> >>>> b) > > > >> >>>> > >> "Get" > > > >> >>>> > >> > > > > > requests - > > > >> >>>> > >> > > > > > >> > >> > > > > List/Describe... > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > a) should only hold request > result > > > >> >>>> (regardless > > > >> >>>> > >> what > > > >> >>>> > >> > > we > > > >> >>>> > >> > > > > > decide > > > >> >>>> > >> > > > > > >> > >> about > > > >> >>>> > >> > > > > > >> > >> > > > > blocking/non-blocking commands > > > >> execution). > > > >> >>>> > >> > > > > > >> > >> > > > > Usually we provide error code in > > > >> response > > > >> >>>> but > > > >> >>>> > >> since > > > >> >>>> > >> > > we > > > >> >>>> > >> > > > > will > > > >> >>>> > >> > > > > > >> use > > > >> >>>> > >> > > > > > >> > >> this > > > >> >>>> > >> > > > > > >> > >> > in > > > >> >>>> > >> > > > > > >> > >> > > > > interactive shell we need some > > human > > > >> >>>> readable > > > >> >>>> > >> error > > > >> >>>> > >> > > > > > >> description > > > >> >>>> > >> > > > > > >> > - > > > >> >>>> > >> > > > > > >> > >> so > > > >> >>>> > >> > > > > > >> > >> > I > > > >> >>>> > >> > > > > > >> > >> > > > > added errorDesription field > where > > > you > > > >> can > > > >> >>>> at > > > >> >>>> > >> least > > > >> >>>> > >> > > > leave > > > >> >>>> > >> > > > > > >> > >> > > > > exception.getMessage. > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > b) in addition to previous item > > > >> message > > > >> >>>> should > > > >> >>>> > >> hold > > > >> >>>> > >> > > > > command > > > >> >>>> > >> > > > > > >> > >> specific > > > >> >>>> > >> > > > > > >> > >> > > > > response data. We can discuss in > > > >> detail > > > >> >>>> each of > > > >> >>>> > >> > them > > > >> >>>> > >> > > > but > > > >> >>>> > >> > > > > > >> let's > > > >> >>>> > >> > > > > > >> > for > > > >> >>>> > >> > > > > > >> > >> > now > > > >> >>>> > >> > > > > > >> > >> > > > > agree about the overall pattern. > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > Thanks, > > > >> >>>> > >> > > > > > >> > >> > > > > Andrii Biletskyi > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > On Fri, Jan 23, 2015 at 6:59 AM, > > Jay > > > >> Kreps > > > >> >>>> < > > > >> >>>> > >> > > > > > >> jay.kr...@gmail.com > > > >> >>>> > >> > > > > > >> > > > > > >> >>>> > >> > > > > > >> > >> > > wrote: > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > Hey Joe, > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > This is great. A few comments > on > > > >> KIP-4 > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 1. This is much needed > > > >> functionality, > > > >> >>>> but there > > > >> >>>> > >> > > are a > > > >> >>>> > >> > > > > lot > > > >> >>>> > >> > > > > > >> of > > > >> >>>> > >> > > > > > >> > >> the so > > > >> >>>> > >> > > > > > >> > >> > > > let's > > > >> >>>> > >> > > > > > >> > >> > > > > > really think these protocols > > > >> through. We > > > >> >>>> really > > > >> >>>> > >> > > want > > > >> >>>> > >> > > > to > > > >> >>>> > >> > > > > > >> end up > > > >> >>>> > >> > > > > > >> > >> > with a > > > >> >>>> > >> > > > > > >> > >> > > > set > > > >> >>>> > >> > > > > > >> > >> > > > > > of well thought-out, > orthoganol > > > >> apis. > > > >> >>>> For this > > > >> >>>> > >> > > > reason I > > > >> >>>> > >> > > > > > >> think > > > >> >>>> > >> > > > > > >> > >> it is > > > >> >>>> > >> > > > > > >> > >> > > > > really > > > >> >>>> > >> > > > > > >> > >> > > > > > important to think through the > > end > > > >> state > > > >> >>>> even > > > >> >>>> > >> if > > > >> >>>> > >> > > that > > > >> >>>> > >> > > > > > >> includes > > > >> >>>> > >> > > > > > >> > >> APIs > > > >> >>>> > >> > > > > > >> > >> > > we > > > >> >>>> > >> > > > > > >> > >> > > > > > won't implement in the first > > > phase. > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 2. Let's please please please > > wait > > > >> until > > > >> >>>> we > > > >> >>>> > >> have > > > >> >>>> > >> > > > > switched > > > >> >>>> > >> > > > > > >> the > > > >> >>>> > >> > > > > > >> > >> > server > > > >> >>>> > >> > > > > > >> > >> > > > over > > > >> >>>> > >> > > > > > >> > >> > > > > > to the new java protocol > > > >> definitions. If > > > >> >>>> we add > > > >> >>>> > >> > > > upteen > > > >> >>>> > >> > > > > > >> more ad > > > >> >>>> > >> > > > > > >> > >> hoc > > > >> >>>> > >> > > > > > >> > >> > > > scala > > > >> >>>> > >> > > > > > >> > >> > > > > > objects that is just > generating > > > more > > > >> >>>> work for > > > >> >>>> > >> the > > > >> >>>> > >> > > > > > >> conversion > > > >> >>>> > >> > > > > > >> > we > > > >> >>>> > >> > > > > > >> > >> > know > > > >> >>>> > >> > > > > > >> > >> > > we > > > >> >>>> > >> > > > > > >> > >> > > > > > have to do. > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 3. This proposal introduces a > > new > > > >> type of > > > >> >>>> > >> > optional > > > >> >>>> > >> > > > > > >> parameter. > > > >> >>>> > >> > > > > > >> > >> This > > > >> >>>> > >> > > > > > >> > >> > is > > > >> >>>> > >> > > > > > >> > >> > > > > > inconsistent with everything > > else > > > >> in the > > > >> >>>> > >> protocol > > > >> >>>> > >> > > > where > > > >> >>>> > >> > > > > > we > > > >> >>>> > >> > > > > > >> use > > > >> >>>> > >> > > > > > >> > >> -1 > > > >> >>>> > >> > > > > > >> > >> > or > > > >> >>>> > >> > > > > > >> > >> > > > some > > > >> >>>> > >> > > > > > >> > >> > > > > > other marker value. You could > > > argue > > > >> >>>> either way > > > >> >>>> > >> > but > > > >> >>>> > >> > > > > let's > > > >> >>>> > >> > > > > > >> stick > > > >> >>>> > >> > > > > > >> > >> with > > > >> >>>> > >> > > > > > >> > >> > > > that > > > >> >>>> > >> > > > > > >> > >> > > > > > for consistency. For clients > > that > > > >> >>>> implemented > > > >> >>>> > >> the > > > >> >>>> > >> > > > > > protocol > > > >> >>>> > >> > > > > > >> in > > > >> >>>> > >> > > > > > >> > a > > > >> >>>> > >> > > > > > >> > >> > > better > > > >> >>>> > >> > > > > > >> > >> > > > > way > > > >> >>>> > >> > > > > > >> > >> > > > > > than our scala code these > basic > > > >> >>>> primitives are > > > >> >>>> > >> > hard > > > >> >>>> > >> > > > to > > > >> >>>> > >> > > > > > >> change. > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 4. ClusterMetadata: This seems > > to > > > >> >>>> duplicate > > > >> >>>> > >> > > > > > >> > TopicMetadataRequest > > > >> >>>> > >> > > > > > >> > >> > > which > > > >> >>>> > >> > > > > > >> > >> > > > > has > > > >> >>>> > >> > > > > > >> > >> > > > > > brokers, topics, and > > partitions. I > > > >> think > > > >> >>>> we > > > >> >>>> > >> > should > > > >> >>>> > >> > > > > rename > > > >> >>>> > >> > > > > > >> that > > > >> >>>> > >> > > > > > >> > >> > > request > > > >> >>>> > >> > > > > > >> > >> > > > > > ClusterMetadataRequest (or > just > > > >> >>>> > >> MetadataRequest) > > > >> >>>> > >> > > and > > > >> >>>> > >> > > > > > >> include > > > >> >>>> > >> > > > > > >> > >> the id > > > >> >>>> > >> > > > > > >> > >> > > of > > > >> >>>> > >> > > > > > >> > >> > > > > the > > > >> >>>> > >> > > > > > >> > >> > > > > > controller. Or are there other > > > >> things we > > > >> >>>> could > > > >> >>>> > >> > add > > > >> >>>> > >> > > > > here? > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 5. We have a tendency to try > to > > > >> make a > > > >> >>>> lot of > > > >> >>>> > >> > > > requests > > > >> >>>> > >> > > > > > that > > > >> >>>> > >> > > > > > >> > can > > > >> >>>> > >> > > > > > >> > >> > only > > > >> >>>> > >> > > > > > >> > >> > > go > > > >> >>>> > >> > > > > > >> > >> > > > > to > > > >> >>>> > >> > > > > > >> > >> > > > > > particular nodes. This adds a > > lot > > > of > > > >> >>>> burden for > > > >> >>>> > >> > > > client > > > >> >>>> > >> > > > > > >> > >> > > implementations > > > >> >>>> > >> > > > > > >> > >> > > > > (it > > > >> >>>> > >> > > > > > >> > >> > > > > > sounds easy but each discovery > > can > > > >> fail > > > >> >>>> in many > > > >> >>>> > >> > > parts > > > >> >>>> > >> > > > > so > > > >> >>>> > >> > > > > > it > > > >> >>>> > >> > > > > > >> > >> ends up > > > >> >>>> > >> > > > > > >> > >> > > > > being a > > > >> >>>> > >> > > > > > >> > >> > > > > > full state machine to do > > right). I > > > >> think > > > >> >>>> we > > > >> >>>> > >> > should > > > >> >>>> > >> > > > > > consider > > > >> >>>> > >> > > > > > >> > >> making > > > >> >>>> > >> > > > > > >> > >> > > > admin > > > >> >>>> > >> > > > > > >> > >> > > > > > commands and ideally as many > of > > > the > > > >> >>>> other apis > > > >> >>>> > >> as > > > >> >>>> > >> > > > > > possible > > > >> >>>> > >> > > > > > >> > >> > available > > > >> >>>> > >> > > > > > >> > >> > > on > > > >> >>>> > >> > > > > > >> > >> > > > > all > > > >> >>>> > >> > > > > > >> > >> > > > > > brokers and just redirect to > the > > > >> >>>> controller on > > > >> >>>> > >> > the > > > >> >>>> > >> > > > > broker > > > >> >>>> > >> > > > > > >> > side. > > > >> >>>> > >> > > > > > >> > >> > > Perhaps > > > >> >>>> > >> > > > > > >> > >> > > > > > there would be a general way > to > > > >> >>>> encapsulate > > > >> >>>> > >> this > > > >> >>>> > >> > > > > > re-routing > > > >> >>>> > >> > > > > > >> > >> > behavior. > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 6. We should probably > normalize > > > the > > > >> key > > > >> >>>> value > > > >> >>>> > >> > pairs > > > >> >>>> > >> > > > > used > > > >> >>>> > >> > > > > > >> for > > > >> >>>> > >> > > > > > >> > >> > configs > > > >> >>>> > >> > > > > > >> > >> > > > > rather > > > >> >>>> > >> > > > > > >> > >> > > > > > than embedding a new > formatting. > > > So > > > >> two > > > >> >>>> strings > > > >> >>>> > >> > > > rather > > > >> >>>> > >> > > > > > than > > > >> >>>> > >> > > > > > >> > one > > > >> >>>> > >> > > > > > >> > >> > with > > > >> >>>> > >> > > > > > >> > >> > > an > > > >> >>>> > >> > > > > > >> > >> > > > > > internal equals sign. > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 7. Is the postcondition of > these > > > >> APIs > > > >> >>>> that the > > > >> >>>> > >> > > > command > > > >> >>>> > >> > > > > > has > > > >> >>>> > >> > > > > > >> > >> begun or > > > >> >>>> > >> > > > > > >> > >> > > > that > > > >> >>>> > >> > > > > > >> > >> > > > > > the command has been > completed? > > It > > > >> is a > > > >> >>>> lot > > > >> >>>> > >> more > > > >> >>>> > >> > > > usable > > > >> >>>> > >> > > > > > if > > > >> >>>> > >> > > > > > >> the > > > >> >>>> > >> > > > > > >> > >> > > command > > > >> >>>> > >> > > > > > >> > >> > > > > has > > > >> >>>> > >> > > > > > >> > >> > > > > > been completed so you know > that > > if > > > >> you > > > >> >>>> create a > > > >> >>>> > >> > > topic > > > >> >>>> > >> > > > > and > > > >> >>>> > >> > > > > > >> then > > > >> >>>> > >> > > > > > >> > >> > > publish > > > >> >>>> > >> > > > > > >> > >> > > > to > > > >> >>>> > >> > > > > > >> > >> > > > > > it you won't get an exception > > > about > > > >> >>>> there being > > > >> >>>> > >> > no > > > >> >>>> > >> > > > such > > > >> >>>> > >> > > > > > >> topic. > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 8. Describe topic and list > > topics > > > >> >>>> duplicate a > > > >> >>>> > >> lot > > > >> >>>> > >> > > of > > > >> >>>> > >> > > > > > stuff > > > >> >>>> > >> > > > > > >> in > > > >> >>>> > >> > > > > > >> > >> the > > > >> >>>> > >> > > > > > >> > >> > > > > metadata > > > >> >>>> > >> > > > > > >> > >> > > > > > request. Is there a reason to > > give > > > >> back > > > >> >>>> topics > > > >> >>>> > >> > > marked > > > >> >>>> > >> > > > > for > > > >> >>>> > >> > > > > > >> > >> > deletion? I > > > >> >>>> > >> > > > > > >> > >> > > > > feel > > > >> >>>> > >> > > > > > >> > >> > > > > > like if we just make the > > > >> post-condition > > > >> >>>> of the > > > >> >>>> > >> > > delete > > > >> >>>> > >> > > > > > >> command > > > >> >>>> > >> > > > > > >> > be > > > >> >>>> > >> > > > > > >> > >> > that > > > >> >>>> > >> > > > > > >> > >> > > > the > > > >> >>>> > >> > > > > > >> > >> > > > > > topic is deleted that will get > > rid > > > >> of > > > >> >>>> the need > > > >> >>>> > >> > for > > > >> >>>> > >> > > > this > > > >> >>>> > >> > > > > > >> right? > > > >> >>>> > >> > > > > > >> > >> And > > > >> >>>> > >> > > > > > >> > >> > it > > > >> >>>> > >> > > > > > >> > >> > > > > will > > > >> >>>> > >> > > > > > >> > >> > > > > > be much more intuitive. > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 9. Should we consider batching > > > these > > > >> >>>> requests? > > > >> >>>> > >> We > > > >> >>>> > >> > > > have > > > >> >>>> > >> > > > > > >> > generally > > > >> >>>> > >> > > > > > >> > >> > > tried > > > >> >>>> > >> > > > > > >> > >> > > > to > > > >> >>>> > >> > > > > > >> > >> > > > > > allow multiple operations to > be > > > >> batched. > > > >> >>>> My > > > >> >>>> > >> > > suspicion > > > >> >>>> > >> > > > > is > > > >> >>>> > >> > > > > > >> that > > > >> >>>> > >> > > > > > >> > >> > without > > > >> >>>> > >> > > > > > >> > >> > > > > this > > > >> >>>> > >> > > > > > >> > >> > > > > > we will get a lot of code that > > > does > > > >> >>>> something > > > >> >>>> > >> > like > > > >> >>>> > >> > > > > > >> > >> > > > > > for(topic: > > > >> adminClient.listTopics()) > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> adminClient.describeTopic(topic) > > > >> >>>> > >> > > > > > >> > >> > > > > > this code will work great when > > you > > > >> test > > > >> >>>> on 5 > > > >> >>>> > >> > topics > > > >> >>>> > >> > > > but > > > >> >>>> > >> > > > > > >> not do > > > >> >>>> > >> > > > > > >> > >> as > > > >> >>>> > >> > > > > > >> > >> > > well > > > >> >>>> > >> > > > > > >> > >> > > > if > > > >> >>>> > >> > > > > > >> > >> > > > > > you have 50k. > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 10. I think we should also > > discuss > > > >> how > > > >> >>>> we want > > > >> >>>> > >> to > > > >> >>>> > >> > > > > expose > > > >> >>>> > >> > > > > > a > > > >> >>>> > >> > > > > > >> > >> > > programmatic > > > >> >>>> > >> > > > > > >> > >> > > > > JVM > > > >> >>>> > >> > > > > > >> > >> > > > > > client api for these > operations. > > > >> >>>> Currently > > > >> >>>> > >> people > > > >> >>>> > >> > > > rely > > > >> >>>> > >> > > > > on > > > >> >>>> > >> > > > > > >> > >> > AdminUtils > > > >> >>>> > >> > > > > > >> > >> > > > > which > > > >> >>>> > >> > > > > > >> > >> > > > > > is totally sketchy. I think we > > > >> probably > > > >> >>>> need > > > >> >>>> > >> > > another > > > >> >>>> > >> > > > > > client > > > >> >>>> > >> > > > > > >> > >> under > > > >> >>>> > >> > > > > > >> > >> > > > > clients/ > > > >> >>>> > >> > > > > > >> > >> > > > > > that exposes administrative > > > >> >>>> functionality. We > > > >> >>>> > >> > will > > > >> >>>> > >> > > > need > > > >> >>>> > >> > > > > > >> this > > > >> >>>> > >> > > > > > >> > >> just > > > >> >>>> > >> > > > > > >> > >> > to > > > >> >>>> > >> > > > > > >> > >> > > > > > properly test the new apis, I > > > >> suspect. We > > > >> >>>> > >> should > > > >> >>>> > >> > > > figure > > > >> >>>> > >> > > > > > out > > > >> >>>> > >> > > > > > >> > that > > > >> >>>> > >> > > > > > >> > >> > API. > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > 11. The other information that > > > >> would be > > > >> >>>> really > > > >> >>>> > >> > > useful > > > >> >>>> > >> > > > > to > > > >> >>>> > >> > > > > > >> get > > > >> >>>> > >> > > > > > >> > >> would > > > >> >>>> > >> > > > > > >> > >> > be > > > >> >>>> > >> > > > > > >> > >> > > > > > information about > > partitions--how > > > >> much > > > >> >>>> data is > > > >> >>>> > >> in > > > >> >>>> > >> > > the > > > >> >>>> > >> > > > > > >> > partition, > > > >> >>>> > >> > > > > > >> > >> > what > > > >> >>>> > >> > > > > > >> > >> > > > are > > > >> >>>> > >> > > > > > >> > >> > > > > > the segment offsets, what is > the > > > >> log-end > > > >> >>>> offset > > > >> >>>> > >> > > (i.e. > > > >> >>>> > >> > > > > > last > > > >> >>>> > >> > > > > > >> > >> offset), > > > >> >>>> > >> > > > > > >> > >> > > > what > > > >> >>>> > >> > > > > > >> > >> > > > > is > > > >> >>>> > >> > > > > > >> > >> > > > > > the compaction point, etc. I > > think > > > >> that > > > >> >>>> done > > > >> >>>> > >> > right > > > >> >>>> > >> > > > this > > > >> >>>> > >> > > > > > >> would > > > >> >>>> > >> > > > > > >> > be > > > >> >>>> > >> > > > > > >> > >> > the > > > >> >>>> > >> > > > > > >> > >> > > > > > successor to the very awkward > > > >> >>>> OffsetRequest we > > > >> >>>> > >> > have > > > >> >>>> > >> > > > > > today. > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > -Jay > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > On Wed, Jan 21, 2015 at 10:27 > > PM, > > > >> Joe > > > >> >>>> Stein < > > > >> >>>> > >> > > > > > >> > >> joe.st...@stealth.ly> > > > >> >>>> > >> > > > > > >> > >> > > > > wrote: > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > Hi, created a KIP > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > >> >>>> > >> > > > > > >> > >> > > > > >> >>>> > >> > > > > > >> > >> > > > >> >>>> > >> > > > > > >> > > > > >> >>>> > >> > > > > > >> > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > > > > > > > >> >>>> > >> > > > > > > >> >>>> > >> > > > > > >> >>>> > >> > > > > >> >>>> > >> > > > >> >>>> > > > >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > JIRA > > > >> >>>> > >> > > > > https://issues.apache.org/jira/browse/KAFKA-1694 > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> /******************************************* > > > >> >>>> > >> > > > > > >> > >> > > > > > > Joe Stein > > > >> >>>> > >> > > > > > >> > >> > > > > > > Founder, Principal > Consultant > > > >> >>>> > >> > > > > > >> > >> > > > > > > Big Data Open Source > Security > > > LLC > > > >> >>>> > >> > > > > > >> > >> > > > > > > http://www.stealth.ly > > > >> >>>> > >> > > > > > >> > >> > > > > > > Twitter: @allthingshadoop < > > > >> >>>> > >> > > > > > >> > >> > > http://www.twitter.com/allthingshadoop > > > >> >>>> > >> > > > > > >> > >> > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> ********************************************/ > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > > >> >>>> > >> > > > > > >> > >> > > > > > >> >>>> > >> > > > > > >> > >> > > > > >> >>>> > >> > > > > > >> > >> > > > >> >>>> > >> > > > > > >> > >> > > > >> >>>> > >> > > > > > >> > >> > > > >> >>>> > >> > > > > > >> > >> -- > > > >> >>>> > >> > > > > > >> > >> -- Guozhang > > > >> >>>> > >> > > > > > >> > >> > > > >> >>>> > >> > > > > > >> > > > > > >> >>>> > >> > > > > > >> > > > > > >> >>>> > >> > > > > > >> > > > > >> >>>> > >> > > > > > >> > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > > > > > > > > >> >>>> > >> > > > > > > > > >> >>>> > >> > > > > > > > >> >>>> > >> > > > > > > >> >>>> > >> > > > > > >> >>>> > >> > > > > >> >>>> > >> > > > > >> >>>> > >> > > > > >> >>>> > >> > -- > > > >> >>>> > >> > Jeff Holoman > > > >> >>>> > >> > Systems Engineer > > > >> >>>> > >> > > > > >> >>>> > >> > > > >> >>>> > > > >> >>>> > > > >> >>> > > > >> >>> > > > >> >>> -- > > > >> >>> -- Guozhang > > > >> >>> > > > >> >> > > > >> >> > > > >> >> > > > >> >> -- > > > >> >> -- Guozhang > > > >> > > > > > > > > > > > > > >
