Thanks for sending that out Joe - I don't think I will be able to make it today, so if notes can be sent out afterward that would be great.
On Mon, Mar 02, 2015 at 09:16:13AM -0800, Gwen Shapira wrote: > Thanks for sending this out Joe. Looking forward to chatting with everyone :) > > On Mon, Mar 2, 2015 at 6:46 AM, Joe Stein <joe.st...@stealth.ly> wrote: > > Hey, I just sent out a google hangout invite to all pmc, committers and > > everyone I found working on a KIP. If I missed anyone in the invite please > > let me know and can update it, np. > > > > We should do this every Tuesday @ 2pm Eastern Time. Maybe we can get INFRA > > help to make a google account so we can manage better? > > > > To discuss > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > in progress and related JIRA that are interdependent and common work. > > > > ~ Joe Stein > > > > On Tue, Feb 24, 2015 at 2:59 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > >> Let's stay on Google hangouts that will also record and make the sessions > >> available on youtube. > >> > >> -Jay > >> > >> On Tue, Feb 24, 2015 at 11:49 AM, Jeff Holoman <jholo...@cloudera.com> > >> wrote: > >> > >> > Jay / Joe > >> > > >> > We're happy to send out a Webex for this purpose. We could record the > >> > sessions if there is interest and publish them out. > >> > > >> > Thanks > >> > > >> > Jeff > >> > > >> > On Tue, Feb 24, 2015 at 11:28 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > >> > > >> > > Let's try to get the technical hang-ups sorted out, though. I really > >> > think > >> > > there is some benefit to live discussion vs writing. I am hopeful that > >> if > >> > > we post instructions and give ourselves a few attempts we can get it > >> > > working. > >> > > > >> > > Tuesday at that time would work for me...any objections? > >> > > > >> > > -Jay > >> > > > >> > > On Tue, Feb 24, 2015 at 8:18 AM, Joe Stein <joe.st...@stealth.ly> > >> wrote: > >> > > > >> > > > Weekly would be great maybe like every Tuesday ~ 1pm ET / 10am PT > >> ???? > >> > > > > >> > > > I don't mind google hangout but there is always some issue or > >> whatever > >> > so > >> > > > we know the apache irc channel works. We can start there and see how > >> it > >> > > > goes? We can pull transcripts too and associate to tickets if need be > >> > > makes > >> > > > it helpful for things. > >> > > > > >> > > > ~ Joestein > >> > > > > >> > > > On Tue, Feb 24, 2015 at 11:10 AM, Jay Kreps <jay.kr...@gmail.com> > >> > wrote: > >> > > > > >> > > > > We'd talked about doing a Google Hangout to chat about this. What > >> > about > >> > > > > generalizing that a little further...I actually think it would be > >> > good > >> > > > for > >> > > > > everyone spending a reasonable chunk of their week on Kafka stuff > >> to > >> > > > maybe > >> > > > > sync up once a week. I think we could use time to talk through > >> design > >> > > > > stuff, make sure we are on top of code reviews, talk through any > >> > tricky > >> > > > > issues, etc. > >> > > > > > >> > > > > We can make it publicly available so that any one can follow along > >> > who > >> > > > > likes. > >> > > > > > >> > > > > Any interest in doing this? If so I'll try to set it up starting > >> next > >> > > > week. > >> > > > > > >> > > > > -Jay > >> > > > > > >> > > > > On Tue, Feb 24, 2015 at 3:57 AM, Andrii Biletskyi < > >> > > > > andrii.bilets...@stealth.ly> wrote: > >> > > > > > >> > > > > > Hi all, > >> > > > > > > >> > > > > > I've updated KIP page, fixed / aligned document structure. Also I > >> > > added > >> > > > > > some > >> > > > > > very initial proposal for AdminClient so we have something to > >> start > >> > > > from > >> > > > > > while > >> > > > > > discussing the KIP. > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > >> > > > > > > >> > > > > > Thanks, > >> > > > > > Andrii Biletskyi > >> > > > > > > >> > > > > > On Wed, Feb 18, 2015 at 9:01 PM, Andrii Biletskyi < > >> > > > > > andrii.bilets...@stealth.ly> wrote: > >> > > > > > > >> > > > > > > Jay, > >> > > > > > > > >> > > > > > > Re error messages: you are right, in most cases client will > >> have > >> > > > enough > >> > > > > > > context to show descriptive error message. My concern is that > >> we > >> > > will > >> > > > > > have > >> > > > > > > to > >> > > > > > > add lots of new error codes for each possible error. Of course, > >> > we > >> > > > > could > >> > > > > > > reuse > >> > > > > > > some of existing like UknownTopicOrPartitionCode, but we will > >> > also > >> > > > need > >> > > > > > to > >> > > > > > > add smth like: TopicAlreadyExistsCode, TopicConfigInvalid (both > >> > for > >> > > > > topic > >> > > > > > > name and config, and probably user would like to know what > >> > exactly > >> > > > > > > is wrong in his config), InvalidReplicaAssignment, > >> InternalError > >> > > > (e.g. > >> > > > > > > zookeeper failure) etc. > >> > > > > > > And this is only for TopicCommand, we will also need to add > >> > similar > >> > > > > stuff > >> > > > > > > for > >> > > > > > > ReassignPartitions, PreferredReplica. So we'll end up with a > >> > large > >> > > > list > >> > > > > > of > >> > > > > > > error codes, used only in Admin protocol. > >> > > > > > > Having said that, I agree my proposal is not consistent with > >> > other > >> > > > > cases. > >> > > > > > > Maybe we can find better solution or something in-between. > >> > > > > > > > >> > > > > > > Re Hangout chat: I think it is a great idea. This way we can > >> move > >> > > on > >> > > > > > > faster. > >> > > > > > > Let's agree somehow on date/time so people can join. Will work > >> > for > >> > > me > >> > > > > > this > >> > > > > > > and > >> > > > > > > next week almost anytime if agreed in advance. > >> > > > > > > > >> > > > > > > Thanks, > >> > > > > > > Andrii > >> > > > > > > > >> > > > > > > On Wed, Feb 18, 2015 at 7:09 PM, Jay Kreps < > >> jay.kr...@gmail.com> > >> > > > > wrote: > >> > > > > > > > >> > > > > > >> Hey Andrii, > >> > > > > > >> > >> > > > > > >> Generally we can do good error handling without needing custom > >> > > > > > server-side > >> > > > > > >> messages. I.e. generally the client has the context to know > >> that > >> > > if > >> > > > it > >> > > > > > got > >> > > > > > >> an error that the topic doesn't exist to say "Topic X doesn't > >> > > exist" > >> > > > > > >> rather > >> > > > > > >> than "error code 14" (or whatever). Maybe there are specific > >> > cases > >> > > > > where > >> > > > > > >> this is hard? If we want to add server-side error messages we > >> > > really > >> > > > > do > >> > > > > > >> need to do this in a consistent way across the protocol. > >> > > > > > >> > >> > > > > > >> I still have a bunch of open questions here from my previous > >> > > list. I > >> > > > > > will > >> > > > > > >> be out for the next few days for Strata though. Maybe we could > >> > do > >> > > a > >> > > > > > Google > >> > > > > > >> Hangout chat on any open issues some time towards the end of > >> > next > >> > > > week > >> > > > > > for > >> > > > > > >> anyone interested in this ticket? I have a feeling that might > >> > > > progress > >> > > > > > >> things a little faster than email--I think we could talk > >> through > >> > > > those > >> > > > > > >> issues I brought up fairly quickly... > >> > > > > > >> > >> > > > > > >> -Jay > >> > > > > > >> > >> > > > > > >> On Wed, Feb 18, 2015 at 7:27 AM, Andrii Biletskyi < > >> > > > > > >> andrii.bilets...@stealth.ly> wrote: > >> > > > > > >> > >> > > > > > >> > Hi all, > >> > > > > > >> > > >> > > > > > >> > I'm trying to address some of the issues which were > >> mentioned > >> > > > > earlier > >> > > > > > >> about > >> > > > > > >> > Admin RQ/RP format. One of those was about batching > >> > operations. > >> > > > What > >> > > > > > if > >> > > > > > >> we > >> > > > > > >> > follow TopicCommand approach and let people specify > >> topic-name > >> > > by > >> > > > > > >> regexp - > >> > > > > > >> > would that cover most of the use cases? > >> > > > > > >> > > >> > > > > > >> > Secondly, is what information should we generally provide in > >> > > Admin > >> > > > > > >> > responses. > >> > > > > > >> > I realize that Admin commands don't imply they will be used > >> > only > >> > > > in > >> > > > > > CLI > >> > > > > > >> > but, > >> > > > > > >> > it seems to me, CLI is a very important client of this > >> > feature. > >> > > In > >> > > > > > this > >> > > > > > >> > case, > >> > > > > > >> > seems logical, we would like to provide users with rich > >> > > experience > >> > > > > in > >> > > > > > >> terms > >> > > > > > >> > of > >> > > > > > >> > getting results / errors of the executed commands. Usually > >> we > >> > > > supply > >> > > > > > >> with > >> > > > > > >> > responses only errorCode, which looks very limiting, in case > >> > of > >> > > > CLI > >> > > > > we > >> > > > > > >> may > >> > > > > > >> > want to print human readable error description. > >> > > > > > >> > > >> > > > > > >> > So, taking into account previous item about batching, what > >> do > >> > > you > >> > > > > > think > >> > > > > > >> > about > >> > > > > > >> > having smth like: > >> > > > > > >> > > >> > > > > > >> > ('create' doesn't support regexp) > >> > > > > > >> > CreateTopicRequest => TopicName Partitions Replicas > >> > > > > ReplicaAssignment > >> > > > > > >> > [Config] > >> > > > > > >> > CreateTopicResponse => ErrorCode ErrorDescription > >> > > > > > >> > ErrorCode => int16 > >> > > > > > >> > ErrorDescription => string (empty if successful) > >> > > > > > >> > > >> > > > > > >> > AlterTopicRequest -> TopicNameRegexp Partitions > >> > > ReplicaAssignment > >> > > > > > >> > [AddedConfig] [DeletedConfig] > >> > > > > > >> > AlterTopicResponse -> [TopicName ErrorCode ErrorDescription] > >> > > > > > >> > CommandErrorCode CommandErrorDescription > >> > > > > > >> > CommandErrorCode => int16 > >> > > > > > >> > CommandErrorDescription => string (nonempty in case of > >> fatal > >> > > > > error, > >> > > > > > >> e.g. > >> > > > > > >> > we couldn't get topics by regexp) > >> > > > > > >> > > >> > > > > > >> > DescribeTopicRequest -> TopicNameRegexp > >> > > > > > >> > DescribeTopicResponse -> [TopicName TopicDescription > >> ErrorCode > >> > > > > > >> > ErrorDescription] CommandErrorCode CommandErrorDescription > >> > > > > > >> > > >> > > > > > >> > Also, any thoughts about our discussion regarding re-routing > >> > > > > facility? > >> > > > > > >> In > >> > > > > > >> > my > >> > > > > > >> > understanding, it is like between augmenting > >> > > TopicMetadataRequest > >> > > > > > >> > (to include at least controllerId) and implementing new > >> > generic > >> > > > > > >> re-routing > >> > > > > > >> > facility so sending messages to controller will be handled > >> by > >> > > it. > >> > > > > > >> > > >> > > > > > >> > Thanks, > >> > > > > > >> > Andrii Biletskyi > >> > > > > > >> > > >> > > > > > >> > On Mon, Feb 16, 2015 at 5:26 PM, Andrii Biletskyi < > >> > > > > > >> > andrii.bilets...@stealth.ly> wrote: > >> > > > > > >> > > >> > > > > > >> > > @Guozhang: > >> > > > > > >> > > Thanks for your comments, I've answered some of those. The > >> > > main > >> > > > > > thing > >> > > > > > >> is > >> > > > > > >> > > having merged request for create-alter-delete-describe - I > >> > > have > >> > > > > some > >> > > > > > >> > > concerns about this approach. > >> > > > > > >> > > > >> > > > > > >> > > @*Jay*: > >> > > > > > >> > > I see that introduced ClusterMetadaRequest is also one of > >> > the > >> > > > > > >> concerns. > >> > > > > > >> > We > >> > > > > > >> > > can solve it if we implement re-routing facility. But I > >> > agree > >> > > > with > >> > > > > > >> > > Guozhang - it will make clients' internals a little bit > >> > easier > >> > > > but > >> > > > > > >> this > >> > > > > > >> > > seems to be a complex logic to implement and support then. > >> > > > > > Especially > >> > > > > > >> for > >> > > > > > >> > > Fetch and Produce (even if we add re-routing later for > >> these > >> > > > > > >> requests). > >> > > > > > >> > > Also people will tend to avoid this re-routing facility > >> and > >> > > hold > >> > > > > > local > >> > > > > > >> > > cluster cache to ensure their high-priority requests > >> (which > >> > > some > >> > > > > of > >> > > > > > >> the > >> > > > > > >> > > admin requests are) not sent to some busy broker where > >> they > >> > > wait > >> > > > > to > >> > > > > > be > >> > > > > > >> > > routed to the correct one. > >> > > > > > >> > > As pointed out by Jun here ( > >> > > > > > >> > > > >> > > > > > >> > > >> > > > > > >> > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> https://issues.apache.org/jira/browse/KAFKA-1772?focusedCommentId=14234530&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14234530 > >> > > > > > >> > ) > >> > > > > > >> > > to solve the issue we might introduce a message type to > >> get > >> > > > > cluster > >> > > > > > >> > state. > >> > > > > > >> > > But I agree we can just update TopicMetadataResponse to > >> > > include > >> > > > > > >> > > controllerId (and probably smth else). > >> > > > > > >> > > What are you thougths? > >> > > > > > >> > > > >> > > > > > >> > > Thanks, > >> > > > > > >> > > Andrii > >> > > > > > >> > > > >> > > > > > >> > > On Thu, Feb 12, 2015 at 8:31 AM, Guozhang Wang < > >> > > > > wangg...@gmail.com> > >> > > > > > >> > wrote: > >> > > > > > >> > > > >> > > > > > >> > >> I think for the topics commands we can actually merge > >> > > > > > >> > >> create/alter/delete/describe as one request type since > >> > their > >> > > > > > formats > >> > > > > > >> are > >> > > > > > >> > >> very much similar, and keep list-topics and others like > >> > > > > > >> > >> partition-reassignment / preferred-leader-election as > >> > > separate > >> > > > > > >> request > >> > > > > > >> > >> types, I also left some other comments on the RB ( > >> > > > > > >> > >> https://reviews.apache.org/r/29301/). > >> > > > > > >> > >> > >> > > > > > >> > >> On Wed, Feb 11, 2015 at 2:04 PM, Jay Kreps < > >> > > > jay.kr...@gmail.com> > >> > > > > > >> wrote: > >> > > > > > >> > >> > >> > > > > > >> > >> > Yeah I totally agree that we don't want to just have > >> one > >> > > "do > >> > > > > > admin > >> > > > > > >> > >> stuff" > >> > > > > > >> > >> > command that has the union of all parameters. > >> > > > > > >> > >> > > >> > > > > > >> > >> > What I am saying is that command line tools are one > >> > client > >> > > of > >> > > > > the > >> > > > > > >> > >> > administrative apis, but these will be used in a number > >> > of > >> > > > > > >> scenarios > >> > > > > > >> > so > >> > > > > > >> > >> > they should make logical sense even in the absence of > >> the > >> > > > > command > >> > > > > > >> line > >> > > > > > >> > >> > tool. Hence comments like trying to clarify the > >> > > relationship > >> > > > > > >> between > >> > > > > > >> > >> > ClusterMetadata and TopicMetadata...these kinds of > >> things > >> > > > > really > >> > > > > > >> need > >> > > > > > >> > >> to be > >> > > > > > >> > >> > thought through. > >> > > > > > >> > >> > > >> > > > > > >> > >> > Hope that makes sense. > >> > > > > > >> > >> > > >> > > > > > >> > >> > -Jay > >> > > > > > >> > >> > > >> > > > > > >> > >> > On Wed, Feb 11, 2015 at 1:41 PM, Andrii Biletskyi < > >> > > > > > >> > >> > andrii.bilets...@stealth.ly> wrote: > >> > > > > > >> > >> > > >> > > > > > >> > >> > > Jay, > >> > > > > > >> > >> > > > >> > > > > > >> > >> > > Thanks for answering. You understood correctly, most > >> of > >> > > my > >> > > > > > >> comments > >> > > > > > >> > >> were > >> > > > > > >> > >> > > related to your point 1) - about "well thought-out" > >> > apis. > >> > > > > Also, > >> > > > > > >> yes, > >> > > > > > >> > >> as I > >> > > > > > >> > >> > > understood we would like to introduce a single > >> unified > >> > > CLI > >> > > > > tool > >> > > > > > >> with > >> > > > > > >> > >> > > centralized server-side request handling for lots of > >> > > > existing > >> > > > > > >> ones > >> > > > > > >> > >> (incl. > >> > > > > > >> > >> > > TopicCommand, CommitOffsetChecker, > >> ReassignPartitions, > >> > > smth > >> > > > > > else > >> > > > > > >> if > >> > > > > > >> > >> added > >> > > > > > >> > >> > > in future). In our previous discussion ( > >> > > > > > >> > >> > > https://issues.apache.org/jira/browse/KAFKA-1694) > >> > people > >> > > > > said > >> > > > > > >> > they'd > >> > > > > > >> > >> > > rather > >> > > > > > >> > >> > > have a separate message for each command, so, yes, > >> this > >> > > > way I > >> > > > > > >> came > >> > > > > > >> > to > >> > > > > > >> > >> 1-1 > >> > > > > > >> > >> > > mapping between commands in the tool and protocol > >> > > > additions. > >> > > > > > But > >> > > > > > >> I > >> > > > > > >> > >> might > >> > > > > > >> > >> > be > >> > > > > > >> > >> > > wrong. > >> > > > > > >> > >> > > At the end I just try to start discussion how at > >> least > >> > > > > > generally > >> > > > > > >> > this > >> > > > > > >> > >> > > protocol should look like. > >> > > > > > >> > >> > > > >> > > > > > >> > >> > > Thanks, > >> > > > > > >> > >> > > Andrii > >> > > > > > >> > >> > > > >> > > > > > >> > >> > > On Wed, Feb 11, 2015 at 11:10 PM, Jay Kreps < > >> > > > > > jay.kr...@gmail.com > >> > > > > > >> > > >> > > > > > >> > >> wrote: > >> > > > > > >> > >> > > > >> > > > > > >> > >> > > > Hey Andrii, > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > To answer your earlier question we just really > >> can't > >> > be > >> > > > > > adding > >> > > > > > >> any > >> > > > > > >> > >> more > >> > > > > > >> > >> > > > scala protocol objects. These things are super hard > >> > to > >> > > > > > maintain > >> > > > > > >> > >> because > >> > > > > > >> > >> > > > they hand code the byte parsing and don't have good > >> > > > > > versioning > >> > > > > > >> > >> support. > >> > > > > > >> > >> > > > Since we are already planning on converting we > >> > > definitely > >> > > > > > don't > >> > > > > > >> > >> want to > >> > > > > > >> > >> > > add > >> > > > > > >> > >> > > > a ton more of these--they are total tech debt. > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > What does it mean that the changes are isolated > >> from > >> > > the > >> > > > > > >> current > >> > > > > > >> > >> code > >> > > > > > >> > >> > > base? > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > I actually didn't understand the remaining > >> comments, > >> > > > which > >> > > > > of > >> > > > > > >> the > >> > > > > > >> > >> > points > >> > > > > > >> > >> > > > are you responding to? > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > Maybe one sticking point here is that it seems like > >> > you > >> > > > > want > >> > > > > > to > >> > > > > > >> > make > >> > > > > > >> > >> > some > >> > > > > > >> > >> > > > kind of tool, and you have made a 1-1 mapping > >> between > >> > > > > > commands > >> > > > > > >> you > >> > > > > > >> > >> > > imagine > >> > > > > > >> > >> > > > in the tool and protocol additions. I want to make > >> > sure > >> > > > we > >> > > > > > >> don't > >> > > > > > >> > do > >> > > > > > >> > >> > that. > >> > > > > > >> > >> > > > The protocol needs to be really really well thought > >> > out > >> > > > > > against > >> > > > > > >> > many > >> > > > > > >> > >> > use > >> > > > > > >> > >> > > > cases so it should make perfect logical sense in > >> the > >> > > > > absence > >> > > > > > of > >> > > > > > >> > >> knowing > >> > > > > > >> > >> > > the > >> > > > > > >> > >> > > > command line tool, right? > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > -Jay > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > On Wed, Feb 11, 2015 at 11:57 AM, Andrii Biletskyi > >> < > >> > > > > > >> > >> > > > andrii.bilets...@stealth.ly> wrote: > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > > Hey Jay, > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > I would like to continue this discussion as it > >> seem > >> > > > there > >> > > > > > is > >> > > > > > >> no > >> > > > > > >> > >> > > progress > >> > > > > > >> > >> > > > > here. > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > First of all, could you please explain what did > >> you > >> > > > mean > >> > > > > in > >> > > > > > >> 2? > >> > > > > > >> > How > >> > > > > > >> > >> > > > exactly > >> > > > > > >> > >> > > > > are we going to migrate to the new java protocol > >> > > > > > definitions. > >> > > > > > >> > And > >> > > > > > >> > >> why > >> > > > > > >> > >> > > > it's > >> > > > > > >> > >> > > > > a blocker for centralized CLI? > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > I agree with you, this feature includes lots of > >> > > stuff, > >> > > > > but > >> > > > > > >> > >> thankfully > >> > > > > > >> > >> > > > > almost all changes are isolated from the current > >> > code > >> > > > > base, > >> > > > > > >> > >> > > > > so the main thing, I think, we need to agree is > >> > RQ/RP > >> > > > > > format. > >> > > > > > >> > >> > > > > So how can we start discussion about the concrete > >> > > > > messages > >> > > > > > >> > format? > >> > > > > > >> > >> > > > > Can we take ( > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > >> > > > > > >> > >> > > >> > > > > > >> > >> > >> > > > > > >> > > >> > > > > > >> > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-ProposedRQ/RPFormat > >> > > > > > >> > >> > > > > ) > >> > > > > > >> > >> > > > > as starting point? > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > We had some doubts earlier whether it worth > >> > > introducing > >> > > > > one > >> > > > > > >> > >> generic > >> > > > > > >> > >> > > Admin > >> > > > > > >> > >> > > > > Request for all commands ( > >> > > > > > >> > >> > > > https://issues.apache.org/jira/browse/KAFKA-1694 > >> > > > > > >> > >> > > > > ) > >> > > > > > >> > >> > > > > but then everybody agreed it would be better to > >> > have > >> > > > > > separate > >> > > > > > >> > >> message > >> > > > > > >> > >> > > for > >> > > > > > >> > >> > > > > each admin command. The Request part is really > >> > > dictated > >> > > > > > from > >> > > > > > >> the > >> > > > > > >> > >> > > command > >> > > > > > >> > >> > > > > (e.g. TopicCommand) arguments itself, so the > >> > proposed > >> > > > > > version > >> > > > > > >> > >> should > >> > > > > > >> > >> > be > >> > > > > > >> > >> > > > > fine (let's put aside for now remarks about > >> > Optional > >> > > > > type, > >> > > > > > >> > >> batching, > >> > > > > > >> > >> > > > > configs normalization - I agree with all of > >> them). > >> > > > > > >> > >> > > > > So the second part is Response. I see there are > >> two > >> > > > cases > >> > > > > > >> here. > >> > > > > > >> > >> > > > > a) "Mutate" requests - Create/Alter/... ; b) > >> "Get" > >> > > > > > requests - > >> > > > > > >> > >> > > > > List/Describe... > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > a) should only hold request result (regardless > >> what > >> > > we > >> > > > > > decide > >> > > > > > >> > >> about > >> > > > > > >> > >> > > > > blocking/non-blocking commands execution). > >> > > > > > >> > >> > > > > Usually we provide error code in response but > >> since > >> > > we > >> > > > > will > >> > > > > > >> use > >> > > > > > >> > >> this > >> > > > > > >> > >> > in > >> > > > > > >> > >> > > > > interactive shell we need some human readable > >> error > >> > > > > > >> description > >> > > > > > >> > - > >> > > > > > >> > >> so > >> > > > > > >> > >> > I > >> > > > > > >> > >> > > > > added errorDesription field where you can at > >> least > >> > > > leave > >> > > > > > >> > >> > > > > exception.getMessage. > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > b) in addition to previous item message should > >> hold > >> > > > > command > >> > > > > > >> > >> specific > >> > > > > > >> > >> > > > > response data. We can discuss in detail each of > >> > them > >> > > > but > >> > > > > > >> let's > >> > > > > > >> > for > >> > > > > > >> > >> > now > >> > > > > > >> > >> > > > > agree about the overall pattern. > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > Thanks, > >> > > > > > >> > >> > > > > Andrii Biletskyi > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > On Fri, Jan 23, 2015 at 6:59 AM, Jay Kreps < > >> > > > > > >> jay.kr...@gmail.com > >> > > > > > >> > > > >> > > > > > >> > >> > > wrote: > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > > Hey Joe, > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > This is great. A few comments on KIP-4 > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 1. This is much needed functionality, but there > >> > > are a > >> > > > > lot > >> > > > > > >> of > >> > > > > > >> > >> the so > >> > > > > > >> > >> > > > let's > >> > > > > > >> > >> > > > > > really think these protocols through. We really > >> > > want > >> > > > to > >> > > > > > >> end up > >> > > > > > >> > >> > with a > >> > > > > > >> > >> > > > set > >> > > > > > >> > >> > > > > > of well thought-out, orthoganol apis. For this > >> > > > reason I > >> > > > > > >> think > >> > > > > > >> > >> it is > >> > > > > > >> > >> > > > > really > >> > > > > > >> > >> > > > > > important to think through the end state even > >> if > >> > > that > >> > > > > > >> includes > >> > > > > > >> > >> APIs > >> > > > > > >> > >> > > we > >> > > > > > >> > >> > > > > > won't implement in the first phase. > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 2. Let's please please please wait until we > >> have > >> > > > > switched > >> > > > > > >> the > >> > > > > > >> > >> > server > >> > > > > > >> > >> > > > over > >> > > > > > >> > >> > > > > > to the new java protocol definitions. If we add > >> > > > upteen > >> > > > > > >> more ad > >> > > > > > >> > >> hoc > >> > > > > > >> > >> > > > scala > >> > > > > > >> > >> > > > > > objects that is just generating more work for > >> the > >> > > > > > >> conversion > >> > > > > > >> > we > >> > > > > > >> > >> > know > >> > > > > > >> > >> > > we > >> > > > > > >> > >> > > > > > have to do. > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 3. This proposal introduces a new type of > >> > optional > >> > > > > > >> parameter. > >> > > > > > >> > >> This > >> > > > > > >> > >> > is > >> > > > > > >> > >> > > > > > inconsistent with everything else in the > >> protocol > >> > > > where > >> > > > > > we > >> > > > > > >> use > >> > > > > > >> > >> -1 > >> > > > > > >> > >> > or > >> > > > > > >> > >> > > > some > >> > > > > > >> > >> > > > > > other marker value. You could argue either way > >> > but > >> > > > > let's > >> > > > > > >> stick > >> > > > > > >> > >> with > >> > > > > > >> > >> > > > that > >> > > > > > >> > >> > > > > > for consistency. For clients that implemented > >> the > >> > > > > > protocol > >> > > > > > >> in > >> > > > > > >> > a > >> > > > > > >> > >> > > better > >> > > > > > >> > >> > > > > way > >> > > > > > >> > >> > > > > > than our scala code these basic primitives are > >> > hard > >> > > > to > >> > > > > > >> change. > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 4. ClusterMetadata: This seems to duplicate > >> > > > > > >> > TopicMetadataRequest > >> > > > > > >> > >> > > which > >> > > > > > >> > >> > > > > has > >> > > > > > >> > >> > > > > > brokers, topics, and partitions. I think we > >> > should > >> > > > > rename > >> > > > > > >> that > >> > > > > > >> > >> > > request > >> > > > > > >> > >> > > > > > ClusterMetadataRequest (or just > >> MetadataRequest) > >> > > and > >> > > > > > >> include > >> > > > > > >> > >> the id > >> > > > > > >> > >> > > of > >> > > > > > >> > >> > > > > the > >> > > > > > >> > >> > > > > > controller. Or are there other things we could > >> > add > >> > > > > here? > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 5. We have a tendency to try to make a lot of > >> > > > requests > >> > > > > > that > >> > > > > > >> > can > >> > > > > > >> > >> > only > >> > > > > > >> > >> > > go > >> > > > > > >> > >> > > > > to > >> > > > > > >> > >> > > > > > particular nodes. This adds a lot of burden for > >> > > > client > >> > > > > > >> > >> > > implementations > >> > > > > > >> > >> > > > > (it > >> > > > > > >> > >> > > > > > sounds easy but each discovery can fail in many > >> > > parts > >> > > > > so > >> > > > > > it > >> > > > > > >> > >> ends up > >> > > > > > >> > >> > > > > being a > >> > > > > > >> > >> > > > > > full state machine to do right). I think we > >> > should > >> > > > > > consider > >> > > > > > >> > >> making > >> > > > > > >> > >> > > > admin > >> > > > > > >> > >> > > > > > commands and ideally as many of the other apis > >> as > >> > > > > > possible > >> > > > > > >> > >> > available > >> > > > > > >> > >> > > on > >> > > > > > >> > >> > > > > all > >> > > > > > >> > >> > > > > > brokers and just redirect to the controller on > >> > the > >> > > > > broker > >> > > > > > >> > side. > >> > > > > > >> > >> > > Perhaps > >> > > > > > >> > >> > > > > > there would be a general way to encapsulate > >> this > >> > > > > > re-routing > >> > > > > > >> > >> > behavior. > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 6. We should probably normalize the key value > >> > pairs > >> > > > > used > >> > > > > > >> for > >> > > > > > >> > >> > configs > >> > > > > > >> > >> > > > > rather > >> > > > > > >> > >> > > > > > than embedding a new formatting. So two strings > >> > > > rather > >> > > > > > than > >> > > > > > >> > one > >> > > > > > >> > >> > with > >> > > > > > >> > >> > > an > >> > > > > > >> > >> > > > > > internal equals sign. > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 7. Is the postcondition of these APIs that the > >> > > > command > >> > > > > > has > >> > > > > > >> > >> begun or > >> > > > > > >> > >> > > > that > >> > > > > > >> > >> > > > > > the command has been completed? It is a lot > >> more > >> > > > usable > >> > > > > > if > >> > > > > > >> the > >> > > > > > >> > >> > > command > >> > > > > > >> > >> > > > > has > >> > > > > > >> > >> > > > > > been completed so you know that if you create a > >> > > topic > >> > > > > and > >> > > > > > >> then > >> > > > > > >> > >> > > publish > >> > > > > > >> > >> > > > to > >> > > > > > >> > >> > > > > > it you won't get an exception about there being > >> > no > >> > > > such > >> > > > > > >> topic. > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 8. Describe topic and list topics duplicate a > >> lot > >> > > of > >> > > > > > stuff > >> > > > > > >> in > >> > > > > > >> > >> the > >> > > > > > >> > >> > > > > metadata > >> > > > > > >> > >> > > > > > request. Is there a reason to give back topics > >> > > marked > >> > > > > for > >> > > > > > >> > >> > deletion? I > >> > > > > > >> > >> > > > > feel > >> > > > > > >> > >> > > > > > like if we just make the post-condition of the > >> > > delete > >> > > > > > >> command > >> > > > > > >> > be > >> > > > > > >> > >> > that > >> > > > > > >> > >> > > > the > >> > > > > > >> > >> > > > > > topic is deleted that will get rid of the need > >> > for > >> > > > this > >> > > > > > >> right? > >> > > > > > >> > >> And > >> > > > > > >> > >> > it > >> > > > > > >> > >> > > > > will > >> > > > > > >> > >> > > > > > be much more intuitive. > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 9. Should we consider batching these requests? > >> We > >> > > > have > >> > > > > > >> > generally > >> > > > > > >> > >> > > tried > >> > > > > > >> > >> > > > to > >> > > > > > >> > >> > > > > > allow multiple operations to be batched. My > >> > > suspicion > >> > > > > is > >> > > > > > >> that > >> > > > > > >> > >> > without > >> > > > > > >> > >> > > > > this > >> > > > > > >> > >> > > > > > we will get a lot of code that does something > >> > like > >> > > > > > >> > >> > > > > > for(topic: adminClient.listTopics()) > >> > > > > > >> > >> > > > > > adminClient.describeTopic(topic) > >> > > > > > >> > >> > > > > > this code will work great when you test on 5 > >> > topics > >> > > > but > >> > > > > > >> not do > >> > > > > > >> > >> as > >> > > > > > >> > >> > > well > >> > > > > > >> > >> > > > if > >> > > > > > >> > >> > > > > > you have 50k. > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 10. I think we should also discuss how we want > >> to > >> > > > > expose > >> > > > > > a > >> > > > > > >> > >> > > programmatic > >> > > > > > >> > >> > > > > JVM > >> > > > > > >> > >> > > > > > client api for these operations. Currently > >> people > >> > > > rely > >> > > > > on > >> > > > > > >> > >> > AdminUtils > >> > > > > > >> > >> > > > > which > >> > > > > > >> > >> > > > > > is totally sketchy. I think we probably need > >> > > another > >> > > > > > client > >> > > > > > >> > >> under > >> > > > > > >> > >> > > > > clients/ > >> > > > > > >> > >> > > > > > that exposes administrative functionality. We > >> > will > >> > > > need > >> > > > > > >> this > >> > > > > > >> > >> just > >> > > > > > >> > >> > to > >> > > > > > >> > >> > > > > > properly test the new apis, I suspect. We > >> should > >> > > > figure > >> > > > > > out > >> > > > > > >> > that > >> > > > > > >> > >> > API. > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > 11. The other information that would be really > >> > > useful > >> > > > > to > >> > > > > > >> get > >> > > > > > >> > >> would > >> > > > > > >> > >> > be > >> > > > > > >> > >> > > > > > information about partitions--how much data is > >> in > >> > > the > >> > > > > > >> > partition, > >> > > > > > >> > >> > what > >> > > > > > >> > >> > > > are > >> > > > > > >> > >> > > > > > the segment offsets, what is the log-end offset > >> > > (i.e. > >> > > > > > last > >> > > > > > >> > >> offset), > >> > > > > > >> > >> > > > what > >> > > > > > >> > >> > > > > is > >> > > > > > >> > >> > > > > > the compaction point, etc. I think that done > >> > right > >> > > > this > >> > > > > > >> would > >> > > > > > >> > be > >> > > > > > >> > >> > the > >> > > > > > >> > >> > > > > > successor to the very awkward OffsetRequest we > >> > have > >> > > > > > today. > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > -Jay > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > On Wed, Jan 21, 2015 at 10:27 PM, Joe Stein < > >> > > > > > >> > >> joe.st...@stealth.ly> > >> > > > > > >> > >> > > > > wrote: > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > > Hi, created a KIP > >> > > > > > >> > >> > > > > > > > >> > > > > > >> > >> > > > > > > > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > >> > > > > > >> > >> > > >> > > > > > >> > >> > >> > > > > > >> > > >> > > > > > >> > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > >> > > > > > >> > >> > > > > > > > >> > > > > > >> > >> > > > > > > JIRA > >> > > > > https://issues.apache.org/jira/browse/KAFKA-1694 > >> > > > > > >> > >> > > > > > > > >> > > > > > >> > >> > > > > > > /******************************************* > >> > > > > > >> > >> > > > > > > Joe Stein > >> > > > > > >> > >> > > > > > > Founder, Principal Consultant > >> > > > > > >> > >> > > > > > > Big Data Open Source Security LLC > >> > > > > > >> > >> > > > > > > http://www.stealth.ly > >> > > > > > >> > >> > > > > > > Twitter: @allthingshadoop < > >> > > > > > >> > >> > http://www.twitter.com/allthingshadoop > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > > > > ********************************************/ > >> > > > > > >> > >> > > > > > > > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > >> > > > > > >> > >> > > >> > > > > > >> > >> > >> > > > > > >> > >> > >> > > > > > >> > >> > >> > > > > > >> > >> -- > >> > > > > > >> > >> -- Guozhang > >> > > > > > >> > >> > >> > > > > > >> > > > >> > > > > > >> > > > >> > > > > > >> > > >> > > > > > >> > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > >> > > >> > -- > >> > Jeff Holoman > >> > Systems Engineer > >> > > >>
