Thanks for sending this out Joe. Looking forward to chatting with everyone :)
On Mon, Mar 2, 2015 at 6:46 AM, Joe Stein <joe.st...@stealth.ly> wrote: > Hey, I just sent out a google hangout invite to all pmc, committers and > everyone I found working on a KIP. If I missed anyone in the invite please > let me know and can update it, np. > > We should do this every Tuesday @ 2pm Eastern Time. Maybe we can get INFRA > help to make a google account so we can manage better? > > To discuss > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > in progress and related JIRA that are interdependent and common work. > > ~ Joe Stein > > On Tue, Feb 24, 2015 at 2:59 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > >> Let's stay on Google hangouts that will also record and make the sessions >> available on youtube. >> >> -Jay >> >> On Tue, Feb 24, 2015 at 11:49 AM, Jeff Holoman <jholo...@cloudera.com> >> wrote: >> >> > Jay / Joe >> > >> > We're happy to send out a Webex for this purpose. We could record the >> > sessions if there is interest and publish them out. >> > >> > Thanks >> > >> > Jeff >> > >> > On Tue, Feb 24, 2015 at 11:28 AM, Jay Kreps <jay.kr...@gmail.com> wrote: >> > >> > > Let's try to get the technical hang-ups sorted out, though. I really >> > think >> > > there is some benefit to live discussion vs writing. I am hopeful that >> if >> > > we post instructions and give ourselves a few attempts we can get it >> > > working. >> > > >> > > Tuesday at that time would work for me...any objections? >> > > >> > > -Jay >> > > >> > > On Tue, Feb 24, 2015 at 8:18 AM, Joe Stein <joe.st...@stealth.ly> >> wrote: >> > > >> > > > Weekly would be great maybe like every Tuesday ~ 1pm ET / 10am PT >> ???? >> > > > >> > > > I don't mind google hangout but there is always some issue or >> whatever >> > so >> > > > we know the apache irc channel works. We can start there and see how >> it >> > > > goes? We can pull transcripts too and associate to tickets if need be >> > > makes >> > > > it helpful for things. >> > > > >> > > > ~ Joestein >> > > > >> > > > On Tue, Feb 24, 2015 at 11:10 AM, Jay Kreps <jay.kr...@gmail.com> >> > wrote: >> > > > >> > > > > We'd talked about doing a Google Hangout to chat about this. What >> > about >> > > > > generalizing that a little further...I actually think it would be >> > good >> > > > for >> > > > > everyone spending a reasonable chunk of their week on Kafka stuff >> to >> > > > maybe >> > > > > sync up once a week. I think we could use time to talk through >> design >> > > > > stuff, make sure we are on top of code reviews, talk through any >> > tricky >> > > > > issues, etc. >> > > > > >> > > > > We can make it publicly available so that any one can follow along >> > who >> > > > > likes. >> > > > > >> > > > > Any interest in doing this? If so I'll try to set it up starting >> next >> > > > week. >> > > > > >> > > > > -Jay >> > > > > >> > > > > On Tue, Feb 24, 2015 at 3:57 AM, Andrii Biletskyi < >> > > > > andrii.bilets...@stealth.ly> wrote: >> > > > > >> > > > > > Hi all, >> > > > > > >> > > > > > I've updated KIP page, fixed / aligned document structure. Also I >> > > added >> > > > > > some >> > > > > > very initial proposal for AdminClient so we have something to >> start >> > > > from >> > > > > > while >> > > > > > discussing the KIP. >> > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations >> > > > > > >> > > > > > Thanks, >> > > > > > Andrii Biletskyi >> > > > > > >> > > > > > On Wed, Feb 18, 2015 at 9:01 PM, Andrii Biletskyi < >> > > > > > andrii.bilets...@stealth.ly> wrote: >> > > > > > >> > > > > > > Jay, >> > > > > > > >> > > > > > > Re error messages: you are right, in most cases client will >> have >> > > > enough >> > > > > > > context to show descriptive error message. My concern is that >> we >> > > will >> > > > > > have >> > > > > > > to >> > > > > > > add lots of new error codes for each possible error. Of course, >> > we >> > > > > could >> > > > > > > reuse >> > > > > > > some of existing like UknownTopicOrPartitionCode, but we will >> > also >> > > > need >> > > > > > to >> > > > > > > add smth like: TopicAlreadyExistsCode, TopicConfigInvalid (both >> > for >> > > > > topic >> > > > > > > name and config, and probably user would like to know what >> > exactly >> > > > > > > is wrong in his config), InvalidReplicaAssignment, >> InternalError >> > > > (e.g. >> > > > > > > zookeeper failure) etc. >> > > > > > > And this is only for TopicCommand, we will also need to add >> > similar >> > > > > stuff >> > > > > > > for >> > > > > > > ReassignPartitions, PreferredReplica. So we'll end up with a >> > large >> > > > list >> > > > > > of >> > > > > > > error codes, used only in Admin protocol. >> > > > > > > Having said that, I agree my proposal is not consistent with >> > other >> > > > > cases. >> > > > > > > Maybe we can find better solution or something in-between. >> > > > > > > >> > > > > > > Re Hangout chat: I think it is a great idea. This way we can >> move >> > > on >> > > > > > > faster. >> > > > > > > Let's agree somehow on date/time so people can join. Will work >> > for >> > > me >> > > > > > this >> > > > > > > and >> > > > > > > next week almost anytime if agreed in advance. >> > > > > > > >> > > > > > > Thanks, >> > > > > > > Andrii >> > > > > > > >> > > > > > > On Wed, Feb 18, 2015 at 7:09 PM, Jay Kreps < >> jay.kr...@gmail.com> >> > > > > wrote: >> > > > > > > >> > > > > > >> Hey Andrii, >> > > > > > >> >> > > > > > >> Generally we can do good error handling without needing custom >> > > > > > server-side >> > > > > > >> messages. I.e. generally the client has the context to know >> that >> > > if >> > > > it >> > > > > > got >> > > > > > >> an error that the topic doesn't exist to say "Topic X doesn't >> > > exist" >> > > > > > >> rather >> > > > > > >> than "error code 14" (or whatever). Maybe there are specific >> > cases >> > > > > where >> > > > > > >> this is hard? If we want to add server-side error messages we >> > > really >> > > > > do >> > > > > > >> need to do this in a consistent way across the protocol. >> > > > > > >> >> > > > > > >> I still have a bunch of open questions here from my previous >> > > list. I >> > > > > > will >> > > > > > >> be out for the next few days for Strata though. Maybe we could >> > do >> > > a >> > > > > > Google >> > > > > > >> Hangout chat on any open issues some time towards the end of >> > next >> > > > week >> > > > > > for >> > > > > > >> anyone interested in this ticket? I have a feeling that might >> > > > progress >> > > > > > >> things a little faster than email--I think we could talk >> through >> > > > those >> > > > > > >> issues I brought up fairly quickly... >> > > > > > >> >> > > > > > >> -Jay >> > > > > > >> >> > > > > > >> On Wed, Feb 18, 2015 at 7:27 AM, Andrii Biletskyi < >> > > > > > >> andrii.bilets...@stealth.ly> wrote: >> > > > > > >> >> > > > > > >> > Hi all, >> > > > > > >> > >> > > > > > >> > I'm trying to address some of the issues which were >> mentioned >> > > > > earlier >> > > > > > >> about >> > > > > > >> > Admin RQ/RP format. One of those was about batching >> > operations. >> > > > What >> > > > > > if >> > > > > > >> we >> > > > > > >> > follow TopicCommand approach and let people specify >> topic-name >> > > by >> > > > > > >> regexp - >> > > > > > >> > would that cover most of the use cases? >> > > > > > >> > >> > > > > > >> > Secondly, is what information should we generally provide in >> > > Admin >> > > > > > >> > responses. >> > > > > > >> > I realize that Admin commands don't imply they will be used >> > only >> > > > in >> > > > > > CLI >> > > > > > >> > but, >> > > > > > >> > it seems to me, CLI is a very important client of this >> > feature. >> > > In >> > > > > > this >> > > > > > >> > case, >> > > > > > >> > seems logical, we would like to provide users with rich >> > > experience >> > > > > in >> > > > > > >> terms >> > > > > > >> > of >> > > > > > >> > getting results / errors of the executed commands. Usually >> we >> > > > supply >> > > > > > >> with >> > > > > > >> > responses only errorCode, which looks very limiting, in case >> > of >> > > > CLI >> > > > > we >> > > > > > >> may >> > > > > > >> > want to print human readable error description. >> > > > > > >> > >> > > > > > >> > So, taking into account previous item about batching, what >> do >> > > you >> > > > > > think >> > > > > > >> > about >> > > > > > >> > having smth like: >> > > > > > >> > >> > > > > > >> > ('create' doesn't support regexp) >> > > > > > >> > CreateTopicRequest => TopicName Partitions Replicas >> > > > > ReplicaAssignment >> > > > > > >> > [Config] >> > > > > > >> > CreateTopicResponse => ErrorCode ErrorDescription >> > > > > > >> > ErrorCode => int16 >> > > > > > >> > ErrorDescription => string (empty if successful) >> > > > > > >> > >> > > > > > >> > AlterTopicRequest -> TopicNameRegexp Partitions >> > > ReplicaAssignment >> > > > > > >> > [AddedConfig] [DeletedConfig] >> > > > > > >> > AlterTopicResponse -> [TopicName ErrorCode ErrorDescription] >> > > > > > >> > CommandErrorCode CommandErrorDescription >> > > > > > >> > CommandErrorCode => int16 >> > > > > > >> > CommandErrorDescription => string (nonempty in case of >> fatal >> > > > > error, >> > > > > > >> e.g. >> > > > > > >> > we couldn't get topics by regexp) >> > > > > > >> > >> > > > > > >> > DescribeTopicRequest -> TopicNameRegexp >> > > > > > >> > DescribeTopicResponse -> [TopicName TopicDescription >> ErrorCode >> > > > > > >> > ErrorDescription] CommandErrorCode CommandErrorDescription >> > > > > > >> > >> > > > > > >> > Also, any thoughts about our discussion regarding re-routing >> > > > > facility? >> > > > > > >> In >> > > > > > >> > my >> > > > > > >> > understanding, it is like between augmenting >> > > TopicMetadataRequest >> > > > > > >> > (to include at least controllerId) and implementing new >> > generic >> > > > > > >> re-routing >> > > > > > >> > facility so sending messages to controller will be handled >> by >> > > it. >> > > > > > >> > >> > > > > > >> > Thanks, >> > > > > > >> > Andrii Biletskyi >> > > > > > >> > >> > > > > > >> > On Mon, Feb 16, 2015 at 5:26 PM, Andrii Biletskyi < >> > > > > > >> > andrii.bilets...@stealth.ly> wrote: >> > > > > > >> > >> > > > > > >> > > @Guozhang: >> > > > > > >> > > Thanks for your comments, I've answered some of those. The >> > > main >> > > > > > thing >> > > > > > >> is >> > > > > > >> > > having merged request for create-alter-delete-describe - I >> > > have >> > > > > some >> > > > > > >> > > concerns about this approach. >> > > > > > >> > > >> > > > > > >> > > @*Jay*: >> > > > > > >> > > I see that introduced ClusterMetadaRequest is also one of >> > the >> > > > > > >> concerns. >> > > > > > >> > We >> > > > > > >> > > can solve it if we implement re-routing facility. But I >> > agree >> > > > with >> > > > > > >> > > Guozhang - it will make clients' internals a little bit >> > easier >> > > > but >> > > > > > >> this >> > > > > > >> > > seems to be a complex logic to implement and support then. >> > > > > > Especially >> > > > > > >> for >> > > > > > >> > > Fetch and Produce (even if we add re-routing later for >> these >> > > > > > >> requests). >> > > > > > >> > > Also people will tend to avoid this re-routing facility >> and >> > > hold >> > > > > > local >> > > > > > >> > > cluster cache to ensure their high-priority requests >> (which >> > > some >> > > > > of >> > > > > > >> the >> > > > > > >> > > admin requests are) not sent to some busy broker where >> they >> > > wait >> > > > > to >> > > > > > be >> > > > > > >> > > routed to the correct one. >> > > > > > >> > > As pointed out by Jun here ( >> > > > > > >> > > >> > > > > > >> > >> > > > > > >> >> > > > > > >> > > > > >> > > > >> > > >> > >> https://issues.apache.org/jira/browse/KAFKA-1772?focusedCommentId=14234530&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14234530 >> > > > > > >> > ) >> > > > > > >> > > to solve the issue we might introduce a message type to >> get >> > > > > cluster >> > > > > > >> > state. >> > > > > > >> > > But I agree we can just update TopicMetadataResponse to >> > > include >> > > > > > >> > > controllerId (and probably smth else). >> > > > > > >> > > What are you thougths? >> > > > > > >> > > >> > > > > > >> > > Thanks, >> > > > > > >> > > Andrii >> > > > > > >> > > >> > > > > > >> > > On Thu, Feb 12, 2015 at 8:31 AM, Guozhang Wang < >> > > > > wangg...@gmail.com> >> > > > > > >> > wrote: >> > > > > > >> > > >> > > > > > >> > >> I think for the topics commands we can actually merge >> > > > > > >> > >> create/alter/delete/describe as one request type since >> > their >> > > > > > formats >> > > > > > >> are >> > > > > > >> > >> very much similar, and keep list-topics and others like >> > > > > > >> > >> partition-reassignment / preferred-leader-election as >> > > separate >> > > > > > >> request >> > > > > > >> > >> types, I also left some other comments on the RB ( >> > > > > > >> > >> https://reviews.apache.org/r/29301/). >> > > > > > >> > >> >> > > > > > >> > >> On Wed, Feb 11, 2015 at 2:04 PM, Jay Kreps < >> > > > jay.kr...@gmail.com> >> > > > > > >> wrote: >> > > > > > >> > >> >> > > > > > >> > >> > Yeah I totally agree that we don't want to just have >> one >> > > "do >> > > > > > admin >> > > > > > >> > >> stuff" >> > > > > > >> > >> > command that has the union of all parameters. >> > > > > > >> > >> > >> > > > > > >> > >> > What I am saying is that command line tools are one >> > client >> > > of >> > > > > the >> > > > > > >> > >> > administrative apis, but these will be used in a number >> > of >> > > > > > >> scenarios >> > > > > > >> > so >> > > > > > >> > >> > they should make logical sense even in the absence of >> the >> > > > > command >> > > > > > >> line >> > > > > > >> > >> > tool. Hence comments like trying to clarify the >> > > relationship >> > > > > > >> between >> > > > > > >> > >> > ClusterMetadata and TopicMetadata...these kinds of >> things >> > > > > really >> > > > > > >> need >> > > > > > >> > >> to be >> > > > > > >> > >> > thought through. >> > > > > > >> > >> > >> > > > > > >> > >> > Hope that makes sense. >> > > > > > >> > >> > >> > > > > > >> > >> > -Jay >> > > > > > >> > >> > >> > > > > > >> > >> > On Wed, Feb 11, 2015 at 1:41 PM, Andrii Biletskyi < >> > > > > > >> > >> > andrii.bilets...@stealth.ly> wrote: >> > > > > > >> > >> > >> > > > > > >> > >> > > Jay, >> > > > > > >> > >> > > >> > > > > > >> > >> > > Thanks for answering. You understood correctly, most >> of >> > > my >> > > > > > >> comments >> > > > > > >> > >> were >> > > > > > >> > >> > > related to your point 1) - about "well thought-out" >> > apis. >> > > > > Also, >> > > > > > >> yes, >> > > > > > >> > >> as I >> > > > > > >> > >> > > understood we would like to introduce a single >> unified >> > > CLI >> > > > > tool >> > > > > > >> with >> > > > > > >> > >> > > centralized server-side request handling for lots of >> > > > existing >> > > > > > >> ones >> > > > > > >> > >> (incl. >> > > > > > >> > >> > > TopicCommand, CommitOffsetChecker, >> ReassignPartitions, >> > > smth >> > > > > > else >> > > > > > >> if >> > > > > > >> > >> added >> > > > > > >> > >> > > in future). In our previous discussion ( >> > > > > > >> > >> > > https://issues.apache.org/jira/browse/KAFKA-1694) >> > people >> > > > > said >> > > > > > >> > they'd >> > > > > > >> > >> > > rather >> > > > > > >> > >> > > have a separate message for each command, so, yes, >> this >> > > > way I >> > > > > > >> came >> > > > > > >> > to >> > > > > > >> > >> 1-1 >> > > > > > >> > >> > > mapping between commands in the tool and protocol >> > > > additions. >> > > > > > But >> > > > > > >> I >> > > > > > >> > >> might >> > > > > > >> > >> > be >> > > > > > >> > >> > > wrong. >> > > > > > >> > >> > > At the end I just try to start discussion how at >> least >> > > > > > generally >> > > > > > >> > this >> > > > > > >> > >> > > protocol should look like. >> > > > > > >> > >> > > >> > > > > > >> > >> > > Thanks, >> > > > > > >> > >> > > Andrii >> > > > > > >> > >> > > >> > > > > > >> > >> > > On Wed, Feb 11, 2015 at 11:10 PM, Jay Kreps < >> > > > > > jay.kr...@gmail.com >> > > > > > >> > >> > > > > > >> > >> wrote: >> > > > > > >> > >> > > >> > > > > > >> > >> > > > Hey Andrii, >> > > > > > >> > >> > > > >> > > > > > >> > >> > > > To answer your earlier question we just really >> can't >> > be >> > > > > > adding >> > > > > > >> any >> > > > > > >> > >> more >> > > > > > >> > >> > > > scala protocol objects. These things are super hard >> > to >> > > > > > maintain >> > > > > > >> > >> because >> > > > > > >> > >> > > > they hand code the byte parsing and don't have good >> > > > > > versioning >> > > > > > >> > >> support. >> > > > > > >> > >> > > > Since we are already planning on converting we >> > > definitely >> > > > > > don't >> > > > > > >> > >> want to >> > > > > > >> > >> > > add >> > > > > > >> > >> > > > a ton more of these--they are total tech debt. >> > > > > > >> > >> > > > >> > > > > > >> > >> > > > What does it mean that the changes are isolated >> from >> > > the >> > > > > > >> current >> > > > > > >> > >> code >> > > > > > >> > >> > > base? >> > > > > > >> > >> > > > >> > > > > > >> > >> > > > I actually didn't understand the remaining >> comments, >> > > > which >> > > > > of >> > > > > > >> the >> > > > > > >> > >> > points >> > > > > > >> > >> > > > are you responding to? >> > > > > > >> > >> > > > >> > > > > > >> > >> > > > Maybe one sticking point here is that it seems like >> > you >> > > > > want >> > > > > > to >> > > > > > >> > make >> > > > > > >> > >> > some >> > > > > > >> > >> > > > kind of tool, and you have made a 1-1 mapping >> between >> > > > > > commands >> > > > > > >> you >> > > > > > >> > >> > > imagine >> > > > > > >> > >> > > > in the tool and protocol additions. I want to make >> > sure >> > > > we >> > > > > > >> don't >> > > > > > >> > do >> > > > > > >> > >> > that. >> > > > > > >> > >> > > > The protocol needs to be really really well thought >> > out >> > > > > > against >> > > > > > >> > many >> > > > > > >> > >> > use >> > > > > > >> > >> > > > cases so it should make perfect logical sense in >> the >> > > > > absence >> > > > > > of >> > > > > > >> > >> knowing >> > > > > > >> > >> > > the >> > > > > > >> > >> > > > command line tool, right? >> > > > > > >> > >> > > > >> > > > > > >> > >> > > > -Jay >> > > > > > >> > >> > > > >> > > > > > >> > >> > > > On Wed, Feb 11, 2015 at 11:57 AM, Andrii Biletskyi >> < >> > > > > > >> > >> > > > andrii.bilets...@stealth.ly> wrote: >> > > > > > >> > >> > > > >> > > > > > >> > >> > > > > Hey Jay, >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > > I would like to continue this discussion as it >> seem >> > > > there >> > > > > > is >> > > > > > >> no >> > > > > > >> > >> > > progress >> > > > > > >> > >> > > > > here. >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > > First of all, could you please explain what did >> you >> > > > mean >> > > > > in >> > > > > > >> 2? >> > > > > > >> > How >> > > > > > >> > >> > > > exactly >> > > > > > >> > >> > > > > are we going to migrate to the new java protocol >> > > > > > definitions. >> > > > > > >> > And >> > > > > > >> > >> why >> > > > > > >> > >> > > > it's >> > > > > > >> > >> > > > > a blocker for centralized CLI? >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > > I agree with you, this feature includes lots of >> > > stuff, >> > > > > but >> > > > > > >> > >> thankfully >> > > > > > >> > >> > > > > almost all changes are isolated from the current >> > code >> > > > > base, >> > > > > > >> > >> > > > > so the main thing, I think, we need to agree is >> > RQ/RP >> > > > > > format. >> > > > > > >> > >> > > > > So how can we start discussion about the concrete >> > > > > messages >> > > > > > >> > format? >> > > > > > >> > >> > > > > Can we take ( >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > >> > > > > > >> > >> > > >> > > > > > >> > >> > >> > > > > > >> > >> >> > > > > > >> > >> > > > > > >> >> > > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-ProposedRQ/RPFormat >> > > > > > >> > >> > > > > ) >> > > > > > >> > >> > > > > as starting point? >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > > We had some doubts earlier whether it worth >> > > introducing >> > > > > one >> > > > > > >> > >> generic >> > > > > > >> > >> > > Admin >> > > > > > >> > >> > > > > Request for all commands ( >> > > > > > >> > >> > > > https://issues.apache.org/jira/browse/KAFKA-1694 >> > > > > > >> > >> > > > > ) >> > > > > > >> > >> > > > > but then everybody agreed it would be better to >> > have >> > > > > > separate >> > > > > > >> > >> message >> > > > > > >> > >> > > for >> > > > > > >> > >> > > > > each admin command. The Request part is really >> > > dictated >> > > > > > from >> > > > > > >> the >> > > > > > >> > >> > > command >> > > > > > >> > >> > > > > (e.g. TopicCommand) arguments itself, so the >> > proposed >> > > > > > version >> > > > > > >> > >> should >> > > > > > >> > >> > be >> > > > > > >> > >> > > > > fine (let's put aside for now remarks about >> > Optional >> > > > > type, >> > > > > > >> > >> batching, >> > > > > > >> > >> > > > > configs normalization - I agree with all of >> them). >> > > > > > >> > >> > > > > So the second part is Response. I see there are >> two >> > > > cases >> > > > > > >> here. >> > > > > > >> > >> > > > > a) "Mutate" requests - Create/Alter/... ; b) >> "Get" >> > > > > > requests - >> > > > > > >> > >> > > > > List/Describe... >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > > a) should only hold request result (regardless >> what >> > > we >> > > > > > decide >> > > > > > >> > >> about >> > > > > > >> > >> > > > > blocking/non-blocking commands execution). >> > > > > > >> > >> > > > > Usually we provide error code in response but >> since >> > > we >> > > > > will >> > > > > > >> use >> > > > > > >> > >> this >> > > > > > >> > >> > in >> > > > > > >> > >> > > > > interactive shell we need some human readable >> error >> > > > > > >> description >> > > > > > >> > - >> > > > > > >> > >> so >> > > > > > >> > >> > I >> > > > > > >> > >> > > > > added errorDesription field where you can at >> least >> > > > leave >> > > > > > >> > >> > > > > exception.getMessage. >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > > b) in addition to previous item message should >> hold >> > > > > command >> > > > > > >> > >> specific >> > > > > > >> > >> > > > > response data. We can discuss in detail each of >> > them >> > > > but >> > > > > > >> let's >> > > > > > >> > for >> > > > > > >> > >> > now >> > > > > > >> > >> > > > > agree about the overall pattern. >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > > Thanks, >> > > > > > >> > >> > > > > Andrii Biletskyi >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > > On Fri, Jan 23, 2015 at 6:59 AM, Jay Kreps < >> > > > > > >> jay.kr...@gmail.com >> > > > > > >> > > >> > > > > > >> > >> > > wrote: >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > > > Hey Joe, >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > > This is great. A few comments on KIP-4 >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > > 1. This is much needed functionality, but there >> > > are a >> > > > > lot >> > > > > > >> of >> > > > > > >> > >> the so >> > > > > > >> > >> > > > let's >> > > > > > >> > >> > > > > > really think these protocols through. We really >> > > want >> > > > to >> > > > > > >> end up >> > > > > > >> > >> > with a >> > > > > > >> > >> > > > set >> > > > > > >> > >> > > > > > of well thought-out, orthoganol apis. For this >> > > > reason I >> > > > > > >> think >> > > > > > >> > >> it is >> > > > > > >> > >> > > > > really >> > > > > > >> > >> > > > > > important to think through the end state even >> if >> > > that >> > > > > > >> includes >> > > > > > >> > >> APIs >> > > > > > >> > >> > > we >> > > > > > >> > >> > > > > > won't implement in the first phase. >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > > 2. Let's please please please wait until we >> have >> > > > > switched >> > > > > > >> the >> > > > > > >> > >> > server >> > > > > > >> > >> > > > over >> > > > > > >> > >> > > > > > to the new java protocol definitions. If we add >> > > > upteen >> > > > > > >> more ad >> > > > > > >> > >> hoc >> > > > > > >> > >> > > > scala >> > > > > > >> > >> > > > > > objects that is just generating more work for >> the >> > > > > > >> conversion >> > > > > > >> > we >> > > > > > >> > >> > know >> > > > > > >> > >> > > we >> > > > > > >> > >> > > > > > have to do. >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > > 3. This proposal introduces a new type of >> > optional >> > > > > > >> parameter. >> > > > > > >> > >> This >> > > > > > >> > >> > is >> > > > > > >> > >> > > > > > inconsistent with everything else in the >> protocol >> > > > where >> > > > > > we >> > > > > > >> use >> > > > > > >> > >> -1 >> > > > > > >> > >> > or >> > > > > > >> > >> > > > some >> > > > > > >> > >> > > > > > other marker value. You could argue either way >> > but >> > > > > let's >> > > > > > >> stick >> > > > > > >> > >> with >> > > > > > >> > >> > > > that >> > > > > > >> > >> > > > > > for consistency. For clients that implemented >> the >> > > > > > protocol >> > > > > > >> in >> > > > > > >> > a >> > > > > > >> > >> > > better >> > > > > > >> > >> > > > > way >> > > > > > >> > >> > > > > > than our scala code these basic primitives are >> > hard >> > > > to >> > > > > > >> change. >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > > 4. ClusterMetadata: This seems to duplicate >> > > > > > >> > TopicMetadataRequest >> > > > > > >> > >> > > which >> > > > > > >> > >> > > > > has >> > > > > > >> > >> > > > > > brokers, topics, and partitions. I think we >> > should >> > > > > rename >> > > > > > >> that >> > > > > > >> > >> > > request >> > > > > > >> > >> > > > > > ClusterMetadataRequest (or just >> MetadataRequest) >> > > and >> > > > > > >> include >> > > > > > >> > >> the id >> > > > > > >> > >> > > of >> > > > > > >> > >> > > > > the >> > > > > > >> > >> > > > > > controller. Or are there other things we could >> > add >> > > > > here? >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > > 5. We have a tendency to try to make a lot of >> > > > requests >> > > > > > that >> > > > > > >> > can >> > > > > > >> > >> > only >> > > > > > >> > >> > > go >> > > > > > >> > >> > > > > to >> > > > > > >> > >> > > > > > particular nodes. This adds a lot of burden for >> > > > client >> > > > > > >> > >> > > implementations >> > > > > > >> > >> > > > > (it >> > > > > > >> > >> > > > > > sounds easy but each discovery can fail in many >> > > parts >> > > > > so >> > > > > > it >> > > > > > >> > >> ends up >> > > > > > >> > >> > > > > being a >> > > > > > >> > >> > > > > > full state machine to do right). I think we >> > should >> > > > > > consider >> > > > > > >> > >> making >> > > > > > >> > >> > > > admin >> > > > > > >> > >> > > > > > commands and ideally as many of the other apis >> as >> > > > > > possible >> > > > > > >> > >> > available >> > > > > > >> > >> > > on >> > > > > > >> > >> > > > > all >> > > > > > >> > >> > > > > > brokers and just redirect to the controller on >> > the >> > > > > broker >> > > > > > >> > side. >> > > > > > >> > >> > > Perhaps >> > > > > > >> > >> > > > > > there would be a general way to encapsulate >> this >> > > > > > re-routing >> > > > > > >> > >> > behavior. >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > > 6. We should probably normalize the key value >> > pairs >> > > > > used >> > > > > > >> for >> > > > > > >> > >> > configs >> > > > > > >> > >> > > > > rather >> > > > > > >> > >> > > > > > than embedding a new formatting. So two strings >> > > > rather >> > > > > > than >> > > > > > >> > one >> > > > > > >> > >> > with >> > > > > > >> > >> > > an >> > > > > > >> > >> > > > > > internal equals sign. >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > > 7. Is the postcondition of these APIs that the >> > > > command >> > > > > > has >> > > > > > >> > >> begun or >> > > > > > >> > >> > > > that >> > > > > > >> > >> > > > > > the command has been completed? It is a lot >> more >> > > > usable >> > > > > > if >> > > > > > >> the >> > > > > > >> > >> > > command >> > > > > > >> > >> > > > > has >> > > > > > >> > >> > > > > > been completed so you know that if you create a >> > > topic >> > > > > and >> > > > > > >> then >> > > > > > >> > >> > > publish >> > > > > > >> > >> > > > to >> > > > > > >> > >> > > > > > it you won't get an exception about there being >> > no >> > > > such >> > > > > > >> topic. >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > > 8. Describe topic and list topics duplicate a >> lot >> > > of >> > > > > > stuff >> > > > > > >> in >> > > > > > >> > >> the >> > > > > > >> > >> > > > > metadata >> > > > > > >> > >> > > > > > request. Is there a reason to give back topics >> > > marked >> > > > > for >> > > > > > >> > >> > deletion? I >> > > > > > >> > >> > > > > feel >> > > > > > >> > >> > > > > > like if we just make the post-condition of the >> > > delete >> > > > > > >> command >> > > > > > >> > be >> > > > > > >> > >> > that >> > > > > > >> > >> > > > the >> > > > > > >> > >> > > > > > topic is deleted that will get rid of the need >> > for >> > > > this >> > > > > > >> right? >> > > > > > >> > >> And >> > > > > > >> > >> > it >> > > > > > >> > >> > > > > will >> > > > > > >> > >> > > > > > be much more intuitive. >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > > 9. Should we consider batching these requests? >> We >> > > > have >> > > > > > >> > generally >> > > > > > >> > >> > > tried >> > > > > > >> > >> > > > to >> > > > > > >> > >> > > > > > allow multiple operations to be batched. My >> > > suspicion >> > > > > is >> > > > > > >> that >> > > > > > >> > >> > without >> > > > > > >> > >> > > > > this >> > > > > > >> > >> > > > > > we will get a lot of code that does something >> > like >> > > > > > >> > >> > > > > > for(topic: adminClient.listTopics()) >> > > > > > >> > >> > > > > > adminClient.describeTopic(topic) >> > > > > > >> > >> > > > > > this code will work great when you test on 5 >> > topics >> > > > but >> > > > > > >> not do >> > > > > > >> > >> as >> > > > > > >> > >> > > well >> > > > > > >> > >> > > > if >> > > > > > >> > >> > > > > > you have 50k. >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > > 10. I think we should also discuss how we want >> to >> > > > > expose >> > > > > > a >> > > > > > >> > >> > > programmatic >> > > > > > >> > >> > > > > JVM >> > > > > > >> > >> > > > > > client api for these operations. Currently >> people >> > > > rely >> > > > > on >> > > > > > >> > >> > AdminUtils >> > > > > > >> > >> > > > > which >> > > > > > >> > >> > > > > > is totally sketchy. I think we probably need >> > > another >> > > > > > client >> > > > > > >> > >> under >> > > > > > >> > >> > > > > clients/ >> > > > > > >> > >> > > > > > that exposes administrative functionality. We >> > will >> > > > need >> > > > > > >> this >> > > > > > >> > >> just >> > > > > > >> > >> > to >> > > > > > >> > >> > > > > > properly test the new apis, I suspect. We >> should >> > > > figure >> > > > > > out >> > > > > > >> > that >> > > > > > >> > >> > API. >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > > 11. The other information that would be really >> > > useful >> > > > > to >> > > > > > >> get >> > > > > > >> > >> would >> > > > > > >> > >> > be >> > > > > > >> > >> > > > > > information about partitions--how much data is >> in >> > > the >> > > > > > >> > partition, >> > > > > > >> > >> > what >> > > > > > >> > >> > > > are >> > > > > > >> > >> > > > > > the segment offsets, what is the log-end offset >> > > (i.e. >> > > > > > last >> > > > > > >> > >> offset), >> > > > > > >> > >> > > > what >> > > > > > >> > >> > > > > is >> > > > > > >> > >> > > > > > the compaction point, etc. I think that done >> > right >> > > > this >> > > > > > >> would >> > > > > > >> > be >> > > > > > >> > >> > the >> > > > > > >> > >> > > > > > successor to the very awkward OffsetRequest we >> > have >> > > > > > today. >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > > -Jay >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > > On Wed, Jan 21, 2015 at 10:27 PM, Joe Stein < >> > > > > > >> > >> joe.st...@stealth.ly> >> > > > > > >> > >> > > > > wrote: >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > > > Hi, created a KIP >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > >> > > > > > >> > >> > > >> > > > > > >> > >> > >> > > > > > >> > >> >> > > > > > >> > >> > > > > > >> >> > > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > > JIRA >> > > > > https://issues.apache.org/jira/browse/KAFKA-1694 >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > > /******************************************* >> > > > > > >> > >> > > > > > > Joe Stein >> > > > > > >> > >> > > > > > > Founder, Principal Consultant >> > > > > > >> > >> > > > > > > Big Data Open Source Security LLC >> > > > > > >> > >> > > > > > > http://www.stealth.ly >> > > > > > >> > >> > > > > > > Twitter: @allthingshadoop < >> > > > > > >> > >> > http://www.twitter.com/allthingshadoop >> > > > > > >> > >> > > > >> > > > > > >> > >> > > > > > > ********************************************/ >> > > > > > >> > >> > > > > > > >> > > > > > >> > >> > > > > > >> > > > > > >> > >> > > > > >> > > > > > >> > >> > > > >> > > > > > >> > >> > > >> > > > > > >> > >> > >> > > > > > >> > >> >> > > > > > >> > >> >> > > > > > >> > >> >> > > > > > >> > >> -- >> > > > > > >> > >> -- Guozhang >> > > > > > >> > >> >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > >> > > > > > >> >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >> > >> > -- >> > Jeff Holoman >> > Systems Engineer >> > >>
