Found KIP-11 (https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface) It actually specifies changes to the Metadata protocol, so making sure both KIPs are consistent in this regard will be good.
On Thu, Mar 12, 2015 at 12:21 PM, Gwen Shapira <gshap...@cloudera.com> wrote: > Specifically for ownership, I think the plan is to add ACL (it sounds > like you are describing ACL) via an external system (Argus, Sentry). > I remember KIP-11 described this, but I can't find the KIP any longer. > > Regardless, I think KIP-4 focuses on getting information that already > exists from Kafka brokers, not on adding information that perhaps > should exist but doesn't yet? > > Gwen > > > > > > On Thu, Mar 12, 2015 at 6:37 AM, Guozhang Wang <wangg...@gmail.com> wrote: >> Folks, >> >> Just want to elaborate a bit more on the create-topic metadata and batching >> describe-topic based on config / metadata in my previous email as we work >> on KAFKA-1694. The main motivation is to have some sort of topic management >> mechanisms, which I think is quite important in a multi-tenant / cloud >> architecture: today anyone can create topics in a shared Kafka cluster, but >> there is no concept or "ownership" of topics that are created by different >> users. For example, at LinkedIn we basically distinguish topic owners via >> some casual topic name prefix, which is a bit awkward and does not fly as >> we scale our customers. It would be great to use describe-topics such as: >> >> Describe all topics that is created by me. >> >> Describe all topics whose retention time is overriden to X. >> >> Describe all topics whose writable group include user Y (this is related to >> authorization), etc.. >> >> One possible way to achieve this is to add a metadata file in the >> create-topic request, whose value will also be written ZK as we create the >> topic; then describe-topics can choose to batch topics based on 1) name >> regex, 2) config K-V matching, 3) metadata regex, etc. >> >> Thoughts? >> >> Guozhang >> >> On Thu, Mar 5, 2015 at 4:37 PM, Guozhang Wang <wangg...@gmail.com> wrote: >> >>> Thanks for the updated wiki. A few comments below: >>> >>> 1. Error description in response: I think if some errorCode could indicate >>> several different error cases then we should really change it to multiple >>> codes. In general the errorCode itself would be precise and sufficient for >>> describing the server side errors. >>> >>> 2. Describe topic request: it would be great to go beyond just batching on >>> topic name regex for this request. For example, a very common use case of >>> the topic command is to list all topics whose config A's value is B. With >>> topic name regex then we have to first retrieve __all__ topics's >>> description info and then filter at the client end, which will be a huge >>> burden on ZK. >>> >>> 3. Config K-Vs in create topic: this is related to the previous point; >>> maybe we can add another metadata K-V or just a metadata string along side >>> with config K-V in create topic like we did for offset commit request. This >>> field can be quite useful in storing information like "owner" of the topic >>> who issue the create command, etc, which is quite important for a >>> multi-tenant setting. Then in the describe topic request we can also batch >>> on regex of the metadata field. >>> >>> 4. Today all the admin operations are async in the sense that command will >>> return once it is written in ZK, and that is why we need extra verification >>> like testUtil.waitForTopicCreated() / verify partition reassignment >>> request, etc. With admin requests we could add a flag to enable / disable >>> synchronous requests; when it is turned on, the response will not return >>> until the request has been completed. And for async requests we can add a >>> "token" field in the response, and then only need a general "admin >>> verification request" with the given token to check if the async request >>> has been completed. >>> >>> 5. +1 for extending Metadata request to include controller / coordinator >>> information, and then we can remove the ConsumerMetadata / ClusterMetadata >>> requests. >>> >>> Guozhang >>> >>> On Tue, Mar 3, 2015 at 10:23 AM, Joel Koshy <jjkosh...@gmail.com> wrote: >>> >>>> Thanks for sending that out Joe - I don't think I will be able to make >>>> it today, so if notes can be sent out afterward that would be great. >>>> >>>> On Mon, Mar 02, 2015 at 09:16:13AM -0800, Gwen Shapira wrote: >>>> > Thanks for sending this out Joe. Looking forward to chatting with >>>> everyone :) >>>> > >>>> > On Mon, Mar 2, 2015 at 6:46 AM, Joe Stein <joe.st...@stealth.ly> wrote: >>>> > > Hey, I just sent out a google hangout invite to all pmc, committers >>>> and >>>> > > everyone I found working on a KIP. If I missed anyone in the invite >>>> please >>>> > > let me know and can update it, np. >>>> > > >>>> > > We should do this every Tuesday @ 2pm Eastern Time. Maybe we can get >>>> INFRA >>>> > > help to make a google account so we can manage better? >>>> > > >>>> > > To discuss >>>> > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals >>>> > > in progress and related JIRA that are interdependent and common work. >>>> > > >>>> > > ~ Joe Stein >>>> > > >>>> > > On Tue, Feb 24, 2015 at 2:59 PM, Jay Kreps <jay.kr...@gmail.com> >>>> wrote: >>>> > > >>>> > >> Let's stay on Google hangouts that will also record and make the >>>> sessions >>>> > >> available on youtube. >>>> > >> >>>> > >> -Jay >>>> > >> >>>> > >> On Tue, Feb 24, 2015 at 11:49 AM, Jeff Holoman < >>>> jholo...@cloudera.com> >>>> > >> wrote: >>>> > >> >>>> > >> > Jay / Joe >>>> > >> > >>>> > >> > We're happy to send out a Webex for this purpose. We could record >>>> the >>>> > >> > sessions if there is interest and publish them out. >>>> > >> > >>>> > >> > Thanks >>>> > >> > >>>> > >> > Jeff >>>> > >> > >>>> > >> > On Tue, Feb 24, 2015 at 11:28 AM, Jay Kreps <jay.kr...@gmail.com> >>>> wrote: >>>> > >> > >>>> > >> > > Let's try to get the technical hang-ups sorted out, though. I >>>> really >>>> > >> > think >>>> > >> > > there is some benefit to live discussion vs writing. I am >>>> hopeful that >>>> > >> if >>>> > >> > > we post instructions and give ourselves a few attempts we can >>>> get it >>>> > >> > > working. >>>> > >> > > >>>> > >> > > Tuesday at that time would work for me...any objections? >>>> > >> > > >>>> > >> > > -Jay >>>> > >> > > >>>> > >> > > On Tue, Feb 24, 2015 at 8:18 AM, Joe Stein <joe.st...@stealth.ly >>>> > >>>> > >> wrote: >>>> > >> > > >>>> > >> > > > Weekly would be great maybe like every Tuesday ~ 1pm ET / 10am >>>> PT >>>> > >> ???? >>>> > >> > > > >>>> > >> > > > I don't mind google hangout but there is always some issue or >>>> > >> whatever >>>> > >> > so >>>> > >> > > > we know the apache irc channel works. We can start there and >>>> see how >>>> > >> it >>>> > >> > > > goes? We can pull transcripts too and associate to tickets if >>>> need be >>>> > >> > > makes >>>> > >> > > > it helpful for things. >>>> > >> > > > >>>> > >> > > > ~ Joestein >>>> > >> > > > >>>> > >> > > > On Tue, Feb 24, 2015 at 11:10 AM, Jay Kreps < >>>> jay.kr...@gmail.com> >>>> > >> > wrote: >>>> > >> > > > >>>> > >> > > > > We'd talked about doing a Google Hangout to chat about this. >>>> What >>>> > >> > about >>>> > >> > > > > generalizing that a little further...I actually think it >>>> would be >>>> > >> > good >>>> > >> > > > for >>>> > >> > > > > everyone spending a reasonable chunk of their week on Kafka >>>> stuff >>>> > >> to >>>> > >> > > > maybe >>>> > >> > > > > sync up once a week. I think we could use time to talk >>>> through >>>> > >> design >>>> > >> > > > > stuff, make sure we are on top of code reviews, talk through >>>> any >>>> > >> > tricky >>>> > >> > > > > issues, etc. >>>> > >> > > > > >>>> > >> > > > > We can make it publicly available so that any one can follow >>>> along >>>> > >> > who >>>> > >> > > > > likes. >>>> > >> > > > > >>>> > >> > > > > Any interest in doing this? If so I'll try to set it up >>>> starting >>>> > >> next >>>> > >> > > > week. >>>> > >> > > > > >>>> > >> > > > > -Jay >>>> > >> > > > > >>>> > >> > > > > On Tue, Feb 24, 2015 at 3:57 AM, Andrii Biletskyi < >>>> > >> > > > > andrii.bilets...@stealth.ly> wrote: >>>> > >> > > > > >>>> > >> > > > > > Hi all, >>>> > >> > > > > > >>>> > >> > > > > > I've updated KIP page, fixed / aligned document structure. >>>> Also I >>>> > >> > > added >>>> > >> > > > > > some >>>> > >> > > > > > very initial proposal for AdminClient so we have something >>>> to >>>> > >> start >>>> > >> > > > from >>>> > >> > > > > > while >>>> > >> > > > > > discussing the KIP. >>>> > >> > > > > > >>>> > >> > > > > > >>>> > >> > > > > > >>>> > >> > > > > >>>> > >> > > > >>>> > >> > > >>>> > >> > >>>> > >> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations >>>> > >> > > > > > >>>> > >> > > > > > Thanks, >>>> > >> > > > > > Andrii Biletskyi >>>> > >> > > > > > >>>> > >> > > > > > On Wed, Feb 18, 2015 at 9:01 PM, Andrii Biletskyi < >>>> > >> > > > > > andrii.bilets...@stealth.ly> wrote: >>>> > >> > > > > > >>>> > >> > > > > > > Jay, >>>> > >> > > > > > > >>>> > >> > > > > > > Re error messages: you are right, in most cases client >>>> will >>>> > >> have >>>> > >> > > > enough >>>> > >> > > > > > > context to show descriptive error message. My concern is >>>> that >>>> > >> we >>>> > >> > > will >>>> > >> > > > > > have >>>> > >> > > > > > > to >>>> > >> > > > > > > add lots of new error codes for each possible error. Of >>>> course, >>>> > >> > we >>>> > >> > > > > could >>>> > >> > > > > > > reuse >>>> > >> > > > > > > some of existing like UknownTopicOrPartitionCode, but we >>>> will >>>> > >> > also >>>> > >> > > > need >>>> > >> > > > > > to >>>> > >> > > > > > > add smth like: TopicAlreadyExistsCode, >>>> TopicConfigInvalid (both >>>> > >> > for >>>> > >> > > > > topic >>>> > >> > > > > > > name and config, and probably user would like to know >>>> what >>>> > >> > exactly >>>> > >> > > > > > > is wrong in his config), InvalidReplicaAssignment, >>>> > >> InternalError >>>> > >> > > > (e.g. >>>> > >> > > > > > > zookeeper failure) etc. >>>> > >> > > > > > > And this is only for TopicCommand, we will also need to >>>> add >>>> > >> > similar >>>> > >> > > > > stuff >>>> > >> > > > > > > for >>>> > >> > > > > > > ReassignPartitions, PreferredReplica. So we'll end up >>>> with a >>>> > >> > large >>>> > >> > > > list >>>> > >> > > > > > of >>>> > >> > > > > > > error codes, used only in Admin protocol. >>>> > >> > > > > > > Having said that, I agree my proposal is not consistent >>>> with >>>> > >> > other >>>> > >> > > > > cases. >>>> > >> > > > > > > Maybe we can find better solution or something >>>> in-between. >>>> > >> > > > > > > >>>> > >> > > > > > > Re Hangout chat: I think it is a great idea. This way we >>>> can >>>> > >> move >>>> > >> > > on >>>> > >> > > > > > > faster. >>>> > >> > > > > > > Let's agree somehow on date/time so people can join. >>>> Will work >>>> > >> > for >>>> > >> > > me >>>> > >> > > > > > this >>>> > >> > > > > > > and >>>> > >> > > > > > > next week almost anytime if agreed in advance. >>>> > >> > > > > > > >>>> > >> > > > > > > Thanks, >>>> > >> > > > > > > Andrii >>>> > >> > > > > > > >>>> > >> > > > > > > On Wed, Feb 18, 2015 at 7:09 PM, Jay Kreps < >>>> > >> jay.kr...@gmail.com> >>>> > >> > > > > wrote: >>>> > >> > > > > > > >>>> > >> > > > > > >> Hey Andrii, >>>> > >> > > > > > >> >>>> > >> > > > > > >> Generally we can do good error handling without needing >>>> custom >>>> > >> > > > > > server-side >>>> > >> > > > > > >> messages. I.e. generally the client has the context to >>>> know >>>> > >> that >>>> > >> > > if >>>> > >> > > > it >>>> > >> > > > > > got >>>> > >> > > > > > >> an error that the topic doesn't exist to say "Topic X >>>> doesn't >>>> > >> > > exist" >>>> > >> > > > > > >> rather >>>> > >> > > > > > >> than "error code 14" (or whatever). Maybe there are >>>> specific >>>> > >> > cases >>>> > >> > > > > where >>>> > >> > > > > > >> this is hard? If we want to add server-side error >>>> messages we >>>> > >> > > really >>>> > >> > > > > do >>>> > >> > > > > > >> need to do this in a consistent way across the protocol. >>>> > >> > > > > > >> >>>> > >> > > > > > >> I still have a bunch of open questions here from my >>>> previous >>>> > >> > > list. I >>>> > >> > > > > > will >>>> > >> > > > > > >> be out for the next few days for Strata though. Maybe >>>> we could >>>> > >> > do >>>> > >> > > a >>>> > >> > > > > > Google >>>> > >> > > > > > >> Hangout chat on any open issues some time towards the >>>> end of >>>> > >> > next >>>> > >> > > > week >>>> > >> > > > > > for >>>> > >> > > > > > >> anyone interested in this ticket? I have a feeling that >>>> might >>>> > >> > > > progress >>>> > >> > > > > > >> things a little faster than email--I think we could talk >>>> > >> through >>>> > >> > > > those >>>> > >> > > > > > >> issues I brought up fairly quickly... >>>> > >> > > > > > >> >>>> > >> > > > > > >> -Jay >>>> > >> > > > > > >> >>>> > >> > > > > > >> On Wed, Feb 18, 2015 at 7:27 AM, Andrii Biletskyi < >>>> > >> > > > > > >> andrii.bilets...@stealth.ly> wrote: >>>> > >> > > > > > >> >>>> > >> > > > > > >> > Hi all, >>>> > >> > > > > > >> > >>>> > >> > > > > > >> > I'm trying to address some of the issues which were >>>> > >> mentioned >>>> > >> > > > > earlier >>>> > >> > > > > > >> about >>>> > >> > > > > > >> > Admin RQ/RP format. One of those was about batching >>>> > >> > operations. >>>> > >> > > > What >>>> > >> > > > > > if >>>> > >> > > > > > >> we >>>> > >> > > > > > >> > follow TopicCommand approach and let people specify >>>> > >> topic-name >>>> > >> > > by >>>> > >> > > > > > >> regexp - >>>> > >> > > > > > >> > would that cover most of the use cases? >>>> > >> > > > > > >> > >>>> > >> > > > > > >> > Secondly, is what information should we generally >>>> provide in >>>> > >> > > Admin >>>> > >> > > > > > >> > responses. >>>> > >> > > > > > >> > I realize that Admin commands don't imply they will >>>> be used >>>> > >> > only >>>> > >> > > > in >>>> > >> > > > > > CLI >>>> > >> > > > > > >> > but, >>>> > >> > > > > > >> > it seems to me, CLI is a very important client of this >>>> > >> > feature. >>>> > >> > > In >>>> > >> > > > > > this >>>> > >> > > > > > >> > case, >>>> > >> > > > > > >> > seems logical, we would like to provide users with >>>> rich >>>> > >> > > experience >>>> > >> > > > > in >>>> > >> > > > > > >> terms >>>> > >> > > > > > >> > of >>>> > >> > > > > > >> > getting results / errors of the executed commands. >>>> Usually >>>> > >> we >>>> > >> > > > supply >>>> > >> > > > > > >> with >>>> > >> > > > > > >> > responses only errorCode, which looks very limiting, >>>> in case >>>> > >> > of >>>> > >> > > > CLI >>>> > >> > > > > we >>>> > >> > > > > > >> may >>>> > >> > > > > > >> > want to print human readable error description. >>>> > >> > > > > > >> > >>>> > >> > > > > > >> > So, taking into account previous item about batching, >>>> what >>>> > >> do >>>> > >> > > you >>>> > >> > > > > > think >>>> > >> > > > > > >> > about >>>> > >> > > > > > >> > having smth like: >>>> > >> > > > > > >> > >>>> > >> > > > > > >> > ('create' doesn't support regexp) >>>> > >> > > > > > >> > CreateTopicRequest => TopicName Partitions Replicas >>>> > >> > > > > ReplicaAssignment >>>> > >> > > > > > >> > [Config] >>>> > >> > > > > > >> > CreateTopicResponse => ErrorCode ErrorDescription >>>> > >> > > > > > >> > ErrorCode => int16 >>>> > >> > > > > > >> > ErrorDescription => string (empty if successful) >>>> > >> > > > > > >> > >>>> > >> > > > > > >> > AlterTopicRequest -> TopicNameRegexp Partitions >>>> > >> > > ReplicaAssignment >>>> > >> > > > > > >> > [AddedConfig] [DeletedConfig] >>>> > >> > > > > > >> > AlterTopicResponse -> [TopicName ErrorCode >>>> ErrorDescription] >>>> > >> > > > > > >> > CommandErrorCode CommandErrorDescription >>>> > >> > > > > > >> > CommandErrorCode => int16 >>>> > >> > > > > > >> > CommandErrorDescription => string (nonempty in case >>>> of >>>> > >> fatal >>>> > >> > > > > error, >>>> > >> > > > > > >> e.g. >>>> > >> > > > > > >> > we couldn't get topics by regexp) >>>> > >> > > > > > >> > >>>> > >> > > > > > >> > DescribeTopicRequest -> TopicNameRegexp >>>> > >> > > > > > >> > DescribeTopicResponse -> [TopicName TopicDescription >>>> > >> ErrorCode >>>> > >> > > > > > >> > ErrorDescription] CommandErrorCode >>>> CommandErrorDescription >>>> > >> > > > > > >> > >>>> > >> > > > > > >> > Also, any thoughts about our discussion regarding >>>> re-routing >>>> > >> > > > > facility? >>>> > >> > > > > > >> In >>>> > >> > > > > > >> > my >>>> > >> > > > > > >> > understanding, it is like between augmenting >>>> > >> > > TopicMetadataRequest >>>> > >> > > > > > >> > (to include at least controllerId) and implementing >>>> new >>>> > >> > generic >>>> > >> > > > > > >> re-routing >>>> > >> > > > > > >> > facility so sending messages to controller will be >>>> handled >>>> > >> by >>>> > >> > > it. >>>> > >> > > > > > >> > >>>> > >> > > > > > >> > Thanks, >>>> > >> > > > > > >> > Andrii Biletskyi >>>> > >> > > > > > >> > >>>> > >> > > > > > >> > On Mon, Feb 16, 2015 at 5:26 PM, Andrii Biletskyi < >>>> > >> > > > > > >> > andrii.bilets...@stealth.ly> wrote: >>>> > >> > > > > > >> > >>>> > >> > > > > > >> > > @Guozhang: >>>> > >> > > > > > >> > > Thanks for your comments, I've answered some of >>>> those. The >>>> > >> > > main >>>> > >> > > > > > thing >>>> > >> > > > > > >> is >>>> > >> > > > > > >> > > having merged request for >>>> create-alter-delete-describe - I >>>> > >> > > have >>>> > >> > > > > some >>>> > >> > > > > > >> > > concerns about this approach. >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > > @*Jay*: >>>> > >> > > > > > >> > > I see that introduced ClusterMetadaRequest is also >>>> one of >>>> > >> > the >>>> > >> > > > > > >> concerns. >>>> > >> > > > > > >> > We >>>> > >> > > > > > >> > > can solve it if we implement re-routing facility. >>>> But I >>>> > >> > agree >>>> > >> > > > with >>>> > >> > > > > > >> > > Guozhang - it will make clients' internals a little >>>> bit >>>> > >> > easier >>>> > >> > > > but >>>> > >> > > > > > >> this >>>> > >> > > > > > >> > > seems to be a complex logic to implement and >>>> support then. >>>> > >> > > > > > Especially >>>> > >> > > > > > >> for >>>> > >> > > > > > >> > > Fetch and Produce (even if we add re-routing later >>>> for >>>> > >> these >>>> > >> > > > > > >> requests). >>>> > >> > > > > > >> > > Also people will tend to avoid this re-routing >>>> facility >>>> > >> and >>>> > >> > > hold >>>> > >> > > > > > local >>>> > >> > > > > > >> > > cluster cache to ensure their high-priority requests >>>> > >> (which >>>> > >> > > some >>>> > >> > > > > of >>>> > >> > > > > > >> the >>>> > >> > > > > > >> > > admin requests are) not sent to some busy broker >>>> where >>>> > >> they >>>> > >> > > wait >>>> > >> > > > > to >>>> > >> > > > > > be >>>> > >> > > > > > >> > > routed to the correct one. >>>> > >> > > > > > >> > > As pointed out by Jun here ( >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > >>>> > >> > > > > > >> >>>> > >> > > > > > >>>> > >> > > > > >>>> > >> > > > >>>> > >> > > >>>> > >> > >>>> > >> >>>> https://issues.apache.org/jira/browse/KAFKA-1772?focusedCommentId=14234530&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14234530 >>>> > >> > > > > > >> > ) >>>> > >> > > > > > >> > > to solve the issue we might introduce a message >>>> type to >>>> > >> get >>>> > >> > > > > cluster >>>> > >> > > > > > >> > state. >>>> > >> > > > > > >> > > But I agree we can just update >>>> TopicMetadataResponse to >>>> > >> > > include >>>> > >> > > > > > >> > > controllerId (and probably smth else). >>>> > >> > > > > > >> > > What are you thougths? >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > > Thanks, >>>> > >> > > > > > >> > > Andrii >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > > On Thu, Feb 12, 2015 at 8:31 AM, Guozhang Wang < >>>> > >> > > > > wangg...@gmail.com> >>>> > >> > > > > > >> > wrote: >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > >> I think for the topics commands we can actually >>>> merge >>>> > >> > > > > > >> > >> create/alter/delete/describe as one request type >>>> since >>>> > >> > their >>>> > >> > > > > > formats >>>> > >> > > > > > >> are >>>> > >> > > > > > >> > >> very much similar, and keep list-topics and others >>>> like >>>> > >> > > > > > >> > >> partition-reassignment / preferred-leader-election >>>> as >>>> > >> > > separate >>>> > >> > > > > > >> request >>>> > >> > > > > > >> > >> types, I also left some other comments on the RB ( >>>> > >> > > > > > >> > >> https://reviews.apache.org/r/29301/). >>>> > >> > > > > > >> > >> >>>> > >> > > > > > >> > >> On Wed, Feb 11, 2015 at 2:04 PM, Jay Kreps < >>>> > >> > > > jay.kr...@gmail.com> >>>> > >> > > > > > >> wrote: >>>> > >> > > > > > >> > >> >>>> > >> > > > > > >> > >> > Yeah I totally agree that we don't want to just >>>> have >>>> > >> one >>>> > >> > > "do >>>> > >> > > > > > admin >>>> > >> > > > > > >> > >> stuff" >>>> > >> > > > > > >> > >> > command that has the union of all parameters. >>>> > >> > > > > > >> > >> > >>>> > >> > > > > > >> > >> > What I am saying is that command line tools are >>>> one >>>> > >> > client >>>> > >> > > of >>>> > >> > > > > the >>>> > >> > > > > > >> > >> > administrative apis, but these will be used in a >>>> number >>>> > >> > of >>>> > >> > > > > > >> scenarios >>>> > >> > > > > > >> > so >>>> > >> > > > > > >> > >> > they should make logical sense even in the >>>> absence of >>>> > >> the >>>> > >> > > > > command >>>> > >> > > > > > >> line >>>> > >> > > > > > >> > >> > tool. Hence comments like trying to clarify the >>>> > >> > > relationship >>>> > >> > > > > > >> between >>>> > >> > > > > > >> > >> > ClusterMetadata and TopicMetadata...these kinds >>>> of >>>> > >> things >>>> > >> > > > > really >>>> > >> > > > > > >> need >>>> > >> > > > > > >> > >> to be >>>> > >> > > > > > >> > >> > thought through. >>>> > >> > > > > > >> > >> > >>>> > >> > > > > > >> > >> > Hope that makes sense. >>>> > >> > > > > > >> > >> > >>>> > >> > > > > > >> > >> > -Jay >>>> > >> > > > > > >> > >> > >>>> > >> > > > > > >> > >> > On Wed, Feb 11, 2015 at 1:41 PM, Andrii >>>> Biletskyi < >>>> > >> > > > > > >> > >> > andrii.bilets...@stealth.ly> wrote: >>>> > >> > > > > > >> > >> > >>>> > >> > > > > > >> > >> > > Jay, >>>> > >> > > > > > >> > >> > > >>>> > >> > > > > > >> > >> > > Thanks for answering. You understood >>>> correctly, most >>>> > >> of >>>> > >> > > my >>>> > >> > > > > > >> comments >>>> > >> > > > > > >> > >> were >>>> > >> > > > > > >> > >> > > related to your point 1) - about "well >>>> thought-out" >>>> > >> > apis. >>>> > >> > > > > Also, >>>> > >> > > > > > >> yes, >>>> > >> > > > > > >> > >> as I >>>> > >> > > > > > >> > >> > > understood we would like to introduce a single >>>> > >> unified >>>> > >> > > CLI >>>> > >> > > > > tool >>>> > >> > > > > > >> with >>>> > >> > > > > > >> > >> > > centralized server-side request handling for >>>> lots of >>>> > >> > > > existing >>>> > >> > > > > > >> ones >>>> > >> > > > > > >> > >> (incl. >>>> > >> > > > > > >> > >> > > TopicCommand, CommitOffsetChecker, >>>> > >> ReassignPartitions, >>>> > >> > > smth >>>> > >> > > > > > else >>>> > >> > > > > > >> if >>>> > >> > > > > > >> > >> added >>>> > >> > > > > > >> > >> > > in future). In our previous discussion ( >>>> > >> > > > > > >> > >> > > >>>> https://issues.apache.org/jira/browse/KAFKA-1694) >>>> > >> > people >>>> > >> > > > > said >>>> > >> > > > > > >> > they'd >>>> > >> > > > > > >> > >> > > rather >>>> > >> > > > > > >> > >> > > have a separate message for each command, so, >>>> yes, >>>> > >> this >>>> > >> > > > way I >>>> > >> > > > > > >> came >>>> > >> > > > > > >> > to >>>> > >> > > > > > >> > >> 1-1 >>>> > >> > > > > > >> > >> > > mapping between commands in the tool and >>>> protocol >>>> > >> > > > additions. >>>> > >> > > > > > But >>>> > >> > > > > > >> I >>>> > >> > > > > > >> > >> might >>>> > >> > > > > > >> > >> > be >>>> > >> > > > > > >> > >> > > wrong. >>>> > >> > > > > > >> > >> > > At the end I just try to start discussion how >>>> at >>>> > >> least >>>> > >> > > > > > generally >>>> > >> > > > > > >> > this >>>> > >> > > > > > >> > >> > > protocol should look like. >>>> > >> > > > > > >> > >> > > >>>> > >> > > > > > >> > >> > > Thanks, >>>> > >> > > > > > >> > >> > > Andrii >>>> > >> > > > > > >> > >> > > >>>> > >> > > > > > >> > >> > > On Wed, Feb 11, 2015 at 11:10 PM, Jay Kreps < >>>> > >> > > > > > jay.kr...@gmail.com >>>> > >> > > > > > >> > >>>> > >> > > > > > >> > >> wrote: >>>> > >> > > > > > >> > >> > > >>>> > >> > > > > > >> > >> > > > Hey Andrii, >>>> > >> > > > > > >> > >> > > > >>>> > >> > > > > > >> > >> > > > To answer your earlier question we just >>>> really >>>> > >> can't >>>> > >> > be >>>> > >> > > > > > adding >>>> > >> > > > > > >> any >>>> > >> > > > > > >> > >> more >>>> > >> > > > > > >> > >> > > > scala protocol objects. These things are >>>> super hard >>>> > >> > to >>>> > >> > > > > > maintain >>>> > >> > > > > > >> > >> because >>>> > >> > > > > > >> > >> > > > they hand code the byte parsing and don't >>>> have good >>>> > >> > > > > > versioning >>>> > >> > > > > > >> > >> support. >>>> > >> > > > > > >> > >> > > > Since we are already planning on converting >>>> we >>>> > >> > > definitely >>>> > >> > > > > > don't >>>> > >> > > > > > >> > >> want to >>>> > >> > > > > > >> > >> > > add >>>> > >> > > > > > >> > >> > > > a ton more of these--they are total tech >>>> debt. >>>> > >> > > > > > >> > >> > > > >>>> > >> > > > > > >> > >> > > > What does it mean that the changes are >>>> isolated >>>> > >> from >>>> > >> > > the >>>> > >> > > > > > >> current >>>> > >> > > > > > >> > >> code >>>> > >> > > > > > >> > >> > > base? >>>> > >> > > > > > >> > >> > > > >>>> > >> > > > > > >> > >> > > > I actually didn't understand the remaining >>>> > >> comments, >>>> > >> > > > which >>>> > >> > > > > of >>>> > >> > > > > > >> the >>>> > >> > > > > > >> > >> > points >>>> > >> > > > > > >> > >> > > > are you responding to? >>>> > >> > > > > > >> > >> > > > >>>> > >> > > > > > >> > >> > > > Maybe one sticking point here is that it >>>> seems like >>>> > >> > you >>>> > >> > > > > want >>>> > >> > > > > > to >>>> > >> > > > > > >> > make >>>> > >> > > > > > >> > >> > some >>>> > >> > > > > > >> > >> > > > kind of tool, and you have made a 1-1 mapping >>>> > >> between >>>> > >> > > > > > commands >>>> > >> > > > > > >> you >>>> > >> > > > > > >> > >> > > imagine >>>> > >> > > > > > >> > >> > > > in the tool and protocol additions. I want >>>> to make >>>> > >> > sure >>>> > >> > > > we >>>> > >> > > > > > >> don't >>>> > >> > > > > > >> > do >>>> > >> > > > > > >> > >> > that. >>>> > >> > > > > > >> > >> > > > The protocol needs to be really really well >>>> thought >>>> > >> > out >>>> > >> > > > > > against >>>> > >> > > > > > >> > many >>>> > >> > > > > > >> > >> > use >>>> > >> > > > > > >> > >> > > > cases so it should make perfect logical >>>> sense in >>>> > >> the >>>> > >> > > > > absence >>>> > >> > > > > > of >>>> > >> > > > > > >> > >> knowing >>>> > >> > > > > > >> > >> > > the >>>> > >> > > > > > >> > >> > > > command line tool, right? >>>> > >> > > > > > >> > >> > > > >>>> > >> > > > > > >> > >> > > > -Jay >>>> > >> > > > > > >> > >> > > > >>>> > >> > > > > > >> > >> > > > On Wed, Feb 11, 2015 at 11:57 AM, Andrii >>>> Biletskyi >>>> > >> < >>>> > >> > > > > > >> > >> > > > andrii.bilets...@stealth.ly> wrote: >>>> > >> > > > > > >> > >> > > > >>>> > >> > > > > > >> > >> > > > > Hey Jay, >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > > I would like to continue this discussion >>>> as it >>>> > >> seem >>>> > >> > > > there >>>> > >> > > > > > is >>>> > >> > > > > > >> no >>>> > >> > > > > > >> > >> > > progress >>>> > >> > > > > > >> > >> > > > > here. >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > > First of all, could you please explain >>>> what did >>>> > >> you >>>> > >> > > > mean >>>> > >> > > > > in >>>> > >> > > > > > >> 2? >>>> > >> > > > > > >> > How >>>> > >> > > > > > >> > >> > > > exactly >>>> > >> > > > > > >> > >> > > > > are we going to migrate to the new java >>>> protocol >>>> > >> > > > > > definitions. >>>> > >> > > > > > >> > And >>>> > >> > > > > > >> > >> why >>>> > >> > > > > > >> > >> > > > it's >>>> > >> > > > > > >> > >> > > > > a blocker for centralized CLI? >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > > I agree with you, this feature includes >>>> lots of >>>> > >> > > stuff, >>>> > >> > > > > but >>>> > >> > > > > > >> > >> thankfully >>>> > >> > > > > > >> > >> > > > > almost all changes are isolated from the >>>> current >>>> > >> > code >>>> > >> > > > > base, >>>> > >> > > > > > >> > >> > > > > so the main thing, I think, we need to >>>> agree is >>>> > >> > RQ/RP >>>> > >> > > > > > format. >>>> > >> > > > > > >> > >> > > > > So how can we start discussion about the >>>> concrete >>>> > >> > > > > messages >>>> > >> > > > > > >> > format? >>>> > >> > > > > > >> > >> > > > > Can we take ( >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > >>>> > >> > > > > > >> > >> > > >>>> > >> > > > > > >> > >> > >>>> > >> > > > > > >> > >> >>>> > >> > > > > > >> > >>>> > >> > > > > > >> >>>> > >> > > > > > >>>> > >> > > > > >>>> > >> > > > >>>> > >> > > >>>> > >> > >>>> > >> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-ProposedRQ/RPFormat >>>> > >> > > > > > >> > >> > > > > ) >>>> > >> > > > > > >> > >> > > > > as starting point? >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > > We had some doubts earlier whether it worth >>>> > >> > > introducing >>>> > >> > > > > one >>>> > >> > > > > > >> > >> generic >>>> > >> > > > > > >> > >> > > Admin >>>> > >> > > > > > >> > >> > > > > Request for all commands ( >>>> > >> > > > > > >> > >> > > > >>>> https://issues.apache.org/jira/browse/KAFKA-1694 >>>> > >> > > > > > >> > >> > > > > ) >>>> > >> > > > > > >> > >> > > > > but then everybody agreed it would be >>>> better to >>>> > >> > have >>>> > >> > > > > > separate >>>> > >> > > > > > >> > >> message >>>> > >> > > > > > >> > >> > > for >>>> > >> > > > > > >> > >> > > > > each admin command. The Request part is >>>> really >>>> > >> > > dictated >>>> > >> > > > > > from >>>> > >> > > > > > >> the >>>> > >> > > > > > >> > >> > > command >>>> > >> > > > > > >> > >> > > > > (e.g. TopicCommand) arguments itself, so >>>> the >>>> > >> > proposed >>>> > >> > > > > > version >>>> > >> > > > > > >> > >> should >>>> > >> > > > > > >> > >> > be >>>> > >> > > > > > >> > >> > > > > fine (let's put aside for now remarks about >>>> > >> > Optional >>>> > >> > > > > type, >>>> > >> > > > > > >> > >> batching, >>>> > >> > > > > > >> > >> > > > > configs normalization - I agree with all of >>>> > >> them). >>>> > >> > > > > > >> > >> > > > > So the second part is Response. I see >>>> there are >>>> > >> two >>>> > >> > > > cases >>>> > >> > > > > > >> here. >>>> > >> > > > > > >> > >> > > > > a) "Mutate" requests - Create/Alter/... ; >>>> b) >>>> > >> "Get" >>>> > >> > > > > > requests - >>>> > >> > > > > > >> > >> > > > > List/Describe... >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > > a) should only hold request result >>>> (regardless >>>> > >> what >>>> > >> > > we >>>> > >> > > > > > decide >>>> > >> > > > > > >> > >> about >>>> > >> > > > > > >> > >> > > > > blocking/non-blocking commands execution). >>>> > >> > > > > > >> > >> > > > > Usually we provide error code in response >>>> but >>>> > >> since >>>> > >> > > we >>>> > >> > > > > will >>>> > >> > > > > > >> use >>>> > >> > > > > > >> > >> this >>>> > >> > > > > > >> > >> > in >>>> > >> > > > > > >> > >> > > > > interactive shell we need some human >>>> readable >>>> > >> error >>>> > >> > > > > > >> description >>>> > >> > > > > > >> > - >>>> > >> > > > > > >> > >> so >>>> > >> > > > > > >> > >> > I >>>> > >> > > > > > >> > >> > > > > added errorDesription field where you can >>>> at >>>> > >> least >>>> > >> > > > leave >>>> > >> > > > > > >> > >> > > > > exception.getMessage. >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > > b) in addition to previous item message >>>> should >>>> > >> hold >>>> > >> > > > > command >>>> > >> > > > > > >> > >> specific >>>> > >> > > > > > >> > >> > > > > response data. We can discuss in detail >>>> each of >>>> > >> > them >>>> > >> > > > but >>>> > >> > > > > > >> let's >>>> > >> > > > > > >> > for >>>> > >> > > > > > >> > >> > now >>>> > >> > > > > > >> > >> > > > > agree about the overall pattern. >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > > Thanks, >>>> > >> > > > > > >> > >> > > > > Andrii Biletskyi >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > > On Fri, Jan 23, 2015 at 6:59 AM, Jay Kreps >>>> < >>>> > >> > > > > > >> jay.kr...@gmail.com >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > >> > > wrote: >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > > > Hey Joe, >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > > This is great. A few comments on KIP-4 >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > > 1. This is much needed functionality, >>>> but there >>>> > >> > > are a >>>> > >> > > > > lot >>>> > >> > > > > > >> of >>>> > >> > > > > > >> > >> the so >>>> > >> > > > > > >> > >> > > > let's >>>> > >> > > > > > >> > >> > > > > > really think these protocols through. We >>>> really >>>> > >> > > want >>>> > >> > > > to >>>> > >> > > > > > >> end up >>>> > >> > > > > > >> > >> > with a >>>> > >> > > > > > >> > >> > > > set >>>> > >> > > > > > >> > >> > > > > > of well thought-out, orthoganol apis. >>>> For this >>>> > >> > > > reason I >>>> > >> > > > > > >> think >>>> > >> > > > > > >> > >> it is >>>> > >> > > > > > >> > >> > > > > really >>>> > >> > > > > > >> > >> > > > > > important to think through the end state >>>> even >>>> > >> if >>>> > >> > > that >>>> > >> > > > > > >> includes >>>> > >> > > > > > >> > >> APIs >>>> > >> > > > > > >> > >> > > we >>>> > >> > > > > > >> > >> > > > > > won't implement in the first phase. >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > > 2. Let's please please please wait until >>>> we >>>> > >> have >>>> > >> > > > > switched >>>> > >> > > > > > >> the >>>> > >> > > > > > >> > >> > server >>>> > >> > > > > > >> > >> > > > over >>>> > >> > > > > > >> > >> > > > > > to the new java protocol definitions. If >>>> we add >>>> > >> > > > upteen >>>> > >> > > > > > >> more ad >>>> > >> > > > > > >> > >> hoc >>>> > >> > > > > > >> > >> > > > scala >>>> > >> > > > > > >> > >> > > > > > objects that is just generating more >>>> work for >>>> > >> the >>>> > >> > > > > > >> conversion >>>> > >> > > > > > >> > we >>>> > >> > > > > > >> > >> > know >>>> > >> > > > > > >> > >> > > we >>>> > >> > > > > > >> > >> > > > > > have to do. >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > > 3. This proposal introduces a new type of >>>> > >> > optional >>>> > >> > > > > > >> parameter. >>>> > >> > > > > > >> > >> This >>>> > >> > > > > > >> > >> > is >>>> > >> > > > > > >> > >> > > > > > inconsistent with everything else in the >>>> > >> protocol >>>> > >> > > > where >>>> > >> > > > > > we >>>> > >> > > > > > >> use >>>> > >> > > > > > >> > >> -1 >>>> > >> > > > > > >> > >> > or >>>> > >> > > > > > >> > >> > > > some >>>> > >> > > > > > >> > >> > > > > > other marker value. You could argue >>>> either way >>>> > >> > but >>>> > >> > > > > let's >>>> > >> > > > > > >> stick >>>> > >> > > > > > >> > >> with >>>> > >> > > > > > >> > >> > > > that >>>> > >> > > > > > >> > >> > > > > > for consistency. For clients that >>>> implemented >>>> > >> the >>>> > >> > > > > > protocol >>>> > >> > > > > > >> in >>>> > >> > > > > > >> > a >>>> > >> > > > > > >> > >> > > better >>>> > >> > > > > > >> > >> > > > > way >>>> > >> > > > > > >> > >> > > > > > than our scala code these basic >>>> primitives are >>>> > >> > hard >>>> > >> > > > to >>>> > >> > > > > > >> change. >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > > 4. ClusterMetadata: This seems to >>>> duplicate >>>> > >> > > > > > >> > TopicMetadataRequest >>>> > >> > > > > > >> > >> > > which >>>> > >> > > > > > >> > >> > > > > has >>>> > >> > > > > > >> > >> > > > > > brokers, topics, and partitions. I think >>>> we >>>> > >> > should >>>> > >> > > > > rename >>>> > >> > > > > > >> that >>>> > >> > > > > > >> > >> > > request >>>> > >> > > > > > >> > >> > > > > > ClusterMetadataRequest (or just >>>> > >> MetadataRequest) >>>> > >> > > and >>>> > >> > > > > > >> include >>>> > >> > > > > > >> > >> the id >>>> > >> > > > > > >> > >> > > of >>>> > >> > > > > > >> > >> > > > > the >>>> > >> > > > > > >> > >> > > > > > controller. Or are there other things we >>>> could >>>> > >> > add >>>> > >> > > > > here? >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > > 5. We have a tendency to try to make a >>>> lot of >>>> > >> > > > requests >>>> > >> > > > > > that >>>> > >> > > > > > >> > can >>>> > >> > > > > > >> > >> > only >>>> > >> > > > > > >> > >> > > go >>>> > >> > > > > > >> > >> > > > > to >>>> > >> > > > > > >> > >> > > > > > particular nodes. This adds a lot of >>>> burden for >>>> > >> > > > client >>>> > >> > > > > > >> > >> > > implementations >>>> > >> > > > > > >> > >> > > > > (it >>>> > >> > > > > > >> > >> > > > > > sounds easy but each discovery can fail >>>> in many >>>> > >> > > parts >>>> > >> > > > > so >>>> > >> > > > > > it >>>> > >> > > > > > >> > >> ends up >>>> > >> > > > > > >> > >> > > > > being a >>>> > >> > > > > > >> > >> > > > > > full state machine to do right). I think >>>> we >>>> > >> > should >>>> > >> > > > > > consider >>>> > >> > > > > > >> > >> making >>>> > >> > > > > > >> > >> > > > admin >>>> > >> > > > > > >> > >> > > > > > commands and ideally as many of the >>>> other apis >>>> > >> as >>>> > >> > > > > > possible >>>> > >> > > > > > >> > >> > available >>>> > >> > > > > > >> > >> > > on >>>> > >> > > > > > >> > >> > > > > all >>>> > >> > > > > > >> > >> > > > > > brokers and just redirect to the >>>> controller on >>>> > >> > the >>>> > >> > > > > broker >>>> > >> > > > > > >> > side. >>>> > >> > > > > > >> > >> > > Perhaps >>>> > >> > > > > > >> > >> > > > > > there would be a general way to >>>> encapsulate >>>> > >> this >>>> > >> > > > > > re-routing >>>> > >> > > > > > >> > >> > behavior. >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > > 6. We should probably normalize the key >>>> value >>>> > >> > pairs >>>> > >> > > > > used >>>> > >> > > > > > >> for >>>> > >> > > > > > >> > >> > configs >>>> > >> > > > > > >> > >> > > > > rather >>>> > >> > > > > > >> > >> > > > > > than embedding a new formatting. So two >>>> strings >>>> > >> > > > rather >>>> > >> > > > > > than >>>> > >> > > > > > >> > one >>>> > >> > > > > > >> > >> > with >>>> > >> > > > > > >> > >> > > an >>>> > >> > > > > > >> > >> > > > > > internal equals sign. >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > > 7. Is the postcondition of these APIs >>>> that the >>>> > >> > > > command >>>> > >> > > > > > has >>>> > >> > > > > > >> > >> begun or >>>> > >> > > > > > >> > >> > > > that >>>> > >> > > > > > >> > >> > > > > > the command has been completed? It is a >>>> lot >>>> > >> more >>>> > >> > > > usable >>>> > >> > > > > > if >>>> > >> > > > > > >> the >>>> > >> > > > > > >> > >> > > command >>>> > >> > > > > > >> > >> > > > > has >>>> > >> > > > > > >> > >> > > > > > been completed so you know that if you >>>> create a >>>> > >> > > topic >>>> > >> > > > > and >>>> > >> > > > > > >> then >>>> > >> > > > > > >> > >> > > publish >>>> > >> > > > > > >> > >> > > > to >>>> > >> > > > > > >> > >> > > > > > it you won't get an exception about >>>> there being >>>> > >> > no >>>> > >> > > > such >>>> > >> > > > > > >> topic. >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > > 8. Describe topic and list topics >>>> duplicate a >>>> > >> lot >>>> > >> > > of >>>> > >> > > > > > stuff >>>> > >> > > > > > >> in >>>> > >> > > > > > >> > >> the >>>> > >> > > > > > >> > >> > > > > metadata >>>> > >> > > > > > >> > >> > > > > > request. Is there a reason to give back >>>> topics >>>> > >> > > marked >>>> > >> > > > > for >>>> > >> > > > > > >> > >> > deletion? I >>>> > >> > > > > > >> > >> > > > > feel >>>> > >> > > > > > >> > >> > > > > > like if we just make the post-condition >>>> of the >>>> > >> > > delete >>>> > >> > > > > > >> command >>>> > >> > > > > > >> > be >>>> > >> > > > > > >> > >> > that >>>> > >> > > > > > >> > >> > > > the >>>> > >> > > > > > >> > >> > > > > > topic is deleted that will get rid of >>>> the need >>>> > >> > for >>>> > >> > > > this >>>> > >> > > > > > >> right? >>>> > >> > > > > > >> > >> And >>>> > >> > > > > > >> > >> > it >>>> > >> > > > > > >> > >> > > > > will >>>> > >> > > > > > >> > >> > > > > > be much more intuitive. >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > > 9. Should we consider batching these >>>> requests? >>>> > >> We >>>> > >> > > > have >>>> > >> > > > > > >> > generally >>>> > >> > > > > > >> > >> > > tried >>>> > >> > > > > > >> > >> > > > to >>>> > >> > > > > > >> > >> > > > > > allow multiple operations to be batched. >>>> My >>>> > >> > > suspicion >>>> > >> > > > > is >>>> > >> > > > > > >> that >>>> > >> > > > > > >> > >> > without >>>> > >> > > > > > >> > >> > > > > this >>>> > >> > > > > > >> > >> > > > > > we will get a lot of code that does >>>> something >>>> > >> > like >>>> > >> > > > > > >> > >> > > > > > for(topic: adminClient.listTopics()) >>>> > >> > > > > > >> > >> > > > > > adminClient.describeTopic(topic) >>>> > >> > > > > > >> > >> > > > > > this code will work great when you test >>>> on 5 >>>> > >> > topics >>>> > >> > > > but >>>> > >> > > > > > >> not do >>>> > >> > > > > > >> > >> as >>>> > >> > > > > > >> > >> > > well >>>> > >> > > > > > >> > >> > > > if >>>> > >> > > > > > >> > >> > > > > > you have 50k. >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > > 10. I think we should also discuss how >>>> we want >>>> > >> to >>>> > >> > > > > expose >>>> > >> > > > > > a >>>> > >> > > > > > >> > >> > > programmatic >>>> > >> > > > > > >> > >> > > > > JVM >>>> > >> > > > > > >> > >> > > > > > client api for these operations. >>>> Currently >>>> > >> people >>>> > >> > > > rely >>>> > >> > > > > on >>>> > >> > > > > > >> > >> > AdminUtils >>>> > >> > > > > > >> > >> > > > > which >>>> > >> > > > > > >> > >> > > > > > is totally sketchy. I think we probably >>>> need >>>> > >> > > another >>>> > >> > > > > > client >>>> > >> > > > > > >> > >> under >>>> > >> > > > > > >> > >> > > > > clients/ >>>> > >> > > > > > >> > >> > > > > > that exposes administrative >>>> functionality. We >>>> > >> > will >>>> > >> > > > need >>>> > >> > > > > > >> this >>>> > >> > > > > > >> > >> just >>>> > >> > > > > > >> > >> > to >>>> > >> > > > > > >> > >> > > > > > properly test the new apis, I suspect. We >>>> > >> should >>>> > >> > > > figure >>>> > >> > > > > > out >>>> > >> > > > > > >> > that >>>> > >> > > > > > >> > >> > API. >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > > 11. The other information that would be >>>> really >>>> > >> > > useful >>>> > >> > > > > to >>>> > >> > > > > > >> get >>>> > >> > > > > > >> > >> would >>>> > >> > > > > > >> > >> > be >>>> > >> > > > > > >> > >> > > > > > information about partitions--how much >>>> data is >>>> > >> in >>>> > >> > > the >>>> > >> > > > > > >> > partition, >>>> > >> > > > > > >> > >> > what >>>> > >> > > > > > >> > >> > > > are >>>> > >> > > > > > >> > >> > > > > > the segment offsets, what is the log-end >>>> offset >>>> > >> > > (i.e. >>>> > >> > > > > > last >>>> > >> > > > > > >> > >> offset), >>>> > >> > > > > > >> > >> > > > what >>>> > >> > > > > > >> > >> > > > > is >>>> > >> > > > > > >> > >> > > > > > the compaction point, etc. I think that >>>> done >>>> > >> > right >>>> > >> > > > this >>>> > >> > > > > > >> would >>>> > >> > > > > > >> > be >>>> > >> > > > > > >> > >> > the >>>> > >> > > > > > >> > >> > > > > > successor to the very awkward >>>> OffsetRequest we >>>> > >> > have >>>> > >> > > > > > today. >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > > -Jay >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > > On Wed, Jan 21, 2015 at 10:27 PM, Joe >>>> Stein < >>>> > >> > > > > > >> > >> joe.st...@stealth.ly> >>>> > >> > > > > > >> > >> > > > > wrote: >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > > > Hi, created a KIP >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > >>>> > >> > > > > > >> > >> > > >>>> > >> > > > > > >> > >> > >>>> > >> > > > > > >> > >> >>>> > >> > > > > > >> > >>>> > >> > > > > > >> >>>> > >> > > > > > >>>> > >> > > > > >>>> > >> > > > >>>> > >> > > >>>> > >> > >>>> > >> >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > > JIRA >>>> > >> > > > > https://issues.apache.org/jira/browse/KAFKA-1694 >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > > >>>> /******************************************* >>>> > >> > > > > > >> > >> > > > > > > Joe Stein >>>> > >> > > > > > >> > >> > > > > > > Founder, Principal Consultant >>>> > >> > > > > > >> > >> > > > > > > Big Data Open Source Security LLC >>>> > >> > > > > > >> > >> > > > > > > http://www.stealth.ly >>>> > >> > > > > > >> > >> > > > > > > Twitter: @allthingshadoop < >>>> > >> > > > > > >> > >> > http://www.twitter.com/allthingshadoop >>>> > >> > > > > > >> > >> > > > >>>> > >> > > > > > >> > >> > > > > > > >>>> ********************************************/ >>>> > >> > > > > > >> > >> > > > > > > >>>> > >> > > > > > >> > >> > > > > > >>>> > >> > > > > > >> > >> > > > > >>>> > >> > > > > > >> > >> > > > >>>> > >> > > > > > >> > >> > > >>>> > >> > > > > > >> > >> > >>>> > >> > > > > > >> > >> >>>> > >> > > > > > >> > >> >>>> > >> > > > > > >> > >> >>>> > >> > > > > > >> > >> -- >>>> > >> > > > > > >> > >> -- Guozhang >>>> > >> > > > > > >> > >> >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > > >>>> > >> > > > > > >> > >>>> > >> > > > > > >> >>>> > >> > > > > > > >>>> > >> > > > > > > >>>> > >> > > > > > >>>> > >> > > > > >>>> > >> > > > >>>> > >> > > >>>> > >> > >>>> > >> > >>>> > >> > >>>> > >> > -- >>>> > >> > Jeff Holoman >>>> > >> > Systems Engineer >>>> > >> > >>>> > >> >>>> >>>> >>> >>> >>> -- >>> -- Guozhang >>> >> >> >> >> -- >> -- Guozhang
