Andrii, I think we agreed on the following.
(a) Admin requests can be sent to and handled by any broker. (b) Admin requests are processed asynchronously, at least for now. That is, when the client gets a response, it just means that the request is initiated, but not necessarily completed. Then, it's up to the client to issue another request to check the status for completion. To support (a), we were thinking of doing request forwarding to the controller (utilizing KAFKA-1912). I am making an alternative proposal. Basically, the broker can just write to ZooKeeper to inform the controller about the request. For example, to handle partitionReassignment, the broker will just write the requested partitions to /admin/reassign_partitions (like what AdminUtils currently does) and then send a response to the client. This shouldn't take long and the implementation will be simpler than forwarding the requests to the controller through RPC. Thanks, Jun On Wed, Mar 18, 2015 at 3:03 PM, Andrii Biletskyi < andrii.bilets...@stealth.ly> wrote: > Jun, > > I might be wrong but didn't we agree we will let any broker from the > cluster handle *long-running* admin requests (at this time preferredReplica > and > reassignPartitions), via zk admin path. Thus CreateTopics etc should be > sent > only to the controller. > > Thanks, > Andrii Biletskyi > > On Wed, Mar 18, 2015 at 11:55 PM, Jun Rao <j...@confluent.io> wrote: > > > Joel, Andril, > > > > I think we agreed that those admin requests can be issued to any broker. > > Because of that, there doesn't seem to be a strong need to know the > > controller. So, perhaps we can proceed by not making any change to the > > format of TMR right now. When we start using create topic request in the > > producer, we will need a new version of TMR that doesn't trigger auto > topic > > creation. But that can be done later. > > > > As a first cut implementation, I think the broker can just write to ZK > > directly for > > createToipic/alterTopic/reassignPartitions/preferredLeaderElection > > requests, instead of forwarding them to the controller. This will > simplify > > the implementation on the broker side. > > > > Thanks, > > > > Jun > > > > > > On Wed, Mar 18, 2015 at 11:58 AM, Joel Koshy <jjkosh...@gmail.com> > wrote: > > > > > (Thanks Andrii for the summary) > > > > > > For (1) yes we will circle back on that shortly after syncing up in > > > person. I think it is close to getting committed although development > > > for KAFKA-1927 can probably begin without it. > > > > > > There is one more item we covered at the hangout. i.e., whether we > > > want to add the coordinator to the topic metadata response or provide > > > a clearer ClusterMetadataRequest. > > > > > > There are two reasons I think we should try and avoid adding the > > > field: > > > - It is irrelevant to topic metadata > > > - If we finally do request rerouting in Kafka then the field would add > > > little to no value. (It still helps to have a separate > > > ClusterMetadataRequest to query for cluster-wide information such as > > > 'which broker is the controller?' as Joe mentioned.) > > > > > > I think it would be cleaner to have an explicit ClusterMetadataRequest > > > that you can send to any broker in order to obtain the controller (and > > > in the future possibly other cluster-wide information). I think the > > > main argument against doing this and instead adding it to the topic > > > metadata response was convenience - i.e., you don't have to discover > > > the controller in advance. However, I don't see much actual > > > benefit/convenience in this and in fact think it is a non-issue. Let > > > me know if I'm overlooking something here. > > > > > > As an example, say we need to initiate partition reassignment by > > > issuing the new ReassignPartitionsRequest to the controller (assume we > > > already have the desired manual partition assignment). If we are to > > > augment topic metadata response then the flow be something like this : > > > > > > - Issue topic metadata request to any broker (and discover the > > > controller > > > - Connect to controller if required (i.e., if the broker above != > > > controller) > > > - Issue the partition reassignment request to the controller. > > > > > > With an explicit cluster metadata request it would be: > > > - Issue cluster metadata request to any broker > > > - Connect to controller if required (i.e., if the broker above != > > > controller) > > > - Issue the partition reassignment request > > > > > > So it seems to add little practical value and bloats topic metadata > > > response with an irrelevant detail. > > > > > > The other angle to this is the following - is it a matter of naming? > > > Should we just rename topic metadata request/response to just > > > MetadataRequest/Response and add cluster metadata to it? By that same > > > token should we also allow querying for the consumer coordinator (and > > > in future transaction coordinator) as well? This leads to a bloated > > > request which isn't very appealing and altogether confusing. > > > > > > Thanks, > > > > > > Joel > > > > > > On Wed, Mar 18, 2015 at 09:34:12AM -0700, Jun Rao wrote: > > > > Andri, > > > > > > > > Thanks for the summary. > > > > > > > > 1. I just realized that in order to start working on KAFKA-1927, we > > will > > > > need to merge the changes to OffsetCommitRequest (from 0.8.2) to > trunk. > > > > This is planned to be done as part of KAFKA-1634. So, we will need > > > Guozhang > > > > and Joel's help to wrap this up. > > > > > > > > 2. Thinking about this a bit more, if the semantic of those "write" > > > > requests is async (i.e., after the client gets a response, it just > > means > > > > that the operation is initiated, but not necessarily completed), we > > don't > > > > really need to forward the requests to the controller. Instead, the > > > > receiving broker can just write the operation to ZK as the admin > > command > > > > line tool previously does. This will simplify the implementation. > > > > > > > > 8. There is another implementation detail for describe topic. > Ideally, > > we > > > > want to read the topic config from the broker cache, instead of > > > ZooKeeper. > > > > Currently, every broker reads the topic-level config for all topics. > > > > However, it ignores those for topics not hosted on itself. So, we may > > > need > > > > to change TopicConfigManager a bit so that it caches the configs for > > all > > > > topics. > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > > > > > On Tue, Mar 17, 2015 at 1:13 PM, Andrii Biletskyi < > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > Guys, > > > > > > > > > > Thanks for a great discussion! > > > > > Here are the actions points: > > > > > > > > > > 1. Q: Get rid of all scala requests objects, use java protocol > > > definitions. > > > > > A: Gwen kindly took that (KAFKA-1927). It's important to speed > up > > > > > review procedure > > > > > there since this ticket blocks other important changes. > > > > > > > > > > 2. Q: Generic re-reroute facility vs client maintaining cluster > > state. > > > > > A: Jay has added pseudo code to KAFKA-1912 - need to consider > > > whether > > > > > this will be > > > > > easy to implement as a server-side feature (comments are > > > > > welcomed!). > > > > > > > > > > 3. Q: Controller field in wire protocol. > > > > > A: This might be useful for clients, add this to > > > TopicMetadataResponse > > > > > (already in KIP). > > > > > > > > > > 4. Q: Decoupling topic creation from TMR. > > > > > A: I will add proposed by Jun solution (using clientId for > that) > > > to the > > > > > KIP. > > > > > > > > > > 5. Q: Bumping new versions of TMR vs grabbing all protocol changes > in > > > one > > > > > version. > > > > > A: It was decided to try to gather all changes to protocol > > (before > > > > > release). > > > > > In case of TMR it worth checking: KAFKA-2020 and KIP-13 > > > (quotas) > > > > > > > > > > 6. Q: JSON lib is needed to deserialize user's input in CLI tool. > > > > > A: Use jackson for that, /tools project is a separate jar so > > > shouldn't > > > > > be a big deal. > > > > > > > > > > 7. Q: VerifyReassingPartitions vs generic status check command. > > > > > A: For long-running requests like reassign partitions > *progress* > > > check > > > > > request is useful, > > > > > it makes sense to introduce it. > > > > > > > > > > Please add, correct me if I missed something. > > > > > > > > > > Thanks, > > > > > Andrii Biletskyi > > > > > > > > > > On Tue, Mar 17, 2015 at 6:20 PM, Andrii Biletskyi < > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > Joel, > > > > > > > > > > > > You are right, I removed ClusterMetadata because we have > partially > > > > > > what we need in TopicMetadata. Also, as Jay pointed out earlier, > we > > > > > > would like to have "orthogonal" API, but at the same time we need > > > > > > to be backward compatible. > > > > > > > > > > > > But I like your idea and even have some other arguments for this > > > option: > > > > > > There is also DescribeTopicRequest which was proposed in this > KIP, > > > > > > it returns topic configs, partitions, replication factor plus > > > partition > > > > > > ISR, ASR, > > > > > > leader replica. The later part is really already there in > > > > > > TopicMetadataRequest. > > > > > > So again we'll have to add stuff to TMR, not to duplicate some > info > > > in > > > > > > newly added requests. However, this way we'll end up with > "monster" > > > > > > request which returns cluster metadata, topic replication and > > config > > > info > > > > > > plus partition replication data. Seems logical to split TMR to > > > > > > - ClusterMetadata (brokers + controller, maybe smth else) > > > > > > - TopicMetadata (topic info + partition details) > > > > > > But since current TMR is involved in lots of places (including > > > network > > > > > > client, > > > > > > as I understand) this might be very serious change and it > probably > > > makes > > > > > > sense to stick with current approach. > > > > > > > > > > > > Thanks, > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 5:29 PM, Joel Koshy <jjkosh...@gmail.com > > > > > wrote: > > > > > > > > > > > >> I may be missing some context but hopefully this will also be > > > covered > > > > > >> today: I thought the earlier proposal where there was an > explicit > > > > > >> ClusterMetadata request was clearer and explicit. During the > > course > > > of > > > > > >> this thread I think the conclusion was that the main need was > for > > > > > >> controller information and that can be rolled into the topic > > > metadata > > > > > >> response but that seems a bit irrelevant to topic metadata. > FWIW I > > > > > >> think the full broker-list is also irrelevant to topic metadata, > > but > > > > > >> it is already there and in use. I think there is still room for > an > > > > > >> explicit ClusterMetadata request since there may be other > > > > > >> cluster-level information that we may want to add over time (and > > > that > > > > > >> have nothing to do with topic metadata). > > > > > >> > > > > > >> On Tue, Mar 17, 2015 at 02:45:30PM +0200, Andrii Biletskyi > wrote: > > > > > >> > Jun, > > > > > >> > > > > > > >> > 101. Okay, if you say that such use case is important. I also > > > think > > > > > >> > using clientId for these purposes is fine - if we already have > > > this > > > > > >> field > > > > > >> > as part of all Wire protocol messages, why not use that. > > > > > >> > I will update KIP-4 page if nobody has other ideas (which may > > > come up > > > > > >> > during the call today). > > > > > >> > > > > > > >> > 102.1 Agree, I'll update the KIP accordingly. I think we can > add > > > new, > > > > > >> > fine-grained error codes if some error code received in > specific > > > case > > > > > >> > won't give enough context to return a descriptive error > message > > > for > > > > > >> user. > > > > > >> > > > > > > >> > Look forward to discussing all outstanding issues in detail > > today > > > > > during > > > > > >> > the call. > > > > > >> > > > > > > >> > Thanks, > > > > > >> > Andrii Biletskyi > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > On Mon, Mar 16, 2015 at 10:59 PM, Jun Rao <j...@confluent.io> > > > wrote: > > > > > >> > > > > > > >> > > 101. There may be a use case where you only want the topics > to > > > be > > > > > >> created > > > > > >> > > manually by admins. Currently, you can do that by disabling > > auto > > > > > topic > > > > > >> > > creation and issue topic creation from the TopicCommand. If > we > > > > > >> disable auto > > > > > >> > > topic creation completely on the broker and don't have a way > > to > > > > > >> distinguish > > > > > >> > > between topic creation requests from the regular clients and > > the > > > > > >> admin, we > > > > > >> > > can't support manual topic creation any more. I was thinking > > > that > > > > > >> another > > > > > >> > > way of distinguishing the clients making the topic creation > > > requests > > > > > >> is > > > > > >> > > using clientId. For example, the admin tool can set it to > > > something > > > > > >> like > > > > > >> > > admin and the broker can treat that clientId specially. > > > > > >> > > > > > > > >> > > Also, there is a related discussion in KAFKA-2020. > Currently, > > > we do > > > > > >> the > > > > > >> > > following in TopicMetadataResponse: > > > > > >> > > > > > > > >> > > 1. If leader is not available, we set the partition level > > error > > > code > > > > > >> to > > > > > >> > > LeaderNotAvailable. > > > > > >> > > 2. If a non-leader replica is not available, we take that > > > replica > > > > > out > > > > > >> of > > > > > >> > > the assigned replica list and isr in the response. As an > > > indication > > > > > >> for > > > > > >> > > doing that, we set the partition level error code to > > > > > >> ReplicaNotAvailable. > > > > > >> > > > > > > > >> > > This has a few problems. First, ReplicaNotAvailable probably > > > > > >> shouldn't be > > > > > >> > > an error, at least for the normal producer/consumer clients > > that > > > > > just > > > > > >> want > > > > > >> > > to find out the leader. Second, it can happen that both the > > > leader > > > > > and > > > > > >> > > another replica are not available at the same time. There is > > no > > > > > error > > > > > >> code > > > > > >> > > to indicate both. Third, even if a replica is not available, > > > it's > > > > > >> still > > > > > >> > > useful to return its replica id since some clients (e.g. > admin > > > tool) > > > > > >> may > > > > > >> > > still make use of it. > > > > > >> > > > > > > > >> > > One way to address this issue is to always return the > replica > > > id for > > > > > >> > > leader, assigned replicas, and isr regardless of whether the > > > > > >> corresponding > > > > > >> > > broker is live or not. Since we also return the list of live > > > > > brokers, > > > > > >> the > > > > > >> > > client can figure out whether a leader or a replica is live > or > > > not > > > > > >> and act > > > > > >> > > accordingly. This way, we don't need to set the partition > > level > > > > > error > > > > > >> code > > > > > >> > > when the leader or a replica is not available. This doesn't > > > change > > > > > >> the wire > > > > > >> > > protocol, but does change the semantics. Since we are > evolving > > > the > > > > > >> protocol > > > > > >> > > of TopicMetadataRequest here, we can potentially piggyback > the > > > > > change. > > > > > >> > > > > > > > >> > > 102.1 For those types of errors due to invalid input, > > shouldn't > > > we > > > > > >> just > > > > > >> > > guard it at parameter validation time and throw > > > > > >> InvalidArgumentException > > > > > >> > > without even sending the request to the broker? > > > > > >> > > > > > > > >> > > Thanks, > > > > > >> > > > > > > > >> > > Jun > > > > > >> > > > > > > > >> > > > > > > > >> > > On Mon, Mar 16, 2015 at 10:37 AM, Andrii Biletskyi < > > > > > >> > > andrii.bilets...@stealth.ly> wrote: > > > > > >> > > > > > > > >> > > > Jun, > > > > > >> > > > > > > > > >> > > > Answering your questions: > > > > > >> > > > > > > > > >> > > > 101. If I understand you correctly, you are saying future > > > producer > > > > > >> > > versions > > > > > >> > > > (which > > > > > >> > > > will be ported to TMR_V1) won't be able to automatically > > > create > > > > > >> topic (if > > > > > >> > > > we > > > > > >> > > > unconditionally remove topic creation from there). But we > > > need to > > > > > >> this > > > > > >> > > > preserve logic. > > > > > >> > > > Ok, about your proposal: I'm not a big fan too, when it > > comes > > > to > > > > > >> > > > differentiating > > > > > >> > > > clients directly in protocol schema. And also I'm not > sure I > > > > > >> understand > > > > > >> > > at > > > > > >> > > > all why > > > > > >> > > > auto.create.topics.enable is a server side configuration. > > Can > > > we > > > > > >> > > deprecate > > > > > >> > > > this setting > > > > > >> > > > in future versions, add this setting to producer and based > > on > > > that > > > > > >> upon > > > > > >> > > > receiving > > > > > >> > > > UnknownTopic create topic explicitly by a separate > producer > > > call > > > > > via > > > > > >> > > > adminClient? > > > > > >> > > > > > > > > >> > > > 102.1. Hm, yes. It's because we want to support batching > and > > > at > > > > > the > > > > > >> same > > > > > >> > > > time we > > > > > >> > > > want to give descriptive error messages for clients. Since > > > > > >> AdminClient > > > > > >> > > > holds the context > > > > > >> > > > to construct such messages (e.g. AdminClient layer can > know > > > that > > > > > >> > > > InvalidArgumentsCode > > > > > >> > > > means two cases: either invalid number - e.g. -1; or > > > > > >> replication-factor > > > > > >> > > was > > > > > >> > > > provided while > > > > > >> > > > partitions argument wasn't) - I wrapped responses in > > > Exceptions. > > > > > >> But I'm > > > > > >> > > > open to any > > > > > >> > > > other ideas, this was just initial version. > > > > > >> > > > 102.2. Yes, I agree. I'll change that to probably some > other > > > dto. > > > > > >> > > > > > > > > >> > > > Thanks, > > > > > >> > > > Andrii Biletskyi > > > > > >> > > > > > > > > >> > > > On Fri, Mar 13, 2015 at 7:16 PM, Jun Rao < > j...@confluent.io> > > > > > wrote: > > > > > >> > > > > > > > > >> > > > > Andrii, > > > > > >> > > > > > > > > > >> > > > > 101. That's what I was thinking too, but it may not be > > that > > > > > >> simple. In > > > > > >> > > > > TopicMetadataRequest_V1, > > > > > >> > > > > we can let it not trigger auto topic creation. Then, in > > the > > > > > >> producer > > > > > >> > > > side, > > > > > >> > > > > if it gets an UnknownTopicException, it can explicitly > > > issue a > > > > > >> > > > > createTopicRequest for auto topic creation. On the > > consumer > > > > > side, > > > > > >> it > > > > > >> > > will > > > > > >> > > > > never issue createTopicRequest. This works when auto > topic > > > > > >> creation is > > > > > >> > > > > enabled on the broker side. However, I am not sure how > > > things > > > > > >> will work > > > > > >> > > > > when auto topic creation is disabled on the broker side. > > In > > > this > > > > > >> case, > > > > > >> > > we > > > > > >> > > > > want to have a way to manually create a topic, > potentially > > > > > through > > > > > >> > > admin > > > > > >> > > > > commands. However, then we need a way to distinguish > > > > > >> createTopicRequest > > > > > >> > > > > issued from the producer clients and the admin tools. > May > > > be we > > > > > >> can > > > > > >> > > add a > > > > > >> > > > > new field in createTopicRequest and set it differently > in > > > the > > > > > >> producer > > > > > >> > > > > client and the admin client. However, I am not sure if > > > that's > > > > > the > > > > > >> best > > > > > >> > > > > approach. > > > > > >> > > > > > > > > > >> > > > > 2. Yes, refactoring existing requests is a non-trivial > > > amount of > > > > > >> work. > > > > > >> > > I > > > > > >> > > > > posted some comments in KAFKA-1927. We will probably > have > > > to fix > > > > > >> > > > KAFKA-1927 > > > > > >> > > > > first, before adding the new logic in KAFKA-1694. > > > Otherwise, the > > > > > >> > > changes > > > > > >> > > > > will be too big. > > > > > >> > > > > > > > > > >> > > > > 102. About the AdminClient: > > > > > >> > > > > 102.1. It's a bit weird that we return exception in the > > > api. It > > > > > >> seems > > > > > >> > > > that > > > > > >> > > > > we should either return error code or throw an exception > > > when > > > > > >> getting > > > > > >> > > the > > > > > >> > > > > response state. > > > > > >> > > > > 102.2. We probably shouldn't explicitly use the request > > > object > > > > > in > > > > > >> the > > > > > >> > > > api. > > > > > >> > > > > Not every request evolution requires an api change. > > > > > >> > > > > > > > > > >> > > > > Thanks, > > > > > >> > > > > > > > > > >> > > > > Jun > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > On Fri, Mar 13, 2015 at 4:08 AM, Andrii Biletskyi < > > > > > >> > > > > andrii.bilets...@stealth.ly> wrote: > > > > > >> > > > > > > > > > >> > > > > > Jun, > > > > > >> > > > > > > > > > > >> > > > > > Thanks for you comments. Answers inline: > > > > > >> > > > > > > > > > > >> > > > > > 100. There are a few fields such as ReplicaAssignment, > > > > > >> > > > > > > ReassignPartitionRequest, > > > > > >> > > > > > > and PartitionsSerialized that are represented as a > > > string, > > > > > but > > > > > >> > > > contain > > > > > >> > > > > > > composite structures in json. Could we flatten them > > out > > > > > >> directly in > > > > > >> > > > the > > > > > >> > > > > > > protocol definition as arrays/records? > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > Yes, now with Admin Client this looks a bit weird. My > > > initial > > > > > >> > > > motivation > > > > > >> > > > > > was: > > > > > >> > > > > > ReassignPartitionCommand accepts input in json, we > want > > to > > > > > >> remain > > > > > >> > > > tools' > > > > > >> > > > > > interfaces unchanged, where possible. > > > > > >> > > > > > If we port it to deserialized format, in CLI (/tools > > > project) > > > > > >> we will > > > > > >> > > > > have > > > > > >> > > > > > to add some > > > > > >> > > > > > json library since /tools is written in java and we'll > > > need to > > > > > >> > > > > deserialize > > > > > >> > > > > > json file > > > > > >> > > > > > provided by a user. Can we quickly agree on what this > > > library > > > > > >> should > > > > > >> > > be > > > > > >> > > > > > (Jackson, GSON, whatever)? > > > > > >> > > > > > > > > > > >> > > > > > 101. Does TopicMetadataRequest v1 still trigger auto > > topic > > > > > >> creation? > > > > > >> > > > This > > > > > >> > > > > > > will be a bit weird now that we have a separate > topic > > > > > >> creation api. > > > > > >> > > > > Have > > > > > >> > > > > > > you thought about how the new createTopicRequest and > > > > > >> > > > > TopicMetadataRequest > > > > > >> > > > > > > v1 will be used in the producer/consumer client, in > > > addition > > > > > >> to > > > > > >> > > admin > > > > > >> > > > > > > tools? For example, ideally, we don't want > > > > > >> TopicMetadataRequest > > > > > >> > > from > > > > > >> > > > > the > > > > > >> > > > > > > consumer to trigger auto topic creation. > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > I agree, this strange logic should be fixed. I'm not > > > confident > > > > > >> in > > > > > >> > > this > > > > > >> > > > > > Kafka part so > > > > > >> > > > > > correct me if I'm wrong, but it doesn't look like a > hard > > > thing > > > > > >> to > > > > > >> > > do, I > > > > > >> > > > > > think we can > > > > > >> > > > > > leverage AdminClient for that in Producer and > > > unconditionally > > > > > >> remove > > > > > >> > > > > topic > > > > > >> > > > > > creation from the TopicMetadataRequest_V1. > > > > > >> > > > > > > > > > > >> > > > > > 2. I think Jay meant getting rid of scala classes > > > > > >> > > > > > > like HeartbeatRequestAndHeader and > > > > > >> HeartbeatResponseAndHeader. We > > > > > >> > > did > > > > > >> > > > > > that > > > > > >> > > > > > > as a stop-gap thing when adding the new requests for > > the > > > > > >> consumers. > > > > > >> > > > > > > However, the long term plan is to get rid of all > those > > > and > > > > > >> just > > > > > >> > > reuse > > > > > >> > > > > the > > > > > >> > > > > > > java request/response in the client. Since this KIP > > > proposes > > > > > >> to > > > > > >> > > add a > > > > > >> > > > > > > significant number of new requests, perhaps we > should > > > bite > > > > > the > > > > > >> > > bullet > > > > > >> > > > > to > > > > > >> > > > > > > clean up the existing scala requests first before > > > adding new > > > > > >> ones? > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > Yes, looks like I misunderstood the point of > > > > > >> ...RequestAndHeader. > > > > > >> > > > Okay, I > > > > > >> > > > > > will > > > > > >> > > > > > rework that. The only thing is that I don't see any > > > example > > > > > how > > > > > >> it > > > > > >> > > was > > > > > >> > > > > done > > > > > >> > > > > > for at > > > > > >> > > > > > least one existing protocol message. Thus, as I > > > understand, I > > > > > >> have to > > > > > >> > > > > think > > > > > >> > > > > > how we > > > > > >> > > > > > are going to do it. > > > > > >> > > > > > Re porting all existing RQ/RP in this patch. Sounds > > > > > reasonable, > > > > > >> but > > > > > >> > > if > > > > > >> > > > > it's > > > > > >> > > > > > an *obligatory* > > > > > >> > > > > > requirement to have Admin KIP done, I'm afraid this > can > > > be a > > > > > >> serious > > > > > >> > > > > > blocker for us. > > > > > >> > > > > > There are 13 protocol messages and all that would > > require > > > not > > > > > >> only > > > > > >> > > unit > > > > > >> > > > > > tests but quite > > > > > >> > > > > > intensive manual testing, no? I'm afraid I'm not the > > > right guy > > > > > >> to > > > > > >> > > cover > > > > > >> > > > > > pretty much all > > > > > >> > > > > > Kafka core internals :). Let me know your thoughts on > > this > > > > > >> item. Btw > > > > > >> > > > > there > > > > > >> > > > > > is a ticket to > > > > > >> > > > > > follow-up this issue ( > > > > > >> > > https://issues.apache.org/jira/browse/KAFKA-2006 > > > > > >> > > > ). > > > > > >> > > > > > > > > > > >> > > > > > Thanks, > > > > > >> > > > > > Andrii Biletskyi > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > On Fri, Mar 13, 2015 at 6:40 AM, Jun Rao < > > > j...@confluent.io> > > > > > >> wrote: > > > > > >> > > > > > > > > > > >> > > > > > > Andrii, > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > A few more comments. > > > > > >> > > > > > > > > > > > >> > > > > > > 100. There are a few fields such as > ReplicaAssignment, > > > > > >> > > > > > > ReassignPartitionRequest, > > > > > >> > > > > > > and PartitionsSerialized that are represented as a > > > string, > > > > > but > > > > > >> > > > contain > > > > > >> > > > > > > composite structures in json. Could we flatten them > > out > > > > > >> directly in > > > > > >> > > > the > > > > > >> > > > > > > protocol definition as arrays/records? > > > > > >> > > > > > > > > > > > >> > > > > > > 101. Does TopicMetadataRequest v1 still trigger auto > > > topic > > > > > >> > > creation? > > > > > >> > > > > This > > > > > >> > > > > > > will be a bit weird now that we have a separate > topic > > > > > >> creation api. > > > > > >> > > > > Have > > > > > >> > > > > > > you thought about how the new createTopicRequest and > > > > > >> > > > > TopicMetadataRequest > > > > > >> > > > > > > v1 will be used in the producer/consumer client, in > > > addition > > > > > >> to > > > > > >> > > admin > > > > > >> > > > > > > tools? For example, ideally, we don't want > > > > > >> TopicMetadataRequest > > > > > >> > > from > > > > > >> > > > > the > > > > > >> > > > > > > consumer to trigger auto topic creation. > > > > > >> > > > > > > > > > > > >> > > > > > > 2. I think Jay meant getting rid of scala classes > > > > > >> > > > > > > like HeartbeatRequestAndHeader and > > > > > >> HeartbeatResponseAndHeader. We > > > > > >> > > did > > > > > >> > > > > > that > > > > > >> > > > > > > as a stop-gap thing when adding the new requests for > > the > > > > > >> consumers. > > > > > >> > > > > > > However, the long term plan is to get rid of all > those > > > and > > > > > >> just > > > > > >> > > reuse > > > > > >> > > > > the > > > > > >> > > > > > > java request/response in the client. Since this KIP > > > proposes > > > > > >> to > > > > > >> > > add a > > > > > >> > > > > > > significant number of new requests, perhaps we > should > > > bite > > > > > the > > > > > >> > > bullet > > > > > >> > > > > to > > > > > >> > > > > > > clean up the existing scala requests first before > > > adding new > > > > > >> ones? > > > > > >> > > > > > > > > > > > >> > > > > > > Thanks, > > > > > >> > > > > > > > > > > > >> > > > > > > Jun > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > On Thu, Mar 12, 2015 at 3:37 PM, Andrii Biletskyi < > > > > > >> > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > >> > > > > > > > > > > > >> > > > > > > > Hi, > > > > > >> > > > > > > > > > > > > >> > > > > > > > As said above - I list again all comments from > this > > > thread > > > > > >> so we > > > > > >> > > > > > > > can see what's left and finalize all pending > issues. > > > > > >> > > > > > > > > > > > > >> > > > > > > > Comments from Jay: > > > > > >> > > > > > > > 1. This is much needed functionality, but there > are > > a > > > lot > > > > > >> of the > > > > > >> > > so > > > > > >> > > > > > let's > > > > > >> > > > > > > > really think these protocols through. We really > want > > > to > > > > > end > > > > > >> up > > > > > >> > > > with a > > > > > >> > > > > > set > > > > > >> > > > > > > > of well thought-out, orthoganol apis. For this > > reason > > > I > > > > > >> think it > > > > > >> > > is > > > > > >> > > > > > > really > > > > > >> > > > > > > > important to think through the end state even if > > that > > > > > >> includes > > > > > >> > > APIs > > > > > >> > > > > we > > > > > >> > > > > > > > won't implement in the first phase. > > > > > >> > > > > > > > > > > > > >> > > > > > > > A: Definitely behind this. Would appreciate if > there > > > are > > > > > >> concrete > > > > > >> > > > > > > comments > > > > > >> > > > > > > > how this can be improved. > > > > > >> > > > > > > > > > > > > >> > > > > > > > 2. Let's please please please wait until we have > > > switched > > > > > >> the > > > > > >> > > > server > > > > > >> > > > > > over > > > > > >> > > > > > > > to the new java protocol definitions. If we add > > upteen > > > > > more > > > > > >> ad > > > > > >> > > hoc > > > > > >> > > > > > scala > > > > > >> > > > > > > > objects that is just generating more work for the > > > > > >> conversion we > > > > > >> > > > know > > > > > >> > > > > we > > > > > >> > > > > > > > have to do. > > > > > >> > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - removed scala > > protocol > > > > > >> classes. > > > > > >> > > > > > > > > > > > > >> > > > > > > > 3. This proposal introduces a new type of optional > > > > > >> parameter. > > > > > >> > > This > > > > > >> > > > is > > > > > >> > > > > > > > inconsistent with everything else in the protocol > > > where we > > > > > >> use -1 > > > > > >> > > > or > > > > > >> > > > > > some > > > > > >> > > > > > > > other marker value. You could argue either way but > > > let's > > > > > >> stick > > > > > >> > > with > > > > > >> > > > > > that > > > > > >> > > > > > > > for consistency. For clients that implemented the > > > protocol > > > > > >> in a > > > > > >> > > > > better > > > > > >> > > > > > > way > > > > > >> > > > > > > > than our scala code these basic primitives are > hard > > to > > > > > >> change. > > > > > >> > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - removed MaybeOf > type > > > and > > > > > >> changed > > > > > >> > > > > > protocol > > > > > >> > > > > > > > accordingly. > > > > > >> > > > > > > > > > > > > >> > > > > > > > 4. ClusterMetadata: This seems to duplicate > > > > > >> TopicMetadataRequest > > > > > >> > > > > which > > > > > >> > > > > > > has > > > > > >> > > > > > > > brokers, topics, and partitions. I think we should > > > rename > > > > > >> that > > > > > >> > > > > request > > > > > >> > > > > > > > ClusterMetadataRequest (or just MetadataRequest) > and > > > > > >> include the > > > > > >> > > id > > > > > >> > > > > of > > > > > >> > > > > > > the > > > > > >> > > > > > > > controller. Or are there other things we could add > > > here? > > > > > >> > > > > > > > > > > > > >> > > > > > > > A: I agree. Updated the KIP. Let's extends > > > TopicMetadata > > > > > to > > > > > >> > > > version 2 > > > > > >> > > > > > and > > > > > >> > > > > > > > include controller. > > > > > >> > > > > > > > > > > > > >> > > > > > > > 5. We have a tendency to try to make a lot of > > requests > > > > > that > > > > > >> can > > > > > >> > > > only > > > > > >> > > > > go > > > > > >> > > > > > > to > > > > > >> > > > > > > > particular nodes. This adds a lot of burden for > > client > > > > > >> > > > > implementations > > > > > >> > > > > > > (it > > > > > >> > > > > > > > sounds easy but each discovery can fail in many > > parts > > > so > > > > > it > > > > > >> ends > > > > > >> > > up > > > > > >> > > > > > > being a > > > > > >> > > > > > > > full state machine to do right). I think we should > > > > > consider > > > > > >> > > making > > > > > >> > > > > > admin > > > > > >> > > > > > > > commands and ideally as many of the other apis as > > > possible > > > > > >> > > > available > > > > > >> > > > > on > > > > > >> > > > > > > all > > > > > >> > > > > > > > brokers and just redirect to the controller on the > > > broker > > > > > >> side. > > > > > >> > > > > Perhaps > > > > > >> > > > > > > > there would be a general way to encapsulate this > > > > > re-routing > > > > > >> > > > behavior. > > > > > >> > > > > > > > > > > > > >> > > > > > > > A: It's a very interesting idea, but seems there > are > > > some > > > > > >> > > concerns > > > > > >> > > > > > about > > > > > >> > > > > > > > this > > > > > >> > > > > > > > feature (like performance considerations, how this > > > will > > > > > >> > > complicate > > > > > >> > > > > > server > > > > > >> > > > > > > > etc). > > > > > >> > > > > > > > I believe this shouldn't be a blocker. If this > > > feature is > > > > > >> > > > implemented > > > > > >> > > > > > at > > > > > >> > > > > > > > some > > > > > >> > > > > > > > point it won't affect Admin changes - at least no > > > changes > > > > > to > > > > > >> > > public > > > > > >> > > > > API > > > > > >> > > > > > > > will be required. > > > > > >> > > > > > > > > > > > > >> > > > > > > > 6. We should probably normalize the key value > pairs > > > used > > > > > for > > > > > >> > > > configs > > > > > >> > > > > > > rather > > > > > >> > > > > > > > than embedding a new formatting. So two strings > > rather > > > > > than > > > > > >> one > > > > > >> > > > with > > > > > >> > > > > an > > > > > >> > > > > > > > internal equals sign. > > > > > >> > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - normalized configs > > and > > > > > >> changed > > > > > >> > > > > protocol > > > > > >> > > > > > > > accordingly. > > > > > >> > > > > > > > > > > > > >> > > > > > > > 7. Is the postcondition of these APIs that the > > > command has > > > > > >> begun > > > > > >> > > or > > > > > >> > > > > > that > > > > > >> > > > > > > > the command has been completed? It is a lot more > > > usable if > > > > > >> the > > > > > >> > > > > command > > > > > >> > > > > > > has > > > > > >> > > > > > > > been completed so you know that if you create a > > topic > > > and > > > > > >> then > > > > > >> > > > > publish > > > > > >> > > > > > to > > > > > >> > > > > > > > it you won't get an exception about there being no > > > such > > > > > >> topic. > > > > > >> > > > > > > > > > > > > >> > > > > > > > A: For long running requests (like reassign > > > partitions) - > > > > > >> the > > > > > >> > > post > > > > > >> > > > > > > > condition is > > > > > >> > > > > > > > command has begun - so we don't block the client. > In > > > case > > > > > >> of your > > > > > >> > > > > > > example - > > > > > >> > > > > > > > topic commands, this will be refactored and topic > > > commands > > > > > >> will > > > > > >> > > be > > > > > >> > > > > > > executed > > > > > >> > > > > > > > immediately, since the Controller will serve Admin > > > > > requests > > > > > >> > > > > > > > (follow-up ticket KAFKA-1777). > > > > > >> > > > > > > > > > > > > >> > > > > > > > 8. Describe topic and list topics duplicate a lot > of > > > stuff > > > > > >> in the > > > > > >> > > > > > > metadata > > > > > >> > > > > > > > request. Is there a reason to give back topics > > marked > > > for > > > > > >> > > > deletion? I > > > > > >> > > > > > > feel > > > > > >> > > > > > > > like if we just make the post-condition of the > > delete > > > > > >> command be > > > > > >> > > > that > > > > > >> > > > > > the > > > > > >> > > > > > > > topic is deleted that will get rid of the need for > > > this > > > > > >> right? > > > > > >> > > And > > > > > >> > > > it > > > > > >> > > > > > > will > > > > > >> > > > > > > > be much more intuitive. > > > > > >> > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - removed topics > marked > > > for > > > > > >> deletion > > > > > >> > > > in > > > > > >> > > > > > > > ListTopicsRequest. > > > > > >> > > > > > > > > > > > > >> > > > > > > > 9. Should we consider batching these requests? We > > have > > > > > >> generally > > > > > >> > > > > tried > > > > > >> > > > > > to > > > > > >> > > > > > > > allow multiple operations to be batched. My > > suspicion > > > is > > > > > >> that > > > > > >> > > > without > > > > > >> > > > > > > this > > > > > >> > > > > > > > we will get a lot of code that does something like > > > > > >> > > > > > > > for(topic: adminClient.listTopics()) > > > > > >> > > > > > > > adminClient.describeTopic(topic) > > > > > >> > > > > > > > this code will work great when you test on 5 > topics > > > but > > > > > not > > > > > >> do as > > > > > >> > > > > well > > > > > >> > > > > > if > > > > > >> > > > > > > > you have 50k. > > > > > >> > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please check "Topic Admin > > Schema" > > > > > >> section. > > > > > >> > > > > > > > > > > > > >> > > > > > > > 10. I think we should also discuss how we want to > > > expose a > > > > > >> > > > > programmatic > > > > > >> > > > > > > JVM > > > > > >> > > > > > > > client api for these operations. Currently people > > > rely on > > > > > >> > > > AdminUtils > > > > > >> > > > > > > which > > > > > >> > > > > > > > is totally sketchy. I think we probably need > another > > > > > client > > > > > >> under > > > > > >> > > > > > > clients/ > > > > > >> > > > > > > > that exposes administrative functionality. We will > > > need > > > > > >> this just > > > > > >> > > > to > > > > > >> > > > > > > > properly test the new apis, I suspect. We should > > > figure > > > > > out > > > > > >> that > > > > > >> > > > API. > > > > > >> > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please check "Admin Client" > > > section > > > > > >> with an > > > > > >> > > > > > initial > > > > > >> > > > > > > > API proposal. > > > > > >> > > > > > > > > > > > > >> > > > > > > > 11. The other information that would be really > > useful > > > to > > > > > get > > > > > >> > > would > > > > > >> > > > be > > > > > >> > > > > > > > information about partitions--how much data is in > > the > > > > > >> partition, > > > > > >> > > > what > > > > > >> > > > > > are > > > > > >> > > > > > > > the segment offsets, what is the log-end offset > > (i.e. > > > last > > > > > >> > > offset), > > > > > >> > > > > > what > > > > > >> > > > > > > is > > > > > >> > > > > > > > the compaction point, etc. I think that done right > > > this > > > > > >> would be > > > > > >> > > > the > > > > > >> > > > > > > > successor to the very awkward OffsetRequest we > have > > > today. > > > > > >> > > > > > > > > > > > > >> > > > > > > > A: I removed ConsumerGroupOffsetsRequest in the > > latest > > > > > >> patch. I > > > > > >> > > > > believe > > > > > >> > > > > > > > this should > > > > > >> > > > > > > > be resolved in a separate KIP / jira ticket. > > > > > >> > > > > > > > > > > > > >> > > > > > > > 12. Generally we can do good error handling > without > > > > > needing > > > > > >> > > custom > > > > > >> > > > > > > > server-side > > > > > >> > > > > > > > messages. I.e. generally the client has the > context > > to > > > > > know > > > > > >> that > > > > > >> > > if > > > > > >> > > > > it > > > > > >> > > > > > > got > > > > > >> > > > > > > > an error that the topic doesn't exist to say > "Topic > > X > > > > > >> doesn't > > > > > >> > > > exist" > > > > > >> > > > > > > rather > > > > > >> > > > > > > > than "error code 14" (or whatever). Maybe there > are > > > > > specific > > > > > >> > > cases > > > > > >> > > > > > where > > > > > >> > > > > > > > this is hard? If we want to add server-side error > > > messages > > > > > >> we > > > > > >> > > > really > > > > > >> > > > > do > > > > > >> > > > > > > > need to do this in a consistent way across the > > > protocol. > > > > > >> > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please check "Protocol > Errors" > > > > > >> section. I > > > > > >> > > > added > > > > > >> > > > > > the > > > > > >> > > > > > > > comprehensive, fine-grained list of error codes. > > > > > >> > > > > > > > > > > > > >> > > > > > > > Comments from Guozhang: > > > > > >> > > > > > > > 13. Describe topic request: it would be great to > go > > > beyond > > > > > >> just > > > > > >> > > > > > batching > > > > > >> > > > > > > on > > > > > >> > > > > > > > topic name regex for this request. For example, a > > very > > > > > >> common use > > > > > >> > > > > case > > > > > >> > > > > > of > > > > > >> > > > > > > > the topic command is to list all topics whose > config > > > A's > > > > > >> value is > > > > > >> > > > B. > > > > > >> > > > > > With > > > > > >> > > > > > > > topic name regex then we have to first retrieve > > > __all__ > > > > > >> topics's > > > > > >> > > > > > > > description info and then filter at the client > end, > > > which > > > > > >> will > > > > > >> > > be a > > > > > >> > > > > > huge > > > > > >> > > > > > > > burden on ZK. > > > > > >> > > > > > > > AND > > > > > >> > > > > > > > 14. Config K-Vs in create topic: this is related > to > > > the > > > > > >> previous > > > > > >> > > > > point; > > > > > >> > > > > > > > maybe we can add another metadata K-V or just a > > > metadata > > > > > >> string > > > > > >> > > > along > > > > > >> > > > > > > side > > > > > >> > > > > > > > with config K-V in create topic like we did for > > offset > > > > > >> commit > > > > > >> > > > > request. > > > > > >> > > > > > > This > > > > > >> > > > > > > > field can be quite useful in storing information > > like > > > > > >> "owner" of > > > > > >> > > > the > > > > > >> > > > > > > topic > > > > > >> > > > > > > > who issue the create command, etc, which is quite > > > > > important > > > > > >> for a > > > > > >> > > > > > > > multi-tenant setting. Then in the describe topic > > > request > > > > > we > > > > > >> can > > > > > >> > > > also > > > > > >> > > > > > > batch > > > > > >> > > > > > > > on regex of the metadata field. > > > > > >> > > > > > > > > > > > > >> > > > > > > > A: As discussed it is very interesting but can be > > > > > >> implemented > > > > > >> > > later > > > > > >> > > > > > after > > > > > >> > > > > > > > we have some basic functionality there. > > > > > >> > > > > > > > > > > > > >> > > > > > > > 15. Today all the admin operations are async in > the > > > sense > > > > > >> that > > > > > >> > > > > command > > > > > >> > > > > > > will > > > > > >> > > > > > > > return once it is written in ZK, and that is why > we > > > need > > > > > >> extra > > > > > >> > > > > > > verification > > > > > >> > > > > > > > like testUtil.waitForTopicCreated() / verify > > partition > > > > > >> > > reassignment > > > > > >> > > > > > > > request, etc. With admin requests we could add a > > flag > > > to > > > > > >> enable / > > > > > >> > > > > > disable > > > > > >> > > > > > > > synchronous requests; when it is turned on, the > > > response > > > > > >> will not > > > > > >> > > > > > return > > > > > >> > > > > > > > until the request has been completed. And for > async > > > > > >> requests we > > > > > >> > > can > > > > > >> > > > > > add a > > > > > >> > > > > > > > "token" field in the response, and then only need > a > > > > > general > > > > > >> > > "admin > > > > > >> > > > > > > > verification request" with the given token to > check > > > if the > > > > > >> async > > > > > >> > > > > > request > > > > > >> > > > > > > > has been completed. > > > > > >> > > > > > > > > > > > > >> > > > > > > > A: I see your point. My idea was to provide > specific > > > > > >> > > > Verify...Request > > > > > >> > > > > > per > > > > > >> > > > > > > > each > > > > > >> > > > > > > > long running request, where needed. We can do it > the > > > way > > > > > you > > > > > >> > > > suggest. > > > > > >> > > > > > The > > > > > >> > > > > > > > only > > > > > >> > > > > > > > concern is that introducing a token we again will > > make > > > > > >> schema > > > > > >> > > > > > "dynamic". > > > > > >> > > > > > > We > > > > > >> > > > > > > > wanted > > > > > >> > > > > > > > to do similar thing introducing single > AdminRequest > > > for > > > > > all > > > > > >> topic > > > > > >> > > > > > > commands > > > > > >> > > > > > > > but rejected > > > > > >> > > > > > > > this idea because we wanted to have schema > defined. > > So > > > > > this > > > > > >> is > > > > > >> > > > more a > > > > > >> > > > > > > > choice between: > > > > > >> > > > > > > > a) have fixed schema but introduce each time new > > > > > >> Verify...Request > > > > > >> > > > for > > > > > >> > > > > > > > long-running requests > > > > > >> > > > > > > > b) use one request for verification but generalize > > it > > > with > > > > > >> token > > > > > >> > > > > > > > I'm fine with whatever decision community come to. > > > Just > > > > > let > > > > > >> me > > > > > >> > > know > > > > > >> > > > > > your > > > > > >> > > > > > > > thoughts. > > > > > >> > > > > > > > > > > > > >> > > > > > > > Comment from Gwen: > > > > > >> > > > > > > > 16. Specifically for ownership, I think the plan > is > > > to add > > > > > >> ACL > > > > > >> > > (it > > > > > >> > > > > > sounds > > > > > >> > > > > > > > like you are describing ACL) via an external > system > > > > > (Argus, > > > > > >> > > > Sentry). > > > > > >> > > > > > > > I remember KIP-11 described this, but I can't find > > > the KIP > > > > > >> any > > > > > >> > > > > longer. > > > > > >> > > > > > > > > > > > > >> > > > > > > > A: Okay, no problem. Not sure though how we are > > going > > > to > > > > > >> handle > > > > > >> > > it. > > > > > >> > > > > > Wait > > > > > >> > > > > > > > which KIP > > > > > >> > > > > > > > will be committed first and include changes to > > > > > >> TopicMetadata from > > > > > >> > > > the > > > > > >> > > > > > > later > > > > > >> > > > > > > > one? > > > > > >> > > > > > > > Anyway, I added this note to "Open Questions" > > section > > > so > > > > > we > > > > > >> don't > > > > > >> > > > > miss > > > > > >> > > > > > > this > > > > > >> > > > > > > > piece. > > > > > >> > > > > > > > > > > > > >> > > > > > > > Thanks, > > > > > >> > > > > > > > Andrii Biletskyi > > > > > >> > > > > > > > > > > > > >> > > > > > > > On Fri, Mar 13, 2015 at 12:34 AM, Andrii > Biletskyi < > > > > > >> > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > >> > > > > > > > > > > > > >> > > > > > > > > Hi all, > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > Today I uploaded the patch that covers some of > the > > > > > >> discussed > > > > > >> > > and > > > > > >> > > > > > agreed > > > > > >> > > > > > > > > items: > > > > > >> > > > > > > > > - removed MaybeOf optional type > > > > > >> > > > > > > > > - switched to java protocol definitions > > > > > >> > > > > > > > > - simplified messages (normalized configs, > removed > > > topic > > > > > >> marked > > > > > >> > > > for > > > > > >> > > > > > > > > deletion) > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > I also updated the KIP-4 with respective changes > > and > > > > > >> wrote down > > > > > >> > > > my > > > > > >> > > > > > > > > proposal for > > > > > >> > > > > > > > > pending items: > > > > > >> > > > > > > > > - Batch Admin Operations -> updated Wire > Protocol > > > schema > > > > > >> > > proposal > > > > > >> > > > > > > > > - Remove ClusterMetadata -> changed to extend > > > > > >> > > > TopicMetadataRequest > > > > > >> > > > > > > > > - Admin Client -> updated my initial proposal to > > > reflect > > > > > >> > > batching > > > > > >> > > > > > > > > - Error codes -> proposed fine-grained error > code > > > > > instead > > > > > >> of > > > > > >> > > > > > > > > AdminRequestFailed > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > I will also send a separate email to cover all > > > comments > > > > > >> from > > > > > >> > > this > > > > > >> > > > > > > thread. > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > Thanks, > > > > > >> > > > > > > > > Andrii Biletskyi > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > On Thu, Mar 12, 2015 at 9:26 PM, Gwen Shapira < > > > > > >> > > > > gshap...@cloudera.com > > > > > >> > > > > > > > > > > > >> > > > > > > > > wrote: > > > > > >> > > > > > > > > > > > > > >> > > > > > > > >> Found KIP-11 ( > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface > > > > > >> > > > > > > > >> ) > > > > > >> > > > > > > > >> It actually specifies changes to the Metadata > > > protocol, > > > > > >> so > > > > > >> > > > making > > > > > >> > > > > > sure > > > > > >> > > > > > > > >> both KIPs are consistent in this regard will be > > > good. > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > >> On Thu, Mar 12, 2015 at 12:21 PM, Gwen Shapira > < > > > > > >> > > > > > gshap...@cloudera.com > > > > > >> > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > >> > > > > > > > >> > Specifically for ownership, I think the plan > is > > > to > > > > > add > > > > > >> ACL > > > > > >> > > (it > > > > > >> > > > > > > sounds > > > > > >> > > > > > > > >> > like you are describing ACL) via an external > > > system > > > > > >> (Argus, > > > > > >> > > > > > Sentry). > > > > > >> > > > > > > > >> > I remember KIP-11 described this, but I can't > > > find > > > > > the > > > > > >> KIP > > > > > >> > > any > > > > > >> > > > > > > longer. > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > >> > Regardless, I think KIP-4 focuses on getting > > > > > >> information > > > > > >> > > that > > > > > >> > > > > > > already > > > > > >> > > > > > > > >> > exists from Kafka brokers, not on adding > > > information > > > > > >> that > > > > > >> > > > > perhaps > > > > > >> > > > > > > > >> > should exist but doesn't yet? > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > >> > Gwen > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > >> > On Thu, Mar 12, 2015 at 6:37 AM, Guozhang > Wang > > < > > > > > >> > > > > > wangg...@gmail.com> > > > > > >> > > > > > > > >> wrote: > > > > > >> > > > > > > > >> >> Folks, > > > > > >> > > > > > > > >> >> > > > > > >> > > > > > > > >> >> Just want to elaborate a bit more on the > > > > > create-topic > > > > > >> > > > metadata > > > > > >> > > > > > and > > > > > >> > > > > > > > >> batching > > > > > >> > > > > > > > >> >> describe-topic based on config / metadata in > > my > > > > > >> previous > > > > > >> > > > email > > > > > >> > > > > as > > > > > >> > > > > > > we > > > > > >> > > > > > > > >> work > > > > > >> > > > > > > > >> >> on KAFKA-1694. The main motivation is to > have > > > some > > > > > >> sort of > > > > > >> > > > > topic > > > > > >> > > > > > > > >> management > > > > > >> > > > > > > > >> >> mechanisms, which I think is quite important > > in > > > a > > > > > >> > > > multi-tenant > > > > > >> > > > > / > > > > > >> > > > > > > > cloud > > > > > >> > > > > > > > >> >> architecture: today anyone can create topics > > in > > > a > > > > > >> shared > > > > > >> > > > Kafka > > > > > >> > > > > > > > >> cluster, but > > > > > >> > > > > > > > >> >> there is no concept or "ownership" of topics > > > that > > > > > are > > > > > >> > > created > > > > > >> > > > > by > > > > > >> > > > > > > > >> different > > > > > >> > > > > > > > >> >> users. For example, at LinkedIn we basically > > > > > >> distinguish > > > > > >> > > > topic > > > > > >> > > > > > > owners > > > > > >> > > > > > > > >> via > > > > > >> > > > > > > > >> >> some casual topic name prefix, which is a > bit > > > > > awkward > > > > > >> and > > > > > >> > > > does > > > > > >> > > > > > not > > > > > >> > > > > > > > fly > > > > > >> > > > > > > > >> as > > > > > >> > > > > > > > >> >> we scale our customers. It would be great to > > use > > > > > >> > > > > describe-topics > > > > > >> > > > > > > such > > > > > >> > > > > > > > >> as: > > > > > >> > > > > > > > >> >> > > > > > >> > > > > > > > >> >> Describe all topics that is created by me. > > > > > >> > > > > > > > >> >> > > > > > >> > > > > > > > >> >> Describe all topics whose retention time is > > > > > overriden > > > > > >> to X. > > > > > >> > > > > > > > >> >> > > > > > >> > > > > > > > >> >> Describe all topics whose writable group > > include > > > > > user > > > > > >> Y > > > > > >> > > (this > > > > > >> > > > > is > > > > > >> > > > > > > > >> related to > > > > > >> > > > > > > > >> >> authorization), etc.. > > > > > >> > > > > > > > >> >> > > > > > >> > > > > > > > >> >> One possible way to achieve this is to add a > > > > > metadata > > > > > >> file > > > > > >> > > in > > > > > >> > > > > the > > > > > >> > > > > > > > >> >> create-topic request, whose value will also > be > > > > > >> written ZK > > > > > >> > > as > > > > > >> > > > we > > > > > >> > > > > > > > create > > > > > >> > > > > > > > >> the > > > > > >> > > > > > > > >> >> topic; then describe-topics can choose to > > batch > > > > > topics > > > > > >> > > based > > > > > >> > > > on > > > > > >> > > > > > 1) > > > > > >> > > > > > > > name > > > > > >> > > > > > > > >> >> regex, 2) config K-V matching, 3) metadata > > > regex, > > > > > etc. > > > > > >> > > > > > > > >> >> > > > > > >> > > > > > > > >> >> Thoughts? > > > > > >> > > > > > > > >> >> > > > > > >> > > > > > > > >> >> Guozhang > > > > > >> > > > > > > > >> >> > > > > > >> > > > > > > > >> >> On Thu, Mar 5, 2015 at 4:37 PM, Guozhang > Wang > > < > > > > > >> > > > > > wangg...@gmail.com> > > > > > >> > > > > > > > >> wrote: > > > > > >> > > > > > > > >> >> > > > > > >> > > > > > > > >> >>> Thanks for the updated wiki. A few comments > > > below: > > > > > >> > > > > > > > >> >>> > > > > > >> > > > > > > > >> >>> 1. Error description in response: I think > if > > > some > > > > > >> > > errorCode > > > > > >> > > > > > could > > > > > >> > > > > > > > >> indicate > > > > > >> > > > > > > > >> >>> several different error cases then we > should > > > really > > > > > >> change > > > > > >> > > > it > > > > > >> > > > > to > > > > > >> > > > > > > > >> multiple > > > > > >> > > > > > > > >> >>> codes. In general the errorCode itself > would > > be > > > > > >> precise > > > > > >> > > and > > > > > >> > > > > > > > >> sufficient for > > > > > >> > > > > > > > >> >>> describing the server side errors. > > > > > >> > > > > > > > >> >>> > > > > > >> > > > > > > > >> >>> 2. Describe topic request: it would be > great > > > to go > > > > > >> beyond > > > > > >> > > > just > > > > > >> > > > > > > > >> batching on > > > > > >> > > > > > > > >> >>> topic name regex for this request. For > > > example, a > > > > > >> very > > > > > >> > > > common > > > > > >> > > > > > use > > > > > >> > > > > > > > >> case of > > > > > >> > > > > > > > >> >>> the topic command is to list all topics > whose > > > > > config > > > > > >> A's > > > > > >> > > > value > > > > > >> > > > > > is > > > > > >> > > > > > > B. > > > > > >> > > > > > > > >> With > > > > > >> > > > > > > > >> >>> topic name regex then we have to first > > retrieve > > > > > >> __all__ > > > > > >> > > > > topics's > > > > > >> > > > > > > > >> >>> description info and then filter at the > > client > > > end, > > > > > >> which > > > > > >> > > > will > > > > > >> > > > > > be > > > > > >> > > > > > > a > > > > > >> > > > > > > > >> huge > > > > > >> > > > > > > > >> >>> burden on ZK. > > > > > >> > > > > > > > >> >>> > > > > > >> > > > > > > > >> >>> 3. Config K-Vs in create topic: this is > > > related to > > > > > >> the > > > > > >> > > > > previous > > > > > >> > > > > > > > point; > > > > > >> > > > > > > > >> >>> maybe we can add another metadata K-V or > > just a > > > > > >> metadata > > > > > >> > > > > string > > > > > >> > > > > > > > along > > > > > >> > > > > > > > >> side > > > > > >> > > > > > > > >> >>> with config K-V in create topic like we did > > for > > > > > >> offset > > > > > >> > > > commit > > > > > >> > > > > > > > >> request. This > > > > > >> > > > > > > > >> >>> field can be quite useful in storing > > > information > > > > > like > > > > > >> > > > "owner" > > > > > >> > > > > of > > > > > >> > > > > > > the > > > > > >> > > > > > > > >> topic > > > > > >> > > > > > > > >> >>> who issue the create command, etc, which is > > > quite > > > > > >> > > important > > > > > >> > > > > for > > > > > >> > > > > > a > > > > > >> > > > > > > > >> >>> multi-tenant setting. Then in the describe > > > topic > > > > > >> request > > > > > >> > > we > > > > > >> > > > > can > > > > > >> > > > > > > also > > > > > >> > > > > > > > >> batch > > > > > >> > > > > > > > >> >>> on regex of the metadata field. > > > > > >> > > > > > > > >> >>> > > > > > >> > > > > > > > >> >>> 4. Today all the admin operations are async > > in > > > the > > > > > >> sense > > > > > >> > > > that > > > > > >> > > > > > > > command > > > > > >> > > > > > > > >> will > > > > > >> > > > > > > > >> >>> return once it is written in ZK, and that > is > > > why we > > > > > >> need > > > > > >> > > > extra > > > > > >> > > > > > > > >> verification > > > > > >> > > > > > > > >> >>> like testUtil.waitForTopicCreated() / > verify > > > > > >> partition > > > > > >> > > > > > > reassignment > > > > > >> > > > > > > > >> >>> request, etc. With admin requests we could > > add > > > a > > > > > >> flag to > > > > > >> > > > > enable > > > > > >> > > > > > / > > > > > >> > > > > > > > >> disable > > > > > >> > > > > > > > >> >>> synchronous requests; when it is turned on, > > the > > > > > >> response > > > > > >> > > > will > > > > > >> > > > > > not > > > > > >> > > > > > > > >> return > > > > > >> > > > > > > > >> >>> until the request has been completed. And > for > > > async > > > > > >> > > requests > > > > > >> > > > > we > > > > > >> > > > > > > can > > > > > >> > > > > > > > >> add a > > > > > >> > > > > > > > >> >>> "token" field in the response, and then > only > > > need a > > > > > >> > > general > > > > > >> > > > > > "admin > > > > > >> > > > > > > > >> >>> verification request" with the given token > to > > > check > > > > > >> if the > > > > > >> > > > > async > > > > > >> > > > > > > > >> request > > > > > >> > > > > > > > >> >>> has been completed. > > > > > >> > > > > > > > >> >>> > > > > > >> > > > > > > > >> >>> 5. +1 for extending Metadata request to > > include > > > > > >> > > controller / > > > > > >> > > > > > > > >> coordinator > > > > > >> > > > > > > > >> >>> information, and then we can remove the > > > > > >> ConsumerMetadata / > > > > > >> > > > > > > > >> ClusterMetadata > > > > > >> > > > > > > > >> >>> requests. > > > > > >> > > > > > > > >> >>> > > > > > >> > > > > > > > >> >>> Guozhang > > > > > >> > > > > > > > >> >>> > > > > > >> > > > > > > > >> >>> On Tue, Mar 3, 2015 at 10:23 AM, Joel > Koshy < > > > > > >> > > > > > jjkosh...@gmail.com> > > > > > >> > > > > > > > >> wrote: > > > > > >> > > > > > > > >> >>> > > > > > >> > > > > > > > >> >>>> Thanks for sending that out Joe - I don't > > > think I > > > > > >> will be > > > > > >> > > > > able > > > > > >> > > > > > to > > > > > >> > > > > > > > >> make > > > > > >> > > > > > > > >> >>>> it today, so if notes can be sent out > > > afterward > > > > > that > > > > > >> > > would > > > > > >> > > > be > > > > > >> > > > > > > > great. > > > > > >> > > > > > > > >> >>>> > > > > > >> > > > > > > > >> >>>> On Mon, Mar 02, 2015 at 09:16:13AM -0800, > > Gwen > > > > > >> Shapira > > > > > >> > > > wrote: > > > > > >> > > > > > > > >> >>>> > Thanks for sending this out Joe. Looking > > > forward > > > > > >> to > > > > > >> > > > > chatting > > > > > >> > > > > > > with > > > > > >> > > > > > > > >> >>>> everyone :) > > > > > >> > > > > > > > >> >>>> > > > > > > >> > > > > > > > >> >>>> > On Mon, Mar 2, 2015 at 6:46 AM, Joe > Stein > > < > > > > > >> > > > > > > joe.st...@stealth.ly> > > > > > >> > > > > > > > >> wrote: > > > > > >> > > > > > > > >> >>>> > > Hey, I just sent out a google hangout > > > invite > > > > > to > > > > > >> all > > > > > >> > > > pmc, > > > > > >> > > > > > > > >> committers > > > > > >> > > > > > > > >> >>>> and > > > > > >> > > > > > > > >> >>>> > > everyone I found working on a KIP. If > I > > > missed > > > > > >> anyone > > > > > >> > > > in > > > > > >> > > > > > the > > > > > >> > > > > > > > >> invite > > > > > >> > > > > > > > >> >>>> please > > > > > >> > > > > > > > >> >>>> > > let me know and can update it, np. > > > > > >> > > > > > > > >> >>>> > > > > > > > >> > > > > > > > >> >>>> > > We should do this every Tuesday @ 2pm > > > Eastern > > > > > >> Time. > > > > > >> > > > Maybe > > > > > >> > > > > > we > > > > > >> > > > > > > > can > > > > > >> > > > > > > > >> get > > > > > >> > > > > > > > >> >>>> INFRA > > > > > >> > > > > > > > >> >>>> > > help to make a google account so we > can > > > manage > > > > > >> > > better? > > > > > >> > > > > > > > >> >>>> > > > > > > > >> > > > > > > > >> >>>> > > To discuss > > > > > >> > > > > > > > >> >>>> > > > > > > > >> > > > > > > > >> >>>> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > > >> > > > > > > > >> >>>> > > in progress and related JIRA that are > > > > > >> interdependent > > > > > >> > > > and > > > > > >> > > > > > > common > > > > > >> > > > > > > > >> work. > > > > > >> > > > > > > > >> >>>> > > > > > > > >> > > > > > > > >> >>>> > > ~ Joe Stein > > > > > >> > > > > > > > >> >>>> > > > > > > > >> > > > > > > > >> >>>> > > On Tue, Feb 24, 2015 at 2:59 PM, Jay > > > Kreps < > > > > > >> > > > > > > > jay.kr...@gmail.com> > > > > > >> > > > > > > > >> >>>> wrote: > > > > > >> > > > > > > > >> >>>> > > > > > > > >> > > > > > > > >> >>>> > >> Let's stay on Google hangouts that > will > > > also > > > > > >> record > > > > > >> > > > and > > > > > >> > > > > > make > > > > > >> > > > > > > > the > > > > > >> > > > > > > > >> >>>> sessions > > > > > >> > > > > > > > >> >>>> > >> available on youtube. > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> >>>> > >> -Jay > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> >>>> > >> On Tue, Feb 24, 2015 at 11:49 AM, > Jeff > > > > > Holoman > > > > > >> < > > > > > >> > > > > > > > >> >>>> jholo...@cloudera.com> > > > > > >> > > > > > > > >> >>>> > >> wrote: > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> >>>> > >> > Jay / Joe > > > > > >> > > > > > > > >> >>>> > >> > > > > > > >> > > > > > > > >> >>>> > >> > We're happy to send out a Webex for > > > this > > > > > >> purpose. > > > > > >> > > We > > > > > >> > > > > > could > > > > > >> > > > > > > > >> record > > > > > >> > > > > > > > >> >>>> the > > > > > >> > > > > > > > >> >>>> > >> > sessions if there is interest and > > > publish > > > > > >> them > > > > > >> > > out. > > > > > >> > > > > > > > >> >>>> > >> > > > > > > >> > > > > > > > >> >>>> > >> > Thanks > > > > > >> > > > > > > > >> >>>> > >> > > > > > > >> > > > > > > > >> >>>> > >> > Jeff > > > > > >> > > > > > > > >> >>>> > >> > > > > > > >> > > > > > > > >> >>>> > >> > On Tue, Feb 24, 2015 at 11:28 AM, > Jay > > > > > Kreps < > > > > > >> > > > > > > > >> jay.kr...@gmail.com> > > > > > >> > > > > > > > >> >>>> wrote: > > > > > >> > > > > > > > >> >>>> > >> > > > > > > >> > > > > > > > >> >>>> > >> > > Let's try to get the technical > > > hang-ups > > > > > >> sorted > > > > > >> > > > out, > > > > > >> > > > > > > > though. > > > > > >> > > > > > > > >> I > > > > > >> > > > > > > > >> >>>> really > > > > > >> > > > > > > > >> >>>> > >> > think > > > > > >> > > > > > > > >> >>>> > >> > > there is some benefit to live > > > discussion > > > > > vs > > > > > >> > > > > writing. I > > > > > >> > > > > > > am > > > > > >> > > > > > > > >> >>>> hopeful that > > > > > >> > > > > > > > >> >>>> > >> if > > > > > >> > > > > > > > >> >>>> > >> > > we post instructions and give > > > ourselves a > > > > > >> few > > > > > >> > > > > attempts > > > > > >> > > > > > > we > > > > > >> > > > > > > > >> can > > > > > >> > > > > > > > >> >>>> get it > > > > > >> > > > > > > > >> >>>> > >> > > working. > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > >> > > > > > > > >> >>>> > >> > > Tuesday at that time would work > for > > > > > >> me...any > > > > > >> > > > > > objections? > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > >> > > > > > > > >> >>>> > >> > > -Jay > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > >> > > > > > > > >> >>>> > >> > > On Tue, Feb 24, 2015 at 8:18 AM, > > Joe > > > > > Stein > > > > > >> < > > > > > >> > > > > > > > >> joe.st...@stealth.ly > > > > > >> > > > > > > > >> >>>> > > > > > > >> > > > > > > > >> >>>> > >> wrote: > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > >> > > > > > > > >> >>>> > >> > > > Weekly would be great maybe > like > > > every > > > > > >> > > Tuesday ~ > > > > > >> > > > > 1pm > > > > > >> > > > > > > ET > > > > > >> > > > > > > > / > > > > > >> > > > > > > > >> 10am > > > > > >> > > > > > > > >> >>>> PT > > > > > >> > > > > > > > >> >>>> > >> ???? > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > > > I don't mind google hangout but > > > there > > > > > is > > > > > >> > > always > > > > > >> > > > > some > > > > > >> > > > > > > > >> issue or > > > > > >> > > > > > > > >> >>>> > >> whatever > > > > > >> > > > > > > > >> >>>> > >> > so > > > > > >> > > > > > > > >> >>>> > >> > > > we know the apache irc channel > > > works. > > > > > We > > > > > >> can > > > > > >> > > > start > > > > > >> > > > > > > there > > > > > >> > > > > > > > >> and > > > > > >> > > > > > > > >> >>>> see how > > > > > >> > > > > > > > >> >>>> > >> it > > > > > >> > > > > > > > >> >>>> > >> > > > goes? We can pull transcripts > too > > > and > > > > > >> > > associate > > > > > >> > > > to > > > > > >> > > > > > > > >> tickets if > > > > > >> > > > > > > > >> >>>> need be > > > > > >> > > > > > > > >> >>>> > >> > > makes > > > > > >> > > > > > > > >> >>>> > >> > > > it helpful for things. > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > > > ~ Joestein > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > > > On Tue, Feb 24, 2015 at 11:10 > AM, > > > Jay > > > > > >> Kreps < > > > > > >> > > > > > > > >> >>>> jay.kr...@gmail.com> > > > > > >> > > > > > > > >> >>>> > >> > wrote: > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > We'd talked about doing a > > Google > > > > > >> Hangout to > > > > > >> > > > chat > > > > > >> > > > > > > about > > > > > >> > > > > > > > >> this. > > > > > >> > > > > > > > >> >>>> What > > > > > >> > > > > > > > >> >>>> > >> > about > > > > > >> > > > > > > > >> >>>> > >> > > > > generalizing that a little > > > > > further...I > > > > > >> > > > actually > > > > > >> > > > > > > think > > > > > >> > > > > > > > it > > > > > >> > > > > > > > >> >>>> would be > > > > > >> > > > > > > > >> >>>> > >> > good > > > > > >> > > > > > > > >> >>>> > >> > > > for > > > > > >> > > > > > > > >> >>>> > >> > > > > everyone spending a > reasonable > > > chunk > > > > > of > > > > > >> > > their > > > > > >> > > > > week > > > > > >> > > > > > > on > > > > > >> > > > > > > > >> Kafka > > > > > >> > > > > > > > >> >>>> stuff > > > > > >> > > > > > > > >> >>>> > >> to > > > > > >> > > > > > > > >> >>>> > >> > > > maybe > > > > > >> > > > > > > > >> >>>> > >> > > > > sync up once a week. I think > we > > > could > > > > > >> use > > > > > >> > > time > > > > > >> > > > > to > > > > > >> > > > > > > talk > > > > > >> > > > > > > > >> >>>> through > > > > > >> > > > > > > > >> >>>> > >> design > > > > > >> > > > > > > > >> >>>> > >> > > > > stuff, make sure we are on > top > > of > > > > > code > > > > > >> > > > reviews, > > > > > >> > > > > > talk > > > > > >> > > > > > > > >> through > > > > > >> > > > > > > > >> >>>> any > > > > > >> > > > > > > > >> >>>> > >> > tricky > > > > > >> > > > > > > > >> >>>> > >> > > > > issues, etc. > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > We can make it publicly > > > available so > > > > > >> that > > > > > >> > > any > > > > > >> > > > > one > > > > > >> > > > > > > can > > > > > >> > > > > > > > >> follow > > > > > >> > > > > > > > >> >>>> along > > > > > >> > > > > > > > >> >>>> > >> > who > > > > > >> > > > > > > > >> >>>> > >> > > > > likes. > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > Any interest in doing this? > If > > so > > > > > I'll > > > > > >> try > > > > > >> > > to > > > > > >> > > > > set > > > > > >> > > > > > it > > > > > >> > > > > > > > up > > > > > >> > > > > > > > >> >>>> starting > > > > > >> > > > > > > > >> >>>> > >> next > > > > > >> > > > > > > > >> >>>> > >> > > > week. > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > -Jay > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > On Tue, Feb 24, 2015 at 3:57 > > AM, > > > > > Andrii > > > > > >> > > > > Biletskyi > > > > > >> > > > > > < > > > > > >> > > > > > > > >> >>>> > >> > > > > andrii.bilets...@stealth.ly> > > > wrote: > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > Hi all, > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > I've updated KIP page, > fixed > > / > > > > > >> aligned > > > > > >> > > > > document > > > > > >> > > > > > > > >> structure. > > > > > >> > > > > > > > >> >>>> Also I > > > > > >> > > > > > > > >> >>>> > >> > > added > > > > > >> > > > > > > > >> >>>> > >> > > > > > some > > > > > >> > > > > > > > >> >>>> > >> > > > > > very initial proposal for > > > > > >> AdminClient so > > > > > >> > > we > > > > > >> > > > > have > > > > > >> > > > > > > > >> something > > > > > >> > > > > > > > >> >>>> to > > > > > >> > > > > > > > >> >>>> > >> start > > > > > >> > > > > > > > >> >>>> > >> > > > from > > > > > >> > > > > > > > >> >>>> > >> > > > > > while > > > > > >> > > > > > > > >> >>>> > >> > > > > > discussing the KIP. > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> >>>> > > > > > >> > > > > > > > >> > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > Thanks, > > > > > >> > > > > > > > >> >>>> > >> > > > > > Andrii Biletskyi > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > On Wed, Feb 18, 2015 at > 9:01 > > > PM, > > > > > >> Andrii > > > > > >> > > > > > Biletskyi > > > > > >> > > > > > > < > > > > > >> > > > > > > > >> >>>> > >> > > > > > > andrii.bilets...@stealth.ly> > > > > > wrote: > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Jay, > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Re error messages: you > are > > > right, > > > > > >> in > > > > > >> > > most > > > > > >> > > > > > cases > > > > > >> > > > > > > > >> client > > > > > >> > > > > > > > >> >>>> will > > > > > >> > > > > > > > >> >>>> > >> have > > > > > >> > > > > > > > >> >>>> > >> > > > enough > > > > > >> > > > > > > > >> >>>> > >> > > > > > > context to show > descriptive > > > error > > > > > >> > > message. > > > > > >> > > > > My > > > > > >> > > > > > > > >> concern is > > > > > >> > > > > > > > >> >>>> that > > > > > >> > > > > > > > >> >>>> > >> we > > > > > >> > > > > > > > >> >>>> > >> > > will > > > > > >> > > > > > > > >> >>>> > >> > > > > > have > > > > > >> > > > > > > > >> >>>> > >> > > > > > > to > > > > > >> > > > > > > > >> >>>> > >> > > > > > > add lots of new error > codes > > > for > > > > > >> each > > > > > >> > > > > possible > > > > > >> > > > > > > > >> error. Of > > > > > >> > > > > > > > >> >>>> course, > > > > > >> > > > > > > > >> >>>> > >> > we > > > > > >> > > > > > > > >> >>>> > >> > > > > could > > > > > >> > > > > > > > >> >>>> > >> > > > > > > reuse > > > > > >> > > > > > > > >> >>>> > >> > > > > > > some of existing like > > > > > >> > > > > > > UknownTopicOrPartitionCode, > > > > > >> > > > > > > > >> but we > > > > > >> > > > > > > > >> >>>> will > > > > > >> > > > > > > > >> >>>> > >> > also > > > > > >> > > > > > > > >> >>>> > >> > > > need > > > > > >> > > > > > > > >> >>>> > >> > > > > > to > > > > > >> > > > > > > > >> >>>> > >> > > > > > > add smth like: > > > > > >> TopicAlreadyExistsCode, > > > > > >> > > > > > > > >> >>>> TopicConfigInvalid (both > > > > > >> > > > > > > > >> >>>> > >> > for > > > > > >> > > > > > > > >> >>>> > >> > > > > topic > > > > > >> > > > > > > > >> >>>> > >> > > > > > > name and config, and > > probably > > > > > user > > > > > >> would > > > > > >> > > > > like > > > > > >> > > > > > to > > > > > >> > > > > > > > >> know > > > > > >> > > > > > > > >> >>>> what > > > > > >> > > > > > > > >> >>>> > >> > exactly > > > > > >> > > > > > > > >> >>>> > >> > > > > > > is wrong in his config), > > > > > >> > > > > > > InvalidReplicaAssignment, > > > > > >> > > > > > > > >> >>>> > >> InternalError > > > > > >> > > > > > > > >> >>>> > >> > > > (e.g. > > > > > >> > > > > > > > >> >>>> > >> > > > > > > zookeeper failure) etc. > > > > > >> > > > > > > > >> >>>> > >> > > > > > > And this is only for > > > > > TopicCommand, > > > > > >> we > > > > > >> > > will > > > > > >> > > > > > also > > > > > >> > > > > > > > >> need to > > > > > >> > > > > > > > >> >>>> add > > > > > >> > > > > > > > >> >>>> > >> > similar > > > > > >> > > > > > > > >> >>>> > >> > > > > stuff > > > > > >> > > > > > > > >> >>>> > >> > > > > > > for > > > > > >> > > > > > > > >> >>>> > >> > > > > > > ReassignPartitions, > > > > > >> PreferredReplica. So > > > > > >> > > > > we'll > > > > > >> > > > > > > end > > > > > >> > > > > > > > >> up > > > > > >> > > > > > > > >> >>>> with a > > > > > >> > > > > > > > >> >>>> > >> > large > > > > > >> > > > > > > > >> >>>> > >> > > > list > > > > > >> > > > > > > > >> >>>> > >> > > > > > of > > > > > >> > > > > > > > >> >>>> > >> > > > > > > error codes, used only in > > > Admin > > > > > >> > > protocol. > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Having said that, I agree > > my > > > > > >> proposal is > > > > > >> > > > not > > > > > >> > > > > > > > >> consistent > > > > > >> > > > > > > > >> >>>> with > > > > > >> > > > > > > > >> >>>> > >> > other > > > > > >> > > > > > > > >> >>>> > >> > > > > cases. > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Maybe we can find better > > > solution > > > > > >> or > > > > > >> > > > > something > > > > > >> > > > > > > > >> >>>> in-between. > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Re Hangout chat: I think > it > > > is a > > > > > >> great > > > > > >> > > > idea. > > > > > >> > > > > > > This > > > > > >> > > > > > > > >> way we > > > > > >> > > > > > > > >> >>>> can > > > > > >> > > > > > > > >> >>>> > >> move > > > > > >> > > > > > > > >> >>>> > >> > > on > > > > > >> > > > > > > > >> >>>> > >> > > > > > > faster. > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Let's agree somehow on > > > date/time > > > > > so > > > > > >> > > people > > > > > >> > > > > can > > > > > >> > > > > > > > join. > > > > > >> > > > > > > > >> >>>> Will work > > > > > >> > > > > > > > >> >>>> > >> > for > > > > > >> > > > > > > > >> >>>> > >> > > me > > > > > >> > > > > > > > >> >>>> > >> > > > > > this > > > > > >> > > > > > > > >> >>>> > >> > > > > > > and > > > > > >> > > > > > > > >> >>>> > >> > > > > > > next week almost anytime > if > > > > > agreed > > > > > >> in > > > > > >> > > > > advance. > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Thanks, > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Andrii > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > On Wed, Feb 18, 2015 at > > 7:09 > > > PM, > > > > > >> Jay > > > > > >> > > > Kreps < > > > > > >> > > > > > > > >> >>>> > >> jay.kr...@gmail.com> > > > > > >> > > > > > > > >> >>>> > >> > > > > wrote: > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> Hey Andrii, > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> Generally we can do good > > > error > > > > > >> handling > > > > > >> > > > > > without > > > > > >> > > > > > > > >> needing > > > > > >> > > > > > > > >> >>>> custom > > > > > >> > > > > > > > >> >>>> > >> > > > > > server-side > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> messages. I.e. generally > > the > > > > > >> client has > > > > > >> > > > the > > > > > >> > > > > > > > >> context to > > > > > >> > > > > > > > >> >>>> know > > > > > >> > > > > > > > >> >>>> > >> that > > > > > >> > > > > > > > >> >>>> > >> > > if > > > > > >> > > > > > > > >> >>>> > >> > > > it > > > > > >> > > > > > > > >> >>>> > >> > > > > > got > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> an error that the topic > > > doesn't > > > > > >> exist > > > > > >> > > to > > > > > >> > > > > say > > > > > >> > > > > > > > >> "Topic X > > > > > >> > > > > > > > >> >>>> doesn't > > > > > >> > > > > > > > >> >>>> > >> > > exist" > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> rather > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> than "error code 14" (or > > > > > >> whatever). > > > > > >> > > Maybe > > > > > >> > > > > > there > > > > > >> > > > > > > > are > > > > > >> > > > > > > > >> >>>> specific > > > > > >> > > > > > > > >> >>>> > >> > cases > > > > > >> > > > > > > > >> >>>> > >> > > > > where > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> this is hard? If we want > > to > > > add > > > > > >> > > > server-side > > > > > >> > > > > > > error > > > > > >> > > > > > > > >> >>>> messages we > > > > > >> > > > > > > > >> >>>> > >> > > really > > > > > >> > > > > > > > >> >>>> > >> > > > > do > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> need to do this in a > > > consistent > > > > > >> way > > > > > >> > > > across > > > > > >> > > > > > the > > > > > >> > > > > > > > >> protocol. > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> I still have a bunch of > > open > > > > > >> questions > > > > > >> > > > here > > > > > >> > > > > > > from > > > > > >> > > > > > > > my > > > > > >> > > > > > > > >> >>>> previous > > > > > >> > > > > > > > >> >>>> > >> > > list. I > > > > > >> > > > > > > > >> >>>> > >> > > > > > will > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> be out for the next few > > > days for > > > > > >> Strata > > > > > >> > > > > > though. > > > > > >> > > > > > > > >> Maybe > > > > > >> > > > > > > > >> >>>> we could > > > > > >> > > > > > > > >> >>>> > >> > do > > > > > >> > > > > > > > >> >>>> > >> > > a > > > > > >> > > > > > > > >> >>>> > >> > > > > > Google > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> Hangout chat on any open > > > issues > > > > > >> some > > > > > >> > > time > > > > > >> > > > > > > towards > > > > > >> > > > > > > > >> the > > > > > >> > > > > > > > >> >>>> end of > > > > > >> > > > > > > > >> >>>> > >> > next > > > > > >> > > > > > > > >> >>>> > >> > > > week > > > > > >> > > > > > > > >> >>>> > >> > > > > > for > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> anyone interested in > this > > > > > ticket? > > > > > >> I > > > > > >> > > have > > > > > >> > > > a > > > > > >> > > > > > > > feeling > > > > > >> > > > > > > > >> that > > > > > >> > > > > > > > >> >>>> might > > > > > >> > > > > > > > >> >>>> > >> > > > progress > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> things a little faster > > than > > > > > >> email--I > > > > > >> > > > think > > > > > >> > > > > we > > > > > >> > > > > > > > >> could talk > > > > > >> > > > > > > > >> >>>> > >> through > > > > > >> > > > > > > > >> >>>> > >> > > > those > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> issues I brought up > fairly > > > > > >> quickly... > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> -Jay > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> On Wed, Feb 18, 2015 at > > > 7:27 AM, > > > > > >> Andrii > > > > > >> > > > > > > > Biletskyi < > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > andrii.bilets...@stealth.ly > > > > > > > > > >> wrote: > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > Hi all, > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > I'm trying to address > > > some of > > > > > >> the > > > > > >> > > > issues > > > > > >> > > > > > > which > > > > > >> > > > > > > > >> were > > > > > >> > > > > > > > >> >>>> > >> mentioned > > > > > >> > > > > > > > >> >>>> > >> > > > > earlier > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> about > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > Admin RQ/RP format. > One > > of > > > > > >> those was > > > > > >> > > > > about > > > > > >> > > > > > > > >> batching > > > > > >> > > > > > > > >> >>>> > >> > operations. > > > > > >> > > > > > > > >> >>>> > >> > > > What > > > > > >> > > > > > > > >> >>>> > >> > > > > > if > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> we > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > follow TopicCommand > > > approach > > > > > >> and let > > > > > >> > > > > people > > > > > >> > > > > > > > >> specify > > > > > >> > > > > > > > >> >>>> > >> topic-name > > > > > >> > > > > > > > >> >>>> > >> > > by > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> regexp - > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > would that cover most > of > > > the > > > > > use > > > > > >> > > cases? > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > Secondly, is what > > > information > > > > > >> should > > > > > >> > > we > > > > > >> > > > > > > > generally > > > > > >> > > > > > > > >> >>>> provide in > > > > > >> > > > > > > > >> >>>> > >> > > Admin > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > responses. > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > I realize that Admin > > > commands > > > > > >> don't > > > > > >> > > > imply > > > > > >> > > > > > > they > > > > > >> > > > > > > > >> will > > > > > >> > > > > > > > >> >>>> be used > > > > > >> > > > > > > > >> >>>> > >> > only > > > > > >> > > > > > > > >> >>>> > >> > > > in > > > > > >> > > > > > > > >> >>>> > >> > > > > > CLI > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > but, > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > it seems to me, CLI > is a > > > very > > > > > >> > > important > > > > > >> > > > > > > client > > > > > >> > > > > > > > >> of this > > > > > >> > > > > > > > >> >>>> > >> > feature. > > > > > >> > > > > > > > >> >>>> > >> > > In > > > > > >> > > > > > > > >> >>>> > >> > > > > > this > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > case, > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > seems logical, we > would > > > like > > > > > to > > > > > >> > > provide > > > > > >> > > > > > users > > > > > >> > > > > > > > >> with > > > > > >> > > > > > > > >> >>>> rich > > > > > >> > > > > > > > >> >>>> > >> > > experience > > > > > >> > > > > > > > >> >>>> > >> > > > > in > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> terms > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > of > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > getting results / > errors > > > of > > > > > the > > > > > >> > > > executed > > > > > >> > > > > > > > >> commands. > > > > > >> > > > > > > > >> >>>> Usually > > > > > >> > > > > > > > >> >>>> > >> we > > > > > >> > > > > > > > >> >>>> > >> > > > supply > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> with > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > responses only > > errorCode, > > > > > which > > > > > >> looks > > > > > >> > > > > very > > > > > >> > > > > > > > >> limiting, > > > > > >> > > > > > > > >> >>>> in case > > > > > >> > > > > > > > >> >>>> > >> > of > > > > > >> > > > > > > > >> >>>> > >> > > > CLI > > > > > >> > > > > > > > >> >>>> > >> > > > > we > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> may > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > want to print human > > > readable > > > > > >> error > > > > > >> > > > > > > description. > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > So, taking into > account > > > > > >> previous item > > > > > >> > > > > about > > > > > >> > > > > > > > >> batching, > > > > > >> > > > > > > > >> >>>> what > > > > > >> > > > > > > > >> >>>> > >> do > > > > > >> > > > > > > > >> >>>> > >> > > you > > > > > >> > > > > > > > >> >>>> > >> > > > > > think > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > about > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > having smth like: > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > ('create' doesn't > > support > > > > > >> regexp) > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > CreateTopicRequest => > > > > > TopicName > > > > > >> > > > > Partitions > > > > > >> > > > > > > > >> Replicas > > > > > >> > > > > > > > >> >>>> > >> > > > > ReplicaAssignment > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > [Config] > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > CreateTopicResponse => > > > > > ErrorCode > > > > > >> > > > > > > > ErrorDescription > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > ErrorCode => int16 > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > ErrorDescription => > > > string > > > > > >> (empty > > > > > >> > > if > > > > > >> > > > > > > > >> successful) > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > AlterTopicRequest -> > > > > > >> TopicNameRegexp > > > > > >> > > > > > > > > > > > > > > > ... > > > > > > > > > > > > [Message clipped] > > > > > > > > > > > > > >
