+1 as well. I think it helps to keep the rerouting approach orthogonal to this KIP.
On Wed, Mar 18, 2015 at 03:40:48PM -0700, Jay Kreps wrote: > I'm +1 on Jun's suggestion as long as it can work for all the requests. > > On Wed, Mar 18, 2015 at 3:35 PM, Jun Rao <j...@confluent.io> wrote: > > > Andrii, > > > > I think we agreed on the following. > > > > (a) Admin requests can be sent to and handled by any broker. > > (b) Admin requests are processed asynchronously, at least for now. That is, > > when the client gets a response, it just means that the request is > > initiated, but not necessarily completed. Then, it's up to the client to > > issue another request to check the status for completion. > > > > To support (a), we were thinking of doing request forwarding to the > > controller (utilizing KAFKA-1912). I am making an alternative proposal. > > Basically, the broker can just write to ZooKeeper to inform the controller > > about the request. For example, to handle partitionReassignment, the broker > > will just write the requested partitions to /admin/reassign_partitions > > (like what AdminUtils currently does) and then send a response to the > > client. This shouldn't take long and the implementation will be simpler > > than forwarding the requests to the controller through RPC. > > > > Thanks, > > > > Jun > > > > > > On Wed, Mar 18, 2015 at 3:03 PM, Andrii Biletskyi < > > andrii.bilets...@stealth.ly> wrote: > > > > > Jun, > > > > > > I might be wrong but didn't we agree we will let any broker from the > > > cluster handle *long-running* admin requests (at this time > > preferredReplica > > > and > > > reassignPartitions), via zk admin path. Thus CreateTopics etc should be > > > sent > > > only to the controller. > > > > > > Thanks, > > > Andrii Biletskyi > > > > > > On Wed, Mar 18, 2015 at 11:55 PM, Jun Rao <j...@confluent.io> wrote: > > > > > > > Joel, Andril, > > > > > > > > I think we agreed that those admin requests can be issued to any > > broker. > > > > Because of that, there doesn't seem to be a strong need to know the > > > > controller. So, perhaps we can proceed by not making any change to the > > > > format of TMR right now. When we start using create topic request in > > the > > > > producer, we will need a new version of TMR that doesn't trigger auto > > > topic > > > > creation. But that can be done later. > > > > > > > > As a first cut implementation, I think the broker can just write to ZK > > > > directly for > > > > createToipic/alterTopic/reassignPartitions/preferredLeaderElection > > > > requests, instead of forwarding them to the controller. This will > > > simplify > > > > the implementation on the broker side. > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > > > > > On Wed, Mar 18, 2015 at 11:58 AM, Joel Koshy <jjkosh...@gmail.com> > > > wrote: > > > > > > > > > (Thanks Andrii for the summary) > > > > > > > > > > For (1) yes we will circle back on that shortly after syncing up in > > > > > person. I think it is close to getting committed although development > > > > > for KAFKA-1927 can probably begin without it. > > > > > > > > > > There is one more item we covered at the hangout. i.e., whether we > > > > > want to add the coordinator to the topic metadata response or provide > > > > > a clearer ClusterMetadataRequest. > > > > > > > > > > There are two reasons I think we should try and avoid adding the > > > > > field: > > > > > - It is irrelevant to topic metadata > > > > > - If we finally do request rerouting in Kafka then the field would > > add > > > > > little to no value. (It still helps to have a separate > > > > > ClusterMetadataRequest to query for cluster-wide information such > > as > > > > > 'which broker is the controller?' as Joe mentioned.) > > > > > > > > > > I think it would be cleaner to have an explicit > > ClusterMetadataRequest > > > > > that you can send to any broker in order to obtain the controller > > (and > > > > > in the future possibly other cluster-wide information). I think the > > > > > main argument against doing this and instead adding it to the topic > > > > > metadata response was convenience - i.e., you don't have to discover > > > > > the controller in advance. However, I don't see much actual > > > > > benefit/convenience in this and in fact think it is a non-issue. Let > > > > > me know if I'm overlooking something here. > > > > > > > > > > As an example, say we need to initiate partition reassignment by > > > > > issuing the new ReassignPartitionsRequest to the controller (assume > > we > > > > > already have the desired manual partition assignment). If we are to > > > > > augment topic metadata response then the flow be something like this > > : > > > > > > > > > > - Issue topic metadata request to any broker (and discover the > > > > > controller > > > > > - Connect to controller if required (i.e., if the broker above != > > > > > controller) > > > > > - Issue the partition reassignment request to the controller. > > > > > > > > > > With an explicit cluster metadata request it would be: > > > > > - Issue cluster metadata request to any broker > > > > > - Connect to controller if required (i.e., if the broker above != > > > > > controller) > > > > > - Issue the partition reassignment request > > > > > > > > > > So it seems to add little practical value and bloats topic metadata > > > > > response with an irrelevant detail. > > > > > > > > > > The other angle to this is the following - is it a matter of naming? > > > > > Should we just rename topic metadata request/response to just > > > > > MetadataRequest/Response and add cluster metadata to it? By that same > > > > > token should we also allow querying for the consumer coordinator (and > > > > > in future transaction coordinator) as well? This leads to a bloated > > > > > request which isn't very appealing and altogether confusing. > > > > > > > > > > Thanks, > > > > > > > > > > Joel > > > > > > > > > > On Wed, Mar 18, 2015 at 09:34:12AM -0700, Jun Rao wrote: > > > > > > Andri, > > > > > > > > > > > > Thanks for the summary. > > > > > > > > > > > > 1. I just realized that in order to start working on KAFKA-1927, we > > > > will > > > > > > need to merge the changes to OffsetCommitRequest (from 0.8.2) to > > > trunk. > > > > > > This is planned to be done as part of KAFKA-1634. So, we will need > > > > > Guozhang > > > > > > and Joel's help to wrap this up. > > > > > > > > > > > > 2. Thinking about this a bit more, if the semantic of those "write" > > > > > > requests is async (i.e., after the client gets a response, it just > > > > means > > > > > > that the operation is initiated, but not necessarily completed), we > > > > don't > > > > > > really need to forward the requests to the controller. Instead, the > > > > > > receiving broker can just write the operation to ZK as the admin > > > > command > > > > > > line tool previously does. This will simplify the implementation. > > > > > > > > > > > > 8. There is another implementation detail for describe topic. > > > Ideally, > > > > we > > > > > > want to read the topic config from the broker cache, instead of > > > > > ZooKeeper. > > > > > > Currently, every broker reads the topic-level config for all > > topics. > > > > > > However, it ignores those for topics not hosted on itself. So, we > > may > > > > > need > > > > > > to change TopicConfigManager a bit so that it caches the configs > > for > > > > all > > > > > > topics. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 1:13 PM, Andrii Biletskyi < > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > Guys, > > > > > > > > > > > > > > Thanks for a great discussion! > > > > > > > Here are the actions points: > > > > > > > > > > > > > > 1. Q: Get rid of all scala requests objects, use java protocol > > > > > definitions. > > > > > > > A: Gwen kindly took that (KAFKA-1927). It's important to > > speed > > > up > > > > > > > review procedure > > > > > > > there since this ticket blocks other important changes. > > > > > > > > > > > > > > 2. Q: Generic re-reroute facility vs client maintaining cluster > > > > state. > > > > > > > A: Jay has added pseudo code to KAFKA-1912 - need to consider > > > > > whether > > > > > > > this will be > > > > > > > easy to implement as a server-side feature (comments are > > > > > > > welcomed!). > > > > > > > > > > > > > > 3. Q: Controller field in wire protocol. > > > > > > > A: This might be useful for clients, add this to > > > > > TopicMetadataResponse > > > > > > > (already in KIP). > > > > > > > > > > > > > > 4. Q: Decoupling topic creation from TMR. > > > > > > > A: I will add proposed by Jun solution (using clientId for > > > that) > > > > > to the > > > > > > > KIP. > > > > > > > > > > > > > > 5. Q: Bumping new versions of TMR vs grabbing all protocol > > changes > > > in > > > > > one > > > > > > > version. > > > > > > > A: It was decided to try to gather all changes to protocol > > > > (before > > > > > > > release). > > > > > > > In case of TMR it worth checking: KAFKA-2020 and KIP-13 > > > > > (quotas) > > > > > > > > > > > > > > 6. Q: JSON lib is needed to deserialize user's input in CLI tool. > > > > > > > A: Use jackson for that, /tools project is a separate jar so > > > > > shouldn't > > > > > > > be a big deal. > > > > > > > > > > > > > > 7. Q: VerifyReassingPartitions vs generic status check command. > > > > > > > A: For long-running requests like reassign partitions > > > *progress* > > > > > check > > > > > > > request is useful, > > > > > > > it makes sense to introduce it. > > > > > > > > > > > > > > Please add, correct me if I missed something. > > > > > > > > > > > > > > Thanks, > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 6:20 PM, Andrii Biletskyi < > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > > > > > > > > > Joel, > > > > > > > > > > > > > > > > You are right, I removed ClusterMetadata because we have > > > partially > > > > > > > > what we need in TopicMetadata. Also, as Jay pointed out > > earlier, > > > we > > > > > > > > would like to have "orthogonal" API, but at the same time we > > need > > > > > > > > to be backward compatible. > > > > > > > > > > > > > > > > But I like your idea and even have some other arguments for > > this > > > > > option: > > > > > > > > There is also DescribeTopicRequest which was proposed in this > > > KIP, > > > > > > > > it returns topic configs, partitions, replication factor plus > > > > > partition > > > > > > > > ISR, ASR, > > > > > > > > leader replica. The later part is really already there in > > > > > > > > TopicMetadataRequest. > > > > > > > > So again we'll have to add stuff to TMR, not to duplicate some > > > info > > > > > in > > > > > > > > newly added requests. However, this way we'll end up with > > > "monster" > > > > > > > > request which returns cluster metadata, topic replication and > > > > config > > > > > info > > > > > > > > plus partition replication data. Seems logical to split TMR to > > > > > > > > - ClusterMetadata (brokers + controller, maybe smth else) > > > > > > > > - TopicMetadata (topic info + partition details) > > > > > > > > But since current TMR is involved in lots of places (including > > > > > network > > > > > > > > client, > > > > > > > > as I understand) this might be very serious change and it > > > probably > > > > > makes > > > > > > > > sense to stick with current approach. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Andrii Biletskyi > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2015 at 5:29 PM, Joel Koshy < > > jjkosh...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > > > > > >> I may be missing some context but hopefully this will also be > > > > > covered > > > > > > > >> today: I thought the earlier proposal where there was an > > > explicit > > > > > > > >> ClusterMetadata request was clearer and explicit. During the > > > > course > > > > > of > > > > > > > >> this thread I think the conclusion was that the main need was > > > for > > > > > > > >> controller information and that can be rolled into the topic > > > > > metadata > > > > > > > >> response but that seems a bit irrelevant to topic metadata. > > > FWIW I > > > > > > > >> think the full broker-list is also irrelevant to topic > > metadata, > > > > but > > > > > > > >> it is already there and in use. I think there is still room > > for > > > an > > > > > > > >> explicit ClusterMetadata request since there may be other > > > > > > > >> cluster-level information that we may want to add over time > > (and > > > > > that > > > > > > > >> have nothing to do with topic metadata). > > > > > > > >> > > > > > > > >> On Tue, Mar 17, 2015 at 02:45:30PM +0200, Andrii Biletskyi > > > wrote: > > > > > > > >> > Jun, > > > > > > > >> > > > > > > > > >> > 101. Okay, if you say that such use case is important. I > > also > > > > > think > > > > > > > >> > using clientId for these purposes is fine - if we already > > have > > > > > this > > > > > > > >> field > > > > > > > >> > as part of all Wire protocol messages, why not use that. > > > > > > > >> > I will update KIP-4 page if nobody has other ideas (which > > may > > > > > come up > > > > > > > >> > during the call today). > > > > > > > >> > > > > > > > > >> > 102.1 Agree, I'll update the KIP accordingly. I think we can > > > add > > > > > new, > > > > > > > >> > fine-grained error codes if some error code received in > > > specific > > > > > case > > > > > > > >> > won't give enough context to return a descriptive error > > > message > > > > > for > > > > > > > >> user. > > > > > > > >> > > > > > > > > >> > Look forward to discussing all outstanding issues in detail > > > > today > > > > > > > during > > > > > > > >> > the call. > > > > > > > >> > > > > > > > > >> > Thanks, > > > > > > > >> > Andrii Biletskyi > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > On Mon, Mar 16, 2015 at 10:59 PM, Jun Rao <j...@confluent.io > > > > > > > > wrote: > > > > > > > >> > > > > > > > > >> > > 101. There may be a use case where you only want the > > topics > > > to > > > > > be > > > > > > > >> created > > > > > > > >> > > manually by admins. Currently, you can do that by > > disabling > > > > auto > > > > > > > topic > > > > > > > >> > > creation and issue topic creation from the TopicCommand. > > If > > > we > > > > > > > >> disable auto > > > > > > > >> > > topic creation completely on the broker and don't have a > > way > > > > to > > > > > > > >> distinguish > > > > > > > >> > > between topic creation requests from the regular clients > > and > > > > the > > > > > > > >> admin, we > > > > > > > >> > > can't support manual topic creation any more. I was > > thinking > > > > > that > > > > > > > >> another > > > > > > > >> > > way of distinguishing the clients making the topic > > creation > > > > > requests > > > > > > > >> is > > > > > > > >> > > using clientId. For example, the admin tool can set it to > > > > > something > > > > > > > >> like > > > > > > > >> > > admin and the broker can treat that clientId specially. > > > > > > > >> > > > > > > > > > >> > > Also, there is a related discussion in KAFKA-2020. > > > Currently, > > > > > we do > > > > > > > >> the > > > > > > > >> > > following in TopicMetadataResponse: > > > > > > > >> > > > > > > > > > >> > > 1. If leader is not available, we set the partition level > > > > error > > > > > code > > > > > > > >> to > > > > > > > >> > > LeaderNotAvailable. > > > > > > > >> > > 2. If a non-leader replica is not available, we take that > > > > > replica > > > > > > > out > > > > > > > >> of > > > > > > > >> > > the assigned replica list and isr in the response. As an > > > > > indication > > > > > > > >> for > > > > > > > >> > > doing that, we set the partition level error code to > > > > > > > >> ReplicaNotAvailable. > > > > > > > >> > > > > > > > > > >> > > This has a few problems. First, ReplicaNotAvailable > > probably > > > > > > > >> shouldn't be > > > > > > > >> > > an error, at least for the normal producer/consumer > > clients > > > > that > > > > > > > just > > > > > > > >> want > > > > > > > >> > > to find out the leader. Second, it can happen that both > > the > > > > > leader > > > > > > > and > > > > > > > >> > > another replica are not available at the same time. There > > is > > > > no > > > > > > > error > > > > > > > >> code > > > > > > > >> > > to indicate both. Third, even if a replica is not > > available, > > > > > it's > > > > > > > >> still > > > > > > > >> > > useful to return its replica id since some clients (e.g. > > > admin > > > > > tool) > > > > > > > >> may > > > > > > > >> > > still make use of it. > > > > > > > >> > > > > > > > > > >> > > One way to address this issue is to always return the > > > replica > > > > > id for > > > > > > > >> > > leader, assigned replicas, and isr regardless of whether > > the > > > > > > > >> corresponding > > > > > > > >> > > broker is live or not. Since we also return the list of > > live > > > > > > > brokers, > > > > > > > >> the > > > > > > > >> > > client can figure out whether a leader or a replica is > > live > > > or > > > > > not > > > > > > > >> and act > > > > > > > >> > > accordingly. This way, we don't need to set the partition > > > > level > > > > > > > error > > > > > > > >> code > > > > > > > >> > > when the leader or a replica is not available. This > > doesn't > > > > > change > > > > > > > >> the wire > > > > > > > >> > > protocol, but does change the semantics. Since we are > > > evolving > > > > > the > > > > > > > >> protocol > > > > > > > >> > > of TopicMetadataRequest here, we can potentially piggyback > > > the > > > > > > > change. > > > > > > > >> > > > > > > > > > >> > > 102.1 For those types of errors due to invalid input, > > > > shouldn't > > > > > we > > > > > > > >> just > > > > > > > >> > > guard it at parameter validation time and throw > > > > > > > >> InvalidArgumentException > > > > > > > >> > > without even sending the request to the broker? > > > > > > > >> > > > > > > > > > >> > > Thanks, > > > > > > > >> > > > > > > > > > >> > > Jun > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > On Mon, Mar 16, 2015 at 10:37 AM, Andrii Biletskyi < > > > > > > > >> > > andrii.bilets...@stealth.ly> wrote: > > > > > > > >> > > > > > > > > > >> > > > Jun, > > > > > > > >> > > > > > > > > > > >> > > > Answering your questions: > > > > > > > >> > > > > > > > > > > >> > > > 101. If I understand you correctly, you are saying > > future > > > > > producer > > > > > > > >> > > versions > > > > > > > >> > > > (which > > > > > > > >> > > > will be ported to TMR_V1) won't be able to automatically > > > > > create > > > > > > > >> topic (if > > > > > > > >> > > > we > > > > > > > >> > > > unconditionally remove topic creation from there). But > > we > > > > > need to > > > > > > > >> this > > > > > > > >> > > > preserve logic. > > > > > > > >> > > > Ok, about your proposal: I'm not a big fan too, when it > > > > comes > > > > > to > > > > > > > >> > > > differentiating > > > > > > > >> > > > clients directly in protocol schema. And also I'm not > > > sure I > > > > > > > >> understand > > > > > > > >> > > at > > > > > > > >> > > > all why > > > > > > > >> > > > auto.create.topics.enable is a server side > > configuration. > > > > Can > > > > > we > > > > > > > >> > > deprecate > > > > > > > >> > > > this setting > > > > > > > >> > > > in future versions, add this setting to producer and > > based > > > > on > > > > > that > > > > > > > >> upon > > > > > > > >> > > > receiving > > > > > > > >> > > > UnknownTopic create topic explicitly by a separate > > > producer > > > > > call > > > > > > > via > > > > > > > >> > > > adminClient? > > > > > > > >> > > > > > > > > > > >> > > > 102.1. Hm, yes. It's because we want to support batching > > > and > > > > > at > > > > > > > the > > > > > > > >> same > > > > > > > >> > > > time we > > > > > > > >> > > > want to give descriptive error messages for clients. > > Since > > > > > > > >> AdminClient > > > > > > > >> > > > holds the context > > > > > > > >> > > > to construct such messages (e.g. AdminClient layer can > > > know > > > > > that > > > > > > > >> > > > InvalidArgumentsCode > > > > > > > >> > > > means two cases: either invalid number - e.g. -1; or > > > > > > > >> replication-factor > > > > > > > >> > > was > > > > > > > >> > > > provided while > > > > > > > >> > > > partitions argument wasn't) - I wrapped responses in > > > > > Exceptions. > > > > > > > >> But I'm > > > > > > > >> > > > open to any > > > > > > > >> > > > other ideas, this was just initial version. > > > > > > > >> > > > 102.2. Yes, I agree. I'll change that to probably some > > > other > > > > > dto. > > > > > > > >> > > > > > > > > > > >> > > > Thanks, > > > > > > > >> > > > Andrii Biletskyi > > > > > > > >> > > > > > > > > > > >> > > > On Fri, Mar 13, 2015 at 7:16 PM, Jun Rao < > > > j...@confluent.io> > > > > > > > wrote: > > > > > > > >> > > > > > > > > > > >> > > > > Andrii, > > > > > > > >> > > > > > > > > > > > >> > > > > 101. That's what I was thinking too, but it may not be > > > > that > > > > > > > >> simple. In > > > > > > > >> > > > > TopicMetadataRequest_V1, > > > > > > > >> > > > > we can let it not trigger auto topic creation. Then, > > in > > > > the > > > > > > > >> producer > > > > > > > >> > > > side, > > > > > > > >> > > > > if it gets an UnknownTopicException, it can explicitly > > > > > issue a > > > > > > > >> > > > > createTopicRequest for auto topic creation. On the > > > > consumer > > > > > > > side, > > > > > > > >> it > > > > > > > >> > > will > > > > > > > >> > > > > never issue createTopicRequest. This works when auto > > > topic > > > > > > > >> creation is > > > > > > > >> > > > > enabled on the broker side. However, I am not sure how > > > > > things > > > > > > > >> will work > > > > > > > >> > > > > when auto topic creation is disabled on the broker > > side. > > > > In > > > > > this > > > > > > > >> case, > > > > > > > >> > > we > > > > > > > >> > > > > want to have a way to manually create a topic, > > > potentially > > > > > > > through > > > > > > > >> > > admin > > > > > > > >> > > > > commands. However, then we need a way to distinguish > > > > > > > >> createTopicRequest > > > > > > > >> > > > > issued from the producer clients and the admin tools. > > > May > > > > > be we > > > > > > > >> can > > > > > > > >> > > add a > > > > > > > >> > > > > new field in createTopicRequest and set it differently > > > in > > > > > the > > > > > > > >> producer > > > > > > > >> > > > > client and the admin client. However, I am not sure if > > > > > that's > > > > > > > the > > > > > > > >> best > > > > > > > >> > > > > approach. > > > > > > > >> > > > > > > > > > > > >> > > > > 2. Yes, refactoring existing requests is a non-trivial > > > > > amount of > > > > > > > >> work. > > > > > > > >> > > I > > > > > > > >> > > > > posted some comments in KAFKA-1927. We will probably > > > have > > > > > to fix > > > > > > > >> > > > KAFKA-1927 > > > > > > > >> > > > > first, before adding the new logic in KAFKA-1694. > > > > > Otherwise, the > > > > > > > >> > > changes > > > > > > > >> > > > > will be too big. > > > > > > > >> > > > > > > > > > > > >> > > > > 102. About the AdminClient: > > > > > > > >> > > > > 102.1. It's a bit weird that we return exception in > > the > > > > > api. It > > > > > > > >> seems > > > > > > > >> > > > that > > > > > > > >> > > > > we should either return error code or throw an > > exception > > > > > when > > > > > > > >> getting > > > > > > > >> > > the > > > > > > > >> > > > > response state. > > > > > > > >> > > > > 102.2. We probably shouldn't explicitly use the > > request > > > > > object > > > > > > > in > > > > > > > >> the > > > > > > > >> > > > api. > > > > > > > >> > > > > Not every request evolution requires an api change. > > > > > > > >> > > > > > > > > > > > >> > > > > Thanks, > > > > > > > >> > > > > > > > > > > > >> > > > > Jun > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > On Fri, Mar 13, 2015 at 4:08 AM, Andrii Biletskyi < > > > > > > > >> > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > >> > > > > > > > > > > > >> > > > > > Jun, > > > > > > > >> > > > > > > > > > > > > >> > > > > > Thanks for you comments. Answers inline: > > > > > > > >> > > > > > > > > > > > > >> > > > > > 100. There are a few fields such as > > ReplicaAssignment, > > > > > > > >> > > > > > > ReassignPartitionRequest, > > > > > > > >> > > > > > > and PartitionsSerialized that are represented as a > > > > > string, > > > > > > > but > > > > > > > >> > > > contain > > > > > > > >> > > > > > > composite structures in json. Could we flatten > > them > > > > out > > > > > > > >> directly in > > > > > > > >> > > > the > > > > > > > >> > > > > > > protocol definition as arrays/records? > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > Yes, now with Admin Client this looks a bit weird. > > My > > > > > initial > > > > > > > >> > > > motivation > > > > > > > >> > > > > > was: > > > > > > > >> > > > > > ReassignPartitionCommand accepts input in json, we > > > want > > > > to > > > > > > > >> remain > > > > > > > >> > > > tools' > > > > > > > >> > > > > > interfaces unchanged, where possible. > > > > > > > >> > > > > > If we port it to deserialized format, in CLI (/tools > > > > > project) > > > > > > > >> we will > > > > > > > >> > > > > have > > > > > > > >> > > > > > to add some > > > > > > > >> > > > > > json library since /tools is written in java and > > we'll > > > > > need to > > > > > > > >> > > > > deserialize > > > > > > > >> > > > > > json file > > > > > > > >> > > > > > provided by a user. Can we quickly agree on what > > this > > > > > library > > > > > > > >> should > > > > > > > >> > > be > > > > > > > >> > > > > > (Jackson, GSON, whatever)? > > > > > > > >> > > > > > > > > > > > > >> > > > > > 101. Does TopicMetadataRequest v1 still trigger auto > > > > topic > > > > > > > >> creation? > > > > > > > >> > > > This > > > > > > > >> > > > > > > will be a bit weird now that we have a separate > > > topic > > > > > > > >> creation api. > > > > > > > >> > > > > Have > > > > > > > >> > > > > > > you thought about how the new createTopicRequest > > and > > > > > > > >> > > > > TopicMetadataRequest > > > > > > > >> > > > > > > v1 will be used in the producer/consumer client, > > in > > > > > addition > > > > > > > >> to > > > > > > > >> > > admin > > > > > > > >> > > > > > > tools? For example, ideally, we don't want > > > > > > > >> TopicMetadataRequest > > > > > > > >> > > from > > > > > > > >> > > > > the > > > > > > > >> > > > > > > consumer to trigger auto topic creation. > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > I agree, this strange logic should be fixed. I'm not > > > > > confident > > > > > > > >> in > > > > > > > >> > > this > > > > > > > >> > > > > > Kafka part so > > > > > > > >> > > > > > correct me if I'm wrong, but it doesn't look like a > > > hard > > > > > thing > > > > > > > >> to > > > > > > > >> > > do, I > > > > > > > >> > > > > > think we can > > > > > > > >> > > > > > leverage AdminClient for that in Producer and > > > > > unconditionally > > > > > > > >> remove > > > > > > > >> > > > > topic > > > > > > > >> > > > > > creation from the TopicMetadataRequest_V1. > > > > > > > >> > > > > > > > > > > > > >> > > > > > 2. I think Jay meant getting rid of scala classes > > > > > > > >> > > > > > > like HeartbeatRequestAndHeader and > > > > > > > >> HeartbeatResponseAndHeader. We > > > > > > > >> > > did > > > > > > > >> > > > > > that > > > > > > > >> > > > > > > as a stop-gap thing when adding the new requests > > for > > > > the > > > > > > > >> consumers. > > > > > > > >> > > > > > > However, the long term plan is to get rid of all > > > those > > > > > and > > > > > > > >> just > > > > > > > >> > > reuse > > > > > > > >> > > > > the > > > > > > > >> > > > > > > java request/response in the client. Since this > > KIP > > > > > proposes > > > > > > > >> to > > > > > > > >> > > add a > > > > > > > >> > > > > > > significant number of new requests, perhaps we > > > should > > > > > bite > > > > > > > the > > > > > > > >> > > bullet > > > > > > > >> > > > > to > > > > > > > >> > > > > > > clean up the existing scala requests first before > > > > > adding new > > > > > > > >> ones? > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > Yes, looks like I misunderstood the point of > > > > > > > >> ...RequestAndHeader. > > > > > > > >> > > > Okay, I > > > > > > > >> > > > > > will > > > > > > > >> > > > > > rework that. The only thing is that I don't see any > > > > > example > > > > > > > how > > > > > > > >> it > > > > > > > >> > > was > > > > > > > >> > > > > done > > > > > > > >> > > > > > for at > > > > > > > >> > > > > > least one existing protocol message. Thus, as I > > > > > understand, I > > > > > > > >> have to > > > > > > > >> > > > > think > > > > > > > >> > > > > > how we > > > > > > > >> > > > > > are going to do it. > > > > > > > >> > > > > > Re porting all existing RQ/RP in this patch. Sounds > > > > > > > reasonable, > > > > > > > >> but > > > > > > > >> > > if > > > > > > > >> > > > > it's > > > > > > > >> > > > > > an *obligatory* > > > > > > > >> > > > > > requirement to have Admin KIP done, I'm afraid this > > > can > > > > > be a > > > > > > > >> serious > > > > > > > >> > > > > > blocker for us. > > > > > > > >> > > > > > There are 13 protocol messages and all that would > > > > require > > > > > not > > > > > > > >> only > > > > > > > >> > > unit > > > > > > > >> > > > > > tests but quite > > > > > > > >> > > > > > intensive manual testing, no? I'm afraid I'm not the > > > > > right guy > > > > > > > >> to > > > > > > > >> > > cover > > > > > > > >> > > > > > pretty much all > > > > > > > >> > > > > > Kafka core internals :). Let me know your thoughts > > on > > > > this > > > > > > > >> item. Btw > > > > > > > >> > > > > there > > > > > > > >> > > > > > is a ticket to > > > > > > > >> > > > > > follow-up this issue ( > > > > > > > >> > > https://issues.apache.org/jira/browse/KAFKA-2006 > > > > > > > >> > > > ). > > > > > > > >> > > > > > > > > > > > > >> > > > > > Thanks, > > > > > > > >> > > > > > Andrii Biletskyi > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > On Fri, Mar 13, 2015 at 6:40 AM, Jun Rao < > > > > > j...@confluent.io> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > > > > > > >> > > > > > > Andrii, > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > A few more comments. > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > 100. There are a few fields such as > > > ReplicaAssignment, > > > > > > > >> > > > > > > ReassignPartitionRequest, > > > > > > > >> > > > > > > and PartitionsSerialized that are represented as a > > > > > string, > > > > > > > but > > > > > > > >> > > > contain > > > > > > > >> > > > > > > composite structures in json. Could we flatten > > them > > > > out > > > > > > > >> directly in > > > > > > > >> > > > the > > > > > > > >> > > > > > > protocol definition as arrays/records? > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > 101. Does TopicMetadataRequest v1 still trigger > > auto > > > > > topic > > > > > > > >> > > creation? > > > > > > > >> > > > > This > > > > > > > >> > > > > > > will be a bit weird now that we have a separate > > > topic > > > > > > > >> creation api. > > > > > > > >> > > > > Have > > > > > > > >> > > > > > > you thought about how the new createTopicRequest > > and > > > > > > > >> > > > > TopicMetadataRequest > > > > > > > >> > > > > > > v1 will be used in the producer/consumer client, > > in > > > > > addition > > > > > > > >> to > > > > > > > >> > > admin > > > > > > > >> > > > > > > tools? For example, ideally, we don't want > > > > > > > >> TopicMetadataRequest > > > > > > > >> > > from > > > > > > > >> > > > > the > > > > > > > >> > > > > > > consumer to trigger auto topic creation. > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > 2. I think Jay meant getting rid of scala classes > > > > > > > >> > > > > > > like HeartbeatRequestAndHeader and > > > > > > > >> HeartbeatResponseAndHeader. We > > > > > > > >> > > did > > > > > > > >> > > > > > that > > > > > > > >> > > > > > > as a stop-gap thing when adding the new requests > > for > > > > the > > > > > > > >> consumers. > > > > > > > >> > > > > > > However, the long term plan is to get rid of all > > > those > > > > > and > > > > > > > >> just > > > > > > > >> > > reuse > > > > > > > >> > > > > the > > > > > > > >> > > > > > > java request/response in the client. Since this > > KIP > > > > > proposes > > > > > > > >> to > > > > > > > >> > > add a > > > > > > > >> > > > > > > significant number of new requests, perhaps we > > > should > > > > > bite > > > > > > > the > > > > > > > >> > > bullet > > > > > > > >> > > > > to > > > > > > > >> > > > > > > clean up the existing scala requests first before > > > > > adding new > > > > > > > >> ones? > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > Thanks, > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > Jun > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > On Thu, Mar 12, 2015 at 3:37 PM, Andrii Biletskyi > > < > > > > > > > >> > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > Hi, > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > As said above - I list again all comments from > > > this > > > > > thread > > > > > > > >> so we > > > > > > > >> > > > > > > > can see what's left and finalize all pending > > > issues. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > Comments from Jay: > > > > > > > >> > > > > > > > 1. This is much needed functionality, but there > > > are > > > > a > > > > > lot > > > > > > > >> of the > > > > > > > >> > > so > > > > > > > >> > > > > > let's > > > > > > > >> > > > > > > > really think these protocols through. We really > > > want > > > > > to > > > > > > > end > > > > > > > >> up > > > > > > > >> > > > with a > > > > > > > >> > > > > > set > > > > > > > >> > > > > > > > of well thought-out, orthoganol apis. For this > > > > reason > > > > > I > > > > > > > >> think it > > > > > > > >> > > is > > > > > > > >> > > > > > > really > > > > > > > >> > > > > > > > important to think through the end state even if > > > > that > > > > > > > >> includes > > > > > > > >> > > APIs > > > > > > > >> > > > > we > > > > > > > >> > > > > > > > won't implement in the first phase. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > A: Definitely behind this. Would appreciate if > > > there > > > > > are > > > > > > > >> concrete > > > > > > > >> > > > > > > comments > > > > > > > >> > > > > > > > how this can be improved. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > 2. Let's please please please wait until we have > > > > > switched > > > > > > > >> the > > > > > > > >> > > > server > > > > > > > >> > > > > > over > > > > > > > >> > > > > > > > to the new java protocol definitions. If we add > > > > upteen > > > > > > > more > > > > > > > >> ad > > > > > > > >> > > hoc > > > > > > > >> > > > > > scala > > > > > > > >> > > > > > > > objects that is just generating more work for > > the > > > > > > > >> conversion we > > > > > > > >> > > > know > > > > > > > >> > > > > we > > > > > > > >> > > > > > > > have to do. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - removed scala > > > > protocol > > > > > > > >> classes. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > 3. This proposal introduces a new type of > > optional > > > > > > > >> parameter. > > > > > > > >> > > This > > > > > > > >> > > > is > > > > > > > >> > > > > > > > inconsistent with everything else in the > > protocol > > > > > where we > > > > > > > >> use -1 > > > > > > > >> > > > or > > > > > > > >> > > > > > some > > > > > > > >> > > > > > > > other marker value. You could argue either way > > but > > > > > let's > > > > > > > >> stick > > > > > > > >> > > with > > > > > > > >> > > > > > that > > > > > > > >> > > > > > > > for consistency. For clients that implemented > > the > > > > > protocol > > > > > > > >> in a > > > > > > > >> > > > > better > > > > > > > >> > > > > > > way > > > > > > > >> > > > > > > > than our scala code these basic primitives are > > > hard > > > > to > > > > > > > >> change. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - removed MaybeOf > > > type > > > > > and > > > > > > > >> changed > > > > > > > >> > > > > > protocol > > > > > > > >> > > > > > > > accordingly. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > 4. ClusterMetadata: This seems to duplicate > > > > > > > >> TopicMetadataRequest > > > > > > > >> > > > > which > > > > > > > >> > > > > > > has > > > > > > > >> > > > > > > > brokers, topics, and partitions. I think we > > should > > > > > rename > > > > > > > >> that > > > > > > > >> > > > > request > > > > > > > >> > > > > > > > ClusterMetadataRequest (or just MetadataRequest) > > > and > > > > > > > >> include the > > > > > > > >> > > id > > > > > > > >> > > > > of > > > > > > > >> > > > > > > the > > > > > > > >> > > > > > > > controller. Or are there other things we could > > add > > > > > here? > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > A: I agree. Updated the KIP. Let's extends > > > > > TopicMetadata > > > > > > > to > > > > > > > >> > > > version 2 > > > > > > > >> > > > > > and > > > > > > > >> > > > > > > > include controller. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > 5. We have a tendency to try to make a lot of > > > > requests > > > > > > > that > > > > > > > >> can > > > > > > > >> > > > only > > > > > > > >> > > > > go > > > > > > > >> > > > > > > to > > > > > > > >> > > > > > > > particular nodes. This adds a lot of burden for > > > > client > > > > > > > >> > > > > implementations > > > > > > > >> > > > > > > (it > > > > > > > >> > > > > > > > sounds easy but each discovery can fail in many > > > > parts > > > > > so > > > > > > > it > > > > > > > >> ends > > > > > > > >> > > up > > > > > > > >> > > > > > > being a > > > > > > > >> > > > > > > > full state machine to do right). I think we > > should > > > > > > > consider > > > > > > > >> > > making > > > > > > > >> > > > > > admin > > > > > > > >> > > > > > > > commands and ideally as many of the other apis > > as > > > > > possible > > > > > > > >> > > > available > > > > > > > >> > > > > on > > > > > > > >> > > > > > > all > > > > > > > >> > > > > > > > brokers and just redirect to the controller on > > the > > > > > broker > > > > > > > >> side. > > > > > > > >> > > > > Perhaps > > > > > > > >> > > > > > > > there would be a general way to encapsulate this > > > > > > > re-routing > > > > > > > >> > > > behavior. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > A: It's a very interesting idea, but seems there > > > are > > > > > some > > > > > > > >> > > concerns > > > > > > > >> > > > > > about > > > > > > > >> > > > > > > > this > > > > > > > >> > > > > > > > feature (like performance considerations, how > > this > > > > > will > > > > > > > >> > > complicate > > > > > > > >> > > > > > server > > > > > > > >> > > > > > > > etc). > > > > > > > >> > > > > > > > I believe this shouldn't be a blocker. If this > > > > > feature is > > > > > > > >> > > > implemented > > > > > > > >> > > > > > at > > > > > > > >> > > > > > > > some > > > > > > > >> > > > > > > > point it won't affect Admin changes - at least > > no > > > > > changes > > > > > > > to > > > > > > > >> > > public > > > > > > > >> > > > > API > > > > > > > >> > > > > > > > will be required. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > 6. We should probably normalize the key value > > > pairs > > > > > used > > > > > > > for > > > > > > > >> > > > configs > > > > > > > >> > > > > > > rather > > > > > > > >> > > > > > > > than embedding a new formatting. So two strings > > > > rather > > > > > > > than > > > > > > > >> one > > > > > > > >> > > > with > > > > > > > >> > > > > an > > > > > > > >> > > > > > > > internal equals sign. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - normalized > > configs > > > > and > > > > > > > >> changed > > > > > > > >> > > > > protocol > > > > > > > >> > > > > > > > accordingly. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > 7. Is the postcondition of these APIs that the > > > > > command has > > > > > > > >> begun > > > > > > > >> > > or > > > > > > > >> > > > > > that > > > > > > > >> > > > > > > > the command has been completed? It is a lot more > > > > > usable if > > > > > > > >> the > > > > > > > >> > > > > command > > > > > > > >> > > > > > > has > > > > > > > >> > > > > > > > been completed so you know that if you create a > > > > topic > > > > > and > > > > > > > >> then > > > > > > > >> > > > > publish > > > > > > > >> > > > > > to > > > > > > > >> > > > > > > > it you won't get an exception about there being > > no > > > > > such > > > > > > > >> topic. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > A: For long running requests (like reassign > > > > > partitions) - > > > > > > > >> the > > > > > > > >> > > post > > > > > > > >> > > > > > > > condition is > > > > > > > >> > > > > > > > command has begun - so we don't block the > > client. > > > In > > > > > case > > > > > > > >> of your > > > > > > > >> > > > > > > example - > > > > > > > >> > > > > > > > topic commands, this will be refactored and > > topic > > > > > commands > > > > > > > >> will > > > > > > > >> > > be > > > > > > > >> > > > > > > executed > > > > > > > >> > > > > > > > immediately, since the Controller will serve > > Admin > > > > > > > requests > > > > > > > >> > > > > > > > (follow-up ticket KAFKA-1777). > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > 8. Describe topic and list topics duplicate a > > lot > > > of > > > > > stuff > > > > > > > >> in the > > > > > > > >> > > > > > > metadata > > > > > > > >> > > > > > > > request. Is there a reason to give back topics > > > > marked > > > > > for > > > > > > > >> > > > deletion? I > > > > > > > >> > > > > > > feel > > > > > > > >> > > > > > > > like if we just make the post-condition of the > > > > delete > > > > > > > >> command be > > > > > > > >> > > > that > > > > > > > >> > > > > > the > > > > > > > >> > > > > > > > topic is deleted that will get rid of the need > > for > > > > > this > > > > > > > >> right? > > > > > > > >> > > And > > > > > > > >> > > > it > > > > > > > >> > > > > > > will > > > > > > > >> > > > > > > > be much more intuitive. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > A: Fixed in the latest patch - removed topics > > > marked > > > > > for > > > > > > > >> deletion > > > > > > > >> > > > in > > > > > > > >> > > > > > > > ListTopicsRequest. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > 9. Should we consider batching these requests? > > We > > > > have > > > > > > > >> generally > > > > > > > >> > > > > tried > > > > > > > >> > > > > > to > > > > > > > >> > > > > > > > allow multiple operations to be batched. My > > > > suspicion > > > > > is > > > > > > > >> that > > > > > > > >> > > > without > > > > > > > >> > > > > > > this > > > > > > > >> > > > > > > > we will get a lot of code that does something > > like > > > > > > > >> > > > > > > > for(topic: adminClient.listTopics()) > > > > > > > >> > > > > > > > adminClient.describeTopic(topic) > > > > > > > >> > > > > > > > this code will work great when you test on 5 > > > topics > > > > > but > > > > > > > not > > > > > > > >> do as > > > > > > > >> > > > > well > > > > > > > >> > > > > > if > > > > > > > >> > > > > > > > you have 50k. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please check "Topic Admin > > > > Schema" > > > > > > > >> section. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > 10. I think we should also discuss how we want > > to > > > > > expose a > > > > > > > >> > > > > programmatic > > > > > > > >> > > > > > > JVM > > > > > > > >> > > > > > > > client api for these operations. Currently > > people > > > > > rely on > > > > > > > >> > > > AdminUtils > > > > > > > >> > > > > > > which > > > > > > > >> > > > > > > > is totally sketchy. I think we probably need > > > another > > > > > > > client > > > > > > > >> under > > > > > > > >> > > > > > > clients/ > > > > > > > >> > > > > > > > that exposes administrative functionality. We > > will > > > > > need > > > > > > > >> this just > > > > > > > >> > > > to > > > > > > > >> > > > > > > > properly test the new apis, I suspect. We should > > > > > figure > > > > > > > out > > > > > > > >> that > > > > > > > >> > > > API. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please check "Admin Client" > > > > > section > > > > > > > >> with an > > > > > > > >> > > > > > initial > > > > > > > >> > > > > > > > API proposal. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > 11. The other information that would be really > > > > useful > > > > > to > > > > > > > get > > > > > > > >> > > would > > > > > > > >> > > > be > > > > > > > >> > > > > > > > information about partitions--how much data is > > in > > > > the > > > > > > > >> partition, > > > > > > > >> > > > what > > > > > > > >> > > > > > are > > > > > > > >> > > > > > > > the segment offsets, what is the log-end offset > > > > (i.e. > > > > > last > > > > > > > >> > > offset), > > > > > > > >> > > > > > what > > > > > > > >> > > > > > > is > > > > > > > >> > > > > > > > the compaction point, etc. I think that done > > right > > > > > this > > > > > > > >> would be > > > > > > > >> > > > the > > > > > > > >> > > > > > > > successor to the very awkward OffsetRequest we > > > have > > > > > today. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > A: I removed ConsumerGroupOffsetsRequest in the > > > > latest > > > > > > > >> patch. I > > > > > > > >> > > > > believe > > > > > > > >> > > > > > > > this should > > > > > > > >> > > > > > > > be resolved in a separate KIP / jira ticket. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > 12. Generally we can do good error handling > > > without > > > > > > > needing > > > > > > > >> > > custom > > > > > > > >> > > > > > > > server-side > > > > > > > >> > > > > > > > messages. I.e. generally the client has the > > > context > > > > to > > > > > > > know > > > > > > > >> that > > > > > > > >> > > if > > > > > > > >> > > > > it > > > > > > > >> > > > > > > got > > > > > > > >> > > > > > > > an error that the topic doesn't exist to say > > > "Topic > > > > X > > > > > > > >> doesn't > > > > > > > >> > > > exist" > > > > > > > >> > > > > > > rather > > > > > > > >> > > > > > > > than "error code 14" (or whatever). Maybe there > > > are > > > > > > > specific > > > > > > > >> > > cases > > > > > > > >> > > > > > where > > > > > > > >> > > > > > > > this is hard? If we want to add server-side > > error > > > > > messages > > > > > > > >> we > > > > > > > >> > > > really > > > > > > > >> > > > > do > > > > > > > >> > > > > > > > need to do this in a consistent way across the > > > > > protocol. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > A: Updated the KIP - please check "Protocol > > > Errors" > > > > > > > >> section. I > > > > > > > >> > > > added > > > > > > > >> > > > > > the > > > > > > > >> > > > > > > > comprehensive, fine-grained list of error codes. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > Comments from Guozhang: > > > > > > > >> > > > > > > > 13. Describe topic request: it would be great to > > > go > > > > > beyond > > > > > > > >> just > > > > > > > >> > > > > > batching > > > > > > > >> > > > > > > on > > > > > > > >> > > > > > > > topic name regex for this request. For example, > > a > > > > very > > > > > > > >> common use > > > > > > > >> > > > > case > > > > > > > >> > > > > > of > > > > > > > >> > > > > > > > the topic command is to list all topics whose > > > config > > > > > A's > > > > > > > >> value is > > > > > > > >> > > > B. > > > > > > > >> > > > > > With > > > > > > > >> > > > > > > > topic name regex then we have to first retrieve > > > > > __all__ > > > > > > > >> topics's > > > > > > > >> > > > > > > > description info and then filter at the client > > > end, > > > > > which > > > > > > > >> will > > > > > > > >> > > be a > > > > > > > >> > > > > > huge > > > > > > > >> > > > > > > > burden on ZK. > > > > > > > >> > > > > > > > AND > > > > > > > >> > > > > > > > 14. Config K-Vs in create topic: this is related > > > to > > > > > the > > > > > > > >> previous > > > > > > > >> > > > > point; > > > > > > > >> > > > > > > > maybe we can add another metadata K-V or just a > > > > > metadata > > > > > > > >> string > > > > > > > >> > > > along > > > > > > > >> > > > > > > side > > > > > > > >> > > > > > > > with config K-V in create topic like we did for > > > > offset > > > > > > > >> commit > > > > > > > >> > > > > request. > > > > > > > >> > > > > > > This > > > > > > > >> > > > > > > > field can be quite useful in storing information > > > > like > > > > > > > >> "owner" of > > > > > > > >> > > > the > > > > > > > >> > > > > > > topic > > > > > > > >> > > > > > > > who issue the create command, etc, which is > > quite > > > > > > > important > > > > > > > >> for a > > > > > > > >> > > > > > > > multi-tenant setting. Then in the describe topic > > > > > request > > > > > > > we > > > > > > > >> can > > > > > > > >> > > > also > > > > > > > >> > > > > > > batch > > > > > > > >> > > > > > > > on regex of the metadata field. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > A: As discussed it is very interesting but can > > be > > > > > > > >> implemented > > > > > > > >> > > later > > > > > > > >> > > > > > after > > > > > > > >> > > > > > > > we have some basic functionality there. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > 15. Today all the admin operations are async in > > > the > > > > > sense > > > > > > > >> that > > > > > > > >> > > > > command > > > > > > > >> > > > > > > will > > > > > > > >> > > > > > > > return once it is written in ZK, and that is why > > > we > > > > > need > > > > > > > >> extra > > > > > > > >> > > > > > > verification > > > > > > > >> > > > > > > > like testUtil.waitForTopicCreated() / verify > > > > partition > > > > > > > >> > > reassignment > > > > > > > >> > > > > > > > request, etc. With admin requests we could add a > > > > flag > > > > > to > > > > > > > >> enable / > > > > > > > >> > > > > > disable > > > > > > > >> > > > > > > > synchronous requests; when it is turned on, the > > > > > response > > > > > > > >> will not > > > > > > > >> > > > > > return > > > > > > > >> > > > > > > > until the request has been completed. And for > > > async > > > > > > > >> requests we > > > > > > > >> > > can > > > > > > > >> > > > > > add a > > > > > > > >> > > > > > > > "token" field in the response, and then only > > need > > > a > > > > > > > general > > > > > > > >> > > "admin > > > > > > > >> > > > > > > > verification request" with the given token to > > > check > > > > > if the > > > > > > > >> async > > > > > > > >> > > > > > request > > > > > > > >> > > > > > > > has been completed. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > A: I see your point. My idea was to provide > > > specific > > > > > > > >> > > > Verify...Request > > > > > > > >> > > > > > per > > > > > > > >> > > > > > > > each > > > > > > > >> > > > > > > > long running request, where needed. We can do it > > > the > > > > > way > > > > > > > you > > > > > > > >> > > > suggest. > > > > > > > >> > > > > > The > > > > > > > >> > > > > > > > only > > > > > > > >> > > > > > > > concern is that introducing a token we again > > will > > > > make > > > > > > > >> schema > > > > > > > >> > > > > > "dynamic". > > > > > > > >> > > > > > > We > > > > > > > >> > > > > > > > wanted > > > > > > > >> > > > > > > > to do similar thing introducing single > > > AdminRequest > > > > > for > > > > > > > all > > > > > > > >> topic > > > > > > > >> > > > > > > commands > > > > > > > >> > > > > > > > but rejected > > > > > > > >> > > > > > > > this idea because we wanted to have schema > > > defined. > > > > So > > > > > > > this > > > > > > > >> is > > > > > > > >> > > > more a > > > > > > > >> > > > > > > > choice between: > > > > > > > >> > > > > > > > a) have fixed schema but introduce each time new > > > > > > > >> Verify...Request > > > > > > > >> > > > for > > > > > > > >> > > > > > > > long-running requests > > > > > > > >> > > > > > > > b) use one request for verification but > > generalize > > > > it > > > > > with > > > > > > > >> token > > > > > > > >> > > > > > > > I'm fine with whatever decision community come > > to. > > > > > Just > > > > > > > let > > > > > > > >> me > > > > > > > >> > > know > > > > > > > >> > > > > > your > > > > > > > >> > > > > > > > thoughts. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > Comment from Gwen: > > > > > > > >> > > > > > > > 16. Specifically for ownership, I think the plan > > > is > > > > > to add > > > > > > > >> ACL > > > > > > > >> > > (it > > > > > > > >> > > > > > sounds > > > > > > > >> > > > > > > > like you are describing ACL) via an external > > > system > > > > > > > (Argus, > > > > > > > >> > > > Sentry). > > > > > > > >> > > > > > > > I remember KIP-11 described this, but I can't > > find > > > > > the KIP > > > > > > > >> any > > > > > > > >> > > > > longer. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > A: Okay, no problem. Not sure though how we are > > > > going > > > > > to > > > > > > > >> handle > > > > > > > >> > > it. > > > > > > > >> > > > > > Wait > > > > > > > >> > > > > > > > which KIP > > > > > > > >> > > > > > > > will be committed first and include changes to > > > > > > > >> TopicMetadata from > > > > > > > >> > > > the > > > > > > > >> > > > > > > later > > > > > > > >> > > > > > > > one? > > > > > > > >> > > > > > > > Anyway, I added this note to "Open Questions" > > > > section > > > > > so > > > > > > > we > > > > > > > >> don't > > > > > > > >> > > > > miss > > > > > > > >> > > > > > > this > > > > > > > >> > > > > > > > piece. > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > Thanks, > > > > > > > >> > > > > > > > Andrii Biletskyi > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > On Fri, Mar 13, 2015 at 12:34 AM, Andrii > > > Biletskyi < > > > > > > > >> > > > > > > > andrii.bilets...@stealth.ly> wrote: > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > Hi all, > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > Today I uploaded the patch that covers some of > > > the > > > > > > > >> discussed > > > > > > > >> > > and > > > > > > > >> > > > > > agreed > > > > > > > >> > > > > > > > > items: > > > > > > > >> > > > > > > > > - removed MaybeOf optional type > > > > > > > >> > > > > > > > > - switched to java protocol definitions > > > > > > > >> > > > > > > > > - simplified messages (normalized configs, > > > removed > > > > > topic > > > > > > > >> marked > > > > > > > >> > > > for > > > > > > > >> > > > > > > > > deletion) > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > I also updated the KIP-4 with respective > > changes > > > > and > > > > > > > >> wrote down > > > > > > > >> > > > my > > > > > > > >> > > > > > > > > proposal for > > > > > > > >> > > > > > > > > pending items: > > > > > > > >> > > > > > > > > - Batch Admin Operations -> updated Wire > > > Protocol > > > > > schema > > > > > > > >> > > proposal > > > > > > > >> > > > > > > > > - Remove ClusterMetadata -> changed to extend > > > > > > > >> > > > TopicMetadataRequest > > > > > > > >> > > > > > > > > - Admin Client -> updated my initial proposal > > to > > > > > reflect > > > > > > > >> > > batching > > > > > > > >> > > > > > > > > - Error codes -> proposed fine-grained error > > > code > > > > > > > instead > > > > > > > >> of > > > > > > > >> > > > > > > > > AdminRequestFailed > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > I will also send a separate email to cover all > > > > > comments > > > > > > > >> from > > > > > > > >> > > this > > > > > > > >> > > > > > > thread. > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > Thanks, > > > > > > > >> > > > > > > > > Andrii Biletskyi > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > On Thu, Mar 12, 2015 at 9:26 PM, Gwen Shapira > > < > > > > > > > >> > > > > gshap...@cloudera.com > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > wrote: > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > >> Found KIP-11 ( > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface > > > > > > > >> > > > > > > > >> ) > > > > > > > >> > > > > > > > >> It actually specifies changes to the Metadata > > > > > protocol, > > > > > > > >> so > > > > > > > >> > > > making > > > > > > > >> > > > > > sure > > > > > > > >> > > > > > > > >> both KIPs are consistent in this regard will > > be > > > > > good. > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> On Thu, Mar 12, 2015 at 12:21 PM, Gwen > > Shapira > > > < > > > > > > > >> > > > > > gshap...@cloudera.com > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > >> > Specifically for ownership, I think the > > plan > > > is > > > > > to > > > > > > > add > > > > > > > >> ACL > > > > > > > >> > > (it > > > > > > > >> > > > > > > sounds > > > > > > > >> > > > > > > > >> > like you are describing ACL) via an > > external > > > > > system > > > > > > > >> (Argus, > > > > > > > >> > > > > > Sentry). > > > > > > > >> > > > > > > > >> > I remember KIP-11 described this, but I > > can't > > > > > find > > > > > > > the > > > > > > > >> KIP > > > > > > > >> > > any > > > > > > > >> > > > > > > longer. > > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > >> > Regardless, I think KIP-4 focuses on > > getting > > > > > > > >> information > > > > > > > >> > > that > > > > > > > >> > > > > > > already > > > > > > > >> > > > > > > > >> > exists from Kafka brokers, not on adding > > > > > information > > > > > > > >> that > > > > > > > >> > > > > perhaps > > > > > > > >> > > > > > > > >> > should exist but doesn't yet? > > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > >> > Gwen > > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > >> > On Thu, Mar 12, 2015 at 6:37 AM, Guozhang > > > Wang > > > > < > > > > > > > >> > > > > > wangg...@gmail.com> > > > > > > > >> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > >> >> Folks, > > > > > > > >> > > > > > > > >> >> > > > > > > > >> > > > > > > > >> >> Just want to elaborate a bit more on the > > > > > > > create-topic > > > > > > > >> > > > metadata > > > > > > > >> > > > > > and > > > > > > > >> > > > > > > > >> batching > > > > > > > >> > > > > > > > >> >> describe-topic based on config / metadata > > in > > > > my > > > > > > > >> previous > > > > > > > >> > > > email > > > > > > > >> > > > > as > > > > > > > >> > > > > > > we > > > > > > > >> > > > > > > > >> work > > > > > > > >> > > > > > > > >> >> on KAFKA-1694. The main motivation is to > > > have > > > > > some > > > > > > > >> sort of > > > > > > > >> > > > > topic > > > > > > > >> > > > > > > > >> management > > > > > > > >> > > > > > > > >> >> mechanisms, which I think is quite > > important > > > > in > > > > > a > > > > > > > >> > > > multi-tenant > > > > > > > >> > > > > / > > > > > > > >> > > > > > > > cloud > > > > > > > >> > > > > > > > >> >> architecture: today anyone can create > > topics > > > > in > > > > > a > > > > > > > >> shared > > > > > > > >> > > > Kafka > > > > > > > >> > > > > > > > >> cluster, but > > > > > > > >> > > > > > > > >> >> there is no concept or "ownership" of > > topics > > > > > that > > > > > > > are > > > > > > > >> > > created > > > > > > > >> > > > > by > > > > > > > >> > > > > > > > >> different > > > > > > > >> > > > > > > > >> >> users. For example, at LinkedIn we > > basically > > > > > > > >> distinguish > > > > > > > >> > > > topic > > > > > > > >> > > > > > > owners > > > > > > > >> > > > > > > > >> via > > > > > > > >> > > > > > > > >> >> some casual topic name prefix, which is a > > > bit > > > > > > > awkward > > > > > > > >> and > > > > > > > >> > > > does > > > > > > > >> > > > > > not > > > > > > > >> > > > > > > > fly > > > > > > > >> > > > > > > > >> as > > > > > > > >> > > > > > > > >> >> we scale our customers. It would be great > > to > > > > use > > > > > > > >> > > > > describe-topics > > > > > > > >> > > > > > > such > > > > > > > >> > > > > > > > >> as: > > > > > > > >> > > > > > > > >> >> > > > > > > > >> > > > > > > > >> >> Describe all topics that is created by me. > > > > > > > >> > > > > > > > >> >> > > > > > > > >> > > > > > > > >> >> Describe all topics whose retention time > > is > > > > > > > overriden > > > > > > > >> to X. > > > > > > > >> > > > > > > > >> >> > > > > > > > >> > > > > > > > >> >> Describe all topics whose writable group > > > > include > > > > > > > user > > > > > > > >> Y > > > > > > > >> > > (this > > > > > > > >> > > > > is > > > > > > > >> > > > > > > > >> related to > > > > > > > >> > > > > > > > >> >> authorization), etc.. > > > > > > > >> > > > > > > > >> >> > > > > > > > >> > > > > > > > >> >> One possible way to achieve this is to > > add a > > > > > > > metadata > > > > > > > >> file > > > > > > > >> > > in > > > > > > > >> > > > > the > > > > > > > >> > > > > > > > >> >> create-topic request, whose value will > > also > > > be > > > > > > > >> written ZK > > > > > > > >> > > as > > > > > > > >> > > > we > > > > > > > >> > > > > > > > create > > > > > > > >> > > > > > > > >> the > > > > > > > >> > > > > > > > >> >> topic; then describe-topics can choose to > > > > batch > > > > > > > topics > > > > > > > >> > > based > > > > > > > >> > > > on > > > > > > > >> > > > > > 1) > > > > > > > >> > > > > > > > name > > > > > > > >> > > > > > > > >> >> regex, 2) config K-V matching, 3) metadata > > > > > regex, > > > > > > > etc. > > > > > > > >> > > > > > > > >> >> > > > > > > > >> > > > > > > > >> >> Thoughts? > > > > > > > >> > > > > > > > >> >> > > > > > > > >> > > > > > > > >> >> Guozhang > > > > > > > >> > > > > > > > >> >> > > > > > > > >> > > > > > > > >> >> On Thu, Mar 5, 2015 at 4:37 PM, Guozhang > > > Wang > > > > < > > > > > > > >> > > > > > wangg...@gmail.com> > > > > > > > >> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > >> >> > > > > > > > >> > > > > > > > >> >>> Thanks for the updated wiki. A few > > comments > > > > > below: > > > > > > > >> > > > > > > > >> >>> > > > > > > > >> > > > > > > > >> >>> 1. Error description in response: I think > > > if > > > > > some > > > > > > > >> > > errorCode > > > > > > > >> > > > > > could > > > > > > > >> > > > > > > > >> indicate > > > > > > > >> > > > > > > > >> >>> several different error cases then we > > > should > > > > > really > > > > > > > >> change > > > > > > > >> > > > it > > > > > > > >> > > > > to > > > > > > > >> > > > > > > > >> multiple > > > > > > > >> > > > > > > > >> >>> codes. In general the errorCode itself > > > would > > > > be > > > > > > > >> precise > > > > > > > >> > > and > > > > > > > >> > > > > > > > >> sufficient for > > > > > > > >> > > > > > > > >> >>> describing the server side errors. > > > > > > > >> > > > > > > > >> >>> > > > > > > > >> > > > > > > > >> >>> 2. Describe topic request: it would be > > > great > > > > > to go > > > > > > > >> beyond > > > > > > > >> > > > just > > > > > > > >> > > > > > > > >> batching on > > > > > > > >> > > > > > > > >> >>> topic name regex for this request. For > > > > > example, a > > > > > > > >> very > > > > > > > >> > > > common > > > > > > > >> > > > > > use > > > > > > > >> > > > > > > > >> case of > > > > > > > >> > > > > > > > >> >>> the topic command is to list all topics > > > whose > > > > > > > config > > > > > > > >> A's > > > > > > > >> > > > value > > > > > > > >> > > > > > is > > > > > > > >> > > > > > > B. > > > > > > > >> > > > > > > > >> With > > > > > > > >> > > > > > > > >> >>> topic name regex then we have to first > > > > retrieve > > > > > > > >> __all__ > > > > > > > >> > > > > topics's > > > > > > > >> > > > > > > > >> >>> description info and then filter at the > > > > client > > > > > end, > > > > > > > >> which > > > > > > > >> > > > will > > > > > > > >> > > > > > be > > > > > > > >> > > > > > > a > > > > > > > >> > > > > > > > >> huge > > > > > > > >> > > > > > > > >> >>> burden on ZK. > > > > > > > >> > > > > > > > >> >>> > > > > > > > >> > > > > > > > >> >>> 3. Config K-Vs in create topic: this is > > > > > related to > > > > > > > >> the > > > > > > > >> > > > > previous > > > > > > > >> > > > > > > > point; > > > > > > > >> > > > > > > > >> >>> maybe we can add another metadata K-V or > > > > just a > > > > > > > >> metadata > > > > > > > >> > > > > string > > > > > > > >> > > > > > > > along > > > > > > > >> > > > > > > > >> side > > > > > > > >> > > > > > > > >> >>> with config K-V in create topic like we > > did > > > > for > > > > > > > >> offset > > > > > > > >> > > > commit > > > > > > > >> > > > > > > > >> request. This > > > > > > > >> > > > > > > > >> >>> field can be quite useful in storing > > > > > information > > > > > > > like > > > > > > > >> > > > "owner" > > > > > > > >> > > > > of > > > > > > > >> > > > > > > the > > > > > > > >> > > > > > > > >> topic > > > > > > > >> > > > > > > > >> >>> who issue the create command, etc, which > > is > > > > > quite > > > > > > > >> > > important > > > > > > > >> > > > > for > > > > > > > >> > > > > > a > > > > > > > >> > > > > > > > >> >>> multi-tenant setting. Then in the > > describe > > > > > topic > > > > > > > >> request > > > > > > > >> > > we > > > > > > > >> > > > > can > > > > > > > >> > > > > > > also > > > > > > > >> > > > > > > > >> batch > > > > > > > >> > > > > > > > >> >>> on regex of the metadata field. > > > > > > > >> > > > > > > > >> >>> > > > > > > > >> > > > > > > > >> >>> 4. Today all the admin operations are > > async > > > > in > > > > > the > > > > > > > >> sense > > > > > > > >> > > > that > > > > > > > >> > > > > > > > command > > > > > > > >> > > > > > > > >> will > > > > > > > >> > > > > > > > >> >>> return once it is written in ZK, and that > > > is > > > > > why we > > > > > > > >> need > > > > > > > >> > > > extra > > > > > > > >> > > > > > > > >> verification > > > > > > > >> > > > > > > > >> >>> like testUtil.waitForTopicCreated() / > > > verify > > > > > > > >> partition > > > > > > > >> > > > > > > reassignment > > > > > > > >> > > > > > > > >> >>> request, etc. With admin requests we > > could > > > > add > > > > > a > > > > > > > >> flag to > > > > > > > >> > > > > enable > > > > > > > >> > > > > > / > > > > > > > >> > > > > > > > >> disable > > > > > > > >> > > > > > > > >> >>> synchronous requests; when it is turned > > on, > > > > the > > > > > > > >> response > > > > > > > >> > > > will > > > > > > > >> > > > > > not > > > > > > > >> > > > > > > > >> return > > > > > > > >> > > > > > > > >> >>> until the request has been completed. And > > > for > > > > > async > > > > > > > >> > > requests > > > > > > > >> > > > > we > > > > > > > >> > > > > > > can > > > > > > > >> > > > > > > > >> add a > > > > > > > >> > > > > > > > >> >>> "token" field in the response, and then > > > only > > > > > need a > > > > > > > >> > > general > > > > > > > >> > > > > > "admin > > > > > > > >> > > > > > > > >> >>> verification request" with the given > > token > > > to > > > > > check > > > > > > > >> if the > > > > > > > >> > > > > async > > > > > > > >> > > > > > > > >> request > > > > > > > >> > > > > > > > >> >>> has been completed. > > > > > > > >> > > > > > > > >> >>> > > > > > > > >> > > > > > > > >> >>> 5. +1 for extending Metadata request to > > > > include > > > > > > > >> > > controller / > > > > > > > >> > > > > > > > >> coordinator > > > > > > > >> > > > > > > > >> >>> information, and then we can remove the > > > > > > > >> ConsumerMetadata / > > > > > > > >> > > > > > > > >> ClusterMetadata > > > > > > > >> > > > > > > > >> >>> requests. > > > > > > > >> > > > > > > > >> >>> > > > > > > > >> > > > > > > > >> >>> Guozhang > > > > > > > >> > > > > > > > >> >>> > > > > > > > >> > > > > > > > >> >>> On Tue, Mar 3, 2015 at 10:23 AM, Joel > > > Koshy < > > > > > > > >> > > > > > jjkosh...@gmail.com> > > > > > > > >> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > >> >>> > > > > > > > >> > > > > > > > >> >>>> Thanks for sending that out Joe - I > > don't > > > > > think I > > > > > > > >> will be > > > > > > > >> > > > > able > > > > > > > >> > > > > > to > > > > > > > >> > > > > > > > >> make > > > > > > > >> > > > > > > > >> >>>> it today, so if notes can be sent out > > > > > afterward > > > > > > > that > > > > > > > >> > > would > > > > > > > >> > > > be > > > > > > > >> > > > > > > > great. > > > > > > > >> > > > > > > > >> >>>> > > > > > > > >> > > > > > > > >> >>>> On Mon, Mar 02, 2015 at 09:16:13AM > > -0800, > > > > Gwen > > > > > > > >> Shapira > > > > > > > >> > > > wrote: > > > > > > > >> > > > > > > > >> >>>> > Thanks for sending this out Joe. > > Looking > > > > > forward > > > > > > > >> to > > > > > > > >> > > > > chatting > > > > > > > >> > > > > > > with > > > > > > > >> > > > > > > > >> >>>> everyone :) > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > >> > > > > > > > >> >>>> > On Mon, Mar 2, 2015 at 6:46 AM, Joe > > > Stein > > > > < > > > > > > > >> > > > > > > joe.st...@stealth.ly> > > > > > > > >> > > > > > > > >> wrote: > > > > > > > >> > > > > > > > >> >>>> > > Hey, I just sent out a google > > hangout > > > > > invite > > > > > > > to > > > > > > > >> all > > > > > > > >> > > > pmc, > > > > > > > >> > > > > > > > >> committers > > > > > > > >> > > > > > > > >> >>>> and > > > > > > > >> > > > > > > > >> >>>> > > everyone I found working on a KIP. > > If > > > I > > > > > missed > > > > > > > >> anyone > > > > > > > >> > > > in > > > > > > > >> > > > > > the > > > > > > > >> > > > > > > > >> invite > > > > > > > >> > > > > > > > >> >>>> please > > > > > > > >> > > > > > > > >> >>>> > > let me know and can update it, np. > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > >> > > > > > > > >> >>>> > > We should do this every Tuesday @ > > 2pm > > > > > Eastern > > > > > > > >> Time. > > > > > > > >> > > > Maybe > > > > > > > >> > > > > > we > > > > > > > >> > > > > > > > can > > > > > > > >> > > > > > > > >> get > > > > > > > >> > > > > > > > >> >>>> INFRA > > > > > > > >> > > > > > > > >> >>>> > > help to make a google account so we > > > can > > > > > manage > > > > > > > >> > > better? > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > >> > > > > > > > >> >>>> > > To discuss > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > >> > > > > > > > >> >>>> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > > > > > > >> > > > > > > > >> >>>> > > in progress and related JIRA that > > are > > > > > > > >> interdependent > > > > > > > >> > > > and > > > > > > > >> > > > > > > common > > > > > > > >> > > > > > > > >> work. > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > >> > > > > > > > >> >>>> > > ~ Joe Stein > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > >> > > > > > > > >> >>>> > > On Tue, Feb 24, 2015 at 2:59 PM, Jay > > > > > Kreps < > > > > > > > >> > > > > > > > jay.kr...@gmail.com> > > > > > > > >> > > > > > > > >> >>>> wrote: > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > > >> > > > > > > > >> >>>> > >> Let's stay on Google hangouts that > > > will > > > > > also > > > > > > > >> record > > > > > > > >> > > > and > > > > > > > >> > > > > > make > > > > > > > >> > > > > > > > the > > > > > > > >> > > > > > > > >> >>>> sessions > > > > > > > >> > > > > > > > >> >>>> > >> available on youtube. > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > >> > > > > > > > >> >>>> > >> -Jay > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > >> > > > > > > > >> >>>> > >> On Tue, Feb 24, 2015 at 11:49 AM, > > > Jeff > > > > > > > Holoman > > > > > > > >> < > > > > > > > >> > > > > > > > >> >>>> jholo...@cloudera.com> > > > > > > > >> > > > > > > > >> >>>> > >> wrote: > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > >> > > > > > > > >> >>>> > >> > Jay / Joe > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > We're happy to send out a Webex > > for > > > > > this > > > > > > > >> purpose. > > > > > > > >> > > We > > > > > > > >> > > > > > could > > > > > > > >> > > > > > > > >> record > > > > > > > >> > > > > > > > >> >>>> the > > > > > > > >> > > > > > > > >> >>>> > >> > sessions if there is interest and > > > > > publish > > > > > > > >> them > > > > > > > >> > > out. > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > Thanks > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > Jeff > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > On Tue, Feb 24, 2015 at 11:28 AM, > > > Jay > > > > > > > Kreps < > > > > > > > >> > > > > > > > >> jay.kr...@gmail.com> > > > > > > > >> > > > > > > > >> >>>> wrote: > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > > Let's try to get the technical > > > > > hang-ups > > > > > > > >> sorted > > > > > > > >> > > > out, > > > > > > > >> > > > > > > > though. > > > > > > > >> > > > > > > > >> I > > > > > > > >> > > > > > > > >> >>>> really > > > > > > > >> > > > > > > > >> >>>> > >> > think > > > > > > > >> > > > > > > > >> >>>> > >> > > there is some benefit to live > > > > > discussion > > > > > > > vs > > > > > > > >> > > > > writing. I > > > > > > > >> > > > > > > am > > > > > > > >> > > > > > > > >> >>>> hopeful that > > > > > > > >> > > > > > > > >> >>>> > >> if > > > > > > > >> > > > > > > > >> >>>> > >> > > we post instructions and give > > > > > ourselves a > > > > > > > >> few > > > > > > > >> > > > > attempts > > > > > > > >> > > > > > > we > > > > > > > >> > > > > > > > >> can > > > > > > > >> > > > > > > > >> >>>> get it > > > > > > > >> > > > > > > > >> >>>> > >> > > working. > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > Tuesday at that time would work > > > for > > > > > > > >> me...any > > > > > > > >> > > > > > objections? > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > -Jay > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > On Tue, Feb 24, 2015 at 8:18 > > AM, > > > > Joe > > > > > > > Stein > > > > > > > >> < > > > > > > > >> > > > > > > > >> joe.st...@stealth.ly > > > > > > > >> > > > > > > > >> >>>> > > > > > > > > >> > > > > > > > >> >>>> > >> wrote: > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > Weekly would be great maybe > > > like > > > > > every > > > > > > > >> > > Tuesday ~ > > > > > > > >> > > > > 1pm > > > > > > > >> > > > > > > ET > > > > > > > >> > > > > > > > / > > > > > > > >> > > > > > > > >> 10am > > > > > > > >> > > > > > > > >> >>>> PT > > > > > > > >> > > > > > > > >> >>>> > >> ???? > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > I don't mind google hangout > > but > > > > > there > > > > > > > is > > > > > > > >> > > always > > > > > > > >> > > > > some > > > > > > > >> > > > > > > > >> issue or > > > > > > > >> > > > > > > > >> >>>> > >> whatever > > > > > > > >> > > > > > > > >> >>>> > >> > so > > > > > > > >> > > > > > > > >> >>>> > >> > > > we know the apache irc > > channel > > > > > works. > > > > > > > We > > > > > > > >> can > > > > > > > >> > > > start > > > > > > > >> > > > > > > there > > > > > > > >> > > > > > > > >> and > > > > > > > >> > > > > > > > >> >>>> see how > > > > > > > >> > > > > > > > >> >>>> > >> it > > > > > > > >> > > > > > > > >> >>>> > >> > > > goes? We can pull transcripts > > > too > > > > > and > > > > > > > >> > > associate > > > > > > > >> > > > to > > > > > > > >> > > > > > > > >> tickets if > > > > > > > >> > > > > > > > >> >>>> need be > > > > > > > >> > > > > > > > >> >>>> > >> > > makes > > > > > > > >> > > > > > > > >> >>>> > >> > > > it helpful for things. > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > ~ Joestein > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > On Tue, Feb 24, 2015 at 11:10 > > > AM, > > > > > Jay > > > > > > > >> Kreps < > > > > > > > >> > > > > > > > >> >>>> jay.kr...@gmail.com> > > > > > > > >> > > > > > > > >> >>>> > >> > wrote: > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > We'd talked about doing a > > > > Google > > > > > > > >> Hangout to > > > > > > > >> > > > chat > > > > > > > >> > > > > > > about > > > > > > > >> > > > > > > > >> this. > > > > > > > >> > > > > > > > >> >>>> What > > > > > > > >> > > > > > > > >> >>>> > >> > about > > > > > > > >> > > > > > > > >> >>>> > >> > > > > generalizing that a little > > > > > > > further...I > > > > > > > >> > > > actually > > > > > > > >> > > > > > > think > > > > > > > >> > > > > > > > it > > > > > > > >> > > > > > > > >> >>>> would be > > > > > > > >> > > > > > > > >> >>>> > >> > good > > > > > > > >> > > > > > > > >> >>>> > >> > > > for > > > > > > > >> > > > > > > > >> >>>> > >> > > > > everyone spending a > > > reasonable > > > > > chunk > > > > > > > of > > > > > > > >> > > their > > > > > > > >> > > > > week > > > > > > > >> > > > > > > on > > > > > > > >> > > > > > > > >> Kafka > > > > > > > >> > > > > > > > >> >>>> stuff > > > > > > > >> > > > > > > > >> >>>> > >> to > > > > > > > >> > > > > > > > >> >>>> > >> > > > maybe > > > > > > > >> > > > > > > > >> >>>> > >> > > > > sync up once a week. I > > think > > > we > > > > > could > > > > > > > >> use > > > > > > > >> > > time > > > > > > > >> > > > > to > > > > > > > >> > > > > > > talk > > > > > > > >> > > > > > > > >> >>>> through > > > > > > > >> > > > > > > > >> >>>> > >> design > > > > > > > >> > > > > > > > >> >>>> > >> > > > > stuff, make sure we are on > > > top > > > > of > > > > > > > code > > > > > > > >> > > > reviews, > > > > > > > >> > > > > > talk > > > > > > > >> > > > > > > > >> through > > > > > > > >> > > > > > > > >> >>>> any > > > > > > > >> > > > > > > > >> >>>> > >> > tricky > > > > > > > >> > > > > > > > >> >>>> > >> > > > > issues, etc. > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > We can make it publicly > > > > > available so > > > > > > > >> that > > > > > > > >> > > any > > > > > > > >> > > > > one > > > > > > > >> > > > > > > can > > > > > > > >> > > > > > > > >> follow > > > > > > > >> > > > > > > > >> >>>> along > > > > > > > >> > > > > > > > >> >>>> > >> > who > > > > > > > >> > > > > > > > >> >>>> > >> > > > > likes. > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > Any interest in doing this? > > > If > > > > so > > > > > > > I'll > > > > > > > >> try > > > > > > > >> > > to > > > > > > > >> > > > > set > > > > > > > >> > > > > > it > > > > > > > >> > > > > > > > up > > > > > > > >> > > > > > > > >> >>>> starting > > > > > > > >> > > > > > > > >> >>>> > >> next > > > > > > > >> > > > > > > > >> >>>> > >> > > > week. > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > -Jay > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > On Tue, Feb 24, 2015 at > > 3:57 > > > > AM, > > > > > > > Andrii > > > > > > > >> > > > > Biletskyi > > > > > > > >> > > > > > < > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > andrii.bilets...@stealth.ly> > > > > > wrote: > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > Hi all, > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > I've updated KIP page, > > > fixed > > > > / > > > > > > > >> aligned > > > > > > > >> > > > > document > > > > > > > >> > > > > > > > >> structure. > > > > > > > >> > > > > > > > >> >>>> Also I > > > > > > > >> > > > > > > > >> >>>> > >> > > added > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > some > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > very initial proposal for > > > > > > > >> AdminClient so > > > > > > > >> > > we > > > > > > > >> > > > > have > > > > > > > >> > > > > > > > >> something > > > > > > > >> > > > > > > > >> >>>> to > > > > > > > >> > > > > > > > >> >>>> > >> start > > > > > > > >> > > > > > > > >> >>>> > >> > > > from > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > while > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > discussing the KIP. > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > >> > > > > > > > >> >>>> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > Thanks, > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > Andrii Biletskyi > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > On Wed, Feb 18, 2015 at > > > 9:01 > > > > > PM, > > > > > > > >> Andrii > > > > > > > >> > > > > > Biletskyi > > > > > > > >> > > > > > > < > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > andrii.bilets...@stealth.ly> > > > > > > > wrote: > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Jay, > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Re error messages: you > > > are > > > > > right, > > > > > > > >> in > > > > > > > >> > > most > > > > > > > >> > > > > > cases > > > > > > > >> > > > > > > > >> client > > > > > > > >> > > > > > > > >> >>>> will > > > > > > > >> > > > > > > > >> >>>> > >> have > > > > > > > >> > > > > > > > >> >>>> > >> > > > enough > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > context to show > > > descriptive > > > > > error > > > > > > > >> > > message. > > > > > > > >> > > > > My > > > > > > > >> > > > > > > > >> concern is > > > > > > > >> > > > > > > > >> >>>> that > > > > > > > >> > > > > > > > >> >>>> > >> we > > > > > > > >> > > > > > > > >> >>>> > >> > > will > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > have > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > to > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > add lots of new error > > > codes > > > > > for > > > > > > > >> each > > > > > > > >> > > > > possible > > > > > > > >> > > > > > > > >> error. Of > > > > > > > >> > > > > > > > >> >>>> course, > > > > > > > >> > > > > > > > >> >>>> > >> > we > > > > > > > >> > > > > > > > >> >>>> > >> > > > > could > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > reuse > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > some of existing like > > > > > > > >> > > > > > > UknownTopicOrPartitionCode, > > > > > > > >> > > > > > > > >> but we > > > > > > > >> > > > > > > > >> >>>> will > > > > > > > >> > > > > > > > >> >>>> > >> > also > > > > > > > >> > > > > > > > >> >>>> > >> > > > need > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > to > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > add smth like: > > > > > > > >> TopicAlreadyExistsCode, > > > > > > > >> > > > > > > > >> >>>> TopicConfigInvalid (both > > > > > > > >> > > > > > > > >> >>>> > >> > for > > > > > > > >> > > > > > > > >> >>>> > >> > > > > topic > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > name and config, and > > > > probably > > > > > > > user > > > > > > > >> would > > > > > > > >> > > > > like > > > > > > > >> > > > > > to > > > > > > > >> > > > > > > > >> know > > > > > > > >> > > > > > > > >> >>>> what > > > > > > > >> > > > > > > > >> >>>> > >> > exactly > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > is wrong in his > > config), > > > > > > > >> > > > > > > InvalidReplicaAssignment, > > > > > > > >> > > > > > > > >> >>>> > >> InternalError > > > > > > > >> > > > > > > > >> >>>> > >> > > > (e.g. > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > zookeeper failure) etc. > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > And this is only for > > > > > > > TopicCommand, > > > > > > > >> we > > > > > > > >> > > will > > > > > > > >> > > > > > also > > > > > > > >> > > > > > > > >> need to > > > > > > > >> > > > > > > > >> >>>> add > > > > > > > >> > > > > > > > >> >>>> > >> > similar > > > > > > > >> > > > > > > > >> >>>> > >> > > > > stuff > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > for > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > ReassignPartitions, > > > > > > > >> PreferredReplica. So > > > > > > > >> > > > > we'll > > > > > > > >> > > > > > > end > > > > > > > >> > > > > > > > >> up > > > > > > > >> > > > > > > > >> >>>> with a > > > > > > > >> > > > > > > > >> >>>> > >> > large > > > > > > > >> > > > > > > > >> >>>> > >> > > > list > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > of > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > error codes, used only > > in > > > > > Admin > > > > > > > >> > > protocol. > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Having said that, I > > agree > > > > my > > > > > > > >> proposal is > > > > > > > >> > > > not > > > > > > > >> > > > > > > > >> consistent > > > > > > > >> > > > > > > > >> >>>> with > > > > > > > >> > > > > > > > >> >>>> > >> > other > > > > > > > >> > > > > > > > >> >>>> > >> > > > > cases. > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Maybe we can find > > better > > > > > solution > > > > > > > >> or > > > > > > > >> > > > > something > > > > > > > >> > > > > > > > >> >>>> in-between. > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Re Hangout chat: I > > think > > > it > > > > > is a > > > > > > > >> great > > > > > > > >> > > > idea. > > > > > > > >> > > > > > > This > > > > > > > >> > > > > > > > >> way we > > > > > > > >> > > > > > > > >> >>>> can > > > > > > > >> > > > > > > > >> >>>> > >> move > > > > > > > >> > > > > > > > >> >>>> > >> > > on > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > faster. > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Let's agree somehow on > > > > > date/time > > > > > > > so > > > > > > > >> > > people > > > > > > > >> > > > > can > > > > > > > >> > > > > > > > join. > > > > > > > >> > > > > > > > >> >>>> Will work > > > > > > > >> > > > > > > > >> >>>> > >> > for > > > > > > > >> > > > > > > > >> >>>> > >> > > me > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > this > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > and > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > next week almost > > anytime > > > if > > > > > > > agreed > > > > > > > >> in > > > > > > > >> > > > > advance. > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Thanks, > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > Andrii > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > On Wed, Feb 18, 2015 at > > > > 7:09 > > > > > PM, > > > > > > > >> Jay > > > > > > > >> > > > Kreps < > > > > > > > >> > > > > > > > >> >>>> > >> jay.kr...@gmail.com> > > > > > > > >> > > > > > > > >> >>>> > >> > > > > wrote: > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> Hey Andrii, > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> Generally we can do > > good > > > > > error > > > > > > > >> handling > > > > > > > >> > > > > > without > > > > > > > >> > > > > > > > >> needing > > > > > > > >> > > > > > > > >> >>>> custom > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > server-side > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> messages. I.e. > > generally > > > > the > > > > > > > >> client has > > > > > > > >> > > > the > > > > > > > >> > > > > > > > >> context to > > > > > > > >> > > > > > > > >> >>>> know > > > > > > > >> > > > > > > > >> >>>> > >> that > > > > > > > >> > > > > > > > >> >>>> > >> > > if > > > > > > > >> > > > > > > > >> >>>> > >> > > > it > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > got > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> an error that the > > topic > > > > > doesn't > > > > > > > >> exist > > > > > > > >> > > to > > > > > > > >> > > > > say > > > > > > > >> > > > > > > > >> "Topic X > > > > > > > >> > > > > > > > >> >>>> doesn't > > > > > > > >> > > > > > > > >> >>>> > >> > > exist" > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> rather > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> than "error code 14" > > (or > > > > > > > >> whatever). > > > > > > > >> > > Maybe > > > > > > > >> > > > > > there > > > > > > > >> > > > > > > > are > > > > > > > >> > > > > > > > >> >>>> specific > > > > > > > >> > > > > > > > >> >>>> > >> > cases > > > > > > > >> > > > > > > > >> >>>> > >> > > > > where > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> this is hard? If we > > want > > > > to > > > > > add > > > > > > > >> > > > server-side > > > > > > > >> > > > > > > error > > > > > > > >> > > > > > > > >> >>>> messages we > > > > > > > >> > > > > > > > >> >>>> > >> > > really > > > > > > > >> > > > > > > > >> >>>> > >> > > > > do > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> need to do this in a > > > > > consistent > > > > > > > >> way > > > > > > > >> > > > across > > > > > > > >> > > > > > the > > > > > > > >> > > > > > > > >> protocol. > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> I still have a bunch > > of > > > > open > > > > > > > >> questions > > > > > > > >> > > > here > > > > > > > >> > > > > > > from > > > > > > > >> > > > > > > > my > > > > > > > >> > > > > > > > >> >>>> previous > > > > > > > >> > > > > > > > >> >>>> > >> > > list. I > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > will > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> be out for the next > > few > > > > > days for > > > > > > > >> Strata > > > > > > > >> > > > > > though. > > > > > > > >> > > > > > > > >> Maybe > > > > > > > >> > > > > > > > >> >>>> we could > > > > > > > >> > > > > > > > >> >>>> > >> > do > > > > > > > >> > > > > > > > >> >>>> > >> > > a > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > Google > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> Hangout chat on any > > open > > > > > issues > > > > > > > >> some > > > > > > > >> > > time > > > > > > > >> > > > > > > towards > > > > > > > >> > > > > > > > >> the > > > > > > > >> > > > > > > > >> >>>> end of > > > > > > > >> > > > > > > > >> >>>> > >> > next > > > > > > > >> > > > > > > > >> >>>> > >> > > > week > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > for > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> anyone interested in > > > this > > > > > > > ticket? > > > > > > > >> I > > > > > > > >> > > have > > > > > > > >> > > > a > > > > > > > >> > > > > > > > feeling > > > > > > > >> > > > > > > > >> that > > > > > > > >> > > > > > > > >> >>>> might > > > > > > > >> > > > > > > > >> >>>> > >> > > > progress > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> things a little faster > > > > than > > > > > > > >> email--I > > > > > > > >> > > > think > > > > > > > >> > > > > we > > > > > > > >> > > > > > > > >> could talk > > > > > > > >> > > > > > > > >> >>>> > >> through > > > > > > > >> > > > > > > > >> >>>> > >> > > > those > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> issues I brought up > > > fairly > > > > > > > >> quickly... > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> -Jay > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> On Wed, Feb 18, 2015 > > at > > > > > 7:27 AM, > > > > > > > >> Andrii > > > > > > > >> > > > > > > > Biletskyi < > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > andrii.bilets...@stealth.ly > > > > > > > > > > > > > >> wrote: > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > Hi all, > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > I'm trying to > > address > > > > > some of > > > > > > > >> the > > > > > > > >> > > > issues > > > > > > > >> > > > > > > which > > > > > > > >> > > > > > > > >> were > > > > > > > >> > > > > > > > >> >>>> > >> mentioned > > > > > > > >> > > > > > > > >> >>>> > >> > > > > earlier > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> about > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > Admin RQ/RP format. > > > One > > > > of > > > > > > > >> those was > > > > > > > >> > > > > about > > > > > > > >> > > > > > > > >> batching > > > > > > > >> > > > > > > > >> >>>> > >> > operations. > > > > > > > >> > > > > > > > >> >>>> > >> > > > What > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > if > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> we > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > follow TopicCommand > > > > > approach > > > > > > > >> and let > > > > > > > >> > > > > people > > > > > > > >> > > > > > > > >> specify > > > > > > > >> > > > > > > > >> >>>> > >> topic-name > > > > > > > >> > > > > > > > >> >>>> > >> > > by > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> regexp - > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > would that cover > > most > > > of > > > > > the > > > > > > > use > > > > > > > >> > > cases? > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > Secondly, is what > > > > > information > > > > > > > >> should > > > > > > > >> > > we > > > > > > > >> > > > > > > > generally > > > > > > > >> > > > > > > > >> >>>> provide in > > > > > > > >> > > > > > > > >> >>>> > >> > > Admin > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > responses. > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > I realize that Admin > > > > > commands > > > > > > > >> don't > > > > > > > >> > > > imply > > > > > > > >> > > > > > > they > > > > > > > >> > > > > > > > >> will > > > > > > > >> > > > > > > > >> >>>> be used > > > > > > > >> > > > > > > > >> >>>> > >> > only > > > > > > > >> > > > > > > > >> >>>> > >> > > > in > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > CLI > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > but, > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > it seems to me, CLI > > > is a > > > > > very > > > > > > > >> > > important > > > > > > > >> > > > > > > client > > > > > > > >> > > > > > > > >> of this > > > > > > > >> > > > > > > > >> >>>> > >> > feature. > > > > > > > >> > > > > > > > >> >>>> > >> > > In > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > this > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > case, > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > seems logical, we > > > would > > > > > like > > > > > > > to > > > > > > > >> > > provide > > > > > > > >> > > > > > users > > > > > > > >> > > > > > > > >> with > > > > > > > >> > > > > > > > >> >>>> rich > > > > > > > >> > > > > > > > >> >>>> > >> > > experience > > > > > > > >> > > > > > > > >> >>>> > >> > > > > in > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> terms > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > of > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > getting results / > > > errors > > > > > of > > > > > > > the > > > > > > > >> > > > executed > > > > > > > >> > > > > > > > >> commands. > > > > > > > >> > > > > > > > >> >>>> Usually > > > > > > > >> > > > > > > > >> >>>> > >> we > > > > > > > >> > > > > > > > >> >>>> > >> > > > supply > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> with > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > responses only > > > > errorCode, > > > > > > > which > > > > > > > >> looks > > > > > > > >> > > > > very > > > > > > > >> > > > > > > > >> limiting, > > > > > > > >> > > > > > > > >> >>>> in case > > > > > > > >> > > > > > > > >> >>>> > >> > of > > > > > > > >> > > > > > > > >> >>>> > >> > > > CLI > > > > > > > >> > > > > > > > >> >>>> > >> > > > > we > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> may > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > want to print human > > > > > readable > > > > > > > >> error > > > > > > > >> > > > > > > description. > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > So, taking into > > > account > > > > > > > >> previous item > > > > > > > >> > > > > about > > > > > > > >> > > > > > > > >> batching, > > > > > > > >> > > > > > > > >> >>>> what > > > > > > > >> > > > > > > > >> >>>> > >> do > > > > > > > >> > > > > > > > >> >>>> > >> > > you > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > think > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > about > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > having smth like: > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > ('create' doesn't > > > > support > > > > > > > >> regexp) > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > CreateTopicRequest > > => > > > > > > > TopicName > > > > > > > >> > > > > Partitions > > > > > > > >> > > > > > > > >> Replicas > > > > > > > >> > > > > > > > >> >>>> > >> > > > > ReplicaAssignment > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > [Config] > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > CreateTopicResponse > > => > > > > > > > ErrorCode > > > > > > > >> > > > > > > > ErrorDescription > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > ErrorCode => int16 > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > ErrorDescription > > => > > > > > string > > > > > > > >> (empty > > > > > > > >> > > if > > > > > > > >> > > > > > > > >> successful) > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > > > > > > > > >> > > > > > > > >> >>>> > >> > > > > > >> > AlterTopicRequest -> > > > > > > > >> TopicNameRegexp > > > > > > > >> > > > > > > > > > > > > > > > > > > > ... > > > > > > > > > > > > > > > > [Message clipped] > > > > > > > > > > > > > > > > > > > > > > > > > >
