Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-06-12 Thread Andrii Biletskyi
; From: Joel Koshy [jjkosh...@gmail.com] > Sent: Thursday, June 11, 2015 1:28 PM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative > operations (Thread 2) > > Discussion aside, was there any significant material change beside

RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-06-12 Thread Aditya Auradkar
afka.apache.org Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2) Discussion aside, was there any significant material change besides the additions below? If so, then we can avoid the overhead of another vote unless someone wants to down-vote these

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-06-11 Thread Joel Koshy
the admin client spec. (Alter and Describe config). > > Please review. > > Aditya > > > From: Ashish Singh [asi...@cloudera.com] > Sent: Friday, May 29, 2015 8:36 AM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-4 - Comman

RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-06-11 Thread Aditya Auradkar
rg Subject: RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2) I've made two changes to the document: - Removed the TMR evolution piece since we agreed to retain this. - Added two new API's to the admin client spec. (Alter and Describe config). Pl

RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-06-11 Thread Aditya Auradkar
.com] Sent: Friday, May 29, 2015 8:36 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2) +1 on discussing this on next KIP hangout. I will update KIP-24 before that. On Fri, May 29, 2015 at 3:40 AM, Andrii Biletskyi <

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-05-29 Thread Ashish Singh
anks. Perhaps we should leave TMR unchanged for now. Should we discuss > > this during the next hangout? > > > > Aditya > > > > ____ > > From: Jun Rao [j...@confluent.io] > > Sent: Thursday, May 28, 2015 5:32 PM > > To: d

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-05-29 Thread Andrii Biletskyi
x27;ve altered the KIP with the > > assumption that this is a good enough reason by itself to evolve the > > request/response protocol. Any concerns there? > > > > Thanks, > > Aditya > > > > ____________ > > From: Mayuresh Gharat [gharatmayures.

RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-05-28 Thread Aditya Auradkar
___ > From: Mayuresh Gharat [gharatmayures...@gmail.com] > Sent: Thursday, May 21, 2015 8:29 PM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative > operations (Thread 2) > > Hi Jun, > > Thanks a lo

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-05-28 Thread Jun Rao
arat [gharatmayures...@gmail.com] > Sent: Thursday, May 21, 2015 8:29 PM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative > operations (Thread 2) > > Hi Jun, > > Thanks a lot. I get it now. > Point 4) will actually en

RE: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-05-26 Thread Aditya Auradkar
015 8:29 PM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2) Hi Jun, Thanks a lot. I get it now. Point 4) will actually enable clients to who don't want to create a topic with default partitions, if it does not exist an

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-05-21 Thread Mayuresh Gharat
Hi Jun, Thanks a lot. I get it now. Point 4) will actually enable clients to who don't want to create a topic with default partitions, if it does not exist and then can manually create the topic with their own configs(#partitions). Thanks, Mayuresh On Thu, May 21, 2015 at 6:16 PM, Jun Rao wro

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-05-21 Thread Jun Rao
Mayuresh, The current plan is the following. 1. Add TMR v1, which still triggers auto topic creation. 2. Change the consumer client to TMR v1. Change the producer client to use TMR v1 and on UnknownTopicException, issue TopicCreateRequest to explicitly create the topic with the default server sid

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-05-21 Thread Joel Koshy
> Here are some ideas to address this : > > 1) The way this can be addressed is that TopicMetadata request should have > a way to specify whether it should only check if the topic exist or check > and create a topic with given number of partitions. If the number of > partitions is not specified u

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-05-21 Thread Mayuresh Gharat
Hi, I had a question about TopicMetadata Request. Currently the way it works is : 1) Suppose a topic T1 does not exist. 2) Client wants to produce data to T1 using producer P1. 3) Since T1 does not exist, P1 issues a TopicMetadata request to kafka. This in turn creates the default number of partit

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-05-14 Thread Jun Rao
For ListTopics, we decided not to add a ListTopics request for now and just rely on passing in an empty list to TMR. We can revisit this in the future if it becomes an issue. Thanks, Jun On Wed, May 13, 2015 at 3:31 PM, Joel Koshy wrote: > Just had a few minor questions before I join the vote

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-05-14 Thread Andrii Biletskyi
Joel, - DecreasePartitionNotAllowed. Yeah, that's kind of subcase of InvalidPartitions... But since client can always request topic metadata and check what exactly is was wrong with Partitions argument, I think, yes, we can remove DecreasePartitionNotAllowed and use InvalidPartitions instead. I'll

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-05-13 Thread Joel Koshy
Just had a few minor questions before I join the vote thread. Apologies if these have been discussed: - Do we need DecreasePartitionsNotAllowed? i.e., can we just return InvalidPartitions instead? - AdminClient.listTopics: should we allow listing all partitions? Or do you intend for the client

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-05-05 Thread Andrii Biletskyi
Guys, I've updated the wiki to reflect all previously discussed items (regarding the schema only - this is included to phase 1). https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations I think we can have the final discussion today (for pha

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-30 Thread Jun Rao
The following is a description of some of my concerns on allowing the same topic multiple times in AlterTopicRequest. ATP has an array of entries, each corresponding to a topic. We allow multiple changes to a topic in a single entry. Those changes may fail to apply independently (e.g., the config

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-28 Thread Andrii Biletskyi
Guys, A quick summary of our today's meeting. There were no additional issues/questions. The only item about which we are not 100% sure is "multiple instructions for one topic in one request" case. It was proposed by Jun to explain reasons behind not allowing users doing that again here in mailin

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-28 Thread Andrii Biletskyi
Guys, It seems that there are no open questions left so prior to our weekly call let me summarize what I'm going to implement as part of phase one for KIP-4. 1. Add 3 new Wire Protocol requests - Create-, Alter- and DeleteTopicRequest 2. Topic requests are batch requests, errors are returned per

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-27 Thread Jun Rao
Yes, to verify if a partition reassignment completes or not, we just need to make sure AR == RAR. So, we don't need ISR for this. It's probably still useful to know ISR for monitoring in general though. Thanks, Jun On Mon, Apr 27, 2015 at 4:15 AM, Andrii Biletskyi < andrii.bilets...@stealth.ly>

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-27 Thread Andrii Biletskyi
Okay, I had some doubts in terms of reassign-partitions case. I was not sure whether we need ISR to check post condition of partition reassignment. But I think we can rely on assigned replicas - the workflow in reassignPartitions is the following: 1. Update AR in ZK with OAR + RAR. ... 10. Update A

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-26 Thread Jun Rao
Andrii, Another thing. We decided not to add the lag info in TMR. To be consistent, we probably also want to remove ISR from TMR since only the leader knows it. We can punt on adding any new request from getting ISR. ISR is mostly useful for monitoring. Currently, one can determine if a replica is

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-26 Thread Andrii Biletskyi
Jun, I like your approach to AlterTopicReques semantics! Sounds like we linearize all request fields to ReplicaAssignment - I will definitely try this out to ensure there are no other pitfalls. With regards to multiple instructions in one batch per topic. For me this sounds reasonable too. We dis

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-26 Thread Jun Rao
Andrii, Thanks for the update. For your second point, I agree that if a single AlterTopicRequest can make multiple changes, there is no need to support the same topic included more than once in the request. Now about the semantics in your first question. I was thinking that we can do the followi

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-25 Thread Andrii Biletskyi
Guys, Can we come to some agreement in terms of the second item from the email above? This blocks me from updating and uploading the patch. Also the new schedule for the weekly calls doesn't work very well for me - it's 1 am in my timezone :) - so I'd rather we confirm everything that is possible

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-22 Thread Andrii Biletskyi
As said above, I spent some time thinking about AlterTopicRequest semantics and batching. Firstly, about AlterTopicRequest. Our goal here is to see whether we can suggest some simple semantics and at the same time let users change different things in one instruction (hereinafter instruction - is o

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-21 Thread Andrii Biletskyi
Guys, Thank you for your time. A short summary of our discussion. Answering previous items: 1. 2. I will double check existing error codes to align the list of errors that needs to be added. 3. We agreed to think again about the batch requests semantics. The main concern is that users would expe

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-21 Thread Jay Kreps
Hey Andrii, thanks for all the hard work on this, it has come a long way. A couple questions and comments on this. For the errors, can we do the following: 1. Remove IllegalArgument from the name, we haven't used that convention for other errors. 2. Normalize this list with the existing errors. F

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-21 Thread Andrii Biletskyi
Hi all, I've updated KIP-4 page to include all previously discussed items such as: new error codes, merged alter-topic and reassign-partitions requests, added TMR_V1. It'd be great if we concentrate on the Errors+Wire Protocol schema and discuss any remaining issues today, since first patch will

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-20 Thread Andrii Biletskyi
1. Yes, this will be much easier. Okay, let's add it. 2, Okay. This will differ a little bit from the way currently kafka-topics.sh handles alter-topic command, but I think it's a reasonable restriction. I'll update KIP acordingly to our weekly call. Thanks, Andrii Biletskyi On Mon, Apr 20, 201

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-20 Thread Jun Rao
1. Yes, lag is probably only going to be useful for the admin client. However, so is isr. It seems to me that we should get lag and isr from the same request. I was thinking that we can just extend TMR by changing replicas from an array of int to an array of (int, lag) pairs. Is that too complicate

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-19 Thread Andrii Biletskyi
Jun, 1. Yes, seems we can add lag info to the TMR. But before that I wonder whether there are other reasons we need this info except for reassign partition command? As we discussed earlier the problem with poor monitoring capabilities for reassign-partitions (as currently we only inform users Comp

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-16 Thread Jun Rao
1. For the lags, we can add a new field "lags" per partition. It will return for each replica that's not in isr, the replica id and the lag in messages. Also, if TMR is sent to a non-leader, the response can just include an empty array for isr and lags. 2. So, we will just return a topic level err

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-16 Thread Joe Stein
1. agreed 2. agree new error 3. having discrete operations for tasks makes sense, combining them is confusing for users I think. + 1 for "let user change only one thing at a time" 4. lets be consistent both to the new code and existing code. lets not confuse the user but give them the right erro

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-15 Thread Andrii Biletskyi
Guys, Thanks for the discussion! Summary: 1. Q: How KAFKA-1367 (isr is inconsistent in brokers' metadata cache) can affect implementation? A: We can fix this issue for the leading broker - ReplicaManager (or Partition) component should have accurate isr list, then with leadin

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-15 Thread Jun Rao
Andrii, 500. I think what you suggested also sounds reasonable. Since ISR is only maintained accurately at the leader, TMR can return ISR if the broker is the leader of a partition. Otherwise, we can return an empty ISR. For partition reassignment, it would be useful to know the lag of each follow

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-13 Thread Andrii Biletskyi
Jun, 404. Great, thanks! 500. If I understand correctly KAFKA-1367 says ISR part of TMR may be inconsistent. If so, then I believe all admin commands but describeTopic are not affected. Let me emphasize that it's about AdminClient operations, not about Wire Protocol requests. What I mean: To veri

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-12 Thread Jun Rao
Andrii, 404. Jay and I chatted a bit. We agreed to leave createTopicRequest as async for now. There is another thing. 500. Currently, we have this issue (KAFKA-1367) that the ISR in the metadata cache can be out of sync. The reason is that ISR is really maintained at the leader. We can potential

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

2015-04-07 Thread Andrii Biletskyi
Hi all, A summary of our discussion: 201. Q: Cluster updates in backward compatible way. A: Add topicConfigs map property and change constructor, this shouldn't break Consumer/Producer since TMR is used in NetworkClient, not directly by Consumer/Producer. 300. Q: Can we merge AlterTopic