; From: Joel Koshy [jjkosh...@gmail.com]
> Sent: Thursday, June 11, 2015 1:28 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative
> operations (Thread 2)
>
> Discussion aside, was there any significant material change beside
afka.apache.org
Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative
operations (Thread 2)
Discussion aside, was there any significant material change besides
the additions below? If so, then we can avoid the overhead of another
vote unless someone wants to down-vote these
the admin client spec. (Alter and Describe config).
>
> Please review.
>
> Aditya
>
>
> From: Ashish Singh [asi...@cloudera.com]
> Sent: Friday, May 29, 2015 8:36 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-4 - Comman
rg
Subject: RE: [DISCUSS] KIP-4 - Command line and centralized administrative
operations (Thread 2)
I've made two changes to the document:
- Removed the TMR evolution piece since we agreed to retain this.
- Added two new API's to the admin client spec. (Alter and Describe config).
Pl
.com]
Sent: Friday, May 29, 2015 8:36 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative
operations (Thread 2)
+1 on discussing this on next KIP hangout. I will update KIP-24 before that.
On Fri, May 29, 2015 at 3:40 AM, Andrii Biletskyi <
anks. Perhaps we should leave TMR unchanged for now. Should we discuss
> > this during the next hangout?
> >
> > Aditya
> >
> > ____
> > From: Jun Rao [j...@confluent.io]
> > Sent: Thursday, May 28, 2015 5:32 PM
> > To: d
x27;ve altered the KIP with the
> > assumption that this is a good enough reason by itself to evolve the
> > request/response protocol. Any concerns there?
> >
> > Thanks,
> > Aditya
> >
> > ____________
> > From: Mayuresh Gharat [gharatmayures.
___
> From: Mayuresh Gharat [gharatmayures...@gmail.com]
> Sent: Thursday, May 21, 2015 8:29 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative
> operations (Thread 2)
>
> Hi Jun,
>
> Thanks a lo
arat [gharatmayures...@gmail.com]
> Sent: Thursday, May 21, 2015 8:29 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative
> operations (Thread 2)
>
> Hi Jun,
>
> Thanks a lot. I get it now.
> Point 4) will actually en
015 8:29 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-4 - Command line and centralized administrative
operations (Thread 2)
Hi Jun,
Thanks a lot. I get it now.
Point 4) will actually enable clients to who don't want to create a topic
with default partitions, if it does not exist an
Hi Jun,
Thanks a lot. I get it now.
Point 4) will actually enable clients to who don't want to create a topic
with default partitions, if it does not exist and then can manually create
the topic with their own configs(#partitions).
Thanks,
Mayuresh
On Thu, May 21, 2015 at 6:16 PM, Jun Rao wro
Mayuresh,
The current plan is the following.
1. Add TMR v1, which still triggers auto topic creation.
2. Change the consumer client to TMR v1. Change the producer client to use
TMR v1 and on UnknownTopicException, issue TopicCreateRequest to explicitly
create the topic with the default server sid
> Here are some ideas to address this :
>
> 1) The way this can be addressed is that TopicMetadata request should have
> a way to specify whether it should only check if the topic exist or check
> and create a topic with given number of partitions. If the number of
> partitions is not specified u
Hi,
I had a question about TopicMetadata Request.
Currently the way it works is :
1) Suppose a topic T1 does not exist.
2) Client wants to produce data to T1 using producer P1.
3) Since T1 does not exist, P1 issues a TopicMetadata request to kafka.
This in turn creates the default number of partit
For ListTopics, we decided not to add a ListTopics request for now and just
rely on passing in an empty list to TMR. We can revisit this in the future
if it becomes an issue.
Thanks,
Jun
On Wed, May 13, 2015 at 3:31 PM, Joel Koshy wrote:
> Just had a few minor questions before I join the vote
Joel,
- DecreasePartitionNotAllowed. Yeah, that's kind of subcase of
InvalidPartitions...
But since client can always request topic metadata and check what exactly is
was wrong with Partitions argument, I think, yes, we can remove
DecreasePartitionNotAllowed
and use InvalidPartitions instead. I'll
Just had a few minor questions before I join the vote thread.
Apologies if these have been discussed:
- Do we need DecreasePartitionsNotAllowed? i.e., can we just return
InvalidPartitions instead?
- AdminClient.listTopics: should we allow listing all partitions? Or
do you intend for the client
Guys,
I've updated the wiki to reflect all previously discussed items
(regarding the schema only - this is included to phase 1).
https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations
I think we can have the final discussion today (for pha
The following is a description of some of my concerns on allowing the same
topic multiple times in AlterTopicRequest.
ATP has an array of entries, each corresponding to a topic. We allow
multiple changes to a topic in a single entry. Those changes may fail to
apply independently (e.g., the config
Guys,
A quick summary of our today's meeting.
There were no additional issues/questions. The only item about which
we are not 100% sure is "multiple instructions for one topic in one
request" case.
It was proposed by Jun to explain reasons behind not allowing users doing
that again
here in mailin
Guys,
It seems that there are no open questions left so prior to our weekly call
let me summarize what I'm going to implement as part of phase one for KIP-4.
1. Add 3 new Wire Protocol requests - Create-, Alter- and DeleteTopicRequest
2. Topic requests are batch requests, errors are returned per
Yes, to verify if a partition reassignment completes or not, we just need
to make sure AR == RAR. So, we don't need ISR for this. It's probably still
useful to know ISR for monitoring in general though.
Thanks,
Jun
On Mon, Apr 27, 2015 at 4:15 AM, Andrii Biletskyi <
andrii.bilets...@stealth.ly>
Okay, I had some doubts in terms of reassign-partitions case. I was
not sure whether we need ISR to check post condition of partition
reassignment. But I think we can rely on assigned replicas - the workflow
in reassignPartitions is the following:
1. Update AR in ZK with OAR + RAR.
...
10. Update A
Andrii,
Another thing. We decided not to add the lag info in TMR. To be consistent,
we probably also want to remove ISR from TMR since only the leader knows
it. We can punt on adding any new request from getting ISR. ISR is mostly
useful for monitoring. Currently, one can determine if a replica is
Jun,
I like your approach to AlterTopicReques semantics! Sounds like
we linearize all request fields to ReplicaAssignment - I will definitely
try this out to ensure there are no other pitfalls.
With regards to multiple instructions in one batch per topic. For me
this sounds reasonable too. We dis
Andrii,
Thanks for the update.
For your second point, I agree that if a single AlterTopicRequest can make
multiple changes, there is no need to support the same topic included more
than once in the request.
Now about the semantics in your first question. I was thinking that we can
do the followi
Guys,
Can we come to some agreement in terms of the second item from
the email above? This blocks me from updating and uploading the
patch. Also the new schedule for the weekly calls doesn't work very
well for me - it's 1 am in my timezone :) - so I'd rather we confirm
everything that is possible
As said above, I spent some time thinking about AlterTopicRequest
semantics and batching.
Firstly, about AlterTopicRequest. Our goal here is to see whether we
can suggest some simple semantics and at the same time let users
change different things in one instruction (hereinafter instruction - is
o
Guys,
Thank you for your time. A short summary of our discussion.
Answering previous items:
1. 2. I will double check existing error codes to align the list of
errors that needs to be added.
3. We agreed to think again about the batch requests semantics.
The main concern is that users would expe
Hey Andrii, thanks for all the hard work on this, it has come a long way.
A couple questions and comments on this.
For the errors, can we do the following:
1. Remove IllegalArgument from the name, we haven't used that convention
for other errors.
2. Normalize this list with the existing errors. F
Hi all,
I've updated KIP-4 page to include all previously discussed items such as:
new error codes, merged alter-topic and reassign-partitions requests, added
TMR_V1.
It'd be great if we concentrate on the Errors+Wire Protocol schema and
discuss
any remaining issues today, since first patch will
1. Yes, this will be much easier. Okay, let's add it.
2, Okay. This will differ a little bit from the way currently
kafka-topics.sh handles alter-topic command, but I think it's
a reasonable restriction.
I'll update KIP acordingly to our weekly call.
Thanks,
Andrii Biletskyi
On Mon, Apr 20, 201
1. Yes, lag is probably only going to be useful for the admin client.
However, so is isr. It seems to me that we should get lag and isr from the
same request. I was thinking that we can just extend TMR by changing
replicas from an array of int to an array of (int, lag) pairs. Is that too
complicate
Jun,
1. Yes, seems we can add lag info to the TMR. But before that I wonder
whether there are other reasons we need this info except for reassign
partition command? As we discussed earlier the problem with poor
monitoring capabilities for reassign-partitions (as currently we only inform
users Comp
1. For the lags, we can add a new field "lags" per partition. It will
return for each replica that's not in isr, the replica id and the lag in
messages. Also, if TMR is sent to a non-leader, the response can just
include an empty array for isr and lags.
2. So, we will just return a topic level err
1. agreed
2. agree new error
3. having discrete operations for tasks makes sense, combining them is
confusing for users I think. + 1 for "let user change only one thing at a
time"
4. lets be consistent both to the new code and existing code. lets not
confuse the user but give them the right erro
Guys,
Thanks for the discussion!
Summary:
1. Q: How KAFKA-1367 (isr is inconsistent in brokers' metadata cache) can
affect implementation?
A: We can fix this issue for the leading broker - ReplicaManager (or
Partition)
component should have accurate isr list, then with leadin
Andrii,
500. I think what you suggested also sounds reasonable. Since ISR is only
maintained accurately at the leader, TMR can return ISR if the broker is
the leader of a partition. Otherwise, we can return an empty ISR. For
partition reassignment, it would be useful to know the lag of each
follow
Jun,
404. Great, thanks!
500. If I understand correctly KAFKA-1367 says ISR part of TMR may
be inconsistent. If so, then I believe all admin commands but describeTopic
are not affected. Let me emphasize that it's about AdminClient operations,
not about Wire Protocol requests. What I mean:
To veri
Andrii,
404. Jay and I chatted a bit. We agreed to leave createTopicRequest as
async for now.
There is another thing.
500. Currently, we have this issue (KAFKA-1367) that the ISR in the
metadata cache can be out of sync. The reason is that ISR is really
maintained at the leader. We can potential
Hi all,
A summary of our discussion:
201. Q: Cluster updates in backward compatible way.
A: Add topicConfigs map property and change constructor, this
shouldn't break Consumer/Producer since TMR is used in NetworkClient,
not directly by Consumer/Producer.
300. Q: Can we merge AlterTopic
41 matches
Mail list logo