Posted a KIP for --re-balance for partition assignment in reassignment tool.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New+reassignment+partition+logic+for+re-balancing
JIRA https://issues.apache.org/jira/browse/KAFKA-1792
While going through the KIP I thought of one thing from t
Ditto on this one. Can you give the algorithm we want to implement?
Also I think in terms of scope this is just proposing to change the logic
in ReassignPartitionsCommand? I think we've had the discussion various
times on the mailing list that what people really want is just for Kafka to
do it's b
Hey Joe,
1. Could you add details to the Public Interface section of the KIP? This
should include the proposed changes to the partition reassignment tool.
Also, maybe the new option can be named --rebalance instead of
--re-balance?
2. It makes sense to list --decommission-broker as part of this KI
I will go back through the ticket and code and write more up. Should be
able to-do that sometime next week. The intention was to not replace
existing functionality by issue a WARN on use. The following version it is
released we could then deprecate it... I will fix the KIP for that too.
On Fri, Ja
Apologize for the late comment on this...
So fair assignment by count (taking into account the current partition
count of each broker) is very good. However, it's worth noting that all
partitions are not created equal. We have actually been performing more
rebalance work based on the partition siz
Todd,
I think plugable design is good with solid default. The only issue I
feel is when you use one and switch to another, will we end up with some
unread messages hanging around and no one thinks or knows it is their
responsibility to take care of them?
Thanks.
Tong
Sent from my iPhone
>
.
Aditya
From: Tong Li [liton...@us.ibm.com]
Sent: Wednesday, March 04, 2015 7:33 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-6 - New reassignment partition logic for re-balancing
Todd,
I think plugable design is good with solid default. The only
I would not think that partitions moving would cause any orphaned messages
like that. I would be more concerned about what happens when you change the
default on a running cluster from one scheme to another. Would we want to
support some kind of automated reassignment of existing partitions
(person
I am +1 Todd's suggestion, the default reassignment scheme is only used
when a reassignment command is issued with no scheme specified, and
changing this default scheme should not automatically trigger a
reassignment of all existing topics: it will only take effect when the next
reassignment comman
Hey Joe,
This still seems pretty incomplete. It still has most the sections just
containing the default text you are supposed to replace. It is really hard
to understand what is being proposed and why and how much of the problem we
are addressing. For example the motivation section just says "oper
Jay,
That makes sense. I think what folks are bringing up all sounds great but
I feel can/should be done afterwards as further improvements as the scope
for this change has a very specific focus to resolve problems folks have
today with --generate (with a patch tested and ready to go ). I should
Sorry for not catching up on this thread earlier, I wanted to-do this
before the KIP got its updates so we could discuss if need be and not waste
more time re-writing/working things that folks have issues with or such. I
captured all the comments so far here with responses.
<< So fair assignment b
I understand the desire to not bloat this one change with too much more
work, and it's a good change to start with. That said, I have one note on
your comments:
"I don't agree with this because right now you get back "the current state of
the partitions" so you can (today) write whatever logic you
Hi, Joe,
A couple of comments.
1. When creating a new topic, our replica assignment algorithm tries to
achieve a few things: (a) all replicas are spread evenly across brokers;
(b) the preferred replica (first replica in the assigned replica list) of
all partitions are spread evenly across brokers
14 matches
Mail list logo
