On Wed, May 21, 2014 at 9:06 PM, Jun Rao <jun...@gmail.com> wrote:
> Expansion can be done by following
> http://kafka.apache.org/documentation.html#basic_ops_cluster_expansion
>
> If you just want to free up a server, you can stop the broker there and
> start a broker using the same broker id on a new server. Data should be
> automatically replicated to the new server.
>
>
The thing is, I don't want to fully free up the server, I just want it to
stop hosting all the topic,partition sets it originally did.
Here's the flow:
broker_1 hosts topic,{1,2,3,4}
over time load for the partitions overwhelms broker_1
spin up broker_2
migrate topic,{3,4} to broker_2 successfully
broker_1 is only useful if partitions topic,{3,4} are dropped by broker_1
how does one get broker_1 to disassociate, drop, forget, evict...(not sure
the verb) topic,{3,4} and let broker_2 own it.
Thanks,
>
> Jun
>
>
> On Wed, May 21, 2014 at 3:23 PM, David Birdsong <david.birds...@gmail.com
> >wrote:
>
> > Any suggestions? I'm kind of in a bind in that I don't understand how to
> > grow the cluster when more capacity is needed--which happens to be right
> > now for me.
> >
> > The only thing I can think that might work is to create new brokers,
> > cherry-pick topic/partition pairs and move them, then turn off the old
> ones
> > and forever retire their IDs freeing up my old hardware to come back
> online
> > as a new kafka broker ID.
> >
> >
> > On Wed, May 21, 2014 at 9:16 AM, David Birdsong <
> david.birds...@gmail.com
> > >wrote:
> >
> > >
> > >
> > >
> > > On Wed, May 21, 2014 at 9:11 AM, David Birdsong <
> > david.birds...@gmail.com>wrote:
> > >
> > >> Here's the reassignment json and current "describe" output:
> > >> https://gist.github.com/davidbirdsong/32cd0c4f49496a6a32e5
> > >>
> > >>
> > >> In my re-assignment json, I tried to re-assign to 2 when the repl is
> set
> > >> to 3. Once I noticed the the completely new node "133" had appeared in
> > the
> > >> ISR, I tried stopping 224, wiping kafka completely and then brought
> 224
> > >> back up as 224 again. It promptly replicated the topic, but never
> joined
> > >> the ISR.
> > >>
> > >
> > > to be clear, i don't want it to join the ISR. i'm curious how to make
> 224
> > > forget about the partitions for that topic since i have other plans for
> > it.
> > >
> > >
> > >>
> > >> How does one move a replica? This is exactly what I'm trying to do.
> > >>
> > >> My pattern is a common one. I started with a set of 3 kafka brokers.
> The
> > >> load and space is overwhelming them. I'm trying to add new brokers and
> > >> spread the partitions to new nodes while removing some of the
> > partitions on
> > >> the old nodes so as to make room. It's the latter that I don't get how
> > to
> > >> do.
> > >>
> > >> I've conflated two issues here mostly due to needing to get this
> cluster
> > >> stable again.
> > >> - reduce replication
> > >> - remove a partition from a broker, ie. remove the replica
> > >>
> > >> they're very distinct actions, but both would help me in the moment
> > >>
> > >>
> > >>
> > >>
> > >> On Wed, May 21, 2014 at 8:56 AM, Jun Rao <jun...@gmail.com> wrote:
> > >>
> > >>> During the re-assignment, did you move the replica off the old
> broker?
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Jun
> > >>>
> > >>>
> > >>> On Wed, May 21, 2014 at 8:21 AM, David Birdsong <
> > >>> david.birds...@gmail.com>wrote:
> > >>>
> > >>> > I did that and so now the topic has 4 replicas for a repl count of
> 3,
> > >>> but
> > >>> > only the 'new' replicas exist in the ISR.
> > >>> >
> > >>> > The old broker that I want to clear disk space and generally free
> up
> > >>> > resources has fully synced a topic that I want to disassociate from
> > it.
> > >>> >
> > >>> > Is there a way to do this?
> > >>> >
> > >>> >
> > >>> > On Wed, May 21, 2014 at 7:48 AM, Jun Rao <jun...@gmail.com> wrote:
> > >>> >
> > >>> > > We don't have an exact tool for doing this. You may be able to do
> > >>> that
> > >>> > > through
> > >>> > >
> > >>> > >
> > >>> >
> > >>>
> >
> http://kafka.apache.org/documentation.html#basic_ops_increase_replication_factorby
> > >>> > > specifying fewer replicas.
> > >>> > >
> > >>> > > Thanks,
> > >>> > >
> > >>> > > Jun
> > >>> > >
> > >>> > >
> > >>> > > On Wed, May 21, 2014 at 1:23 AM, David Birdsong <
> > >>> > david.birds...@gmail.com
> > >>> > > >wrote:
> > >>> > >
> > >>> > > > Is there a way to reduce the replication count? I'm trying to
> > >>> spread
> > >>> > > > existing partitions across more brokers, but it's hard to
> decomm
> > a
> > >>> > > broker.
> > >>> > > > Reducing repl count would suffice for now.
> > >>> > > >
> > >>> > > > Any tips?
> > >>> > > >
> > >>> > > > I'm running a mix of 0.8.1.1 and 0.8.1 (I'm upgrading now.)
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
