Json file should be correct, and I have just checked the code, it should
send the stop replica request to the old leaders to delete their logs. If
you can consistently re-produce this issue could you file a jira?
Guozhang
On Sat, May 31, 2014 at 7:40 PM, Otis Gospodnetic <
otis.gospodne...@gmail.com> wrote:
> Hi,
>
> No, I don't think I saw anything with "Stop" in the logs, but I may have
> lost those old logs, so I'm not 100% sure.
>
> I just re-ran the same command and I didn't see anything with "Stop" in
> broker 2's (source) log.
>
> I did see log messages like the following one in broker 2's logs as soon as
> I ran this:
>
> $ sh kafka-reassign-partitions.sh --broker-list 1 --zookeeper
> zk1.prod.sematext:3131 --reassignment-json-file /tmp/partition.json
> --execute
>
> [2014-06-01 02:29:31,256] ERROR [KafkaApi-2] Error when processing fetch
> request for partition [spm_new_cluster_free_system_topic,4] offset 0 from
> follower with correlation id 1 (kafka.server.KafkaApis)
> kafka.common.OffsetOutOfRangeException: Request for offset 0 but we only
> have log segments in the range 16400532 to 35860153.
> at kafka.log.Log.read(Log.scala:380)
> at
>
> kafka.server.KafkaApis.kafka$server$KafkaApis$$readMessageSet(KafkaApis.scala:530)
> at
>
> kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:476)
> at
>
> kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:471)
>
> ... but I'm not sure what to make of it. The /tmp/partition.json file was
> generated with --generate and contains:
>
> {"version":1,"partitions":[
>
> {"topic":"spm_new_cluster_free_system_topic","partition":0,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":6,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":11,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":4,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":2,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":9,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":3,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":5,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":7,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":1,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":10,"replicas":[1]},{"topic":"spm_new_cluster_free_system_topic","partition":8,"replicas":[1]}]}
>
> Does the above mean, in plain English, "move partitions 0-11 for
> topic spm_new_cluster_free_system_topic to broker 1"?
>
> Thanks,
> Otis
> --
> Performance Monitoring * Log Analytics * Search Analytics
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Sat, May 31, 2014 at 8:32 PM, Guozhang Wang <wangg...@gmail.com> wrote:
>
> > You may take a look at the broker configs for replication factor:
> >
> > http://kafka.apache.org/documentation.html#brokerconfigs
> >
> > With replication factor 1 (default value) the broker holding the old
> leader
> > should received StopReplica Request to stop the replica and delete the
> > segment files. Do you see this log entries on broker 2's request logs?
> >
> > Guozhang
> >
> >
> > On Sat, May 31, 2014 at 5:15 PM, Otis Gospodnetic <
> > otis.gospodne...@gmail.com> wrote:
> >
> > > Hi Guozhang,
> > >
> > > I don't have any replication-related properties in my config file and I
> > > don't see any mentions of replication at startup in the log, so I don't
> > > really know.
> > >
> > > The only time I saw replication mentioned in the log was during some
> > errors
> > > that said my replication factor was 1 and that was > the number of
> > brokers.
> > > So based on this error message I'd say my replication factor is 1.
> > >
> > > Is there a replication property I could explicitly set?
> > >
> > > Thanks,
> > > Otis
> > > --
> > > Performance Monitoring * Log Analytics * Search Analytics
> > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> > >
> > > On Sat, May 31, 2014 at 7:05 PM, Guozhang Wang <wangg...@gmail.com>
> > wrote:
> > >
> > > > Hello Otis,
> > > >
> > > > What is your replication factor?
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Sat, May 31, 2014 at 12:39 PM, Otis Gospodnetic <
> > > > otis.gospodne...@gmail.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I have 2 pretty unbalanced brokers running 0.8.1.1 and I'm trying
> to
> > > > > balance them (read: free disk space on the overloaded broker).
> > > > >
> > > > > Is there a tool that will just rebalance everything? I couldn't
> find
> > > > one,
> > > > > so I ran this:
> > > > >
> > > > > $ sh kafka-reassign-partitions.sh --broker-list 1 --zookeeper
> > > > > zk1.prod.sematext:3131 --reassignment-json-file /tmp/partition.json
> > > > > --execute
> > > > >
> > > > > This seems to have *copied* about half of the partitions of the
> topic
> > > > > specified in the given JSON file from broker 2 (source) to broker 1
> > > > > (target). Good.
> > > > >
> > > > > I see this in broker 2 (source broker):
> > > > >
> > > > > [2014-05-31 19:24:59,994] INFO [ReplicaFetcherManager on broker 2]
> > > > Removed
> > > > > fetcher for partitions [spm_new_cluster_free_system_topic,1]
> > > > > (kafka.server.ReplicaFetcherManager)
> > > > >
> > > > > ... which looks promising.... but:
> > > > >
> > > > > My broker 2 (source) *still* has all partitions of the problematic
> > > topic.
> > > > > I thought it would remove the moved partitions from the source
> > > broker....
> > > > > no?
> > > > >
> > > > > Thanks,
> > > > > Otis
> > > > > --
> > > > > Performance Monitoring * Log Analytics * Search Analytics
> > > > > Solr & Elasticsearch Support * http://sematext.com/
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
