Do you have a partition key?
IIRC, the DefaultPartitioner works differently if you use a partition key
or not.
If you do have a partition key, it use that algorithm:
Utils.abs(key.hashCode) % numPartitions
(So if your key is the same for all your topic you will always publish in
the same partition)
And if you don't, it will change the partition every "
topic.metadata.refresh.interval.ms"
François Langelier
Étudiant en génie Logiciel - École de Technologie Supérieure
<http://www.etsmtl.ca/>
Capitaine Club Capra <http://capra.etsmtl.ca/>
VP-Communication - CS Games <http://csgames.org> 2014
Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
Comité Organisateur Olympiades ÉTS 2012
Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
On Mon, Aug 25, 2014 at 12:04 PM, Ashutosh Prasad Panigrahi <
ashutosh1...@gmail.com> wrote:
> I am using the default partitioner.
>
> Below are the details,
>
> source and target have auto.create.topics.enabled=true and default number
> of partitions is 2.
>
> bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
> mirror-maker/consumer.
> properties --producer.config mirror-maker/producer.properties
> --whitelist=".*" *--num.streams 2 --num.producers 2*
>
>
> the consumer.properties for mirror maker looks like this
> group.id=test-group
> zookeeper.connect=<ZK_IP_LIST>
>
> and producer.properties for mirror maker looks like this
>
> metadata.broker.list=<KAFKA_IP_LIST>
> producer.type=async
> topic.metadata.refresh.interval.ms=60000
>
>
>
> Ashutosh
>
>
> On Mon, Aug 25, 2014 at 6:38 PM, François Langelier <f.langel...@gmail.com
> > wrote:
>
>> What is your partitioning function?
>>
>>
>>
>> François Langelier
>> Étudiant en génie Logiciel - École de Technologie Supérieure
>> <http://www.etsmtl.ca/>
>> Capitaine Club Capra <http://capra.etsmtl.ca/>
>> VP-Communication - CS Games <http://csgames.org> 2014
>> Jeux de Génie <http://www.jdgets.com/> 2011 à 2014
>> Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/>
>>
>> Comité Organisateur Olympiades ÉTS 2012
>> Compétition Québécoise d'Ingénierie 2012 - Compétition Senior
>>
>>
>> On Mon, Aug 25, 2014 at 6:18 AM, Ashutosh Prasad Panigrahi <
>> ashutosh1...@gmail.com> wrote:
>>
>> > Hi Guozhang,
>> >
>> > I repeated my experiment with topic.metadata.refresh.interval.ms set
>> to 60
>> > seconds, and executed the mirror for more than 24 hours.
>> >
>> > I still see that it only writes to Partition "1" and totally ignores
>> > partition "0".
>> >
>> > Can you please provide me some pointers to debug this.
>> >
>> > Ashutosh
>> >
>> >
>> > On Thu, Aug 21, 2014 at 8:52 PM, Guozhang Wang <wangg...@gmail.com>
>> wrote:
>> >
>> > > Hi Ashutosh,
>> > >
>> > > Maybe this faq can help you?
>> > >
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified
>> > > ?
>> > >
>> > > Guozhang
>> > >
>> > >
>> > > On Thu, Aug 21, 2014 at 12:09 AM, Ashutosh Prasad Panigrahi <
>> > > ashutosh1...@gmail.com> wrote:
>> > >
>> > > > Hello All,
>> > > >
>> > > > When i start the mirroring process, events are written only to
>> > partition
>> > > > "1" on the target cluster and no data is written to partition 0.
>> > > >
>> > > > I am stuck on this and wondering what am i doing wrong .
>> > > >
>> > > > My production setup looks something like this
>> > > > 1. Datacenter1 : 3 kafka nodes/3 zookeeper nodes
>> > > > 2. Datacenter 2 : 3 kafka nodes/3 zookeeper nodes
>> > > >
>> > > > All the kafka nodes are configured with
>> auto.create.topics.enable=true
>> > > and
>> > > > the default partition is 2 .
>> > > >
>> > > > I start the kafka mirroring tool with the following command
>> > > >
>> > > > bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config
>> > > > mirror-maker/consumer.properties --producer.config
>> > > > mirror-maker/producer.properties --whitelist=".*" *--num.streams 2
>> > > > --num.producers 2*
>> > > >
>> > > >
>> > > > the consumer.properties for mirror-config looks like this
>> > > > group.id=test-group
>> > > > zookeeper.connect=<ZK_IP_LIST>
>> > > >
>> > > > and producer.properties for mirror-config looks like this
>> > > >
>> > > > metadata.broker.list=<KAFKA_IP_LIST>
>> > > > producer.type=async
>> > > > topic.metadata.refresh.interval.ms=-1
>> > > >
>> > > >
>> > > >
>> > > > Regards
>> > > > --
>> > > > Ashutosh
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > -- Guozhang
>> > >
>> >
>> >
>> >
>> > --
>> > Ashutosh Prasad Panigrahi
>> >
>> > All you who sleep tonight far from the ones you love
>> > No hands to the left or right and emptiness above
>> > Know that you aren't alone
>> > the whole world shares your tears
>> > some for two nights or one
>> > and some for all their years.......
>> >
>>
>
>
>
> --
> Ashutosh Prasad Panigrahi
>
> All you who sleep tonight far from the ones you love
> No hands to the left or right and emptiness above
> Know that you aren't alone
> the whole world shares your tears
> some for two nights or one
> and some for all their years.......
>
