You don't need to match the number of partitions. Your nine consumers should distribute those 12 partitions amongst themselves. Are you by any chance consuming across a high latency link? Can you see if there are rebalance failures in your consumer logs?
On Thu, Apr 10, 2014 at 12:27:39PM +0530, Arjun wrote: > My consumer configuration has auto.offset.reset=smallest will this > be causing the problem for replaying the messages? > I got to know why some partitions have no owners. It was my mistake > i have 12 partitions but i started only 9 consumers. So when i check > the Consumer Offset checker tool 3 partitions were showing "none" as > the owner. > > If some one can please help why we get this > " > [kafka.utils.ZkUtils$] conflict in > /consumers/group1/owners/testtopic/2 data: > group1_ip-10-164-9-107-1397047622368-c14b108b-1 stored data: > group1_ip-10-168-47-114-1397034341089-3861c6cc-1 > " > and the replay of messages i would be grateful > > Thanks > Arjun Narasimha Kota > > > On Thursday 10 April 2014 10:17 AM, Arjun wrote: > >I see this a lot in the consumer logs > > [kafka.utils.ZkUtils$] conflict in > >/consumers/group1/owners/testtopic/2 data: > >group1_ip-10-164-9-107-1397047622368-c14b108b-1 stored data: > >group1_ip-10-168-47-114-1397034341089-3861c6cc-1 > > > >I am not seeing any "expired" in my consumer logs > > > >Thanks > >Arjun Narasimha Kota > > > >On Thursday 10 April 2014 09:13 AM, Jun Rao wrote: > >>Do you see many rebalances in the consumer log? If so, see > >>https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyaretheremanyrebalancesinmyconsumerlog > >> > >>? > >> > >>Thanks, > >> > >>Jun > >> > >> > >>On Wed, Apr 9, 2014 at 8:38 PM, Arjun <[email protected]> wrote: > >> > >>>Hi , > >>> > >>>I have set up kafka 0.8 in 3 servers. I have pushed some data > >>>into these > >>>servers. The number of partitions i use is 12, with a > >>>replication factor of > >>>2. > >>>We use high level consumer to consume messages. we set auto commit to > >>>false, and we do commits almost after each and every successful message > >>>consumption. > >>>The problem i am facing now is, If i restart my consumers, then the > >>>messages are getting replayed. > >>> > >>>I checked the > >>> > >>>bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group1 > >>>--zkconnect zkhost:zkport --topic testtopic > >>> > >>>I see a high lag in some of the partitions and some partitions > >>>have none > >>>as the owner. > >>>To verify consumer rebalance i used the following command > >>> > >>>bin/kafka-run-class.sh kafka.tools.VerifyConsumerRebalance > >>>--group group1 > >>>--zookeeper.connect zkhost:zkport > >>> > >>>but i see errors > >>> > >>>[2014-04-10 03:24:37,832] ERROR No owner for partition [testtopic,0] > >>>(kafka.tools.VerifyConsumerRebalance$) > >>>[2014-04-10 03:24:37,861] ERROR No owner for partition [testtopic,1] > >>>(kafka.tools.VerifyConsumerRebalance$) > >>>[2014-04-10 03:24:37,862] ERROR No owner for partition [testtopic,2] > >>>(kafka.tools.VerifyConsumerRebalance$) > >>>[2014-04-10 03:24:37,863] ERROR No owner for partition [testtopic,3] > >>>(kafka.tools.VerifyConsumerRebalance$) > >>>[2014-04-10 03:24:37,864] ERROR No owner for partition [testtopic,4] > >>>(kafka.tools.VerifyConsumerRebalance$) > >>>[2014-04-10 03:24:37,870] ERROR No owner for partition [testtopic,5] > >>>(kafka.tools.VerifyConsumerRebalance$) > >>>[2014-04-10 03:24:37,871] ERROR No owner for partition [testtopic,6] > >>>(kafka.tools.VerifyConsumerRebalance$) > >>>[2014-04-10 03:24:37,872] ERROR No owner for partition [testtopic,7] > >>>(kafka.tools.VerifyConsumerRebalance$) > >>>[2014-04-10 03:24:37,873] ERROR No owner for partition [testtopic,8] > >>>(kafka.tools.VerifyConsumerRebalance$) > >>>[2014-04-10 03:24:37,874] ERROR No owner for partition [testtopic,9] > >>>(kafka.tools.VerifyConsumerRebalance$) > >>>[2014-04-10 03:24:37,928] ERROR No owner for partition [testtopic,9] > >>>(kafka.tools.VerifyConsumerRebalance$) > >>>[2014-04-10 03:24:37,928] ERROR No owner for partition [testtopic,10] > >>>(kafka.tools.VerifyConsumerRebalance$) > >>>[2014-04-10 03:24:37,930] ERROR No owner for partition [testtopic,10] > >>>(kafka.tools.VerifyConsumerRebalance$) > >>>[2014-04-10 03:24:37,930] ERROR No owner for partition [testtopic,11] > >>>(kafka.tools.VerifyConsumerRebalance$) > >>>[2014-04-10 03:24:37,931] ERROR No owner for partition [testtopic,11] > >>>(kafka.tools.VerifyConsumerRebalance$) > >>> > >>> > >>>I have seen this ( > >>>http://mail-archives.apache.org/mod_mbox/kafka-users/ > >>>201311.mbox/%3CCA+B+UZsnpuNicHcj6Q6yJaLccyhNmoOieM > >>>[email protected]%3E) in the mail archives and i > >>>checked my > >>>consumer log for those patterns but couldn't find any in my logs. > >>> > >>>Can some one help me with this. > >>> > >>>Thanks > >>>Arjun Narasimha kota > >>> > > >
