I have seen that mail thread. Here is what i tried: In my producer.properties I set topic.metadata.refresh.interval.ms=1000. I guess this means that the a different partition will be selected every second.
Then I restart my producer as : ./kafka-console-producer.sh --broker-list localhost:9092 --topic test2 Then I generate more data for a minute or so, I still see it going to one partition. Is this the right way to use the refresh property ? On Mon, Jun 9, 2014 at 4:56 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Kane is right, please see this FAQ for details: > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified > ? > > > On Mon, Jun 9, 2014 at 4:41 PM, Kane Kane <kane.ist...@gmail.com> wrote: > > > Last time I've checked it, producer sticks to partition for 10 minutes. > > > > On Mon, Jun 9, 2014 at 4:13 PM, Prakash Gowri Shankor > > <prakash.shan...@gmail.com> wrote: > > > Hi, > > > > > > This is with 0.8.1.1 and I ran the command line console consumer. > > > I have one broker, one producer and several consumers. I have one > topic, > > > many partitions m, many consumers n, m=n , one consumer group defined > for > > > all the consumers > > > > > > From using Kafka Monitor, I see that each partition is assigned to one > > > consumer now. However, it seems that there is no parallelism in data > > > consumption. What I see happening is that one consumer gets messages > from > > > time t0 to t1 from partition P1. Then another consumer gets messages > from > > > t1 to t2 from partition P2 and so on. > > > > > > *Why is there no parallel consumption happening ?* It looks to me that > > the > > > producer's data goes into P1 from t0 to t1 and then from t1 to t2 into > > P2. > > > I thought that if I dont specify a partitioning key, the producer's > data > > > will get partitioned randomly. It's just that the randomness seems to > be > > > "delayed". Why is this so ? > > > > > > I tried setting topic.metadata.refresh.interval.ms=100 in the > > > producer.properties. > > > > > > That did not seem to change this strange partitioning behavior. > > > > > > Please help. > > > > > > Thanks > > > > > > -- > -- Guozhang >
