Can you please tell me how to set this property ? topic.metadata.refresh.interval.ms Is a value of 100 low enough to solve this issue ? Im guessing I can set it to 100 and restart the command line producer and the partitioning should work ? Please confirm.
Thanks On Mon, Jun 9, 2014 at 5:09 PM, Prakash Gowri Shankor < prakash.shan...@gmail.com> wrote: > Thank you Guozhang. > I've specified how i set and use the property in my previous mail. Can you > tell me if that is fine ? > I also noticed that the kafka-console-producer.sh takes a custom > property(key-value) on the command line. Would it help to set this property > directly on the command line of the producer script ? > > > On Mon, Jun 9, 2014 at 5:06 PM, Guozhang Wang <wangg...@gmail.com> wrote: > >> In the new producer we are changing the default behavior back to pure >> random partitioning and let users to customize their own partitioning >> schemes if they want. For now reducing topic.metadata.refresh.interval.ms >> should help because the stickiness only persists until a metadata refresh. >> >> Guozhang >> >> >> On Mon, Jun 9, 2014 at 4:54 PM, Prakash Gowri Shankor < >> prakash.shan...@gmail.com> wrote: >> >> > Is there a way to modify this duration ? This is not adhering to the >> > "random" behavior that the documentation talks about. >> > >> > >> > On Mon, Jun 9, 2014 at 4:41 PM, Kane Kane <kane.ist...@gmail.com> >> wrote: >> > >> > > Last time I've checked it, producer sticks to partition for 10 >> minutes. >> > > >> > > On Mon, Jun 9, 2014 at 4:13 PM, Prakash Gowri Shankor >> > > <prakash.shan...@gmail.com> wrote: >> > > > Hi, >> > > > >> > > > This is with 0.8.1.1 and I ran the command line console consumer. >> > > > I have one broker, one producer and several consumers. I have one >> > topic, >> > > > many partitions m, many consumers n, m=n , one consumer group >> defined >> > for >> > > > all the consumers >> > > > >> > > > From using Kafka Monitor, I see that each partition is assigned to >> one >> > > > consumer now. However, it seems that there is no parallelism in data >> > > > consumption. What I see happening is that one consumer gets messages >> > from >> > > > time t0 to t1 from partition P1. Then another consumer gets messages >> > from >> > > > t1 to t2 from partition P2 and so on. >> > > > >> > > > *Why is there no parallel consumption happening ?* It looks to me >> that >> > > the >> > > > producer's data goes into P1 from t0 to t1 and then from t1 to t2 >> into >> > > P2. >> > > > I thought that if I dont specify a partitioning key, the producer's >> > data >> > > > will get partitioned randomly. It's just that the randomness seems >> to >> > be >> > > > "delayed". Why is this so ? >> > > > >> > > > I tried setting topic.metadata.refresh.interval.ms=100 in the >> > > > producer.properties. >> > > > >> > > > That did not seem to change this strange partitioning behavior. >> > > > >> > > > Please help. >> > > > >> > > > Thanks >> > > >> > >> >> >> >> -- >> -- Guozhang >> > >
