Re: partitionsFor method doesn't return latest partition metadata

2015-11-30 Thread Guozhang Wang
Siyuan, This behavior of only periodically update metadata at the consumer is by-design to abstract users from manual metadata refresh, and having a stale metadata would not have incorrectness issues since it will only mean some newly added partitions will not be consumed for at most

partitionsFor method doesn't return latest partition metadata

2015-11-30 Thread hsy...@gmail.com
Hi guys, I want to use partitionsFor method of new consumer API periodically to get the monitor partition metadata change, It seems it only issue remote call to the server for the first time. If I add partitions after that, partitionsFor will return stale value. Is there a way to reuse consumer

Re: partitionsFor method doesn't return latest partition metadata

2015-11-30 Thread Jason Gustafson
Hey Siyuan, Are you also consuming from the topics you are querying with partitionsFor? I'm guessing the problem is that we're pulling the metadata from a local cache, and it's not refreshed as quick as you're expecting (the default is 5 minutes). We only use the cache for topics which have