Hi! > On 12 Aug 2016, at 04:29, Jun Rao <j...@confluent.io> wrote: > > Hi, Andrey, > > One potential benefit of keeping the per partition limit is for Kafka > stream. When reading messages from different partitions, KStream prefers to > read from partitions with smaller timestamps first and only advances the > KStream timestamp when it sees at least one message from every partition. > Being able to fill up multiple partitions in a single fetch response can > help KStream advance the timestamp quicker when there is backlog from the > input. So, it's probably better if we just add a new response limit while > keeping the per partition limit. >
Yes, this makes sense for me. > Also, for fairness across partitions, you mentioned "The solution is to > start fetching from first empty partition in round-robin fashion or to > perform random shuffle of partitions.". It seems the former is more > deterministic. Did you use that in your implementation and should we > recommend that for non-java clients as well? > In my initial implementation I just did random shuffling on server side. Now I plan to use deterministic approach - do round-robin in ReplicaFetcher thread - I believe the client library itself shouldn’t do any form of reordering. But we should definitely recommend some form of shuffling if client wants to limit response size. Not sure which shuffling method is better. > Thanks, > > Jun > Thanks, Andrey.
