Hi, Andrey, Why shouldn't the client library do reordering? It seems that if ReplicaFetcher thread does round-robin, the consumer client should do that too?
Thanks, Jun On Fri, Aug 12, 2016 at 3:56 AM, Andrey L. Neporada < anepor...@yandex-team.ru> wrote: > Hi! > > > On 12 Aug 2016, at 04:29, Jun Rao <j...@confluent.io> wrote: > > > > Hi, Andrey, > > > > One potential benefit of keeping the per partition limit is for Kafka > > stream. When reading messages from different partitions, KStream prefers > to > > read from partitions with smaller timestamps first and only advances the > > KStream timestamp when it sees at least one message from every partition. > > Being able to fill up multiple partitions in a single fetch response can > > help KStream advance the timestamp quicker when there is backlog from the > > input. So, it's probably better if we just add a new response limit while > > keeping the per partition limit. > > > > Yes, this makes sense for me. > > > Also, for fairness across partitions, you mentioned "The solution is to > > start fetching from first empty partition in round-robin fashion or to > > perform random shuffle of partitions.". It seems the former is more > > deterministic. Did you use that in your implementation and should we > > recommend that for non-java clients as well? > > > > In my initial implementation I just did random shuffling on server side. > Now I plan to use deterministic approach - do round-robin in > ReplicaFetcher thread - I believe the client library itself shouldn’t do > any form of reordering. > > But we should definitely recommend some form of shuffling if client wants > to limit response size. > Not sure which shuffling method is better. > > > > Thanks, > > > > Jun > > > > Thanks, > Andrey. > >
