Yes it is related to concurrentJobs, so you need to increase that. Salt that will mean that if you get overlapping batches then those will be executed in parallel too
On Tue, 16 Feb 2016, 18:33 p pathiyil <pathi...@gmail.com> wrote: > Hi, > > I am trying to use Fair Scheduler Pools with Kafka Streaming. I am > assigning each Kafka partition to its own pool. The attempt is to give each > partition an equal share of compute time irrespective of the number of > messages in each time window for each partition. > > However, I do not see fair sharing to be the behavior. When running in > local mode, with some artificial delay in the processing of one of the > partitions, I see that till all the messages of that partition is > processed, the other partitions are not being picked up. The total delay in > processing of all messages in that partition from one time window is bigger > than the time window itself and hence there are messages available from > other partitions to be processed while the 'slow' partition is being > processed. > > How is the fairness calculated in this type of scheduling ? Is it in some > way related to setting for the number of concurrentJobs ? > > Thanks. >
