Hi, I am trying to use Fair Scheduler Pools with Kafka Streaming. I am assigning each Kafka partition to its own pool. The attempt is to give each partition an equal share of compute time irrespective of the number of messages in each time window for each partition.
However, I do not see fair sharing to be the behavior. When running in local mode, with some artificial delay in the processing of one of the partitions, I see that till all the messages of that partition is processed, the other partitions are not being picked up. The total delay in processing of all messages in that partition from one time window is bigger than the time window itself and hence there are messages available from other partitions to be processed while the 'slow' partition is being processed. How is the fairness calculated in this type of scheduling ? Is it in some way related to setting for the number of concurrentJobs ? Thanks.
