[ https://issues.apache.org/jira/browse/KAFKA-8478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17377549#comment-17377549 ]
John Roesler commented on KAFKA-8478: ------------------------------------- Hello again, all, this issue should be resolved in 3.0, via KAFKA-10091. > Poll for more records before forced processing > ---------------------------------------------- > > Key: KAFKA-8478 > URL: https://issues.apache.org/jira/browse/KAFKA-8478 > Project: Kafka > Issue Type: Improvement > Components: streams > Reporter: John Roesler > Assignee: John Roesler > Priority: Major > > While analyzing the algorithm of Streams's poll/process loop, I noticed the > following: > The algorithm of runOnce is: > {code} > loop0: > long poll for records (100ms) > loop1: > loop2: for BATCH_SIZE iterations: > process one record in each task that has data enqueued > adjust BATCH_SIZE > if loop2 processed any records, repeat loop 1 > else, break loop1 and repeat loop0 > {code} > There's potentially an unwanted interaction between "keep processing as long > as any record is processed" and forcing processing after `max.task.idle.ms`. > If there are two tasks, A and B, and A runs out of records on one input > before B, then B could keep the processing loop running, and hence prevent A > from getting any new records, until max.task.idle.ms expires, at which point > A will force processing on its other input partition. The intent of idling is > to at least give A a chance of getting more records on the empty input, but > under this situation, we'd never even check for more records before forcing > processing. > I'm thinking we should only enforce processing if there was a completed poll > since we noticed the task was missing inputs (otherwise, we may as well not > bother idling at all). -- This message was sent by Atlassian Jira (v8.3.4#803005)