Prateek Maheshwari created SAMZA-1644:
-----------------------------------------
Summary: Blocking poll on KafkaConsumer can miss the last batch of
messages
Key: SAMZA-1644
URL: https://issues.apache.org/jira/browse/SAMZA-1644
Project: Samza
Issue Type: Bug
Reporter: Prateek Maheshwari
BrokerProxy is calling setIsAtHighWatermark before adding the new messages to
the BEM (i.e., as the first step in moveMessagesToTheirQueue). This can cause
the caller of blocking poll to miss the latest batch of messages in the current
poll, since the poll logic is:
while (envelope == null && !isAtHead(systemStreamPartition)) {
metrics.incBlockingPoll(systemStreamPartition);
envelope = queue.poll(1000, TimeUnit.MILLISECONDS);
}
We should move the setIsAtHead call to after the messages have been added to
the BEM.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
