Lianet Magrans created KAFKA-20534:
--------------------------------------
Summary: Consider decoupling AsyncPollEvent reconciliation trigger
and position init/reset
Key: KAFKA-20534
URL: https://issues.apache.org/jira/browse/KAFKA-20534
Project: Kafka
Issue Type: Task
Components: clients, consumer
Reporter: Lianet Magrans
Assignee: Lianet Magrans
At the moment the async consumer triggers a single AsyncPollEvent that triggers
2 main actions in the background : check/trigger reconciliation if needed, and
updating position to fetch.
But this event has evolved in a way what the needs for the 2 things are
different.
For reconciliations, we want to check on every poll iteration (HB may have been
received in the background with assignment changes, anytime), also we want to
wait until the check is done (mark partitions pending revocation and trigger
auto-commits) before the app thread carries on to fetch data (this check is
async, does not wait for any response)
As for the init/reset positions, the needs are different, we need to trigger
the requests once (and once they complete, it will update positions as needed
and then the app thread can carry on).
It was with the needs of the reset positions in mind that the inflight event is
reused, but I imagine that may delay reconciliations if it happens that a HB
with an assignment change arrives when there are positions needing reset
(inflight poll reused while the reset happens, so won't check for pending
reconciliation)
One option that comes to mind is to have clear separate events.
A checkAndTrigger reconciliation one, which is straightforward: needs to be
triggered on every iteration of the poll loop, expected to run really quick in
the background (only flips vars and maybe generates a request). The app already
checks if it completed before returning buffered records.
The existing AsyncPollEvent, should remain as it is, just without the
reconciliation trigger part that will be decoupled (and ensured to happen
before, as the events will be genrated in that order in the app thread)
With this, we would ensure no delay to the reconciliation if they land combined
with partitions reset/init.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
