[
https://issues.apache.org/jira/browse/KAFKA-20534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lianet Magrans updated KAFKA-20534:
-----------------------------------
Description:
At the moment the async consumer triggers a single AsyncPollEvent that triggers
2 main actions in the background : check/trigger reconciliation if needed, and
validating/resetting positions to fetch.
But this event has evolved in a way what the needs for the 2 things are
different, which I wonder could introduce delays to revocations under specific
cases (position validation) given that the inflightPollEvent may be reused (not
trigger the reconcilication check)
For reconciliations, we want to check on every poll iteration (HB may have been
received in the background with assignment changes, anytime), also we want to
wait until the check is done (mark partitions pending revocation and trigger
auto-commits) before the app thread carries on to fetch data (this check is
async, does not wait for any response)
As for the validate/reset positions, the needs are different, we need to
trigger the requests once (and once they complete, it will update positions as
needed and then the app thread can carry on).
It was with the needs of the reset positions in mind that the inflight event is
reused, but I imagine that may delay reconciliations if it happens that a HB
with an assignment change arrives when there are positions needing reset
(inflight poll reused while the reset happens, so won't check for pending
reconciliation)
One option that comes to mind is to have clear separate events.
A checkAndTrigger reconciliation one, which is straightforward: needs to be
triggered on every iteration of the poll loop, expected to run really quick in
the background (only flips vars and maybe generates a request). The app already
checks if it completed before returning buffered records.
The existing AsyncPollEvent, should remain as it is, just without the
reconciliation trigger part that will be decoupled (and ensured to happen
before, as the events will be genrated in that order in the app thread)
With this, we would ensure no delay to the reconciliation if they land combined
with partitions validation/reset.
--Update
As part of this will look into the AsyncPollEvent flow/lifecyle in the consumer
aiming to simplify if possible (we've had several issues fixed on top of each
other but over the same inflightPoll that seems to leave to loose ends. Eg.,
https://issues.apache.org/jira/browse/KAFKA-20780,
https://issues.apache.org/jira/browse/KAFKA-19259
was:
At the moment the async consumer triggers a single AsyncPollEvent that triggers
2 main actions in the background : check/trigger reconciliation if needed, and
validating/resetting positions to fetch.
But this event has evolved in a way what the needs for the 2 things are
different, which I wonder could introduce delays to revocations under specific
cases (position validation) given that the inflightPollEvent may be reused (not
trigger the reconcilication check)
For reconciliations, we want to check on every poll iteration (HB may have been
received in the background with assignment changes, anytime), also we want to
wait until the check is done (mark partitions pending revocation and trigger
auto-commits) before the app thread carries on to fetch data (this check is
async, does not wait for any response)
As for the validate/reset positions, the needs are different, we need to
trigger the requests once (and once they complete, it will update positions as
needed and then the app thread can carry on).
It was with the needs of the reset positions in mind that the inflight event is
reused, but I imagine that may delay reconciliations if it happens that a HB
with an assignment change arrives when there are positions needing reset
(inflight poll reused while the reset happens, so won't check for pending
reconciliation)
One option that comes to mind is to have clear separate events.
A checkAndTrigger reconciliation one, which is straightforward: needs to be
triggered on every iteration of the poll loop, expected to run really quick in
the background (only flips vars and maybe generates a request). The app already
checks if it completed before returning buffered records.
The existing AsyncPollEvent, should remain as it is, just without the
reconciliation trigger part that will be decoupled (and ensured to happen
before, as the events will be genrated in that order in the app thread)
With this, we would ensure no delay to the reconciliation if they land combined
with partitions validation/reset.
> Consider decoupling AsyncPollEvent reconciliation trigger and position
> validation/reset
> ---------------------------------------------------------------------------------------
>
> Key: KAFKA-20534
> URL: https://issues.apache.org/jira/browse/KAFKA-20534
> Project: Kafka
> Issue Type: Task
> Components: clients, consumer
> Reporter: Lianet Magrans
> Assignee: Lianet Magrans
> Priority: Major
>
> At the moment the async consumer triggers a single AsyncPollEvent that
> triggers 2 main actions in the background : check/trigger reconciliation if
> needed, and validating/resetting positions to fetch.
> But this event has evolved in a way what the needs for the 2 things are
> different, which I wonder could introduce delays to revocations under
> specific cases (position validation) given that the inflightPollEvent may be
> reused (not trigger the reconcilication check)
> For reconciliations, we want to check on every poll iteration (HB may have
> been received in the background with assignment changes, anytime), also we
> want to wait until the check is done (mark partitions pending revocation and
> trigger auto-commits) before the app thread carries on to fetch data (this
> check is async, does not wait for any response)
> As for the validate/reset positions, the needs are different, we need to
> trigger the requests once (and once they complete, it will update positions
> as needed and then the app thread can carry on).
> It was with the needs of the reset positions in mind that the inflight event
> is reused, but I imagine that may delay reconciliations if it happens that a
> HB with an assignment change arrives when there are positions needing reset
> (inflight poll reused while the reset happens, so won't check for pending
> reconciliation)
> One option that comes to mind is to have clear separate events.
> A checkAndTrigger reconciliation one, which is straightforward: needs to be
> triggered on every iteration of the poll loop, expected to run really quick
> in the background (only flips vars and maybe generates a request). The app
> already checks if it completed before returning buffered records.
> The existing AsyncPollEvent, should remain as it is, just without the
> reconciliation trigger part that will be decoupled (and ensured to happen
> before, as the events will be genrated in that order in the app thread)
> With this, we would ensure no delay to the reconciliation if they land
> combined with partitions validation/reset.
>
> --Update
> As part of this will look into the AsyncPollEvent flow/lifecyle in the
> consumer aiming to simplify if possible (we've had several issues fixed on
> top of each other but over the same inflightPoll that seems to leave to loose
> ends. Eg., https://issues.apache.org/jira/browse/KAFKA-20780,
> https://issues.apache.org/jira/browse/KAFKA-19259
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
