Philip Nee created KAFKA-15173:
----------------------------------
Summary: ApplicationEventQueue and BackgroundEventQueue should be
bounded
Key: KAFKA-15173
URL: https://issues.apache.org/jira/browse/KAFKA-15173
Project: Kafka
Issue Type: Task
Components: consumer
Reporter: Philip Nee
Assignee: Philip Nee
The async consumer uses ApplicationEventQueue and BackgroundEventQueue to
facilitate message passing between the application thread and the background
thread. The current implementation is boundless, which can potentially cause
OOM and other performance-related issues.
I think the queues need a finite bound, and we need to decide how to handle the
situation when the bound is reached. In particular, I would like to answer
these questions:
# What should the upper limit be for both queues: Can this be a configurable,
memory-based bound? Or just an arbitrary number of events as the bound.
# What should happen when the application event queue is filled up? It seems
like we should introduce a new exception type and notify the user that the
consumer is full.
# What should happen when the background event queue is filled up? This seems
less likely to happen, but I imagine it could happen when the user stops
polling the consumer, causing the queue to be filled.
# Is it necessary to introduce a public configuration for the queue? I think
initially we would select an arbitrary constant number and see the community
feedback to make a forward plan accordingly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
