[
https://issues.apache.org/jira/browse/KAFKA-48?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200155#comment-13200155
]
Jay Kreps commented on KAFKA-48:
--------------------------------
Hey Taylor, here are the nitty gritty details:
- When a fetch request comes in we immediately check if we have sufficient data
to satisfy it
- if so we respond immediately
- If not we add a "watch" on the topics that the fetch is for, and add it to
a delay queue to expire it after the given timeout
- There is a background thread that checks the delay queue for expired
requests and responds to them with whatever data is available
- When a produce request comes in we update the watchers for all the topics it
produces to, and increment their byte count. Any requests that have been
satisfied by this produce, are then executed and responses are sent.
So one of the earlier questions was how to support polling on a very large
number of topics AND wants very low latency, I think as you described it would
be possible to implement this by simply multiplexing the requests on the single
socket and letting the server respond to these as possible.
> Implement optional "long poll" support in fetch request
> -------------------------------------------------------
>
> Key: KAFKA-48
> URL: https://issues.apache.org/jira/browse/KAFKA-48
> Project: Kafka
> Issue Type: Bug
> Reporter: Alan Cabrera
> Assignee: Jay Kreps
> Attachments: KAFKA-48.patch
>
>
> Currently, the fetch request is non-blocking. If there is nothing on the
> broker for the consumer to retrieve, the broker simply returns an empty set
> to the consumer. This can be inefficient, if you want to ensure low-latency
> because you keep polling over and over. We should make a blocking version of
> the fetch request so that the fetch request is not returned until the broker
> has at least one message for the fetcher or some timeout passes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
