[ https://issues.apache.org/jira/browse/KAFKA-48?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13140231#comment-13140231 ]
Taylor Gautier commented on KAFKA-48: ------------------------------------- Hi - please keep in mind the use case where a consumer is interested in more than one topic. This feature if implemented only for one topic will not be useful for this use case - assuming it's infeasible to open multiple tcp connections. The first proposal I have is to allow the request to contain a list of topics. However, upon consideration, this would require the response to also be adjusted such that it would contain the name of the topic, otherwise it would be next to impossible to ascertain which topic the response corresponds to - well it could be done such that the response is returned in the same way as the request was requested, and for topics with no messages, an empty response is given, but this seems pretty bad from a network bandwidth standpoint. So my final proposal would be to introduce an epoll like request/response. The consumer would submit a request with a list of interested topics, and the response would be a topic and # of messages available on that topic when the topic(s) have messages. The advantage to this solution is that it would be entirely backward compatible, since you would simply introduce a new request/response pair and it would also allow the consumer to decide which topics to poll (or pull) from first, so that it could prioritize, if it wanted. Finally, I like the idea of allowing the consumer to specify a min # of messages required to trigger the poll, you might want to copy the pattern you already setup for log flushing, e.g. max time and/or min # of messages. So the request might look like: list-of : topic-name:min msgs:max time and the response might be: list-of : topic-name:# msgs available > Implement optional "long poll" support in fetch request > ------------------------------------------------------- > > Key: KAFKA-48 > URL: https://issues.apache.org/jira/browse/KAFKA-48 > Project: Kafka > Issue Type: Bug > Reporter: Alan Cabrera > Assignee: Jay Kreps > Attachments: KAFKA-48-socket-server-refactor-draft.patch > > > Currently, the fetch request is non-blocking. If there is nothing on the > broker for the consumer to retrieve, the broker simply returns an empty set > to the consumer. This can be inefficient, if you want to ensure low-latency > because you keep polling over and over. We should make a blocking version of > the fetch request so that the fetch request is not returned until the broker > has at least one message for the fetcher or some timeout passes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira