[ https://issues.apache.org/jira/browse/KAFKA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151827#comment-13151827 ]
Taylor Gautier commented on KAFKA-208: -------------------------------------- All that being said, if you'd like to combine KAFKA-48 and 208 together I am fine with that - as long as we address the network efficiency issue at the same time in KAFKA-48. > Efficient polling mechanism for many topics > ------------------------------------------- > > Key: KAFKA-208 > URL: https://issues.apache.org/jira/browse/KAFKA-208 > Project: Kafka > Issue Type: New Feature > Reporter: Taylor Gautier > > Currently, the way to poll many topics is to submit a request for each one in > turn, and read the responses. Especially if there are few messages delivered > on many topics, the network overhead to implement this scheme can far > outweigh the bandwidth of the actual messages delivered. > Effectively, for topics A, B, C the request/response scheme is the following: > -> Request A offset a > -> Request B offset b > -> Request C offset c > <- no messages > <- 1 message offset b > <- no messages > -> Request A offset a > -> Request B offset b' > -> Request C offset c > <- no messages > <- no messages > <- no messages > etc. > I propose a more efficient mechanism which works a bit like epoll in that the > client can register interest in a particular topic. There are many options > for the implementation, but the one I suggest goes like so: > -> Register interest A offset a > -> Register interest B offset b > -> Register interest C offset c > -> Next message (null) > <- messages for B (1 message) > -> Next message (topic B offset b') > <- no messages > -> Unregister Interest C > ... > It is possible to implement the "Next Message" request as either blocking or > non-blocking. I suggest that the request format include space for the > timeout, which if set to 0 will be a nonblocking response, and if set to > anything other than 0, will block for at most timeout ms. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira