Efficient polling mechanism for many topics
-------------------------------------------
Key: KAFKA-208
URL: https://issues.apache.org/jira/browse/KAFKA-208
Project: Kafka
Issue Type: New Feature
Reporter: Taylor Gautier
Currently, the way to poll many topics is to submit a request for each one in
turn, and read the responses. Especially if there are few messages delivered
on the many topics, the network overhead to implement this scheme can far
outweigh the bandwidth of the actual messages delivered.
Effectively, for topics A, B, C the request/response scheme is the following:
-> Request A offset a
-> Request B offset b
-> Request C offset c
<- no messages
<- 1 message offset b
<- no messages
-> Request A offset a
-> Request B offset b'
-> Request C offset c
<- no messages
<- no messages
<- no messages
etc.
I propose a more efficient mechanism which works a bit like epoll in that the
client can register interest in a particular topic. There are many options for
the implementation, but the one I suggest goes like so:
-> Register interest A offset a
-> Register interest B offset b
-> Register interest C offset c
-> Next message (null)
<- messages for B (1 message)
-> Next message (topic B offset b')
<- no messages
-> Unregister Interest C
...
It is possible to implement the "Next Message" request as either blocking or
non-blocking. I suggest that the request format include space for the timeout,
which if set to 0 will be a nonblocking response, and if set to anything other
than 0, will block for at most timeout ms. If a message is ready it would act
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
