[
https://issues.apache.org/jira/browse/KAFKA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151907#comment-13151907
]
Taylor Gautier commented on KAFKA-208:
--------------------------------------
Hmm,
Your version seems that it would be inefficient for large # of topics with a
(relative to the overall #) small # of topics that receive messages. The
version I posted is much better in this regard, the amount of data actually
transmitted is relative not to the # of topics requested, but to the # that
receive traffic.
I don't think my version is very hard to implement at all, in fact it would be
some very small changes to the current code I have running in Node (which
admittedly is event based and a natural fit for async behavior)
I don't personally have any current need for min bytes nor does it seem that I
would need it in upcoming use cases.
> Efficient polling mechanism for many topics
> -------------------------------------------
>
> Key: KAFKA-208
> URL: https://issues.apache.org/jira/browse/KAFKA-208
> Project: Kafka
> Issue Type: New Feature
> Reporter: Taylor Gautier
>
> Currently, the way to poll many topics is to submit a request for each one in
> turn, and read the responses. Especially if there are few messages delivered
> on many topics, the network overhead to implement this scheme can far
> outweigh the bandwidth of the actual messages delivered.
> Effectively, for topics A, B, C the request/response scheme is the following:
> -> Request A offset a
> -> Request B offset b
> -> Request C offset c
> <- no messages
> <- 1 message offset b
> <- no messages
> -> Request A offset a
> -> Request B offset b'
> -> Request C offset c
> <- no messages
> <- no messages
> <- no messages
> etc.
> I propose a more efficient mechanism which works a bit like epoll in that the
> client can register interest in a particular topic. There are many options
> for the implementation, but the one I suggest goes like so:
> -> Register interest A offset a
> -> Register interest B offset b
> -> Register interest C offset c
> -> Next message (null)
> <- messages for B (1 message)
> -> Next message (topic B offset b')
> <- no messages
> -> Unregister Interest C
> ...
> It is possible to implement the "Next Message" request as either blocking or
> non-blocking. I suggest that the request format include space for the
> timeout, which if set to 0 will be a nonblocking response, and if set to
> anything other than 0, will block for at most timeout ms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
