[
https://issues.apache.org/jira/browse/KAFKA-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152134#comment-13152134
]
Taylor Gautier commented on KAFKA-208:
--------------------------------------
That's right :) And yes, I was assuming that a single socket would retain an
interest set and de-allocate it on close.
However, it might be easier to use my other proposal most recently stated. It
just builds on the existing polling mechanism but allows for requests to be
blocking.
To implement it, responses have to be able to come back out of order compared
to the requests order, and that requires that the response must contain either
a request id, or the topic name (but this is pretty much required no matter
what we do).
In this case, the client simply re-polls for any responses that are received
(or any new topics, if the client so desires)
So to re-iterate and add a timeout parameter to illustrate how that also works
(where C and S are client and server):
C->S fetch messages topic A offset a timeout 500
C->S fetch messages topic B offset b timeout 500
C->S request messages topic C offset c timeout 500
(250 ms passes, server receives 3 messages for topic B)
C<-S topic B messages size b' (3) messages
C->S request messages topic B offset (b+b') timeout 500
(200 ms passes)
C->S request messages topic D offset d timeout 500
(50 ms passes, no messages received for A or C)
C<-S topic A messages size 0 (0) messages
C<-S topic C messages size 0 (0) messages
C->S fetch messages topic A offset a timeout 500
C->S request messages topic C offset c timeout 500
... and so on.
Both of these solutions are pretty similar to one another, in the above case
there is still pretty much an "interest set" however it's implementation might
differ in that it might be considered as outstanding requests.
What I like about the above solution is that there is no magic going on at the
server side. In the registered interests implementation, the server also has
to keep the current offset of the registered interest set to continue advancing
it on the client's behalf, and (other than the automatic storage of the offset
in ZK by the high level consumer) this idea goes against the design principles
of Kafka as I understand them (move the tracking of offsets to the client to
make Kafka simple).
> Efficient polling mechanism for many topics
> -------------------------------------------
>
> Key: KAFKA-208
> URL: https://issues.apache.org/jira/browse/KAFKA-208
> Project: Kafka
> Issue Type: New Feature
> Reporter: Taylor Gautier
>
> Currently, the way to poll many topics is to submit a request for each one in
> turn, and read the responses. Especially if there are few messages delivered
> on many topics, the network overhead to implement this scheme can far
> outweigh the bandwidth of the actual messages delivered.
> Effectively, for topics A, B, C the request/response scheme is the following:
> -> Request A offset a
> -> Request B offset b
> -> Request C offset c
> <- no messages
> <- 1 message offset b
> <- no messages
> -> Request A offset a
> -> Request B offset b'
> -> Request C offset c
> <- no messages
> <- no messages
> <- no messages
> etc.
> I propose a more efficient mechanism which works a bit like epoll in that the
> client can register interest in a particular topic. There are many options
> for the implementation, but the one I suggest goes like so:
> -> Register interest A offset a
> -> Register interest B offset b
> -> Register interest C offset c
> -> Next message (null)
> <- messages for B (1 message)
> -> Next message (topic B offset b')
> <- no messages
> -> Unregister Interest C
> ...
> It is possible to implement the "Next Message" request as either blocking or
> non-blocking. I suggest that the request format include space for the
> timeout, which if set to 0 will be a nonblocking response, and if set to
> anything other than 0, will block for at most timeout ms.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
