On Mon, Nov 27, 2017, at 22:30, Jan Filipiak wrote: > Hi Colin, thank you for this KIP, it can become a really useful thing. > > I just scanned through the discussion so far and wanted to start a > thread to make as decision about keeping the > cache with the Connection / Session or having some sort of UUID indN exed > global Map. > > Sorry if that has been settled already and I missed it. In this case > could anyone point me to the discussion?
Hi Jan, I don't think anyone has discussed the idea of tying the cache to an individual TCP session yet. I agree that since the cache is intended to be used only by a single follower or client, it's an interesting thing to think about. I guess the obvious disadvantage is that whenever your TCP session drops, you have to make a full fetch request rather than an incremental one. It's not clear to me how often this happens in practice -- it probably depends a lot on the quality of the network. From a code perspective, it might also be a bit difficult to access data associated with the Session from classes like KafkaApis (although we could refactor it to make this easier). It's also clear that even if we tie the cache to the session, we still have to have limits on the number of caches we're willing to create. And probably we should reserve some cache slots for each follower, so that clients don't take all of them. > > Id rather see a protocol in which the client is hinting the broker that, > he is going to use the feature instead of a client > realizing that the broker just offered the feature (regardless of > protocol version which should only indicate that the feature > would be usable). Hmm. I'm not sure what you mean by "hinting." I do think that the server should have the option of not accepting incremental requests from specific clients, in order to save memory space. > This seems to work better with a per > connection/session attached Metadata than with a Map and could allow for > easier client implementations. > It would also make Client-side code easier as there wouldn't be any > Cache-miss error Messages to handle. It is nice not to have to handle cache-miss responses, I agree. However, TCP sessions aren't exposed to most of our client-side code. For example, when the Producer creates a message and hands it off to the NetworkClient, the NC will transparently re-connect and re-send a message if the first send failed. The higher-level code will not be informed about whether the TCP session was re-established, whether an existing TCP session was used, and so on. So overall I would still lean towards not coupling this to the TCP session... best, Colin > > Thank you again for the KIP. And again, if this was clarified already > please drop me a hint where I could read about it. > > Best Jan > > > > > > On 21.11.2017 22:02, Colin McCabe wrote: > > Hi all, > > > > I created a KIP to improve the scalability and latency of FetchRequest: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-227%3A+Introduce+Incremental+FetchRequests+to+Increase+Partition+Scalability > > > > Please take a look. > > > > cheers, > > Colin >
