[
https://issues.apache.org/jira/browse/KAFKA-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14066461#comment-14066461
]
Jay Kreps commented on KAFKA-1282:
----------------------------------
The goal is just to reduce server connection count. In our environment there
might be a single Kafka producer in each process we run publishing to a small
Kafka cluster (say ~20 servers). However there are tens of thousands of client
processes. Connections can end up going unused when leadership migrates and we
should eventually close these out rather than retaining them indefinitely.
As you say it is not critical as the server seems to do a good job of dealing
with high connection counts, but it seems like a good thing to do.
I agree that doing this on the server might be better. This does mean it is
possible that the server will attempt to close the socket while the client is
attempting to send something. But if the timeout is 10 mins, it is unlikely
that this will happen often (i.e. if nothing was sent in the last 10 mins, it
will not likely happen in the 0.5 ms it takes to do the close). The advantage
of doing it on the server is that it will work for all clients.
This change would be in core/.../kafka/network/SocketServer.scala.
The only gotcha is that we likely need to avoid iterating over all connections
to avoid latency impact (there could be 100k connections). One way to do this
would be to use java.util.LinkedHashMap to implement an LRU hash map of the
SelectionKeys, and access this every time the selection key comes up in a
select operation. (There are a ton of details in LinkedHashMap--needs to be
"access order", etc). Then every 5-10 select loop iterations we would iterate
the map expiring connections until we come to a connection that doesn't need
expiring, then stop.
> Disconnect idle socket connection in Selector
> ---------------------------------------------
>
> Key: KAFKA-1282
> URL: https://issues.apache.org/jira/browse/KAFKA-1282
> Project: Kafka
> Issue Type: Bug
> Components: producer
> Affects Versions: 0.8.2
> Reporter: Jun Rao
> Labels: newbie++
> Fix For: 0.9.0
>
>
> To reduce # socket connections, it would be useful for the new producer to
> close socket connections that are idle. We can introduce a new producer
> config for the idle time.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
