The second thing has to do with how Keep-alive connections behave. This is a
multi-threaded app, using MultiThreadedHttpConnectionManager. It works great,
however I don't get much benefit of the shared Connections, because I'm not
connecting to the same site more than once, generally. That's OK, the problem
I run into is that after running for not very long, I suddenly start getting
everything timing out. It's hard to really pinpoint the timing, giving all the
activity, and no thread identifiers in the log messages, but I think what is
happening is that the system is simply running out of file handles or
system-level connections. A quick "netstat -n" shows a whole bunch of open,
TIME_WAIT, and other connections. It seems that the Connection Manager is
keeping them around for re-use, and following HTTP/1.1. One fix was to send
"Connection: close" as a RequestHeader, which really fixed things up, but now
I am running into sites that are not responding, and not timing out. The log
traces into ReadRawLine() and just sits there. I am still tracking this down,
I just wonder if anyone else has seen this also?
The problem, as other have been hinting at, is that connections never get destroyed. By default, a max of two connections are created per HostConfiguration. If you are never connecting to the same host more than once this is a bit of a problem. The MultiThreadedHttpConnectionManager was designed to reuse connections for a small number of hosts. So as you have guessed the connection manager will continue to create connections to new hosts and will never reclaim the old ones. We have a couple of options for fixing this, here are a few:
- create a cap on the max number of connections that can be created. once reached unused connections will be reclaimed
- implement some kind of connection cache/timeout. only connections that have been used in the last X milliseconds will be kept
- implement a new type of connection manager that better fits this kind of process. in particular it would focus less on connections per host, but more on total consecutive connections. in general we could introduce a whole new set of connection managers they are optimized for different use scenarios
These are just a few ideas. What do others think?
Mike
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
