THX
-g-
Michael Becke wrote:
Yea, "good news" :)
So now we know that we're losing connections somewhere, and they are not being garbage collected. There are a bunch of things we can try. Here are a few ideas:
- enable debug logging for the MultiThreadedHttpConnectionManager. You will probably also want to print out the thread name with the log. This will require Log4j or JRE 1.4 logging.
- reduce the number of threads executing methods as well as the max number of connections(MultiThreadedHttpConnectionManager.setMaxTotalConnections( )) . This should cause the error to occur more quickly.
- if possible, try to come up with something standalone that reproduces the problem. If I can reproduce the error I will be better able to debug the problem.
- run the application in debug mode so that when things get locked up we can see where each of the threads are.
Also, any more detail you can provide about your application and environment would be helpful. Please let me know if you have any ideas or questions.
Mike
On Monday, August 25, 2003, at 01:59 PM, Leo Galambos wrote:
Hi Mike,
"good news":
c-0.log:org.apache.commons.httpclient.HttpException: Timeout waiting for connection
c-0.log- at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGe tC onnection(MultiThreadedHttpConnectionManager.java:314)
c-0.log- at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getC on nection(MultiThreadedHttpConnectionManager.java:241)
c-0.log- at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.jav a: 609)
c-0.log- at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.jav a: 500)
c-0.log- at org.egothor.robot.Network.get(Network.java:109)
c-0.log- at org.egothor.robot.Capek$Fetcher.run(Capek.java:186)
What next?
THX
Leo
Michael Becke wrote:
Hi Leo,
These 20 CLOSE_WAIT sockets are from the 20 connections in the pool held by MultiThreadedHttpConnectionManager. When connections are released they are not necessarily closed.
My guess is that connections are being lost somewhere. If all 20 connections are checked out and not released the 21st request for a connection will block until one is available. You can verify this by setting httpConnectionFactoryTimeout on HttpClient. This timeout is how long HttpClient will wait when getting a connection from the connection manager. Setting this timeout to something big is probably best, that way you won't run into it by accident.
If connections are getting lost somewhere and httpConnectionFactoryTimeout is set you will start getting HttpExceptions with a message of "Timeout waiting for connection" on every method execute. Please give this a shot to ensure that the problem is with lost connections. Once verified we can start looking for a cause.
Also, in you example Network.java class I noticed the following lines:
while (r.status == -1 && attempt++ < RETRIES) {
try {
r.status = client.executeMethod(method);
r.msg = null;
} catch (Throwable x) {
method.recycle();
r.msg = url + ": " + x.getMessage();
if (logger.isLoggable(Level.FINE)) {
logger.log(Level.FINE, attempt + " of " + url, x);
}
}
}
HttpMethod.recycle() resets all values in the method, including the URL and request headers. Calling recycle in the exception case and not resetting the various values will make the retry behave quite differently. My guess is that this is not what you were trying to >> do.
Mike
On Saturday, August 23, 2003, at 01:07 PM, Leo Galambos wrote:
Hi.
I am using httpclient (HC) in a webcrawler. After 6 hours of run, HC stops working and I think, it is locked by some lock of a critical section in HC. The "problematic" code, I use, is here: http://www.egothor.org/temp/Network.java
When 20 threads are used, I have this in `netstat -an` after stopping the robot from its console (note: the number of dead connection is really *20*):
tcp 23786 0 censored:60083 censored2:80 CLOSE_WAIT
tcp 24179 0 censored:60080 censored2:80 CLOSE_WAIT
tcp 24143 0 censored:60092 censored2:80 CLOSE_WAIT
tcp 24164 0 censored:60091 censored2:80 CLOSE_WAIT
tcp 24169 0 censored:60090 censored2:80 CLOSE_WAIT
tcp 24189 0 censored:60089 censored2:80 CLOSE_WAIT
tcp 24188 0 censored:60071 censored2:80 CLOSE_WAIT
tcp 11697 0 censored:60066 censored2:80 CLOSE_WAIT
tcp 24152 0 censored:60078 censored2:80 CLOSE_WAIT
tcp 24180 0 censored:60076 censored2:80 CLOSE_WAIT
tcp 24208 0 censored:60074 censored2:80 CLOSE_WAIT
tcp 24154 0 censored:60073 censored2:80 CLOSE_WAIT
tcp 24150 0 censored:60044 censored2:80 CLOSE_WAIT
tcp 24150 0 censored:60041 censored2:80 CLOSE_WAIT
tcp 24194 0 censored:60101 censored2:80 CLOSE_WAIT
tcp 24206 0 censored:60099 censored2:80 CLOSE_WAIT
tcp 24156 0 censored:60097 censored2:80 CLOSE_WAIT
tcp 23643 0 censored:60096 censored2:80 CLOSE_WAIT
tcp 23609 0 censored:60111 censored2:80 CLOSE_WAIT
tcp 23961 0 censored:60110 censored2:80 CLOSE_WAIT
Other connections are not allocated and the robot is waiting for something (O/S does not report that the respective JAVA processes > run).
Any thoughts?
Thank you
Leo
------------------------------------------------------------------- --
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-------------------------------------------------------------------- -
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
