Thanks Oleg, now it works. Earlier I used System.setProperty to enable
logging but I get only WARN level logging, I don't know if the
properties are case-sensitive because I set the same properties. Only
difference was that I wrote "debug" instead of "debug".
Generally, with the debug enabled I don't get the exception. Maybe it is
because, the crawler runs a little slower because of logging operation
and saving them to output?
In the logs I found that the connection is created 3715 times and in the
client I have the limit of 100 connections. Generally I often see
something like that:
[DEBUG] ThreadSafeClientConnManager - Released connection open but not
marked reusable.
[DEBUG] DefaultClientConnection - Connection shut down
[DEBUG] ConnPoolByRoute - Freeing connection
[HttpRoute[{}->http://www.mrw.interscience.wiley.com]][null]
[DEBUG] ConnPoolByRoute - Creating new connection
[HttpRoute[{}->http://www.mrw.interscience.wiley.com]]
[DEBUG] ThreadSafeClientConnManager -
ThreadSafeClientConnManager.getConnection:
HttpRoute[{}->http://www.mrw.interscience.wiley.com], timeout = 0
So the connection is not marked as reusable? Could it be a reason of my
BindException problems?
I execute connections very simultaneously, right now there may be about
600 threads in executors and every of them invokes 'execute method' on
httpclient object.
Tomek
On Tue, 2008-09-02 at 16:12 -0400, Tomek Maciejewski wrote:
Hi Oleg,
I am using Windows XP 32bit. I turned on context logging but I didn't
see anything interesting.
Are you seeing any DEBUG logs? Anyways, pass the following parameters to
the JRE upon the start up:
-Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.SimpleLog
-Dorg.apache.commons.logging.simplelog.log.org.apache.http.impl.conn=DEBUG
-Dorg.apache.commons.logging.simplelog.log.org.apache.http.headers=DEBUG
This should result in DEBUG logs getting printed to the stdout and
should give you enough details to help understand what may be causing
problems with the reuse of connections.
For instance, this is a good example of connection not being reused due
to connections being closed on the server side
[DEBUG] ThreadSafeClientConnManager -
ThreadSafeClientConnManager.getConnection:
HttpRoute[{}->http://www.yahoo.com], timeout = 0
[DEBUG] ConnPoolByRoute - Total connections kept alive: 0
[DEBUG] ConnPoolByRoute - Total issued connections: 0
[DEBUG] ConnPoolByRoute - Total allocated connection: 0 out of 200
[DEBUG] ConnPoolByRoute - No free connections
[HttpRoute[{}->http://www.yahoo.com]][null]
[DEBUG] ConnPoolByRoute - Available capacity: 20 out of 20
[HttpRoute[{}->http://www.yahoo.com]][null]
[DEBUG] ConnPoolByRoute - Creating new connection
[HttpRoute[{}->http://www.yahoo.com]]
[DEBUG] DefaultClientConnection - Connection closed
[DEBUG] headers - >> GET / HTTP/1.1
[DEBUG] headers - >> Host: www.yahoo.com
[DEBUG] headers - >> Connection: Keep-Alive
[DEBUG] headers - >> User-Agent: Apache-HttpClient/4.0
[DEBUG] headers - << HTTP/1.1 200 OK
[DEBUG] headers - << Date: Tue, 02 Sep 2008 20:54:27 GMT
[DEBUG] headers - << P3P: policyref="http://p3p.yahoo.com/w3c/p3p.xml";,
CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR
DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA
POL HEA PRE GOV"
[DEBUG] headers - << Cache-Control: private
[DEBUG] headers - << Vary: User-Agent
[DEBUG] headers - << X-XRDS-Location:
http://open.login.yahooapis.com/openid20/www.yahoo.com/xrds
[DEBUG] headers - << Last-Modified: Tue, 02 Sep 2008 20:50:37 GMT
[DEBUG] headers - << Accept-Ranges: bytes
[DEBUG] headers - << Content-Length: 9562
[DEBUG] headers - << Connection: close
[DEBUG] headers - << Content-Type: text/html; charset=utf-8
[DEBUG] ThreadSafeClientConnManager - Released connection open but not
marked reusable.
[DEBUG] DefaultClientConnection - Connection shut down
[DEBUG] ConnPoolByRoute - Freeing connection
[HttpRoute[{}->http://www.yahoo.com]][null]
[DEBUG] ConnPoolByRoute - Notifying no-one, there are no waiting threads
Hope this helps
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
