Hi Oleg,
Thank for your reply. I have got the latest code from repository, but
the problem didn't disappear. I saved the logs here:
http://pentium.hopto.org/~thinred/files/output.zip
At the end of the crawler execution I started to get BindException. I
would appreciate if you could see this logs, because actually I don't
understand what is a relation between not reused connections and this
bind exception. What is the nature of this problem, and why it tries to
bind to 'address already in use'? Is there any way to solve this problem?
Cheers,
Tomek
Oleg Kalnichevski wrote:
Tomek,
I improved some of the DEBUG log statements and reduced the amount of
noise in the logs. Feel free to get the latest code snapshot from SVN
trunk and retest with your application.
Cheers
Oleg
On Wed, 2008-09-03 at 12:27 +0200, Oleg Kalnichevski wrote:
On Tue, 2008-09-02 at 17:57 -0400, Tomek Maciejewski wrote:
Thanks Oleg, now it works. Earlier I used System.setProperty to enable
logging but I get only WARN level logging, I don't know if the
properties are case-sensitive because I set the same properties. Only
difference was that I wrote "debug" instead of "debug".
Generally, with the debug enabled I don't get the exception. Maybe it is
because, the crawler runs a little slower because of logging operation
and saving them to output?
Can be.
In the logs I found that the connection is created 3715 times and in the
client I have the limit of 100 connections. Generally I often see
something like that:
[DEBUG] ThreadSafeClientConnManager - Released connection open but not
marked reusable.
[DEBUG] DefaultClientConnection - Connection shut down
[DEBUG] ConnPoolByRoute - Freeing connection
[HttpRoute[{}->http://www.mrw.interscience.wiley.com]][null]
[DEBUG] ConnPoolByRoute - Creating new connection
[HttpRoute[{}->http://www.mrw.interscience.wiley.com]]
[DEBUG] ThreadSafeClientConnManager -
ThreadSafeClientConnManager.getConnection:
HttpRoute[{}->http://www.mrw.interscience.wiley.com], timeout = 0
So the connection is not marked as reusable? Could it be a reason of my
BindException problems?
Yes, this is precisely the reason. I cannot say why the connection got
marked as non-reusable in the first place without seeing the complete
context log.
Oleg
I execute connections very simultaneously, right now there may be about
600 threads in executors and every of them invokes 'execute method' on
httpclient object.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
