Hi,
I am trying to write a multithreaded crawler. I would like to execute
http requests simultaneously. So I created a simple HttpClient class:
public class MultiThreadedHttpClient extends DefaultHttpClient {
public MultiThreadedHttpClient() {
super(null, null);
}
protected ClientConnectionManager createClientConnectionManager() {
getParams().setParameter(ClientPNames.CONNECTION_MANAGER_FACTORY,
new ThreadedFactory());
HttpConnectionManagerParams.setMaxConnectionsPerRoute(getParams(),
new ConnPerRouteBean(100));
HttpConnectionManagerParams.setMaxTotalConnections(getParams(), 100);
return super.createClientConnectionManager();
}
protected HttpRequestRetryHandler createHttpRequestRetryHandler() {
return new DefaultHttpRequestRetryHandler(3, true);
}
public class ThreadedFactory implements ClientConnectionManagerFactory {
public ClientConnectionManager newInstance(HttpParams params,
SchemeRegistry schemeRegistry) {
return new ThreadSafeClientConnManager(params,
schemeRegistry);
}
}
}
I use a one instance of this client. The instance is marked as static
and final. But after executing some number of requests I start to get
a java.net.BindException with message "Address already in use" almost
every time I want to execute request. The code responsible for
executing requests is below (HTTP_CLIENT is a instance of the class I
mentioned above):
public InputStream getSiteAsStream(String siteAddress) {
HttpEntity entity = null;
ByteArrayOutputStream convertedData = null;
try {
HttpResponse response = HTTP_CLIENT.execute(new HttpGet(siteAddress));
entity = response.getEntity();
convertedData = new ByteArrayOutputStream();
tidy.parse(entity.getContent(), convertedData); // it's some kind
of http content cleaner (JTidy)
return new ByteArrayInputStream(convertedData.toByteArray());
} catch (URISyntaxException e) {
...
} catch (IOException e) {
...
} catch (HttpException e) {
...
} finally {
try {
if (entity != null) {
entity.consumeContent();
}
} catch (IOException e) {
....
}
try {
if (convertedData != null) {
convertedData.close();
}
} catch (IOException e) {
...
}
}
}
Could you tell me what am I doing wrong? I thought that maybe I don't
release the connection and I don't unbind sockets, but in finally
section I use consumeContent() method which should do that.
Cheers,
Tomek
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
