Created as NUTCH-481.
On 5/11/07, Andrzej Bialecki <[EMAIL PROTECTED]> wrote:
charlie w wrote:
> The answer is that http.content.limit is indeed broken in the
> protocol-httpclient plugin, though it doesn't really look like it's
> entirely Nutch's fault.
>
> The org.apache.nutch.protocol.httpclient.HttpResponse class is doing the
> right thing in trying to abort the GET at the content limit, but when it
> calls close on the input stream of the request at HttpResponse.java:120,
> the org.apache.commons.httpclient.AutoCloseInputStream class goes off
and
> tries to read the entire response anyway.
>
> I found that if get.abort() is called when the content goes over limit,
the
> request is terminated and Nutch is able to do the right thing.
>
> The default protocol-http plugin does not use the apache commons
httpclient
> stuff, and works correctly.
Could you please create a JIRA issue, so that your analysis and the
possible fix is recorded? Thanks!
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
