charlie w wrote: > The answer is that http.content.limit is indeed broken in the > protocol-httpclient plugin, though it doesn't really look like it's > entirely Nutch's fault. > > The org.apache.nutch.protocol.httpclient.HttpResponse class is doing the > right thing in trying to abort the GET at the content limit, but when it > calls close on the input stream of the request at HttpResponse.java:120, > the org.apache.commons.httpclient.AutoCloseInputStream class goes off and > tries to read the entire response anyway. > > I found that if get.abort() is called when the content goes over limit, the > request is terminated and Nutch is able to do the right thing. > > The default protocol-http plugin does not use the apache commons httpclient > stuff, and works correctly.
Could you please create a JIRA issue, so that your analysis and the possible fix is recorded? Thanks! -- Best regards, Andrzej Bialecki <>< ___. ___ ___ ___ _ _ __________________________________ [__ || __|__/|__||\/| Information Retrieval, Semantic Web ___|||__|| \| || | Embedded Unix, System Integration http://www.sigram.com Contact: info at sigram dot com ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
