Hi,
I set the http.content.limit to -1 to not truncate any data being
fetched, however if the fetched data was compressed (http response
header Content-Encoding: gzip) then Nutch was not able to uncompress
this data. If i set http.content.limit to its default value of 65536,
Nutch did not have any problem. I debugged nutch in eclipse and I
think the problem is in GZIPUtils.java in the loop:
if ((written + size) > sizeLimit) {
outStream.write(buf, 0, sizeLimit - written);
break;
}
It should truncate the data only if sizeLimit >=0, so the above loop
should read:
if ((written + size) > sizeLimit && sizeLimit >=0) {
outStream.write(buf, 0, sizeLimit - written);
break;
}
Has anyone seen this before and is this solution correct?
Thanks,
Meghna
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
