Greetings I've been using nutch to crawl and index a rather large and complex website. I discovered that some of the linked pdf files didn't come up when searching for keywords that should've hit something.
I did some digging and found that it's due to the URLs to the pdf files. Some of them contain whitespaces and even characters like "ó","ý","æ","þ" or "ö", all of them not being encoded properly, somehow causing nutch, either with http or httpclient, to fail fetching the document. Do you know if there's a solution to this problem at nutch's end or if I need to take measures myself either by "fixing" this in nutch or venture into getting people to properly encode every url that is linked to on the web? Best regards, Árni Hermann Reynisson [EMAIL PROTECTED] ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
