I am migrating from Nutch version 7 to version 9 and I am not able to get
redirects working property.
The crawl stops with very few URLs and it looks like it is not following any
redirects. I noticed the warning message "Redirect requested but
followRedirects is disabled" - but reading a bit of the newsgroup I think I can
ignore this message.
configuration:
Nutch version 0.9
http.redirect.max set to 5 (I have tried with 0 also with same result)
I checked the crawldb and I see lots of entries like the following:
Status: 4 (db_redir_temp)
Fetch time: Sat Jul 21 13:51:36 PDT 2007
Modified time: Wed Dec 31 16:00:00 PST 1969
Retries since fetch: 0
Retry interval: 30.0 days
Score: 0.02375479
Signature: null
Metadata: _pst_:temp_moved(13), lastModified=0: http://www.mysite.com/about
The segments also have these URLs, listed as
CrawlDatum::
Version: 5
Status: 35 (fetch_redir_temp)
Fetch time: Thu Jun 21 13:51:36 PDT 2007
Modified time: Wed Dec 31 16:00:00 PST 1969
Retries since fetch: 0
Retry interval: 30.0 days
Score: 0.022222223
Signature: null
Metadata: _ngt_:1182459087703 _pst_:temp_moved(13), lastModified=0:
http://www.mysite.com/about/
CrawlDatum::
Version: 5
Status: 1 (db_unfetched)
Fetch time: Thu Jun 21 13:51:24 PDT 2007
Modified time: Wed Dec 31 16:00:00 PST 1969
Retries since fetch: 0
Retry interval: 30.0 days
Score: 0.022222223
Signature: null
Metadata: _ngt_:1182459087703
CrawlDatum::
Version: 5
Status: 67 (linked)
Fetch time: Thu Jun 21 13:51:54 PDT 2007
Modified time: Wed Dec 31 16:00:00 PST 1969
Retries since fetch: 0
Retry interval: 30.0 days
Score: 7.6628354E-4
Signature: null
Metadata: null
Could someone let me know if I need any other new configuration/setup in the
latest version
---------------------------------
Need a vacation? Get great deals to amazing places on Yahoo! Travel. -------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
