[ http://issues.apache.org/jira/browse/NUTCH-289?page=all ]
Stefan Groschupf updated NUTCH-289:
-----------------------------------
Attachment: ipInCrawlDatumDraftV5.patch
Release Candidate 1 of this patch.
This patch contains:
+ add IP Address to CrawlDatum Version 5 (as byte[4])
+ a IpAddress Resolver (map runnable) tool to lookup the IP's multithreaded
+ add a property to define if the IpAddress Resolver should be started as a
part of the crawlDb update tool to update the parseoutput folder (contains
CrawlDatum Status Linked) of a segment before updating the crawlDb.
+ using cached IP during Generation
Please review this patch and give me any improvement suggestion, I think this
is a very important issue, since it helps to do _real_ whole web crawls and not
end up in a honey pot after some fetch iterations.
Also if you like please vote for this issue. :-) Thanks.
> CrawlDatum should store IP address
> ----------------------------------
>
> Key: NUTCH-289
> URL: http://issues.apache.org/jira/browse/NUTCH-289
> Project: Nutch
> Type: Bug
> Components: fetcher
> Versions: 0.8-dev
> Reporter: Doug Cutting
> Attachments: ipInCrawlDatumDraftV1.patch, ipInCrawlDatumDraftV4.patch,
> ipInCrawlDatumDraftV5.patch
>
> If the CrawlDatum stored the IP address of the host of it's URL, then one
> could:
> - partition fetch lists on the basis of IP address, for better politeness;
> - truncate pages to fetch per IP address, rather than just hostname. This
> would be a good way to limit the impact of domain spammers.
> The IP addresses could be resolved when a CrawlDatum is first created for a
> new outlink, or perhaps during CrawlDB update.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
