[ http://issues.apache.org/jira/browse/NUTCH-173?page=all ]
Stefan Neufeind updated NUTCH-173:
----------------------------------
Attachment: patch08-new.patch
Here is the 08-patch, corrected to work against nightly from 2006-05-20.
Also fromHost is now only generated if really needed and nutch-default.xml is
patched as well. By the way: Where should a property for "crawl" be located in
the config-file? In the "fetcher"-section? In that case please somebody move it
up/down or rename the property before including it in the dev-tree.
But could somebody please review it quickly? I'm not sure it's 100% correct.
Still investigating on my side ...
> PerHost Crawling Policy ( crawl.ignore.external.links )
> -------------------------------------------------------
>
> Key: NUTCH-173
> URL: http://issues.apache.org/jira/browse/NUTCH-173
> Project: Nutch
> Type: New Feature
> Components: fetcher
> Versions: 0.7.1, 0.7, 0.8-dev
> Reporter: Philippe EUGENE
> Priority: Minor
> Attachments: patch.txt, patch08-new.patch, patch08.txt
>
> There is two major way of crawl in Nutch.
> Intranet Crawl : forbidden all, allow somes few host
> Whole-web crawl : allow all, forbidden few thinks
> I propose a third type of crawl.
> Directory Crawl : The purpose of this crawl is to manage few thousands of
> host wihtout managing rules pattern in UrlFilterRegexp.
> I made two patch for : 0.7, 0.7.1 and 0.8-dev
> I propose a new boolean property in nutch-site.xml :
> crawl.ignore.external.links, with false value at default.
> By default this new feature don't modify the behavior of nutch crawler.
> When you setup this property to true, the crawler don't fetch external links
> of the host.
> So the crawl is limited to the host that you inject at the beginning at the
> crawl.
> I know there is some proposal of new crawl policy using the CrawlDatum in
> 0.8-dev branch.
> This feature colud be a easiest way to add quickly new crawl feature to
> nutch, waiting for a best way to improve crawl policy.
> I post two patch.
> Sorry for my very poor english
> --
> Philippe
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
