Hi - if you have db.ignore.external.links set to true, and only inject http://127.0.0.1:8080/cocoon into an empty CrawlDB, it should not try other domains. So i am not sure what is really wrong, it should just work. I assume you already have cpsc in your CrawlDB and don't filter it out. You should use -filter (once) on the CrawlDB to remove cpsc.
Markus -----Original message----- > From:Mitch Baker <mitch.ba...@iga.in.gov> > Sent: Wednesday 9th March 2016 22:40 > To: user@nutch.apache.org > Subject: Only fetch 127.0.0.1:8080/* > > I have small setup to index some files on a local box. > > Solr 5 > Nutch 1.11 > > I thought I had it configured to not try any URLs that are not local to > the system but it still seems to look for them. > > fetching > http://www.cpsc.gov/Media/Documents/Regulations-Laws--Standards/Advisory-Opinions/Wheelchairs-145--/ > (queue crawl delay=2000ms) > fetching http://www.cpsc.gov/PageFiles/121846/fuclearance.pdf (queue crawl > delay=2000ms) > fetching > http://www.cpsc.gov/Business--Manufacturing/Business-Education/Business-Guidance/Phthalates-Information/ > (queue crawl delay=2000ms) > -activeThreads=150, spinWaiting=148, fetchQueues.totalSize=2091, > fetchQueues.getQueueCount=1 > fetching http://www.cpsc.gov/es/Research--Statistics/ (queue crawl > delay=2000ms) > > The regex-urlfilter.txt: > > # Each non-comment, non-blank line contains a regular expression > # prefixed by '+' or '-'. The first matching pattern in the file > # determines whether a URL is included or ignored. If no pattern > # matches, the URL is ignored. > > # skip file: ftp: and mailto: urls > -^(file|ftp|mailto): > > # skip image and other suffixes we can't yet parse > # for a more extensive coverage use the urlfilter-suffix plugin > -\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|wmf|WMF|zip|ZIP|ppt|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS)$ > > # skip URLs containing certain characters as probable queries, etc. > #-[?*!@=] > > # skip URLs with slash-delimited segment that repeats 3+ times, to break loops > -.*(/[^/]+)/[^/]+\1/[^/]+\1/ > > # skip specific PDF files in the volumes directory > -.*00(FRONT|INTRO)\.PDF.* > > #skip > #-^(http|https)://www\.*$ > #-^(http|https)://blogs\.*$ > #-^(http|https)://store\.*$ > #-^(http|https)://.*\.google.com/.*$ > #-^(http|https)://nist.gov/.*$ > > > # accept anything else > #+. > +^http://127.0.0.1:8080/cocoon > > I have searched and tried several things, including nutch-site.xml: > > <configuration> > <property> > <name>http.agent.name</name> > <value>nutch-solr-integration</value> > </property> > <property> > <name>generate.max.per.host</name> > <value>1000</value> > </property> > <property> > <name>plugin.includes</name> > <value>protocol-http|urlfilter-regex|parse-(html|tika|metatags)|index-(basic|anchor|metadata)|indexer-solr|query-(basic|site|url)|response-(json|xml)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value> > </property> > <property> > <name>db.ignore.external.links</name> > <value>true</value> > <description>If true, outlinks leading from a page to external hosts or > domain > will be ignored. This is an effective way to limit the crawl to include > only initially injected hosts, without creating complex URLFilters. > See 'db.ignore.external.links.mode'. > </description> > </property> > <property> > <name>db.max.outlinks.per.page</name> > <value>0</value> > <description>The maximum number of outlinks that we'll process for a page. > If this value is nonnegative (>=0), at most db.max.outlinks.per.page > outlinks > will be processed for a page; otherwise, all outlinks will be processed. > </description> > </property> > <property> > <name>fetcher.max.crawl.delay</name> > <value>3</value> > <description> > If the Crawl-Delay in robots.txt is set to greater than this value (in > seconds) then the fetcher will skip this page, generating an error report. > If set to -1 the fetcher will never skip such pages and will wait the > amount of time retrieved from robots.txt Crawl-Delay, however long that > might be. > </description> > </property> > <property> > <name>fetcher.queue.mode</name> > <value>byHost</value> > <description>Determines how to put URLs into queues. Default value is > 'byHost', > also takes 'byDomain' or 'byIP'. > </description> > </property> > > <property> > <name>fetcher.verbose</name> > <value>false</value> > <description>If true, fetcher will log more verbosely.</description> > </property> > > > I inherited this and not that well versed on nutch. Many hours of searching, > and trying what I have found but still no luck. Can't get it to just search > the local > system http://127.0.0.1:8080/cocoon > > > And help would be greatly appreciated. > > -- > Mitch Baker <mitch.ba...@iga.in.gov> > LSA >
