OK, I may not be clear on this , I though this property specifies the max time a given page should not be recrawled from the last time it was fetched. I dont think it specifies recrawl interval for Nutch as such, I dont think there is any parameter to control the recrawl interval other than manually triggering a crawl via a script or Hadoop job.
On Wed, Jun 4, 2014 at 1:32 PM, Ali Nazemian <[email protected]> wrote: > Thank you very much. But it is just a parameter for specifying the interval > between re-crawls. The problem is nutch re-crawl does not works with > default crawl script. > > > On Wed, Jun 4, 2014 at 6:49 PM, S.L <[email protected]> wrote: > > > Ali, > > > > If you have not found this out yet, I was referring to > > db.fetch.interval.max. > > > > Sent from my HTC > > > > ----- Reply message ----- > > From: "Ali Nazemian" <[email protected]> > > To: <[email protected]> > > Subject: Incremental crawling with nutch > > Date: Mon, Jun 2, 2014 4:52 AM > > > > Hi, > > Could you please explain more? > > What parameter? How can I do that?! > > Regards. > > > > > > On Mon, Jun 2, 2014 at 3:42 AM, S.L <[email protected]> wrote: > > > > > Hi Ali > > > > > > Please see the nutch-site.xml parameters one of them does that. > > > > > > Sent from my HTC > > > > > > ----- Reply message ----- > > > From: "Ali Nazemian" <[email protected]> > > > To: <[email protected]> > > > Subject: Incremental crawling with nutch > > > Date: Sun, Jun 1, 2014 10:46 AM > > > > > > Hi everybody, > > > I am going to use nutch for crawling some news web site. These websites > > > will be updated regularly. Therefore I should recrawl them at least > > every 2 > > > hours. But the problem is I want to have incremental re-crawl, it means > > > nutch should crawl only the urls that are new and not fetched before > > > (except the main page of each site for extracting new urls). I want in > > each > > > re-crawling process only the new URLs fetched and send to solr for > > > indexing. Would somebody guide me through this scenario with nutch 1.8? > > > Best regards. > > > > > > -- > > > A.Nazemian > > > > > > > > > > > -- > > A.Nazemian > > > > > > -- > A.Nazemian >
