Hi - conf/regex-url-filter.txt and make sure the urlfilter-regex is enabled in
your nutch-site plugin.includes config.
-----Original message-----
> From:devang pandey <devangpande...@gmail.com>
> Sent: Wednesday 10th July 2013 11:51
> To: user@nutch.apache.org
> Subject: Re: nutch crawling issues
>
> hello markus I have one confusion should i implement changes in crawl-url
> filter or regex filter
>
>
> On Wed, Jul 10, 2013 at 3:12 PM, Markus Jelsma
> <markus.jel...@openindex.io>wrote:
>
> > Hi,
> >
> > Use a regex url filter to filter those URL's and prevent them from being
> > crawled again.
> >
> > Cheers
> >
> > -----Original message-----
> > > From:devang pandey <devangpande...@gmail.com>
> > > Sent: Wednesday 10th July 2013 10:29
> > > To: user@nutch.apache.org
> > > Subject: nutch crawling issues
> > >
> > > I have a website eg . www.example.com. Now when I am crawling this using
> > > nutch 1.4 problem is that of duplicated crawling . There are a number of
> > > pages like www.example.com/s38r84rejkfndn/xyz.aspx . Now this number
> > > s38r84rejkfndn keeps on changing every time you visit this page and hence
> > > crawler is crawling this again and again as for nutch I this this must
> > be a
> > > new url everytime . Please suggest me how to overcome this issue
> > >
> >
>
