Well, you can always the DomainStatistics utilities to get the raw numbers on 
hosts, domains and TLD's but this won't tell you whether a domain has been 
fully crawled because the crawling frontier can always change.

You can be sure that everything (disregarding url filters) has been crawled if 
no more records are selected before fetched records are eligible again for 
refetch (default interval).

NUTCH-1325 does a better job in providing stats for hosts than the current 
DomainStatistics but it's uncommitted. It'll work though.

https://issues.apache.org/jira/browse/NUTCH-1325
 
-----Original message-----
> From:Tejas Patil <tejas.patil...@gmail.com>
> Sent: Mon 25-Feb-2013 20:46
> To: user@nutch.apache.org
> Subject: Re: Nutch status info on each domain individually
> 
> I can't of any existing nutch utility which can be used here. Maybe dumping
> the crawldb and then grepping over it would sound reasonable if the number
> of hosts is large and the crawldb is small. This will be a bad idea if this
> has to be done after every nutch cycle on a large crawldb.
> 
> If you are ready to write some small code, then it can become easy:
> 1. Write some code to query the index so that you need not have to do that
> manually. OR
> 2. Write a map reduce code to read crawdb wherein the mapper emits the
> hosts of the url.
> 
> #1 is better deal in terms of execution time.
> 
> Thanks,
> Tejas Patil
> 
> 
> On Mon, Feb 25, 2013 at 11:28 AM, imehesz <imeh...@gmail.com> wrote:
> 
> > hello,
> >
> > I can finally run Nutch (+Solr) with JAVA, my only question left is, how
> > can
> > I make sure if a particular domain has been crawled?
> >
> > Let's say I have 300 sites to crawl and index.
> > So far my work-around was to execute a simple Solr query for each domain
> > URL, and see if the indexing timestamp in the Solr DB is greater then the
> > Nutch crawling start date-time. It works, but I'm curious if there is a
> > better way to do this.
> >
> > thanks,
> > --iM
> >
> >
> >
> > --
> > View this message in context:
> > http://lucene.472066.n3.nabble.com/Nutch-status-info-on-each-domain-individually-tp4042815.html
> > Sent from the Nutch - User mailing list archive at Nabble.com.
> >
> 

Reply via email to