[ 
https://issues.apache.org/jira/browse/NUTCH-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12565848#action_12565848
 ] 

Susam Pal commented on NUTCH-601:
---------------------------------

The 'if (newIndex != index)' condition is just a check whether this is a new 
crawl directory being constructed, or it is a recrawl on a previous crawl 
directory.

If it is a new crawl directory being constructed, a few lines above this check, 
there is another 'if (!fs.exists(index))' condition which will set the newIndex 
= index = '/index'. So, if both newIndex and index are same, we know that it is 
a new crawl directory and we need not delete old 'index' because it would not 
be present.

If it is a recrawl over a previous crawl directory, newIndex = '/new_index' and 
index = '/index'. Since they are different, it indicates that this is a recrawl 
and thus after '/new_index' is created, it'll quickly replace the old '/index' 
with the '/new_index'.

This seems fine to me.

If you want to avoid the possibility of having new segments with old index if 
something bad happens, then I should delete both 'index' and 'indexes' even 
before the generate call. But I didn't want to delete old 'index' so early. I 
was trying to minimize the time for which 'index' directory is unavailable. 
This would be helpful in case someone is running a recrawl on the same 'crawl' 
directory which the web-gui is using to serve search results.

Please let me know what you feel about this.

> Recrawling on existing crawl directory using force option
> ---------------------------------------------------------
>
>                 Key: NUTCH-601
>                 URL: https://issues.apache.org/jira/browse/NUTCH-601
>             Project: Nutch
>          Issue Type: Improvement
>    Affects Versions: 1.0.0
>            Reporter: Susam Pal
>            Priority: Minor
>         Attachments: NUTCH-601v0.1.patch, NUTCH-601v0.2.patch
>
>
> Added a '-force' option to the 'bin/nutch crawl' command line. With this 
> option, one can crawl and recrawl in the following manner:
> {code}
> bin/nutch crawl urls -dir crawl -depth 2 -topN 10 -threads 5
> bin/nutch crawl urls -dir crawl -depth 2 -topN 10 -threads 5 -force
> {code}
> This option can be used for the first crawl too:
> {code}
> bin/nutch crawl urls -dir crawl -depth 2 -topN 10 -threads 5 -force
> bin/nutch crawl urls -dir crawl -depth 2 -topN 10 -threads 5 -force
> {code}
> If one tries to crawl without the -force option when the crawl directory 
> already exists, he/she finds a small warning along with the error message:
> {code}
> # bin/nutch crawl urls -dir crawl -depth 2 -topN 10 -threads 5
> Exception in thread "main" java.lang.RuntimeException: crawl already
> exists. Add -force option to recrawl.
>        at org.apache.nutch.crawl.Crawl.main(Crawl.java:89)
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Reply via email to