[ https://issues.apache.org/jira/browse/NUTCH-601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12565453#action_12565453 ]
Andrzej Bialecki commented on NUTCH-601: ----------------------------------------- Thank you for creating this issue. I think that perhaps the old behavior should be removed altogether - it was based on an assumption that users don't want by default to recrawl the same pages. Our experience as a community shows that this is usually not the case, and the old behavior is confusing - so why not remove this artificial limitation altogether, and if users do want to keep each cycle in a separate directory they can do this by specifying different output directories. > Recrawling on existing crawl directory using force option > --------------------------------------------------------- > > Key: NUTCH-601 > URL: https://issues.apache.org/jira/browse/NUTCH-601 > Project: Nutch > Issue Type: Improvement > Affects Versions: 1.0.0 > Reporter: Susam Pal > Priority: Minor > Attachments: NUTCH-601v0.1.patch > > > Added a '-force' option to the 'bin/nutch crawl' command line. With this > option, one can crawl and recrawl in the following manner: > {code} > bin/nutch crawl urls -dir crawl -depth 2 -topN 10 -threads 5 > bin/nutch crawl urls -dir crawl -depth 2 -topN 10 -threads 5 -force > {code} > This option can be used for the first crawl too: > {code} > bin/nutch crawl urls -dir crawl -depth 2 -topN 10 -threads 5 -force > bin/nutch crawl urls -dir crawl -depth 2 -topN 10 -threads 5 -force > {code} > If one tries to crawl without the -force option when the crawl directory > already exists, he/she finds a small warning along with the error message: > {code} > # bin/nutch crawl urls -dir crawl -depth 2 -topN 10 -threads 5 > Exception in thread "main" java.lang.RuntimeException: crawl already > exists. Add -force option to recrawl. > at org.apache.nutch.crawl.Crawl.main(Crawl.java:89) > {code} -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.