I use 2.1 and if I dont indicate a crawldb dir it crawls into Hbase. So does it mean that information is stored at Hbase too?
2013/3/21 Tejas Patil [via Lucene] <ml-node+s472066n404959...@n3.nabble.com> > Which version of nutch are you using for this crawl ? (please specify this > info whenever you are asking for help over @user. It would help us > understanding the issue) > > In your "bin/nutch crawl" commands, the "crawldb" ain't specified. So each > invocation will produce a fresh "crawldb" using the timestamp when the > command was invoked. If the seed urls, configs and params are not changed > in between, there should be no difference in those 2 executions. This is > how things are done in 1.x. I am not sure how its done in 2.x. Depending > on > your other mail thread over @user, I think that you are using 2.x. > > On Wed, Mar 20, 2013 at 3:59 PM, kamaci <[hidden > email]<http://user/SendEmail.jtp?type=node&node=4049593&i=0>> > wrote: > > > First I used that command: > > > > bin/nutch crawl /usr/apache-nutch-2.1/urls -depth 3 > > > > than I pressed ctrl + c and fetching has ended as usual. > > > > After that I used that command: > > > > bin/nutch crawl /usr/apache-nutch-2.1/urls -depth 2 -topN 2 > > > > But fetching takes long time than usual. So I think it continues to > > previous fetching is that true? > > > > > > 2013/3/21 Markus Jelsma-2 [via Lucene] < > > [hidden email] <http://user/SendEmail.jtp?type=node&node=4049593&i=1>> > > > > > If Nutch exits with an error then the segment is bad, a failing thread > is > > > not an error that leads to a failed segments. This means the segment > is > > > properly fetched but just that some records failed. Those records will > be > > > eligible for refetch. > > > > > > Assuming you use the crawl command, the updatedb command will be > > > successful so there should be no issue here. What's the problem? > > > > > > > > > -----Original message----- > > > > > > > From:kamaci <[hidden email]< > > http://user/SendEmail.jtp?type=node&node=4049571&i=0>> > > > > > > > Sent: Wed 20-Mar-2013 23:48 > > > > To: [hidden email]< > > http://user/SendEmail.jtp?type=node&node=4049571&i=1> > > > > Subject: How to Continue to Crawl with Nutch Even An Error Occurs? > > > > > > > > When I crawl with Nutch and error occurs (i.e. when one of threads > > > doesn't > > > > come within a time) it stops crawling and exits. > > > > > > > > Is there any configuration to continue crawling even a such kind of > > > error > > > > occurs at Nutch? > > > > > > > > > > > > > > > > -- > > > > View this message in context: > > > > > > http://lucene.472066.n3.nabble.com/How-to-Continue-to-Crawl-with-Nutch-Even-An-Error-Occurs-tp4049567.html > > > > Sent from the Nutch - User mailing list archive at Nabble.com. > > > > > > > > > > > > > ------------------------------ > > > If you reply to this email, your message will be added to the > discussion > > > below: > > > > > > > > > http://lucene.472066.n3.nabble.com/How-to-Continue-to-Crawl-with-Nutch-Even-An-Error-Occurs-tp4049567p4049571.html > > > To unsubscribe from How to Continue to Crawl with Nutch Even An Error > > > Occurs?, click here< > > > > > > > . > > > NAML< > > > http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml > > > > > > > > > > > > > > > > > -- > > View this message in context: > > > http://lucene.472066.n3.nabble.com/How-to-Continue-to-Crawl-with-Nutch-Even-An-Error-Occurs-tp4049567p4049577.html > > > Sent from the Nutch - User mailing list archive at Nabble.com. > > > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > > http://lucene.472066.n3.nabble.com/How-to-Continue-to-Crawl-with-Nutch-Even-An-Error-Occurs-tp4049567p4049593.html > To unsubscribe from How to Continue to Crawl with Nutch Even An Error > Occurs?, click > here<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4049567&code=ZnVya2Fua2FtYWNpQGdtYWlsLmNvbXw0MDQ5NTY3fDEyODM4MDc0Mg==> > . > NAML<http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://lucene.472066.n3.nabble.com/How-to-Continue-to-Crawl-with-Nutch-Even-An-Error-Occurs-tp4049567p4049594.html Sent from the Nutch - User mailing list archive at Nabble.com.
