yes. If you have configured it to use HBase, then the info will be stored in HBase.
On Wed, Mar 20, 2013 at 4:35 PM, kamaci <furkankam...@gmail.com> wrote: > I use 2.1 and if I dont indicate a crawldb dir it crawls into Hbase. So > does it mean that information is stored at Hbase too? > > 2013/3/21 Tejas Patil [via Lucene] < > ml-node+s472066n404959...@n3.nabble.com> > > > Which version of nutch are you using for this crawl ? (please specify > this > > info whenever you are asking for help over @user. It would help us > > understanding the issue) > > > > In your "bin/nutch crawl" commands, the "crawldb" ain't specified. So > each > > invocation will produce a fresh "crawldb" using the timestamp when the > > command was invoked. If the seed urls, configs and params are not changed > > in between, there should be no difference in those 2 executions. This is > > how things are done in 1.x. I am not sure how its done in 2.x. Depending > > on > > your other mail thread over @user, I think that you are using 2.x. > > > > On Wed, Mar 20, 2013 at 3:59 PM, kamaci <[hidden email]< > http://user/SendEmail.jtp?type=node&node=4049593&i=0>> > > wrote: > > > > > First I used that command: > > > > > > bin/nutch crawl /usr/apache-nutch-2.1/urls -depth 3 > > > > > > than I pressed ctrl + c and fetching has ended as usual. > > > > > > After that I used that command: > > > > > > bin/nutch crawl /usr/apache-nutch-2.1/urls -depth 2 -topN 2 > > > > > > But fetching takes long time than usual. So I think it continues to > > > previous fetching is that true? > > > > > > > > > 2013/3/21 Markus Jelsma-2 [via Lucene] < > > > [hidden email] <http://user/SendEmail.jtp?type=node&node=4049593&i=1>> > > > > > > > If Nutch exits with an error then the segment is bad, a failing > thread > > is > > > > not an error that leads to a failed segments. This means the segment > > is > > > > properly fetched but just that some records failed. Those records > will > > be > > > > eligible for refetch. > > > > > > > > Assuming you use the crawl command, the updatedb command will be > > > > successful so there should be no issue here. What's the problem? > > > > > > > > > > > > -----Original message----- > > > > > > > > > From:kamaci <[hidden email]< > > > http://user/SendEmail.jtp?type=node&node=4049571&i=0>> > > > > > > > > > Sent: Wed 20-Mar-2013 23:48 > > > > > To: [hidden email]< > > > http://user/SendEmail.jtp?type=node&node=4049571&i=1> > > > > > Subject: How to Continue to Crawl with Nutch Even An Error Occurs? > > > > > > > > > > When I crawl with Nutch and error occurs (i.e. when one of threads > > > > doesn't > > > > > come within a time) it stops crawling and exits. > > > > > > > > > > Is there any configuration to continue crawling even a such kind of > > > > error > > > > > occurs at Nutch? > > > > > > > > > > > > > > > > > > > > -- > > > > > View this message in context: > > > > > > > > > > http://lucene.472066.n3.nabble.com/How-to-Continue-to-Crawl-with-Nutch-Even-An-Error-Occurs-tp4049567.html > > > > > Sent from the Nutch - User mailing list archive at Nabble.com. > > > > > > > > > > > > > > > > > ------------------------------ > > > > If you reply to this email, your message will be added to the > > discussion > > > > below: > > > > > > > > > > > > > > http://lucene.472066.n3.nabble.com/How-to-Continue-to-Crawl-with-Nutch-Even-An-Error-Occurs-tp4049567p4049571.html > > > > To unsubscribe from How to Continue to Crawl with Nutch Even An > Error > > > > Occurs?, click here< > > > > > > > > > > . > > > > NAML< > > > > > > http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > View this message in context: > > > > > > http://lucene.472066.n3.nabble.com/How-to-Continue-to-Crawl-with-Nutch-Even-An-Error-Occurs-tp4049567p4049577.html > > > > > Sent from the Nutch - User mailing list archive at Nabble.com. > > > > > > > > > ------------------------------ > > If you reply to this email, your message will be added to the discussion > > below: > > > > > http://lucene.472066.n3.nabble.com/How-to-Continue-to-Crawl-with-Nutch-Even-An-Error-Occurs-tp4049567p4049593.html > > To unsubscribe from How to Continue to Crawl with Nutch Even An Error > > Occurs?, click here< > http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4049567&code=ZnVya2Fua2FtYWNpQGdtYWlsLmNvbXw0MDQ5NTY3fDEyODM4MDc0Mg== > > > > . > > NAML< > http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml > > > > > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/How-to-Continue-to-Crawl-with-Nutch-Even-An-Error-Occurs-tp4049567p4049594.html > Sent from the Nutch - User mailing list archive at Nabble.com. >
