Hello Shakiba - please check Nutch' logs. The error is reported there. Markus
-----Original message----- > From:shakiba davari <davari.shak...@gmail.com> > Sent: Thursday 16th June 2016 23:04 > To: user@nutch.apache.org > Subject: Re: Indexing nutch crawled data in “Bluemix” solr > > Thanks so much Lewis. It really helped me. at least now I know that there > is a way to make it work. > I did used the command as you said: > > bin/nutch index -D solr.server.url=" > https://gateway.watsonplatform.net/retrieve-and-rank/api/v1/solr_clusters/CLUSTER-ID/solr/admin/collections > -D solr.auth=true -D solr.auth.username="USERNAME" -D > solr.auth.password="PASS" Crawl/crawldb -linkdb Crawl/linkdb > Crawl/segments/2016* > > and now the result is: > > Indexing 153 documents > Indexing 153 documents > Indexer: java.io.IOException: Job failed! > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:836) > at org.apache.nutch.indexer.IndexingJob.index(IndexingJob.java:145) > at org.apache.nutch.indexer.IndexingJob.run(IndexingJob.java:222) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.nutch.indexer.IndexingJob.main(IndexingJob.java:231) > > > I guess it has something to do with the solr.server.url address, maybe the > end of it. I changed it in different ways > e.g " > https://gateway.watsonplatform.net/retrieve-and-rank/api/v1/solr_clusters/CLUSTER-ID/solr/example_collection/update"; > (since it is used for feeding JSON files the the bluemix solr ) > but no chance to now. > > any Idea what's happening now?? > > > Shakiba <https://ca.linkedin.com/pub/shakiba-davari/84/417/b57>* Davari * > > > On Tue, Jun 14, 2016 at 4:58 PM, Lewis John Mcgibbney < > lewis.mcgibb...@gmail.com> wrote: > > > Hi shakiba, > > > > On Sat, Jun 11, 2016 at 1:48 PM, <user-digest-h...@nutch.apache.org> > > wrote: > > > > > From: shakiba davari <davari.shak...@gmail.com> > > > To: user@nutch.apache.org > > > Cc: > > > Date: Thu, 9 Jun 2016 13:11:43 -0400 > > > Subject: Indexing nutch crawled data in “Bluemix” solr > > > 1down votefavorite > > > < > > > > > http://stackoverflow.com/questions/37731716/indexing-nutch-crawled-data-in-bluemix-solr# > > > > > > > > > > I'm trying to index the nutch crawled data by Bluemix solr and I cannot > > > find anyway to do it. My main question is: Is there anybody that can help > > > me to do so? what should I do to send the result of my nutch crawled data > > > to my Blumix Solr. > > > > > > For the crawling I used nutch 1.11 and here is a part of what I did to > > now > > > and the problems I faced: I thought there may be two possible solutions: > > > > > > 1. By nutch command: > > > > > > “NUTCH_PATH/bin/nutch index crawl/crawldb -linkdb crawl/linkdb crawl/ > > > -Dsolr.server.url="OURSOLRURL"” > > > > > > I can index the nutch crawled data by OURSOLR. However, I found some > > > problem with that. > > > > > > a-Though it sounds really odd, it could not accept the URL. I could > > handle > > > it by using the URL’s Encode instead. > > > > > > b-Since I have to connect to a specific Username and password, nutch > > could > > > not connect to my solr. Considering this: > > > > > > Active IndexWriters : > > > SolrIndexWriter > > > solr.server.type : Type of SolrServer to communicate with (default > > > 'http' however options include 'cloud', 'lb' and 'concurrent') > > > solr.server.url : URL of the Solr instance (mandatory) > > > solr.zookeeper.url : URL of the Zookeeper URL (mandatory if > > > 'cloud' value for solr.server.type) > > > solr.loadbalance.urls : Comma-separated string of Solr server > > > strings to be used (madatory if 'lb' value for solr.server.type) > > > solr.mapping.file : name of the mapping file for fields (default > > > solrindex-mapping.xml) > > > solr.commit.size : buffer size when sending to Solr (default 1000) > > > solr.auth : use authentication (default false) > > > solr.auth.username : username for authentication > > > solr.auth.password : password for authentication > > > > > > in the command line output,I tried to manage this problem by using > > > authentication parameters of the command "solr.auth=true > > > solr.auth.username="SOLR-UserName" solr.auth.password="Pass" to it. > > > > > > So up to now I’ve got to a point to use this command: > > > > > > ”bin/nutch index crawl/crawldb -linkdb crawl/linkdb crawl/segments/2016* > > > solr.server.url="https%3A%2F%2Fgateway.watsonplatform.net > > > > > %2Fretrieve-and-rank%2Fapi%2Fv1%2Fsolr_clusters%2FCLUSTER-ID%2Fsolr%2Fadmin%2Fcollections" > > > solr.auth=true solr.auth.username="USERNAME" solr.auth.password="PASS"“. > > > > > > But for some reason that I couldn’t realize yet, the command considers > > the > > > authentication parameters as crawled data directory and does not work. > > So I > > > guess it is not the right way to "Active IndexWriters" can anyone tell me > > > then how can I?? > > > > > > > Please enter the command line parameters IN FRONT of the Tool arguments > > e.g. bin/nutch index -D solr.server.url="https%3A%2F% > > 2Fgateway.watsonplatform.net > > > > %2Fretrieve-and-rank%2Fapi%2Fv1%2Fsolr_clusters%2FCLUSTER-ID%2Fsolr%2Fadmin%2Fcollections" > > -D solr.auth=true -D solr.auth.username="USERNAME" -D > > solr.auth.password="PASS" crawl/crawldb -linkdb crawl/linkdb > > crawl/segments/ > > > > > > > > > > 1. By curl command: > > > > > > “curl -X POST -H "Content-Type: application/json" -u > > > "BLUEMIXSOLR-USERNAME":"BLUEMIXSOLR-PASS" " > > > > > > > > https://gateway.watsonplatform.net/retrieve-and-rank/api/v1/solr_clusters/CLUSTERS-ID/solr/example_collection/update > > > " > > > --data-binary @{/path_to_file}/FILE.json” > > > > > > I thought maybe I can feed json files created by this command: > > > > > > bin/nutch commoncrawldump -outputDir finalcrawlResult/ -segment > > > crawl/segments -gzip -extension json -SimpleDateFormat -epochFilename > > > -jsonArray -reverseKey but there are some problems here. > > > > > > a. this command provides so many files in complicated Paths which will > > take > > > so much time to manually post all of them.I guess for big cawlings it may > > > be even impossible. Is there any way to POST all the files in a directory > > > and its subdirectories at once by just one command?? > > > > > > > Unfortunately, right now AFAIK you cannot prevent the tool from creating > > the directory hell. You might be better off using the FileDumper tool > > instead > > ./bin/nutch dump > > > > > > > > > > b. there is a weird name "ÙÙ÷y œ" at the start of json files created by > > > commoncrawldump. > > > > > > > The data is encoded as CBOR. This is why the Bytes exist. > > > > > > > > > > c. I removed the name weird name and tried to POST just one of these > > files > > > but here is the result: > > > > > > > > > > > {"responseHeader":{"status":400,"QTime":23},"error":{"metadata":["error-class","org.apache.solr.common.SolrException","root-error-class","org.apache.solr.common.SolrException"],"msg":"Unknown > > > command 'url' at [9]","code":400}} > > > > > > > > No it just means that you are not using the index tool correctly and that > > possibly your input data is not in the correct format. > > Hope this help.s > > Lewis > > >
