DIH problem passing HTTP parameters into data-config

Tue, 11 Aug 2009 16:26:13 -0700

I've read the documentation as carefully as I can, but I must be missing something. I'm running Solr 1.3.
The doc sez that I can pass my own parameters in to DIH via the HTTP request: 

http://wiki.apache.org/solr/DataImportHandler#head-520f8e527d9da55e8ed1e274e29709c8805c8eae


What I'd like to do is pass the URL of an RSS feed into DIH via HTTP (/ 
dataimport), and have it import the items in the feed in "the usual  
way" (i.e. as shown for example in the slashdot example).  (This way I  
don't have to hardcode my list of feeds)


SO if I have a URL like the following to dataimport:

http://localhost:8983/solr/rss/dataimport?command=full-import&clean=false&feed=http%3A%2F%2Ffeeds.wsjonline.com%2Fwsj%2Fxml%2Frss%2F3_7011.xml


(i.e. a normal dataimport command, plus a parameter 'feed' with the  
url-encoded value of an RSS feed)


AND, in my config file I have something like:

<dataConfig>
       <dataSource type="HttpDataSource" name="rss-feed" />
       <document>
               <entity name="article" dataSource="rss-feed"
                               pk="link"
                               url="${dataimporter.request.url}"
                               processor="XPathEntityProcessor"


[blah blah blah...looks pretty much like the slashdot RSS rss-data- 
config.xml example)]


It should just work.

However,

I get the following:

Aug 10, 2009 12:15:55 AM org.apache.solr.core.SolrCore execute

INFO: [rss] webapp=/solr path=/dataimport params={command=full- 
import&feed=http://feeds.wsjonline.com/wsj/xml/rss/3_7011.xml&clean=false&feedname=wsjhome 
} status=0 QTime=0
Aug 10, 2009 12:15:55 AM  
org.apache.solr.handler.dataimport.DataImporter doFullImport

INFO: Starting Full Import

Aug 10, 2009 12:15:55 AM  
org.apache.solr.handler.dataimport.HttpDataSource getData

SEVERE: Exception thrown while getting data
java.net.MalformedURLException: no protocol: null
        at java.net.URL.<init>(URL.java:583)
        at java.net.URL.<init>(URL.java:480)
        at java.net.URL.<init>(URL.java:429)

	at  
org 
.apache 
.solr.handler.dataimport.HttpDataSource.getData(HttpDataSource.java:90)
	at  
org 
.apache 
.solr.handler.dataimport.HttpDataSource.getData(HttpDataSource.java:47)
	at  
org 
.apache 
.solr 
.handler 
.dataimport.XPathEntityProcessor.initQuery(XPathEntityProcessor.java: 
210)
	at  
org 
.apache 
.solr 
.handler 
.dataimport 
.XPathEntityProcessor.fetchNextRow(XPathEntityProcessor.java:180)
	at  
org 
.apache 
.solr 
.handler 
.dataimport.XPathEntityProcessor.nextRow(XPathEntityProcessor.java:163)
	at  
org 
.apache 
.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
	at  
org 
.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java: 
178)
	at  
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java: 
136)
	at  
org 
.apache 
.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java: 
334)
	at  
org 
.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java: 
386)
	at org.apache.solr.handler.dataimport.DataImporter 
$1.run(DataImporter.java:377)
Aug 10, 2009 12:15:55 AM org.apache.solr.handler.dataimport.DocBuilder  
buildDocument
SEVERE: Exception while processing: ${dataimporter.request.feedname}  
document : SolrInputDocumnt[{}]
org.apache.solr.handler.dataimport.DataImportHandlerException:  
Exception in invoking url null Processing Document # 1
	at  
org 
.apache 
.solr.handler.dataimport.HttpDataSource.getData(HttpDataSource.java:115)
	at  
org 
.apache 
.solr.handler.dataimport.HttpDataSource.getData(HttpDataSource.java:47)
	at  
org 
.apache 
.solr 
.handler 
.dataimport.XPathEntityProcessor.initQuery(XPathEntityProcessor.java: 
210)
	at  
org 
.apache 
.solr 
.handler 
.dataimport 
.XPathEntityProcessor.fetchNextRow(XPathEntityProcessor.java:180)
	at  
org 
.apache 
.solr 
.handler 
.dataimport.XPathEntityProcessor.nextRow(XPathEntityProcessor.java:163)
	at  
org 
.apache 
.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
	at  
org 
.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java: 
178)
	at  
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java: 
136)
	at  
org 
.apache 
.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java: 
334)
	at  
org 
.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java: 
386)
	at org.apache.solr.handler.dataimport.DataImporter 
$1.run(DataImporter.java:377)

Caused by: java.net.MalformedURLException: no protocol: null
        at java.net.URL.<init>(URL.java:583)
        at java.net.URL.<init>(URL.java:480)
        at java.net.URL.<init>(URL.java:429)

	at  
org 
.apache 
.solr.handler.dataimport.HttpDataSource.getData(HttpDataSource.java:90)

        ... 10 more

Aug 10, 2009 12:15:55 AM  
org.apache.solr.handler.dataimport.DataImporter doFullImport

SEVERE: Full Import failed


Seems like the HTTP parm is not getting passed. Or I'm not referring  
to it properly. Or something!


Any suggestions welcome...

Cheers, TIA,

John





















					Reply via email to