Attempting dataimport using FileListEntityProcessor
I'm trying to use the fileListEntityProcessor to add some xml documents to a solr index. I'm running a nightly version of solr-1.3 with SOLR-469 and SOLR-563. I've been able to successfuly run the slashdot httpDataSource example. My data-config.xml file loads without errors. When I attempt the full-import command I get the exception below. Thanks for any help. Mike WARNING: No lockType configured for /san/tomcat-services/solr-medline/solr/data/index/ assuming 'simple' Jun 23, 2008 7:59:49 PM org.apache.solr.handler.dataimport.DataImporter doFullImport SEVERE: Full Import failed java.lang.RuntimeException: java.lang.NullPointerException at org.apache.solr.handler.dataimport.XPathRecordReader.streamRecords(XPathRecordReader.java:97) at org.apache.solr.handler.dataimport.XPathEntityProcessor.initQuery(XPathEntityProcessor.java:212) at org.apache.solr.handler.dataimport.XPathEntityProcessor.fetchNextRow(XPathEntityProcessor.java:166) at org.apache.solr.handler.dataimport.XPathEntityProcessor.nextRow(XPathEntityProcessor.java:149) at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:286) at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:312) at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:179) at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:140) at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:335) at org.apache.solr.handler.dataimport.DataImporter.rumCmd(DataImporter.java:386) at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377) Caused by: java.lang.NullPointerException at java.io.Reader.(Reader.java:61) at java.io.BufferedReader.(BufferedReader.java:76) at com.bea.xml.stream.MXParser.checkForXMLDecl(MXParser.java:775) at com.bea.xml.stream.MXParser.setInput(MXParser.java:806) at com.bea.xml.stream.MXParserFactory.createXMLStreamReader(MXParserFactory.java:261) at org.apache.solr.handler.dataimport.XPathRecordReader.streamRecords(XPathRecordReader.java:93) ... 10 more Here is my data-config: And a snippet from an xml file: 12236137 1980 01 03 -- View this message in context: http://www.nabble.com/Attempting-dataimport-using-FileListEntityProcessor-tp18081671p18081671.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Attempting dataimport using FileListEntityProcessor
hi ,
You have not registered any datasources . the second entity needs a datasource.
Remove the dataSource="null" and add a name for the second entity
(good practice). No need for baseDir attribute for second entity .
See the modified xml added below
--Noble
On Tue, Jun 24, 2008 at 6:39 AM, mike segv <[EMAIL PROTECTED]> wrote:
>
> I'm trying to use the fileListEntityProcessor to add some xml documents to a
> solr index. I'm running a nightly version of solr-1.3 with SOLR-469 and
> SOLR-563. I've been able to successfuly run the slashdot httpDataSource
> example. My data-config.xml file loads without errors. When I attempt the
> full-import command I get the exception below. Thanks for any help.
>
> Mike
>
> WARNING: No lockType configured for
> /san/tomcat-services/solr-medline/solr/data/index/ assuming 'simple'
> Jun 23, 2008 7:59:49 PM org.apache.solr.handler.dataimport.DataImporter
> doFullImport
> SEVERE: Full Import failed
> java.lang.RuntimeException: java.lang.NullPointerException
>at
> org.apache.solr.handler.dataimport.XPathRecordReader.streamRecords(XPathRecordReader.java:97)
>at
> org.apache.solr.handler.dataimport.XPathEntityProcessor.initQuery(XPathEntityProcessor.java:212)
>at
> org.apache.solr.handler.dataimport.XPathEntityProcessor.fetchNextRow(XPathEntityProcessor.java:166)
>at
> org.apache.solr.handler.dataimport.XPathEntityProcessor.nextRow(XPathEntityProcessor.java:149)
>at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:286)
>at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:312)
>at
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:179)
>at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:140)
>at
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:335)
>at
> org.apache.solr.handler.dataimport.DataImporter.rumCmd(DataImporter.java:386)
>at
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
> Caused by: java.lang.NullPointerException
>at java.io.Reader.(Reader.java:61)
>at java.io.BufferedReader.(BufferedReader.java:76)
>at com.bea.xml.stream.MXParser.checkForXMLDecl(MXParser.java:775)
>at com.bea.xml.stream.MXParser.setInput(MXParser.java:806)
>at
> com.bea.xml.stream.MXParserFactory.createXMLStreamReader(MXParserFactory.java:261)
>at
> org.apache.solr.handler.dataimport.XPathRecordReader.streamRecords(XPathRecordReader.java:93)
>... 10 more
>
> Here is my data-config:
>
>
>
> newerThan="'NOW-10DAYS'" recursive="true" rootEntity="false"
> dataSource="null" baseDi
> r="/san/tomcat-services/solr-medline">
> url="${f.fileAbsolutePath}" dataSource="null">
>
>
>
>
>
>
> And a snippet from an xml file:
>
> 12236137
>
> 1980
> 01
> 03
>
>
>
> --
> View this message in context:
> http://www.nabble.com/Attempting-dataimport-using-FileListEntityProcessor-tp18081671p18081671.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
--Noble Paul
Re: Attempting dataimport using FileListEntityProcessor
That fixed it.
If I'm inserting millions of documents, how do I control docs/update? E.g.
if there are 50K docs per file, I'm thinking that I should probably code up
my own DataSource that allows me to stipulate docs/update. Like say, 100
instead of 50K. Does this make sense?
Mike
Noble Paul നോബിള് नोब्ळ् wrote:
>
> hi ,
> You have not registered any datasources . the second entity needs a
> datasource.
> Remove the dataSource="null" and add a name for the second entity
> (good practice). No need for baseDir attribute for second entity .
> See the modified xml added below
> --Noble
>
>
>
>
> newerThan="'NOW-10DAYS'" recursive="true" rootEntity="false"
> dataSource="null" baseDir="/san/tomcat-services/solr-medline">
> forEach="/MedlineCitation"
> url="${f.fileAbsolutePath}" >
>
>
>
>
>
>
> On Tue, Jun 24, 2008 at 6:39 AM, mike segv <[EMAIL PROTECTED]> wrote:
>>
>> I'm trying to use the fileListEntityProcessor to add some xml documents
>> to a
>> solr index. I'm running a nightly version of solr-1.3 with SOLR-469 and
>> SOLR-563. I've been able to successfuly run the slashdot httpDataSource
>> example. My data-config.xml file loads without errors. When I attempt
>> the
>> full-import command I get the exception below. Thanks for any help.
>>
>> Mike
>>
>> WARNING: No lockType configured for
>> /san/tomcat-services/solr-medline/solr/data/index/ assuming 'simple'
>> Jun 23, 2008 7:59:49 PM org.apache.solr.handler.dataimport.DataImporter
>> doFullImport
>> SEVERE: Full Import failed
>> java.lang.RuntimeException: java.lang.NullPointerException
>>at
>> org.apache.solr.handler.dataimport.XPathRecordReader.streamRecords(XPathRecordReader.java:97)
>>at
>> org.apache.solr.handler.dataimport.XPathEntityProcessor.initQuery(XPathEntityProcessor.java:212)
>>at
>> org.apache.solr.handler.dataimport.XPathEntityProcessor.fetchNextRow(XPathEntityProcessor.java:166)
>>at
>> org.apache.solr.handler.dataimport.XPathEntityProcessor.nextRow(XPathEntityProcessor.java:149)
>>at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:286)
>>at
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:312)
>>at
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:179)
>>at
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:140)
>>at
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:335)
>>at
>> org.apache.solr.handler.dataimport.DataImporter.rumCmd(DataImporter.java:386)
>>at
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>> Caused by: java.lang.NullPointerException
>>at java.io.Reader.(Reader.java:61)
>>at java.io.BufferedReader.(BufferedReader.java:76)
>>at com.bea.xml.stream.MXParser.checkForXMLDecl(MXParser.java:775)
>>at com.bea.xml.stream.MXParser.setInput(MXParser.java:806)
>>at
>> com.bea.xml.stream.MXParserFactory.createXMLStreamReader(MXParserFactory.java:261)
>>at
>> org.apache.solr.handler.dataimport.XPathRecordReader.streamRecords(XPathRecordReader.java:93)
>>... 10 more
>>
>> Here is my data-config:
>>
>>
>>
>> > newerThan="'NOW-10DAYS'" recursive="true" rootEntity="false"
>> dataSource="null" baseDi
>> r="/san/tomcat-services/solr-medline">
>> > url="${f.fileAbsolutePath}" dataSource="null">
>>
>>
>>
>>
>>
>>
>> And a snippet from an xml file:
>>
>> 12236137
>>
>> 1980
>> 01
>> 03
>>
>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Attempting-dataimport-using-FileListEntityProcessor-tp18081671p18081671.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> --Noble Paul
>
>
--
View this message in context:
http://www.nabble.com/Attempting-dataimport-using-FileListEntityProcessor-tp18081671p18083747.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Attempting dataimport using FileListEntityProcessor
Just extend XPathEntityProcessor override nextRow() after 100 . Use
it as your processor
return null;
On Tue, Jun 24, 2008 at 10:23 AM, mike segv <[EMAIL PROTECTED]> wrote:
>
> That fixed it.
>
> If I'm inserting millions of documents, how do I control docs/update? E.g.
> if there are 50K docs per file, I'm thinking that I should probably code up
> my own DataSource that allows me to stipulate docs/update. Like say, 100
> instead of 50K. Does this make sense?
>
> Mike
>
>
> Noble Paul നോബിള് नोब्ळ् wrote:
>>
>> hi ,
>> You have not registered any datasources . the second entity needs a
>> datasource.
>> Remove the dataSource="null" and add a name for the second entity
>> (good practice). No need for baseDir attribute for second entity .
>> See the modified xml added below
>> --Noble
>>
>>
>>
>>
>> > newerThan="'NOW-10DAYS'" recursive="true" rootEntity="false"
>> dataSource="null" baseDir="/san/tomcat-services/solr-medline">
>> > forEach="/MedlineCitation"
>> url="${f.fileAbsolutePath}" >
>>
>>
>>
>>
>>
>>
>> On Tue, Jun 24, 2008 at 6:39 AM, mike segv <[EMAIL PROTECTED]> wrote:
>>>
>>> I'm trying to use the fileListEntityProcessor to add some xml documents
>>> to a
>>> solr index. I'm running a nightly version of solr-1.3 with SOLR-469 and
>>> SOLR-563. I've been able to successfuly run the slashdot httpDataSource
>>> example. My data-config.xml file loads without errors. When I attempt
>>> the
>>> full-import command I get the exception below. Thanks for any help.
>>>
>>> Mike
>>>
>>> WARNING: No lockType configured for
>>> /san/tomcat-services/solr-medline/solr/data/index/ assuming 'simple'
>>> Jun 23, 2008 7:59:49 PM org.apache.solr.handler.dataimport.DataImporter
>>> doFullImport
>>> SEVERE: Full Import failed
>>> java.lang.RuntimeException: java.lang.NullPointerException
>>>at
>>> org.apache.solr.handler.dataimport.XPathRecordReader.streamRecords(XPathRecordReader.java:97)
>>>at
>>> org.apache.solr.handler.dataimport.XPathEntityProcessor.initQuery(XPathEntityProcessor.java:212)
>>>at
>>> org.apache.solr.handler.dataimport.XPathEntityProcessor.fetchNextRow(XPathEntityProcessor.java:166)
>>>at
>>> org.apache.solr.handler.dataimport.XPathEntityProcessor.nextRow(XPathEntityProcessor.java:149)
>>>at
>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:286)
>>>at
>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:312)
>>>at
>>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:179)
>>>at
>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:140)
>>>at
>>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:335)
>>>at
>>> org.apache.solr.handler.dataimport.DataImporter.rumCmd(DataImporter.java:386)
>>>at
>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>>> Caused by: java.lang.NullPointerException
>>>at java.io.Reader.(Reader.java:61)
>>>at java.io.BufferedReader.(BufferedReader.java:76)
>>>at com.bea.xml.stream.MXParser.checkForXMLDecl(MXParser.java:775)
>>>at com.bea.xml.stream.MXParser.setInput(MXParser.java:806)
>>>at
>>> com.bea.xml.stream.MXParserFactory.createXMLStreamReader(MXParserFactory.java:261)
>>>at
>>> org.apache.solr.handler.dataimport.XPathRecordReader.streamRecords(XPathRecordReader.java:93)
>>>... 10 more
>>>
>>> Here is my data-config:
>>>
>>>
>>>
>>> >> newerThan="'NOW-10DAYS'" recursive="true" rootEntity="false"
>>> dataSource="null" baseDi
>>> r="/san/tomcat-services/solr-medline">
>>> >> url="${f.fileAbsolutePath}" dataSource="null">
>>>
>>>
>>>
>>>
>>>
>>>
>>> And a snippet from an xml file:
>>>
>>> 12236137
>>>
>>> 1980
>>> 01
>>> 03
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Attempting-dataimport-using-FileListEntityProcessor-tp18081671p18081671.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> --Noble Paul
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Attempting-dataimport-using-FileListEntityProcessor-tp18081671p18083747.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
--Noble Paul
Re: Attempting dataimport using FileListEntityProcessor
Just extend XPathEntityProcessor override nextRow() after 100
return null. Use it as your processor
--Noble
On Tue, Jun 24, 2008 at 10:45 AM, Noble Paul നോബിള് नोब्ळ्
<[EMAIL PROTECTED]> wrote:
> Just extend XPathEntityProcessor override nextRow() after 100 . Use
> it as your processor
> return null;
>
> On Tue, Jun 24, 2008 at 10:23 AM, mike segv <[EMAIL PROTECTED]> wrote:
>>
>> That fixed it.
>>
>> If I'm inserting millions of documents, how do I control docs/update? E.g.
>> if there are 50K docs per file, I'm thinking that I should probably code up
>> my own DataSource that allows me to stipulate docs/update. Like say, 100
>> instead of 50K. Does this make sense?
>>
>> Mike
>>
>>
>> Noble Paul നോബിള് नोब्ळ् wrote:
>>>
>>> hi ,
>>> You have not registered any datasources . the second entity needs a
>>> datasource.
>>> Remove the dataSource="null" and add a name for the second entity
>>> (good practice). No need for baseDir attribute for second entity .
>>> See the modified xml added below
>>> --Noble
>>>
>>>
>>>
>>>
>>> >> newerThan="'NOW-10DAYS'" recursive="true" rootEntity="false"
>>> dataSource="null" baseDir="/san/tomcat-services/solr-medline">
>>> >> forEach="/MedlineCitation"
>>> url="${f.fileAbsolutePath}" >
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Jun 24, 2008 at 6:39 AM, mike segv <[EMAIL PROTECTED]> wrote:
>>>>
>>>> I'm trying to use the fileListEntityProcessor to add some xml documents
>>>> to a
>>>> solr index. I'm running a nightly version of solr-1.3 with SOLR-469 and
>>>> SOLR-563. I've been able to successfuly run the slashdot httpDataSource
>>>> example. My data-config.xml file loads without errors. When I attempt
>>>> the
>>>> full-import command I get the exception below. Thanks for any help.
>>>>
>>>> Mike
>>>>
>>>> WARNING: No lockType configured for
>>>> /san/tomcat-services/solr-medline/solr/data/index/ assuming 'simple'
>>>> Jun 23, 2008 7:59:49 PM org.apache.solr.handler.dataimport.DataImporter
>>>> doFullImport
>>>> SEVERE: Full Import failed
>>>> java.lang.RuntimeException: java.lang.NullPointerException
>>>>at
>>>> org.apache.solr.handler.dataimport.XPathRecordReader.streamRecords(XPathRecordReader.java:97)
>>>>at
>>>> org.apache.solr.handler.dataimport.XPathEntityProcessor.initQuery(XPathEntityProcessor.java:212)
>>>>at
>>>> org.apache.solr.handler.dataimport.XPathEntityProcessor.fetchNextRow(XPathEntityProcessor.java:166)
>>>>at
>>>> org.apache.solr.handler.dataimport.XPathEntityProcessor.nextRow(XPathEntityProcessor.java:149)
>>>>at
>>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:286)
>>>>at
>>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:312)
>>>>at
>>>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:179)
>>>>at
>>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:140)
>>>>at
>>>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:335)
>>>>at
>>>> org.apache.solr.handler.dataimport.DataImporter.rumCmd(DataImporter.java:386)
>>>>at
>>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>>>> Caused by: java.lang.NullPointerException
>>>>at java.io.Reader.(Reader.java:61)
>>>>at java.io.BufferedReader.(BufferedReader.java:76)
>>>>at com.bea.xml.stream.MXParser.checkForXMLDecl(MXParser.java:775)
>>>>at com.bea.xml.stream.MXParser.setInput(MXParser.java:806)
>>>> at
>>>> com.bea.xml.stream.MXParserFactory.createXMLStreamReader(MXParserFactory.java:261)
>>>>at
>>>> org.apache.solr.handler.dataimport.XPathRecordReader.streamRecords(XPathRecordReader.java:93)
>>>>... 10 more
>>>>
>>>> Here is my data-config:
>>>>
>>>>
>>>>
>>>> >>> newerThan="'NOW-10DAYS'" recursive="true" rootEntity="false"
>>>> dataSource="null" baseDi
>>>> r="/san/tomcat-services/solr-medline">
>>>> >>> url="${f.fileAbsolutePath}" dataSource="null">
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> And a snippet from an xml file:
>>>>
>>>> 12236137
>>>>
>>>> 1980
>>>> 01
>>>> 03
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Attempting-dataimport-using-FileListEntityProcessor-tp18081671p18081671.html
>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Attempting-dataimport-using-FileListEntityProcessor-tp18081671p18083747.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> --Noble Paul
>
--
--Noble Paul
Re: Attempting dataimport using FileListEntityProcessor
Hi Mike,
Just curious to know the use-case here. Why do you want to limit updates to
100 instead of importing all documents?
On Tue, Jun 24, 2008 at 10:23 AM, mike segv <[EMAIL PROTECTED]> wrote:
>
> That fixed it.
>
> If I'm inserting millions of documents, how do I control docs/update? E.g.
> if there are 50K docs per file, I'm thinking that I should probably code up
> my own DataSource that allows me to stipulate docs/update. Like say, 100
> instead of 50K. Does this make sense?
>
> Mike
>
>
> Noble Paul നോബിള് नोब्ळ् wrote:
> >
> > hi ,
> > You have not registered any datasources . the second entity needs a
> > datasource.
> > Remove the dataSource="null" and add a name for the second entity
> > (good practice). No need for baseDir attribute for second entity .
> > See the modified xml added below
> > --Noble
> >
> >
> >
> >
> > > newerThan="'NOW-10DAYS'" recursive="true" rootEntity="false"
> > dataSource="null" baseDir="/san/tomcat-services/solr-medline">
> > > forEach="/MedlineCitation"
> > url="${f.fileAbsolutePath}" >
> >
> >
> >
> >
> >
> >
> > On Tue, Jun 24, 2008 at 6:39 AM, mike segv <[EMAIL PROTECTED]> wrote:
> >>
> >> I'm trying to use the fileListEntityProcessor to add some xml documents
> >> to a
> >> solr index. I'm running a nightly version of solr-1.3 with SOLR-469 and
> >> SOLR-563. I've been able to successfuly run the slashdot httpDataSource
> >> example. My data-config.xml file loads without errors. When I attempt
> >> the
> >> full-import command I get the exception below. Thanks for any help.
> >>
> >> Mike
> >>
> >> WARNING: No lockType configured for
> >> /san/tomcat-services/solr-medline/solr/data/index/ assuming 'simple'
> >> Jun 23, 2008 7:59:49 PM org.apache.solr.handler.dataimport.DataImporter
> >> doFullImport
> >> SEVERE: Full Import failed
> >> java.lang.RuntimeException: java.lang.NullPointerException
> >>at
> >>
> org.apache.solr.handler.dataimport.XPathRecordReader.streamRecords(XPathRecordReader.java:97)
> >>at
> >>
> org.apache.solr.handler.dataimport.XPathEntityProcessor.initQuery(XPathEntityProcessor.java:212)
> >>at
> >>
> org.apache.solr.handler.dataimport.XPathEntityProcessor.fetchNextRow(XPathEntityProcessor.java:166)
> >>at
> >>
> org.apache.solr.handler.dataimport.XPathEntityProcessor.nextRow(XPathEntityProcessor.java:149)
> >>at
> >>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:286)
> >>at
> >>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:312)
> >>at
> >>
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:179)
> >>at
> >>
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:140)
> >>at
> >>
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:335)
> >>at
> >>
> org.apache.solr.handler.dataimport.DataImporter.rumCmd(DataImporter.java:386)
> >>at
> >>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
> >> Caused by: java.lang.NullPointerException
> >>at java.io.Reader.(Reader.java:61)
> >>at java.io.BufferedReader.(BufferedReader.java:76)
> >>at com.bea.xml.stream.MXParser.checkForXMLDecl(MXParser.java:775)
> >>at com.bea.xml.stream.MXParser.setInput(MXParser.java:806)
> >>at
> >>
> com.bea.xml.stream.MXParserFactory.createXMLStreamReader(MXParserFactory.java:261)
> >>at
> >>
> org.apache.solr.handler.dataimport.XPathRecordReader.streamRecords(XPathRecordReader.java:93)
> >>... 10 more
> >>
> >> Here is my data-config:
> >>
> >>
> >>
> >> >> newerThan="'NOW-10DAYS'" recursive="true" rootEntity="false"
> >> dataSource="null" baseDi
> >> r="/san/tomcat-services/solr-medline">
> >> >> url="${f.fileAbsolutePath}" dataSource="null">
> >>
> >>
> >>
> >>
> >>
> >>
> >> And a snippet from an xml file:
> >>
> >> 12236137
> >>
> >> 1980
> >> 01
> >> 03
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Attempting-dataimport-using-FileListEntityProcessor-tp18081671p18081671.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> >
> > --
> > --Noble Paul
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Attempting-dataimport-using-FileListEntityProcessor-tp18081671p18083747.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Regards,
Shalin Shekhar Mangar.
Re: Attempting dataimport using FileListEntityProcessor
I do want to import all documents. My understanding of the way things work,
correct me if I'm wrong, is that there can be a certain number of documents
included in a single atomic update. Instead of having all my 16 Million
documents be part of a single update (that could more easily fail being so
big), I was thinking that it would be better to be able to stipulate how
many docs are part of an update and my 16 Million doc import would consist
of 16M/100 updates.
Shalin Shekhar Mangar wrote:
>
> Hi Mike,
>
> Just curious to know the use-case here. Why do you want to limit updates
> to
> 100 instead of importing all documents?
>
> On Tue, Jun 24, 2008 at 10:23 AM, mike segv <[EMAIL PROTECTED]> wrote:
>
>>
>> That fixed it.
>>
>> If I'm inserting millions of documents, how do I control docs/update?
>> E.g.
>> if there are 50K docs per file, I'm thinking that I should probably code
>> up
>> my own DataSource that allows me to stipulate docs/update. Like say, 100
>> instead of 50K. Does this make sense?
>>
>> Mike
>>
>>
>> Noble Paul നോബിള് नोब्ळ् wrote:
>> >
>> > hi ,
>> > You have not registered any datasources . the second entity needs a
>> > datasource.
>> > Remove the dataSource="null" and add a name for the second entity
>> > (good practice). No need for baseDir attribute for second entity .
>> > See the modified xml added below
>> > --Noble
>> >
>> >
>> >
>> >
>> > > > newerThan="'NOW-10DAYS'" recursive="true" rootEntity="false"
>> > dataSource="null" baseDir="/san/tomcat-services/solr-medline">
>> > > > forEach="/MedlineCitation"
>> > url="${f.fileAbsolutePath}" >
>> >
>> >
>> >
>> >
>> >
>> >
>> > On Tue, Jun 24, 2008 at 6:39 AM, mike segv <[EMAIL PROTECTED]> wrote:
>> >>
>> >> I'm trying to use the fileListEntityProcessor to add some xml
>> documents
>> >> to a
>> >> solr index. I'm running a nightly version of solr-1.3 with SOLR-469
>> and
>> >> SOLR-563. I've been able to successfuly run the slashdot
>> httpDataSource
>> >> example. My data-config.xml file loads without errors. When I
>> attempt
>> >> the
>> >> full-import command I get the exception below. Thanks for any help.
>> >>
>> >> Mike
>> >>
>> >> WARNING: No lockType configured for
>> >> /san/tomcat-services/solr-medline/solr/data/index/ assuming 'simple'
>> >> Jun 23, 2008 7:59:49 PM
>> org.apache.solr.handler.dataimport.DataImporter
>> >> doFullImport
>> >> SEVERE: Full Import failed
>> >> java.lang.RuntimeException: java.lang.NullPointerException
>> >>at
>> >>
>> org.apache.solr.handler.dataimport.XPathRecordReader.streamRecords(XPathRecordReader.java:97)
>> >>at
>> >>
>> org.apache.solr.handler.dataimport.XPathEntityProcessor.initQuery(XPathEntityProcessor.java:212)
>> >>at
>> >>
>> org.apache.solr.handler.dataimport.XPathEntityProcessor.fetchNextRow(XPathEntityProcessor.java:166)
>> >>at
>> >>
>> org.apache.solr.handler.dataimport.XPathEntityProcessor.nextRow(XPathEntityProcessor.java:149)
>> >>at
>> >>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:286)
>> >>at
>> >>
>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:312)
>> >>at
>> >>
>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:179)
>> >>at
>> >>
>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:140)
>> >>at
>> >>
>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:335)
>> >>at
>> >>
>> org.apache.solr.handler.dataimport.DataImporter.rumCmd(DataImporter.java:386)
>> >>at
>> >>
>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>> >> Caused by: java.lang.NullPointerException
>> >>at java.io.Reader.(Reader.java:61)
>> >>at java.io.BufferedReader.(BufferedReader.java:76)
>> >>
Re: Attempting dataimport using FileListEntityProcessor
r.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:312)
> >> >>at
> >> >>
> >>
> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:179)
> >> >>at
> >> >>
> >>
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:140)
> >> >>at
> >> >>
> >>
> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:335)
> >> >>at
> >> >>
> >>
> org.apache.solr.handler.dataimport.DataImporter.rumCmd(DataImporter.java:386)
> >> >>at
> >> >>
> >>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
> >> >> Caused by: java.lang.NullPointerException
> >> >>at java.io.Reader.(Reader.java:61)
> >> >>at java.io.BufferedReader.(BufferedReader.java:76)
> >> >>at
> >> com.bea.xml.stream.MXParser.checkForXMLDecl(MXParser.java:775)
> >> >> at com.bea.xml.stream.MXParser.setInput(MXParser.java:806)
> >> >>at
> >> >>
> >>
> com.bea.xml.stream.MXParserFactory.createXMLStreamReader(MXParserFactory.java:261)
> >> >>at
> >> >>
> >>
> org.apache.solr.handler.dataimport.XPathRecordReader.streamRecords(XPathRecordReader.java:93)
> >> >>... 10 more
> >> >>
> >> >> Here is my data-config:
> >> >>
> >> >>
> >> >>
> >> >> >> >> newerThan="'NOW-10DAYS'" recursive="true" rootEntity="false"
> >> >> dataSource="null" baseDi
> >> >> r="/san/tomcat-services/solr-medline">
> >> >> >> >> url="${f.fileAbsolutePath}" dataSource="null">
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> And a snippet from an xml file:
> >> >>
> >> >> 12236137
> >> >>
> >> >> 1980
> >> >> 01
> >> >> 03
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> View this message in context:
> >> >>
> >>
> http://www.nabble.com/Attempting-dataimport-using-FileListEntityProcessor-tp18081671p18081671.html
> >> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >> >>
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > --Noble Paul
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Attempting-dataimport-using-FileListEntityProcessor-tp18081671p18083747.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >>
> >>
> >
> >
> > --
> > Regards,
> > Shalin Shekhar Mangar.
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Attempting-dataimport-using-FileListEntityProcessor-tp18081671p18095951.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Regards,
Shalin Shekhar Mangar.
