Which version of Solr are you using? The deltaImportQuery feature was added
recently, you'd need a nightly build for it to work.
On Mon, Dec 29, 2008 at 4:57 PM, Manupriya <manupriya.si...@gmail.com>wrote:
>
> Hi Noble,
>
> I tried with the deltaImportQuery as well. But still I am getting the same
> exception on the server console -
>
> Dec 29, 2008 4:53:52 PM org.apache.solr.handler.dataimport.DataImporter
> doDeltaImport
>
> SEVERE: Delta Import Failed
> java.lang.NullPointerException
> at
>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
> at
>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
> at
>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
> at
>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
> at
> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
> at
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
> at
>
> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
> at
>
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
> at
>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>
> Another thing, I do have proper data in the DB.
>
> Thanks,
> Manu
>
>
> Noble Paul നോബിള് नोब्ळ् wrote:
> >
> > The same example with deltaImportQuery would look as follows
> > <dataConfig>
> > <dataSource driver="com.mysql.jdbc.Driver"
> > url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root" />
> > <document name="products">
> > <entity name="item" pk="ID" query="select * from item"
> > deltaQuery="select id from item where last_modified >
> > '${dataimporter.last_index_time}'"
> > deltaImportQuery="select * from item where ID=${dataimporter.delta.ID}"
> >>
> > <field column="NAME" name="name" />
> > <field column="NAME" name="nameSort" />
> > <field column="NAME" name="alphaNameSort" />
> >
> > <entity name="feature" pk="ITEM_ID"
> > query="select DESCRIPTION from FEATURE where
> > ITEM_ID='${item.ID}'"
> > deltaQuery="select ITEM_ID from FEATURE where
> > last_modified > '${dataimporter.last_index_time}'"
> > parentDeltaQuery="select ID from item where
> > ID=${feature.ITEM_ID}">
> > <field name="features" column="DESCRIPTION" />
> > </entity>
> >
> > <entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
> > query="select CATEGORY_ID from item_category where
> > ITEM_ID='${item.ID}'"
> > deltaQuery="select ITEM_ID, CATEGORY_ID from
> > item_category where last_modified > '${dataimporter.last_index_time}'"
> > parentDeltaQuery="select ID from item where
> > ID=${item_category.ITEM_ID}">
> > <entity name="category" pk="ID"
> > query="select DESCRIPTION from category where ID =
> > '${item_category.CATEGORY_ID}'"
> > deltaQuery="select ID from category where
> > last_modified > '${dataimporter.last_index_time}'"
> > parentDeltaQuery="select ITEM_ID, CATEGORY_ID from
> > item_category where CATEGORY_ID=${category.ID}">
> > <field column="description" name="cat" />
> > </entity>
> > </entity>
> > </entity>
> > </document>
> > </dataConfig>
> >
> > On Mon, Dec 29, 2008 at 3:30 PM, Noble Paul നോബിള് नोब्ळ्
> > <noble.p...@gmail.com> wrote:
> >> did you put in the data?
> >>
> >> On Mon, Dec 29, 2008 at 3:07 PM, Manupriya <manupriya.si...@gmail.com>
> >> wrote:
> >>>
> >>> Hi Shalin,
> >>>
> >>> I am actually trying out the example provided at
> >>> http://wiki.apache.org/solr/DataImportHandler.
> >>>
> >>> My db-data-config.xml is as following -
> >>>
> >>> <dataConfig>
> >>> <dataSource driver="com.mysql.jdbc.Driver"
> >>> url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root"
> />
> >>> <document name="products">
> >>> <entity name="item" pk="ID" query="select * from item"
> >>> deltaQuery="select id from item where last_modified >
> >>> '${dataimporter.last_index_time}'">
> >>> <field column="NAME" name="name" />
> >>> <field column="NAME" name="nameSort" />
> >>> <field column="NAME" name="alphaNameSort" />
> >>>
> >>> <entity name="feature" pk="ITEM_ID"
> >>> query="select DESCRIPTION from FEATURE where
> >>> ITEM_ID='${item.ID}'"
> >>> deltaQuery="select ITEM_ID from FEATURE where
> >>> last_modified > '${dataimporter.last_index_time}'"
> >>> parentDeltaQuery="select ID from item where
> >>> ID=${feature.ITEM_ID}">
> >>> <field name="features" column="DESCRIPTION" />
> >>> </entity>
> >>>
> >>> <entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
> >>> query="select CATEGORY_ID from item_category where
> >>> ITEM_ID='${item.ID}'"
> >>> deltaQuery="select ITEM_ID, CATEGORY_ID from
> >>> item_category where last_modified > '${dataimporter.last_index_time}'"
> >>> parentDeltaQuery="select ID from item where
> >>> ID=${item_category.ITEM_ID}">
> >>> <entity name="category" pk="ID"
> >>> query="select DESCRIPTION from category where ID
> >>> =
> >>> '${item_category.CATEGORY_ID}'"
> >>> deltaQuery="select ID from category where
> >>> last_modified > '${dataimporter.last_index_time}'"
> >>> parentDeltaQuery="select ITEM_ID, CATEGORY_ID
> >>> from
> >>> item_category where CATEGORY_ID=${category.ID}">
> >>> <field column="description" name="cat" />
> >>> </entity>
> >>> </entity>
> >>> </entity>
> >>> </document>
> >>> </dataConfig>
> >>>
> ---------------------------------------------------------------------------------------------------
> >>>
> >>> My DB structure is as following -
> >>>
> >>> DROP TABLE IF EXISTS solr_demo_db.item;
> >>> CREATE TABLE `item` (
> >>> `ID` int(2) NOT NULL auto_increment,
> >>> `name` varchar(100) default NULL,
> >>> `manu` varchar(20) default NULL,
> >>> `weight` varchar(20) default NULL,
> >>> `price` varchar(20) default NULL,
> >>> `popularity` varchar(2) default NULL,
> >>> `includes` varchar(10) default NULL,
> >>> `last_modified` datetime default NULL,
> >>> PRIMARY KEY (`ID`)
> >>> ) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
> >>>
> >>> DROP TABLE IF EXISTS solr_demo_db.feature;
> >>> CREATE TABLE `feature` (
> >>> `DESCRIPTION` varchar(100) default NULL,
> >>> `ITEM_ID` int(2) NOT NULL,
> >>> `last_modified` datetime default NULL,
> >>> PRIMARY KEY (`ITEM_ID`)
> >>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
> >>>
> >>> DROP TABLE IF EXISTS solr_demo_db.category;
> >>> CREATE TABLE `category` (
> >>> `ID` int(2) NOT NULL auto_increment,
> >>> `DESCRIPTION` varchar(100) default NULL,
> >>> `last_modified` date default NULL,
> >>> PRIMARY KEY (`ID`)
> >>> ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
> >>>
> >>> DROP TABLE IF EXISTS solr_demo_db.item_category;
> >>> CREATE TABLE `item_category` (
> >>> `ITEM_ID` int(2) NOT NULL default '0',
> >>> `CATEGORY_ID` int(2) NOT NULL default '0',
> >>> `last_modified` date default NULL,
> >>> PRIMARY KEY (`ITEM_ID`,`CATEGORY_ID`)
> >>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
> >>>
> >>> Am I doing something wrong here?
> >>>
> >>> Thanks,
> >>> Manu
> >>>
> >>>
> >>> Shalin Shekhar Mangar wrote:
> >>>>
> >>>> What does your data-config look like? especially the delta query part.
> >>>>
> >>>> On Mon, Dec 29, 2008 at 2:35 PM, Manupriya
> >>>> <manupriya.si...@gmail.com>wrote:
> >>>>
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> I am using Delta DataImport feature to partially refresh the indexes.
> >>>>>
> >>>>> Note - full-import is working perfectly fine. But I need to do
> >>>>> delta-import
> >>>>> as I do not want to rebuild all the indexes.
> >>>>>
> >>>>> My database structure is same as mentioned in the example at
> >>>>> http://wiki.apache.org/solr/DataImportHandler.
> >>>>>
> >>>>> http://www.nabble.com/file/p21200498/example-schema.png
> >>>>>
> ---------------------------------------------------------------------------
> >>>>> The dataimport.properties has the value as -
> >>>>>
> >>>>> #Mon Dec 29 14:08:12 IST 2008
> >>>>> last_index_time=2008-12-29 14\:08\:12
> >>>>>
> ---------------------------------------------------------------------------
> >>>>> I have changed the last_modified for the 'tem' table as current
> >>>>> timestamp.
> >>>>> And when I query the DB, I get the following result -
> >>>>>
> >>>>> http://www.nabble.com/file/p21200498/data.jpg
> >>>>>
> >>>>>
> -------------------------------------------------------------------------------------------------
> >>>>>
> >>>>> But when I call delta data-import, it returns me the following
> >>>>> response -
> >>>>>
> >>>>> Server Response:
> >>>>>
> >>>>>
> {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
> >>>>> response format is experimental. It is likely to change in the
> >>>>> future.}
> >>>>>
> >>>>>
> >>>>>
> ---------------------------------------------------------------------------------------------------
> >>>>> And when I query for data for '*:*', I get the following -
> >>>>> Note - Data returned by query doesnt reflect the correct timestamp
> >>>>> from
> >>>>> database.
> >>>>>
> >>>>> SolrDocument(1)
> >>>>> Field Name & Value: includes=12
> >>>>> Field Name & Value: features=[demo feature, demo feature 1]
> >>>>> Field Name & Value: price=12.0
> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
> >>>>> Field Name & Value: manu=manu-12
> >>>>> Field Name & Value: sku=1
> >>>>> Field Name & Value: id=1
> >>>>> Field Name & Value: popularity=1
> >>>>> Field Name & Value: weight=12.0
> >>>>> SolrDocument(2)
> >>>>> Field Name & Value: includes=43
> >>>>> Field Name & Value: features=demo feature 2
> >>>>> Field Name & Value: price=10.0
> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
> >>>>> Field Name & Value: manu=Demo - 12
> >>>>> Field Name & Value: sku=2
> >>>>> Field Name & Value: id=2
> >>>>> Field Name & Value: popularity=5
> >>>>> Field Name & Value: weight=12.0
> >>>>> SolrDocument(3)
> >>>>> Field Name & Value: includes=1
> >>>>> Field Name & Value: features=demo -3
> >>>>> Field Name & Value: price=1.0
> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
> >>>>> Field Name & Value: manu=manu - 36
> >>>>> Field Name & Value: sku=3
> >>>>> Field Name & Value: id=3
> >>>>> Field Name & Value: popularity=1
> >>>>> Field Name & Value: weight=1.0
> >>>>> SolrDocument(4)
> >>>>> Field Name & Value: includes=2
> >>>>> Field Name & Value: features=demo - 4
> >>>>> Field Name & Value: price=2.0
> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
> >>>>> Field Name & Value: manu=manu - 46
> >>>>> Field Name & Value: sku=4
> >>>>> Field Name & Value: id=4
> >>>>> Field Name & Value: popularity=2
> >>>>> Field Name & Value: weight=2.0
> >>>>> SolrDocument(5)
> >>>>> Field Name & Value: includes=3
> >>>>> Field Name & Value: features=demo - 5
> >>>>> Field Name & Value: price=3.0
> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
> >>>>> Field Name & Value: manu=manu - 56
> >>>>> Field Name & Value: sku=5
> >>>>> Field Name & Value: id=5
> >>>>> Field Name & Value: popularity=3
> >>>>> Field Name & Value: weight=3.0
> >>>>> SolrDocument(6)
> >>>>> Field Name & Value: includes=4
> >>>>> Field Name & Value: features=demo - 66
> >>>>> Field Name & Value: price=4.0
> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
> >>>>> Field Name & Value: manu=manu - 66
> >>>>> Field Name & Value: sku=6
> >>>>> Field Name & Value: id=6
> >>>>> Field Name & Value: popularity=4
> >>>>> Field Name & Value: weight=4.0
> >>>>>
> >>>>>
> >>>>>
> ------------------------------------------------------------------------------------------
> >>>>> My Java code that calls delta import is as follows -
> >>>>>
> >>>>> public static void main(String[] args) throws IOException,
> >>>>> SolrServerException {
> >>>>> CommonsHttpSolrServer server = new
> >>>>> CommonsHttpSolrServer("http://localhost:8983/solr";);
> >>>>> BinaryResponseParser parser = new BinaryResponseParser();
> >>>>> server.setParser(parser);
> >>>>>
> >>>>> ModifiableSolrParams params = new ModifiableSolrParams();
> >>>>> params.set("qt", "/dataimport");
> >>>>> params.set("command", "delta-import");
> >>>>> params.set("commit", "true");
> >>>>> params.set("wt","json");
> >>>>>
> >>>>> try{
> >>>>>
> >>>>> QueryResponse response = server.query(params);
> >>>>> server.commit();
> >>>>> System.out.println("Server Response: " + response);
> >>>>>
> >>>>> SolrQuery query = new SolrQuery("*:*");
> >>>>> QueryResponse results = server.query(query);
> >>>>>
> >>>>> SolrDocumentList list = results.getResults();
> >>>>>
> >>>>> int index = 1;
> >>>>> for (SolrDocument solrDocument : list) {
> >>>>>
> >>>>> System.out.println("SolrDocument("+index+")");
> >>>>> Iterator<Entry<String, Object>> iterator =
> >>>>> solrDocument.iterator();
> >>>>>
> >>>>> while(iterator.hasNext()){
> >>>>> System.out.println("Field Name & Value:
> >>>>> "+iterator.next());
> >>>>> }
> >>>>>
> >>>>> index++;
> >>>>> }
> >>>>>
> >>>>> }catch(Exception ex){
> >>>>>
> >>>>> System.out.println("Exception Occured:"+ex);
> >>>>> }
> >>>>>
> >>>>> }
> >>>>>
> ---------------------------------------------------------------------------
> >>>>>
> >>>>> Does SOLR do some kind of caching? I dont understand as why the
> >>>>> updated
> >>>>> values are not queried correctly from DB?
> >>>>>
> >>>>> There are no errors in the logs. But I can see an error on the server
> >>>>> console -
> >>>>>
> >>>>> SEVERE: Delta Import Failed
> >>>>> java.lang.NullPointerException
> >>>>> at
> >>>>>
> >>>>>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
> >>>>> at
> >>>>>
> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
> >>>>> at
> >>>>>
> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
> >>>>> at
> >>>>>
> >>>>>
> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
> >>>>>
> >>>>>
> >>>>> What can be the reason for this error?
> >>>>>
> >>>>> Thanks,
> >>>>> Manu
> >>>>>
> >>>>> --
> >>>>> View this message in context:
> >>>>>
> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200498.html
> >>>>> Sent from the Solr - User mailing list archive at Nabble.com.
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> Regards,
> >>>> Shalin Shekhar Mangar.
> >>>>
> >>>>
> >>>
> >>> --
> >>> View this message in context:
> >>>
> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200786.html
> >>> Sent from the Solr - User mailing list archive at Nabble.com.
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> --Noble Paul
> >>
> >
> >
> >
> > --
> > --Noble Paul
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21201980.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
Regards,
Shalin Shekhar Mangar.
