On Mon, Dec 29, 2008 at 5:11 PM, Manupriya <manupriya.si...@gmail.com> wrote:
>
> Hi Shalin,
>
> I am using apache-solr-1.3.0 in my project. I thought it to be the latest
> release. Does it not support delta import?
It should, 1.3 of solr was the first realease of DIH. A lot of
features got added to DIH since then. The nightly build of DIH is
compatible with Solr 1.3. BTW 1.3 supports delta import.
>
> Thanks,
> Manu
>
>
> Shalin Shekhar Mangar wrote:
>>
>> Which version of Solr are you using? The deltaImportQuery feature was
>> added
>> recently, you'd need a nightly build for it to work.
>>
>> On Mon, Dec 29, 2008 at 4:57 PM, Manupriya
>> <manupriya.si...@gmail.com>wrote:
>>
>>>
>>> Hi Noble,
>>>
>>> I tried with the deltaImportQuery as well. But still I am getting the
>>> same
>>> exception on the server console -
>>>
>>> Dec 29, 2008 4:53:52 PM org.apache.solr.handler.dataimport.DataImporter
>>> doDeltaImport
>>>
>>> SEVERE: Delta Import Failed
>>> java.lang.NullPointerException
>>> at
>>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
>>> at
>>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
>>> at
>>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
>>> at
>>>
>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
>>> at
>>> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
>>> at
>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
>>> at
>>>
>>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
>>> at
>>>
>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
>>> at
>>>
>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>>>
>>> Another thing, I do have proper data in the DB.
>>>
>>> Thanks,
>>> Manu
>>>
>>>
>>> Noble Paul നോബിള് नोब्ळ् wrote:
>>> >
>>> > The same example with deltaImportQuery would look as follows
>>> > <dataConfig>
>>> > <dataSource driver="com.mysql.jdbc.Driver"
>>> > url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root"
>>> />
>>> > <document name="products">
>>> > <entity name="item" pk="ID" query="select * from item"
>>> > deltaQuery="select id from item where last_modified >
>>> > '${dataimporter.last_index_time}'"
>>> > deltaImportQuery="select * from item where ID=${dataimporter.delta.ID}"
>>> >>
>>> > <field column="NAME" name="name" />
>>> > <field column="NAME" name="nameSort" />
>>> > <field column="NAME" name="alphaNameSort" />
>>> >
>>> > <entity name="feature" pk="ITEM_ID"
>>> > query="select DESCRIPTION from FEATURE where
>>> > ITEM_ID='${item.ID}'"
>>> > deltaQuery="select ITEM_ID from FEATURE where
>>> > last_modified > '${dataimporter.last_index_time}'"
>>> > parentDeltaQuery="select ID from item where
>>> > ID=${feature.ITEM_ID}">
>>> > <field name="features" column="DESCRIPTION" />
>>> > </entity>
>>> >
>>> > <entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
>>> > query="select CATEGORY_ID from item_category where
>>> > ITEM_ID='${item.ID}'"
>>> > deltaQuery="select ITEM_ID, CATEGORY_ID from
>>> > item_category where last_modified > '${dataimporter.last_index_time}'"
>>> > parentDeltaQuery="select ID from item where
>>> > ID=${item_category.ITEM_ID}">
>>> > <entity name="category" pk="ID"
>>> > query="select DESCRIPTION from category where ID
>>> =
>>> > '${item_category.CATEGORY_ID}'"
>>> > deltaQuery="select ID from category where
>>> > last_modified > '${dataimporter.last_index_time}'"
>>> > parentDeltaQuery="select ITEM_ID, CATEGORY_ID
>>> from
>>> > item_category where CATEGORY_ID=${category.ID}">
>>> > <field column="description" name="cat" />
>>> > </entity>
>>> > </entity>
>>> > </entity>
>>> > </document>
>>> > </dataConfig>
>>> >
>>> > On Mon, Dec 29, 2008 at 3:30 PM, Noble Paul നോബിള് नोब्ळ्
>>> > <noble.p...@gmail.com> wrote:
>>> >> did you put in the data?
>>> >>
>>> >> On Mon, Dec 29, 2008 at 3:07 PM, Manupriya <manupriya.si...@gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> Hi Shalin,
>>> >>>
>>> >>> I am actually trying out the example provided at
>>> >>> http://wiki.apache.org/solr/DataImportHandler.
>>> >>>
>>> >>> My db-data-config.xml is as following -
>>> >>>
>>> >>> <dataConfig>
>>> >>> <dataSource driver="com.mysql.jdbc.Driver"
>>> >>> url="jdbc:mysql://localhost/solr_demo_db" user="root" password="root"
>>> />
>>> >>> <document name="products">
>>> >>> <entity name="item" pk="ID" query="select * from item"
>>> >>> deltaQuery="select id from item where last_modified >
>>> >>> '${dataimporter.last_index_time}'">
>>> >>> <field column="NAME" name="name" />
>>> >>> <field column="NAME" name="nameSort" />
>>> >>> <field column="NAME" name="alphaNameSort" />
>>> >>>
>>> >>> <entity name="feature" pk="ITEM_ID"
>>> >>> query="select DESCRIPTION from FEATURE where
>>> >>> ITEM_ID='${item.ID}'"
>>> >>> deltaQuery="select ITEM_ID from FEATURE where
>>> >>> last_modified > '${dataimporter.last_index_time}'"
>>> >>> parentDeltaQuery="select ID from item where
>>> >>> ID=${feature.ITEM_ID}">
>>> >>> <field name="features" column="DESCRIPTION" />
>>> >>> </entity>
>>> >>>
>>> >>> <entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
>>> >>> query="select CATEGORY_ID from item_category where
>>> >>> ITEM_ID='${item.ID}'"
>>> >>> deltaQuery="select ITEM_ID, CATEGORY_ID from
>>> >>> item_category where last_modified >
>>> '${dataimporter.last_index_time}'"
>>> >>> parentDeltaQuery="select ID from item where
>>> >>> ID=${item_category.ITEM_ID}">
>>> >>> <entity name="category" pk="ID"
>>> >>> query="select DESCRIPTION from category where
>>> ID
>>> >>> =
>>> >>> '${item_category.CATEGORY_ID}'"
>>> >>> deltaQuery="select ID from category where
>>> >>> last_modified > '${dataimporter.last_index_time}'"
>>> >>> parentDeltaQuery="select ITEM_ID, CATEGORY_ID
>>> >>> from
>>> >>> item_category where CATEGORY_ID=${category.ID}">
>>> >>> <field column="description" name="cat" />
>>> >>> </entity>
>>> >>> </entity>
>>> >>> </entity>
>>> >>> </document>
>>> >>> </dataConfig>
>>> >>>
>>> ---------------------------------------------------------------------------------------------------
>>> >>>
>>> >>> My DB structure is as following -
>>> >>>
>>> >>> DROP TABLE IF EXISTS solr_demo_db.item;
>>> >>> CREATE TABLE `item` (
>>> >>> `ID` int(2) NOT NULL auto_increment,
>>> >>> `name` varchar(100) default NULL,
>>> >>> `manu` varchar(20) default NULL,
>>> >>> `weight` varchar(20) default NULL,
>>> >>> `price` varchar(20) default NULL,
>>> >>> `popularity` varchar(2) default NULL,
>>> >>> `includes` varchar(10) default NULL,
>>> >>> `last_modified` datetime default NULL,
>>> >>> PRIMARY KEY (`ID`)
>>> >>> ) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=latin1;
>>> >>>
>>> >>> DROP TABLE IF EXISTS solr_demo_db.feature;
>>> >>> CREATE TABLE `feature` (
>>> >>> `DESCRIPTION` varchar(100) default NULL,
>>> >>> `ITEM_ID` int(2) NOT NULL,
>>> >>> `last_modified` datetime default NULL,
>>> >>> PRIMARY KEY (`ITEM_ID`)
>>> >>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>>> >>>
>>> >>> DROP TABLE IF EXISTS solr_demo_db.category;
>>> >>> CREATE TABLE `category` (
>>> >>> `ID` int(2) NOT NULL auto_increment,
>>> >>> `DESCRIPTION` varchar(100) default NULL,
>>> >>> `last_modified` date default NULL,
>>> >>> PRIMARY KEY (`ID`)
>>> >>> ) ENGINE=InnoDB AUTO_INCREMENT=3 DEFAULT CHARSET=latin1;
>>> >>>
>>> >>> DROP TABLE IF EXISTS solr_demo_db.item_category;
>>> >>> CREATE TABLE `item_category` (
>>> >>> `ITEM_ID` int(2) NOT NULL default '0',
>>> >>> `CATEGORY_ID` int(2) NOT NULL default '0',
>>> >>> `last_modified` date default NULL,
>>> >>> PRIMARY KEY (`ITEM_ID`,`CATEGORY_ID`)
>>> >>> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
>>> >>>
>>> >>> Am I doing something wrong here?
>>> >>>
>>> >>> Thanks,
>>> >>> Manu
>>> >>>
>>> >>>
>>> >>> Shalin Shekhar Mangar wrote:
>>> >>>>
>>> >>>> What does your data-config look like? especially the delta query
>>> part.
>>> >>>>
>>> >>>> On Mon, Dec 29, 2008 at 2:35 PM, Manupriya
>>> >>>> <manupriya.si...@gmail.com>wrote:
>>> >>>>
>>> >>>>>
>>> >>>>> Hi,
>>> >>>>>
>>> >>>>> I am using Delta DataImport feature to partially refresh the
>>> indexes.
>>> >>>>>
>>> >>>>> Note - full-import is working perfectly fine. But I need to do
>>> >>>>> delta-import
>>> >>>>> as I do not want to rebuild all the indexes.
>>> >>>>>
>>> >>>>> My database structure is same as mentioned in the example at
>>> >>>>> http://wiki.apache.org/solr/DataImportHandler.
>>> >>>>>
>>> >>>>> http://www.nabble.com/file/p21200498/example-schema.png
>>> >>>>>
>>> ---------------------------------------------------------------------------
>>> >>>>> The dataimport.properties has the value as -
>>> >>>>>
>>> >>>>> #Mon Dec 29 14:08:12 IST 2008
>>> >>>>> last_index_time=2008-12-29 14\:08\:12
>>> >>>>>
>>> ---------------------------------------------------------------------------
>>> >>>>> I have changed the last_modified for the 'tem' table as current
>>> >>>>> timestamp.
>>> >>>>> And when I query the DB, I get the following result -
>>> >>>>>
>>> >>>>> http://www.nabble.com/file/p21200498/data.jpg
>>> >>>>>
>>> >>>>>
>>> -------------------------------------------------------------------------------------------------
>>> >>>>>
>>> >>>>> But when I call delta data-import, it returns me the following
>>> >>>>> response -
>>> >>>>>
>>> >>>>> Server Response:
>>> >>>>>
>>> >>>>>
>>> {responseHeader={status=0,QTime=16},initArgs={defaults={config=db-data-config.xml}},command=delta-import,status=idle,importResponse=,statusMessages={},WARNING=This
>>> >>>>> response format is experimental. It is likely to change in the
>>> >>>>> future.}
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> ---------------------------------------------------------------------------------------------------
>>> >>>>> And when I query for data for '*:*', I get the following -
>>> >>>>> Note - Data returned by query doesnt reflect the correct timestamp
>>> >>>>> from
>>> >>>>> database.
>>> >>>>>
>>> >>>>> SolrDocument(1)
>>> >>>>> Field Name & Value: includes=12
>>> >>>>> Field Name & Value: features=[demo feature, demo feature 1]
>>> >>>>> Field Name & Value: price=12.0
>>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>> >>>>> Field Name & Value: manu=manu-12
>>> >>>>> Field Name & Value: sku=1
>>> >>>>> Field Name & Value: id=1
>>> >>>>> Field Name & Value: popularity=1
>>> >>>>> Field Name & Value: weight=12.0
>>> >>>>> SolrDocument(2)
>>> >>>>> Field Name & Value: includes=43
>>> >>>>> Field Name & Value: features=demo feature 2
>>> >>>>> Field Name & Value: price=10.0
>>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>> >>>>> Field Name & Value: manu=Demo - 12
>>> >>>>> Field Name & Value: sku=2
>>> >>>>> Field Name & Value: id=2
>>> >>>>> Field Name & Value: popularity=5
>>> >>>>> Field Name & Value: weight=12.0
>>> >>>>> SolrDocument(3)
>>> >>>>> Field Name & Value: includes=1
>>> >>>>> Field Name & Value: features=demo -3
>>> >>>>> Field Name & Value: price=1.0
>>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:54 IST 2008
>>> >>>>> Field Name & Value: manu=manu - 36
>>> >>>>> Field Name & Value: sku=3
>>> >>>>> Field Name & Value: id=3
>>> >>>>> Field Name & Value: popularity=1
>>> >>>>> Field Name & Value: weight=1.0
>>> >>>>> SolrDocument(4)
>>> >>>>> Field Name & Value: includes=2
>>> >>>>> Field Name & Value: features=demo - 4
>>> >>>>> Field Name & Value: price=2.0
>>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>> >>>>> Field Name & Value: manu=manu - 46
>>> >>>>> Field Name & Value: sku=4
>>> >>>>> Field Name & Value: id=4
>>> >>>>> Field Name & Value: popularity=2
>>> >>>>> Field Name & Value: weight=2.0
>>> >>>>> SolrDocument(5)
>>> >>>>> Field Name & Value: includes=3
>>> >>>>> Field Name & Value: features=demo - 5
>>> >>>>> Field Name & Value: price=3.0
>>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>> >>>>> Field Name & Value: manu=manu - 56
>>> >>>>> Field Name & Value: sku=5
>>> >>>>> Field Name & Value: id=5
>>> >>>>> Field Name & Value: popularity=3
>>> >>>>> Field Name & Value: weight=3.0
>>> >>>>> SolrDocument(6)
>>> >>>>> Field Name & Value: includes=4
>>> >>>>> Field Name & Value: features=demo - 66
>>> >>>>> Field Name & Value: price=4.0
>>> >>>>> Field Name & Value: timestamp=Mon Dec 29 13:57:55 IST 2008
>>> >>>>> Field Name & Value: manu=manu - 66
>>> >>>>> Field Name & Value: sku=6
>>> >>>>> Field Name & Value: id=6
>>> >>>>> Field Name & Value: popularity=4
>>> >>>>> Field Name & Value: weight=4.0
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> ------------------------------------------------------------------------------------------
>>> >>>>> My Java code that calls delta import is as follows -
>>> >>>>>
>>> >>>>> public static void main(String[] args) throws IOException,
>>> >>>>> SolrServerException {
>>> >>>>> CommonsHttpSolrServer server = new
>>> >>>>> CommonsHttpSolrServer("http://localhost:8983/solr";);
>>> >>>>> BinaryResponseParser parser = new BinaryResponseParser();
>>> >>>>> server.setParser(parser);
>>> >>>>>
>>> >>>>> ModifiableSolrParams params = new ModifiableSolrParams();
>>> >>>>> params.set("qt", "/dataimport");
>>> >>>>> params.set("command", "delta-import");
>>> >>>>> params.set("commit", "true");
>>> >>>>> params.set("wt","json");
>>> >>>>>
>>> >>>>> try{
>>> >>>>>
>>> >>>>> QueryResponse response = server.query(params);
>>> >>>>> server.commit();
>>> >>>>> System.out.println("Server Response: " + response);
>>> >>>>>
>>> >>>>> SolrQuery query = new SolrQuery("*:*");
>>> >>>>> QueryResponse results = server.query(query);
>>> >>>>>
>>> >>>>> SolrDocumentList list = results.getResults();
>>> >>>>>
>>> >>>>> int index = 1;
>>> >>>>> for (SolrDocument solrDocument : list) {
>>> >>>>>
>>> >>>>> System.out.println("SolrDocument("+index+")");
>>> >>>>> Iterator<Entry<String, Object>> iterator =
>>> >>>>> solrDocument.iterator();
>>> >>>>>
>>> >>>>> while(iterator.hasNext()){
>>> >>>>> System.out.println("Field Name & Value:
>>> >>>>> "+iterator.next());
>>> >>>>> }
>>> >>>>>
>>> >>>>> index++;
>>> >>>>> }
>>> >>>>>
>>> >>>>> }catch(Exception ex){
>>> >>>>>
>>> >>>>> System.out.println("Exception Occured:"+ex);
>>> >>>>> }
>>> >>>>>
>>> >>>>> }
>>> >>>>>
>>> ---------------------------------------------------------------------------
>>> >>>>>
>>> >>>>> Does SOLR do some kind of caching? I dont understand as why the
>>> >>>>> updated
>>> >>>>> values are not queried correctly from DB?
>>> >>>>>
>>> >>>>> There are no errors in the logs. But I can see an error on the
>>> server
>>> >>>>> console -
>>> >>>>>
>>> >>>>> SEVERE: Delta Import Failed
>>> >>>>> java.lang.NullPointerException
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getDeltaImportQuery(SqlEntityProcessor.java:153)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.getQuery(SqlEntityProcessor.java:125)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:73)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
>>> >>>>> at
>>> >>>>>
>>> org.apache.solr.handler.dataimport.DocBuilder.doDelta(DocBuilder.java:211)
>>> >>>>> at
>>> >>>>>
>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:133)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> org.apache.solr.handler.dataimport.DataImporter.doDeltaImport(DataImporter.java:359)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:388)
>>> >>>>> at
>>> >>>>>
>>> >>>>>
>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
>>> >>>>>
>>> >>>>>
>>> >>>>> What can be the reason for this error?
>>> >>>>>
>>> >>>>> Thanks,
>>> >>>>> Manu
>>> >>>>>
>>> >>>>> --
>>> >>>>> View this message in context:
>>> >>>>>
>>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200498.html
>>> >>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>> >>>>>
>>> >>>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> Regards,
>>> >>>> Shalin Shekhar Mangar.
>>> >>>>
>>> >>>>
>>> >>>
>>> >>> --
>>> >>> View this message in context:
>>> >>>
>>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21200786.html
>>> >>> Sent from the Solr - User mailing list archive at Nabble.com.
>>> >>>
>>> >>>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> --Noble Paul
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > --Noble Paul
>>> >
>>> >
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21201980.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/Delta-DataImport-is-not-picking-the-modified-value-in-DB-tp21200498p21202107.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>
--
--Noble Paul
