Re: Need help with delta import

[thammegowda]

I was having similar problem with delta import. 

I am using solr 4.6 and making use of ${dih.last_index_time},
${dih.delta.xxx} shorter variable names.

I think the issue in previously discussed posts in the thread lies in
deltaQuery and deltaImportQuery.

if  deltaQuery="select *rowId* from MyTable" 
then deltaImportQuery="select .. from MyTable where
myId='${dih.delta.*rowId*}'".

Note that *rowId* variable name should be in same case in both select clause
of /deltaQuery/ and where clause of /deltaImportQuery /.



This following import configuration tested in solr 4.6 for delta import.


 

 
 
 









  


  
 




sureshadapa wrote
> I am using below configuration file and The problem is I do not see any
> solr documents committed into Solr Core Selector 'db'
> 
> When i run full-import,Is give me message.
> Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.
> Requests: 1, Fetched: 8, Skipped: 0, Processed: 0
> 
> When i run delta-import,It gives me message.
> Requests: 0, Fetched: 0, Skipped: 0, Processed: 0
> 
> solrconfig.xml
> ==
> 
> 4.4
> 
>  class="org.apache.solr.handler.dataimport.DataImportHandler">
> 
> 
>   
> 
> db1-data-config.xml
> 
> 
> 
>   
> 
> schema.xml
> 
> 
>
>  required="true" />
>  
>
> 
>
> 
>
> 
>
> 
>
>  

>  
> 
> solrp_id
> 
> 
> db1-data-config.xml
> =
> 
>  convertType="true" type="JdbcDataSource" driver="com.mysql.jdbc.Driver"
> url="jdbc:mysql://localhost:3306/suresh" user="suresh"
> password="suresh123"/>
> 
> 
> 
>  deltaImportQuery="select providersid,name,phone,email,smsno from
> providers where PROVIDERSID==${dih.delta.PROVIDERSID}"
> deltaquery="select providersid,name,phone,email,smsno from
> providers where modtime=='${dih.last_index_time}'">
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 


Changing ${dih.delta.PROVIDERSID} to ${dih.delta.providersid} should work. 






--
View this message in context: 
http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4117167.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Need help with delta import

[sureshadapa]

I am using below configuration file and The problem is I do not see any solr
documents committed into Solr Core Selector 'db'

When i run full-import,Is give me message.
Indexing completed. Added/Updated: 0 documents. Deleted 0 documents.
Requests: 1, Fetched: 8, Skipped: 0, Processed: 0

When i run delta-import,It gives me message.
Requests: 0, Fetched: 0, Skipped: 0, Processed: 0

solrconfig.xml
==
4.4


db1-data-config.xml

  

schema.xml



   
   
   
   
   
 
 solrp_id


db1-data-config.xml
=















--
View this message in context: 
http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4090999.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Need help with delta import

[umajava]

Sorry but I gave up on this issue. I could not resolve it.


On Tue, Sep 10, 2013 at 8:24 PM, suren [via Lucene] <
ml-node+s472066n4089093...@n3.nabble.com> wrote:

> Any update? I am also having the same issue. pls reply.
>
> This XML file does not appear to have any style information associated
> with it. The document tree is shown below.
> 
> 
> 0
> 7
> 
> 
> 
> db-data-config.xml
> 
> 
> delta-import
> idle
> 
> 
> 2
> 1
> 0
> 2013-09-10 07:46:34
> 2013-09-10 07:46:34
> 2013-09-10 07:46:35
> 2013-09-10 07:46:35
> 1
> 0
> 0:0:1.30
> 
> 
> This response format is experimental. It is likely to change in the
> future.
> 
> 
>
> --
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4089093.html
>  To unsubscribe from Need help with delta import, click 
> here
> .
> NAML
>




--
View this message in context: 
http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4089714.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Need help with delta import

[suren]

Any update? I am also having the same issue. pls reply.

This XML file does not appear to have any style information associated with
it. The document tree is shown below.


0
7



db-data-config.xml


delta-import
idle


2
1
0
2013-09-10 07:46:34
2013-09-10 07:46:34
2013-09-10 07:46:35
2013-09-10 07:46:35
1
0
0:0:1.30


This response format is experimental. It is likely to change in the future.





--
View this message in context: 
http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4089093.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Need help with delta import

[Xavier Pell]

This is absolutely a sintax error, I had the same problem, and with
dih.delta.id it solves all my problems. Thanks to god and the special
person who post the answer in this page.

You have to revise your sintax in queries for delta import and watch the
catalina (i use tomcat) log file for any errors.

Regards,

RE: Need help with delta import

[umajava]

I have changed to use dih.xx but still no luck. Even with dataimport or
dataimporter the query is able to fetch the delta records but they are not
able to commit to solr. Would there be any other reason why this would fail?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4027297.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Need help with delta import

[Dyer, James]

Shawn,

I think it only is a problem with "dih.delta.xxx" ... the longer version, 
"dataimport.delta.xxx" doesn't work.  This is coded in DocBuilder#doDelta and 
this line:

vri.addNamespace(ConfigNameConstants.IMPORTER_NS_SHORT + ".delta", map);

There is no additional line for:

vri.addNamespace(ConfigNameConstants.IMPORTER_NS + ".delta", map);

This is how it is in 3.5, 4.0 and Trunk, which are all the versions I have been 
testing against.

However, at some point someone did depreciate the longer "dataimporter.", so it 
would be safer to change to "dih." everywhere you use it.  I had thought about 
removing the longer one for 4.0 and then figured most people wouldn't realize 
it had been depreciated (there's no warning logged, etc) and I didn't want to 
give people unnecessary problems.  I guess there is no stopping it being 
removed in 5.0 though.

James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311


-Original Message-
From: Shawn Heisey [mailto:s...@elyograg.org] 
Sent: Friday, December 14, 2012 1:41 PM
To: solr-user@lucene.apache.org
Subject: Re: Need help with delta import

On 12/14/2012 11:39 AM, Dyer, James wrote:
> Try ${dih.delta.ID} instead of ${dataimporter.delta.id}.  Also use 
> ${dih.last_index_time} instead of ${dataimporter.last_index_time} .  I 
> noticed when updating the test cases that the wiki incorrectly used the 
> longer name but with all the versions I tested this on only the short name 
> works.  The wiki has since been changed.

James,

I use DIH for full Solr reindexes.  My dih config makes extensive use of 
${dataimporter.request.XXX} variables for my own custom parameters.  I 
am using branch_4x checked out yesterday on my dev machine, and I did a 
full reindex on that version, which worked.  Three questions: 1) Should 
I be using ${dih.request.XXX} instead?  2) Is the longer syntax going 
away?  3) What issues and/or docs would be good reading material?

Thanks,
Shawn

RE: Need help with delta import

[Swati Swoboda]

I am also confused, as I've been using dataimporter.* and not dih.* and it is 
working fine. 

-Original Message-
From: Shawn Heisey [mailto:s...@elyograg.org] 
Sent: Friday, December 14, 2012 2:41 PM
To: solr-user@lucene.apache.org
Subject: Re: Need help with delta import

On 12/14/2012 11:39 AM, Dyer, James wrote:
> Try ${dih.delta.ID} instead of ${dataimporter.delta.id}.  Also use 
> ${dih.last_index_time} instead of ${dataimporter.last_index_time} .  I 
> noticed when updating the test cases that the wiki incorrectly used the 
> longer name but with all the versions I tested this on only the short name 
> works.  The wiki has since been changed.

James,

I use DIH for full Solr reindexes.  My dih config makes extensive use of 
${dataimporter.request.XXX} variables for my own custom parameters.  I am using 
branch_4x checked out yesterday on my dev machine, and I did a full reindex on 
that version, which worked.  Three questions: 1) Should I be using 
${dih.request.XXX} instead?  2) Is the longer syntax going away?  3) What 
issues and/or docs would be good reading material?

Thanks,
Shawn

Re: Need help with delta import

[Shawn Heisey]

On 12/14/2012 11:39 AM, Dyer, James wrote:

Try ${dih.delta.ID} instead of ${dataimporter.delta.id}.  Also use 
${dih.last_index_time} instead of ${dataimporter.last_index_time} .  I noticed 
when updating the test cases that the wiki incorrectly used the longer name but 
with all the versions I tested this on only the short name works.  The wiki has 
since been changed.


James,

I use DIH for full Solr reindexes.  My dih config makes extensive use of 
${dataimporter.request.XXX} variables for my own custom parameters.  I 
am using branch_4x checked out yesterday on my dev machine, and I did a 
full reindex on that version, which worked.  Three questions: 1) Should 
I be using ${dih.request.XXX} instead?  2) Is the longer syntax going 
away?  3) What issues and/or docs would be good reading material?


Thanks,
Shawn

RE: Need help with delta import

[Dyer, James]

Try ${dih.delta.ID} instead of ${dataimporter.delta.id}.  Also use 
${dih.last_index_time} instead of ${dataimporter.last_index_time} .  I noticed 
when updating the test cases that the wiki incorrectly used the longer name but 
with all the versions I tested this on only the short name works.  The wiki has 
since been changed.

James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311


-Original Message-
From: umajava [mailto:umaj...@gmail.com] 
Sent: Friday, December 14, 2012 11:27 AM
To: solr-user@lucene.apache.org
Subject: RE: Need help with delta import

I tried full import and then delta import but still the issue is same.









--
View this message in context: 
http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4027066.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Need help with delta import

[umajava]

I tried full import and then delta import but still the issue is same.









--
View this message in context: 
http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4027066.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Need help with delta import

[umajava]

Thanks, but this didn't help either. Documents are not getting commited.

3202012-12-14 16:57:252012-12-14 16:57:252012-12-14
16:57:252012-12-14 16:57:25200:0:0.140

Should I do full import again as I have included email and fname in the
query and start again?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4027058.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Need help with delta import

[Swati Swoboda]

If I am not mistaken, it's suppose to be "dataimporter.delta.ID" and 
"dataimporter.last_index_time" You are using dataimport.delta.ID and 
dataimport.last_index_time

http://wiki.apache.org/solr/DataImportHandlerDeltaQueryViaFullImport



-Original Message-
From: umajava [mailto:umaj...@gmail.com] 
Sent: Thursday, December 13, 2012 9:35 PM
To: solr-user@lucene.apache.org
Subject: RE: Need help with delta import

Thanks a lot for your reply.

I have made the changes but it still does not work. I still get the same 
results. Any other suggestions please?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4026910.html
Sent from the Solr - User mailing list archive at Nabble.com.

RE: Need help with delta import

[Dyer, James]

Its surprising that your full import is working for you.  Both your "query" and 
your "deltaImportQuery" have:

SELECT ID FROM...

...So both your full-import ("query" attr) and your delta-import 
("deltaImportQuery" attr) are only getting the ID field from your db.  
Shouldn't you be at least be getting "email" and "fname" to index also?  So by 
changing both these queries to something like:

SELECT ID, EMAIL, FNAME FROM...

...You should see these 3 fields come through after your full-import.  Then, 
after changing data in your rbdms and doing a delta you should see the data 
update.

Besides this, your log looks right:
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder collectDelta
O: Completed ModifiedRowKey for Entity: person rows obtained : 8

...so it looks like it was going to update 8 rows.  But seeing that your 
"deltaImportQuery" is only pulling back the ID, it couldn't possibly change the 
values for fields like "email" and "fname".

Make sense?

James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311


-Original Message-
From: umajava [mailto:umaj...@gmail.com] 
Sent: Thursday, December 06, 2012 8:59 PM
To: solr-user@lucene.apache.org
Subject: Need help with delta import

Hi,

I am trying to do delta import and I am not able to get it to work. How ever
full import does work. Could you please help me figure out what I am
missing?

data-config.xml file









dataimport.properties file

metadataObject.last_index_time=2012-09-20 11\:12\:47
person.last_index_time=2012-11-18 13\:54\:29
interval=10
port=8080
server=localhost
params=/select?qt\=/dataimport&command\=delta-import&clean\=false&commit\=true
webapp=solr
syncEnabled=1
last_index_time=2012-11-18 13\:54\:29
syncCores=coreHr,coreEn


log output

Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DataImporter
maybeReloadConfiguration
O: Loading DIH Configuration:
C://Software//apache-solr-4.0.0//apache-solr-4.0.0//Uma//db//db-data-config.xml
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DataImporter
loadDataConfig
O: Data Configuration loaded successfully
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DataImporter
doDeltaImport
O: Starting Delta Import
Dec-2012 02:49:24 org.apache.solr.core.SolrCore execute
O: [collection1] webapp=/solr path=/dataimport
params={commit=false&command=delta-import} status=0 QTime=16
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.SimplePropertiesWriter
readIndexerProperties
O: Read dataimport.properties
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder doDelta
O: Starting delta collection.
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder collectDelta
O: Running ModifiedRowKey() for Entity: person
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.JdbcDataSource$1 call
O: Creating a connection for entity person with URL:
jdbc:mysql://localhost/test
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.JdbcDataSource$1 call
O: Time taken for getConnection(): 125
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder collectDelta
O: Completed ModifiedRowKey for Entity: person rows obtained : 8
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder collectDelta
O: Completed DeletedRowKey for Entity: person rows obtained : 0
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder collectDelta
O: Completed parentDeltaQuery for Entity: person
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder doDelta
O: Delta Import completed successfully
Dec-2012 02:49:24 org.apache.solr.handler.dataimport.DocBuilder execute
O: Time taken = 0:0:0.156
Dec-2012 02:49:24 org.apache.solr.update.processor.LogUpdateProcessor finish
O: [collection1] webapp=/solr path=/dataimport
params={commit=false&command=delta-import} status=0 QTime=16 {} 0


Thanks,



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Need help with delta import

[umajava]

Ouput in the browser is

idle-9802012-12-07
03:15:362012-12-07 03:15:362012-12-07 03:15:362012-12-07 03:15:36800:0:0.32

I have tried all sorts of URL's to invoke the data import

http://localhost:8080/solr/dataimport?command=delta-import
http://localhost:8080/solr/dataimport?command=delta-import&commit=false
http://localhost:8080/solr/dataimport?command=delta-import&commit=true
http://localhost:8080/solr/dataimport?command=delta-import&commit=true&debug=true
http://localhost:8080/solr/dataimport?command=delta-import&commit=true&debug=true&verbose=true




--
View this message in context: 
http://lucene.472066.n3.nabble.com/Need-help-with-delta-import-tp4025003p4025004.html
Sent from the Solr - User mailing list archive at Nabble.com.