Janne,
You're correct on how the delta import works. You specify 3 queries:
- deletedPkQuery = query should return all "id"s (only) of items that were
deleted since the last run.
- deltaQuery = query should return all "id"s (only) of items that were
added/updated since the last run.
- deltaImportQuery = query should return full data for ONE row with "where
id='${dih.delta.id}'".
When DIH runs, it executes the first 2 queries and puts all of the returned
id's in memory in a Set or something. Then it does N selects on the
deltaImportQuery, executing the query once per id. This is maybe a good way to
do it if you're doing very frequent deltas and you only expect a small number
of changed documents per run, but I personally use the alternate way (as you
noted): http://wiki.apache.org/solr/DataImportHandlerDeltaQueryViaFullImport
I'm not sure why deltas were implemented this way. Possibly it was designed to
behave like some of our object-to-relational libraries? In any case, there are
2 ways to do deltas and you just have to take your pick based on what will work
best for your situation. I wouldn't consider the
"command=full-import&clean=false" method a workaround but just a different way
to tackle the same problem.
James Dyer
E-Commerce Systems
Ingram Content Group
(615) 213-4311
-----Original Message-----
From: janne mattila [mailto:jannepostilis...@gmail.com]
Sent: Tuesday, March 27, 2012 9:25 AM
To: solr-user@lucene.apache.org
Subject: dataImportHandler: delta query fetching data, not just ids?
It seems that delta import works in 2 steps, first query fetches the
ids of the modified entries, then second query fetches the actual
data.
<entity name="item" pk="ID"
query="select * from item"
deltaImportQuery="select * from item where
ID='${dataimporter.delta.id}'"
deltaQuery="select id from item where last_modified
> '${dataimporter.last_index_time}'">
<entity name="feature" pk="ITEM_ID"
query="select description as features from feature
where item_id='${item.ID}'">
</entity>
<entity name="item_category" pk="ITEM_ID, CATEGORY_ID"
query="select CATEGORY_ID from item_category where
ITEM_ID='${item.ID}'">
<entity name="category" pk="ID"
query="select description as cat from category
where id = '${item_category.CATEGORY_ID}'">
</entity>
</entity>
I am aware that there's a workaround:
http://wiki.apache.org/solr/DataImportHandlerDeltaQueryViaFullImport
But still, to clarify, and make sure I have up-to-date info how Solr works:
1. Is it possible to fetch the modified data with a single SQL query
using deltaImportQuery, as in:
deltaImportQuery="select * from item where last_modified >
'${dataimporter.last_index_time}'"?
2. If not - what's the reason delta import is implemented like it is?
Why split it in two queries? I would think having a single delta query
that fetches the data would be kind of an "obvious" design unless
there's something that calls for 2 separate queries...?
