Re: Best way to check Solr index for completeness

2010-09-29 Thread Erick Erickson
primary key > with Solr id field. A variation of that is to calculate some kind of > unique > record hash and store it in the index.Then retrieve id and hash via > TermComponent and compare them with hash calculated on database record. > -- > View this message in context: > http

Re: Best way to check Solr index for completeness

2010-09-29 Thread Walter Underwood
t; under 1 sec. I still like the idea of using TermComponent and will use it > in the future if number of docs in the index will grow. Thanks for all > suggestions. > Dmitriy > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Best-way-to-check-Solr-index-fo

Re: Best way to check Solr index for completeness

2010-09-29 Thread dshvadskiy
ucene.472066.n3.nabble.com/Best-way-to-check-Solr-index-for-completeness-tp1598626p1603108.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Best way to check Solr index for completeness

2010-09-29 Thread dshvadskiy
Regenerating index is a slow operation due to limitation of the source systems. We run several complex SQL statements to generate 1 Solr document. Full reindex takes about 24 hours. -- View this message in context: http://lucene.472066.n3.nabble.com/Best-way-to-check-Solr-index-for

Re: Best way to check Solr index for completeness

2010-09-29 Thread dshvadskiy
TermComponent and compare them with hash calculated on database record. -- View this message in context: http://lucene.472066.n3.nabble.com/Best-way-to-check-Solr-index-for-completeness-tp1598626p1602597.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Best way to check Solr index for completeness

2010-09-29 Thread Peter Karich
How long does it take to get 1000 docs? Why not ensure this while indexing? I think besides your suggestion or the suggestion of Luke there is no other way... Regards, Peter. > Hello, > What would be the best way to check Solr index against original system > (Database) to make sure index is up t

Re: Best way to check Solr index for completeness

2010-09-28 Thread Dennis Gearon
owded' Laugh at http://www.yert.com/film.php --- On Tue, 9/28/10, dshvadskiy wrote: > From: dshvadskiy > Subject: Re: Best way to check Solr index for completeness > To: solr-user@lucene.apache.org > Date: Tuesday, September 28, 2010, 2:11 PM > > That will certainly work for

Re: Best way to check Solr index for completeness

2010-09-28 Thread Erick Erickson
gt; >> documents from Solr index via search (1000 docs at a time) is very slow. > >> Is > >> there a better way to do it? > >> > >> Thanks, > >> Dmitriy > >> > > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Best-way-to-check-Solr-index-for-completeness-tp1598626p1598733.html > Sent from the Solr - User mailing list archive at Nabble.com. >

Re: Best way to check Solr index for completeness

2010-09-28 Thread dshvadskiy
there a better way to do it? >> >> Thanks, >> Dmitriy >> > > -- View this message in context: http://lucene.472066.n3.nabble.com/Best-way-to-check-Solr-index-for-completeness-tp1598626p1598733.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Best way to check Solr index for completeness

2010-09-28 Thread Luke Crouch
Is there a 1:1 ratio of db records to solr documents? If so, couldn't you simply select the most recent updated record from the db and check to make sure the corresponding solr doc has the same timestamp? -L On Tue, Sep 28, 2010 at 3:48 PM, Dmitriy Shvadskiy wrote: > Hello, > What would be the b

Best way to check Solr index for completeness

2010-09-28 Thread Dmitriy Shvadskiy
Hello, What would be the best way to check Solr index against original system (Database) to make sure index is up to date? I can use Solr fields like Id and timestamp to check against appropriate fields in database. Our index currently contains over 2 mln documents across several cores. Pulling all