Re: searching while importing

[Gora Mohanty]

On Thu, Oct 14, 2010 at 4:08 AM, Shawn Heisey  wrote:
>  If you are using the DataImportHandler, you will not be able to search new
> data until the full-import or delta-import is complete and the update is
> committed.  When I do a full reindex, it takes about 5 hours, and until it
> is finished, I cannot search it.
>
> I have not tried to issue a manual commit in the middle of an import to see
> whether that makes data inserted up to that point searchable, but I would
> not expect that to work.
[...]

Just as a data point, we have done this, and yes it is possible to do a commit
in the middle of an import, and have the documents that have already been
indexed be available for search.

Regards,
Gora

Re: searching while importing

[Shawn Heisey]

 If I haven't deleted the index for some reason before doing the full 
import, then I can search the old data.


On 10/13/2010 4:41 PM, Tri Nguyen wrote:

Hi,

As long as I can search on the current ("older") index while importing, I'm
good.  I've tested this and I can search the older index while data-importing
the newer index.

So you can search the older index in your 5 hour wait?

Re: searching while importing

[朱炎詹]

You can build 2 similiar Solr cores. One for service & one for importing. 
When the importing is done, you can do either MERGE or SWAP actions, 
depending on how you put your data on these 2 cores.



- Original Message - 
From: "Tri Nguyen" 

To: 
Sent: Thursday, October 14, 2010 5:51 AM
Subject: searching while importing


Hi,

Can I perform searches against the index while it is being imported?

Does importing add 1 document at a time or will solr make a temporary index 
and

switch to that index when indexing is done?

Thanks,

Tri






%<&b6G$J0T.'$$'d(l/f,r!C
Checked by AVG - www.avg.com
Version: 9.0.862 / Virus Database: 271.1.1/3193 - Release Date: 10/13/10 
02:37:00

Re: searching while importing

[Ken Stanley]

On Wed, Oct 13, 2010 at 6:38 PM, Shawn Heisey  wrote:

>  If you are using the DataImportHandler, you will not be able to search new
> data until the full-import or delta-import is complete and the update is
> committed.  When I do a full reindex, it takes about 5 hours, and until it
> is finished, I cannot search it.
>
> This is not true; when I use the DIH to do a full-import, I (and my team)
are still able to search on the already-indexed data that exists.


> I have not tried to issue a manual commit in the middle of an import to see
> whether that makes data inserted up to that point searchable, but I would
> not expect that to work.
>
> If you set the autoCommit properties maxDocs and maxTime to reasonable
values, then once those limits are reached, I suspect that SOLR would commit
and continue indexing; however, I have not had the chance to use those
features in solrconfig.xml.


> If you need this kind of functionality, you may need to change your build
> system so that a full import clears the index manually and then does a
> series of delta-import batches.
>
> The only time I've had an issue with being able to search while indexing is
when my DIH had mis-configuration that caused the import to finish without
indexing anything, thus wiping out my data. Aside of that, I continually
index and search at the same time almost every day (using 1.4.1).


>
>
> On 10/13/2010 3:51 PM, Tri Nguyen wrote:
>
>> Hi,
>>  Can I perform searches against the index while it is being imported?
>>  Does importing add 1 document at a time or will solr make a temporary
>> index and
>> switch to that index when indexing is done?
>>  Thanks,
>>  Tri
>>
>
>

Re: searching while importing

[Shawn Heisey]

 If I haven't deleted the index for some reason before doing the full 
import, then I can search the old data.



On 10/13/2010 4:41 PM, Tri Nguyen wrote:

Hi,

As long as I can search on the current ("older") index while importing, I'm
good.  I've tested this and I can search the older index while data-importing
the newer index.

So you can search the older index in your 5 hour wait?

Re: searching while importing

[Tri Nguyen]

Hi,

As long as I can search on the current ("older") index while importing, I'm 
good.  I've tested this and I can search the older index while data-importing 
the newer index.

So you can search the older index in your 5 hour wait?

Thanks,

Tri





From: Shawn Heisey 
To: solr-user@lucene.apache.org
Sent: Wed, October 13, 2010 3:38:48 PM
Subject: Re: searching while importing

If you are using the DataImportHandler, you will not be able to search new data 
until the full-import or delta-import is complete and the update is committed.  
When I do a full reindex, it takes about 5 hours, and until it is finished, I 
cannot search it.

I have not tried to issue a manual commit in the middle of an import to see 
whether that makes data inserted up to that point searchable, but I would not 
expect that to work.

If you need this kind of functionality, you may need to change your build 
system 
so that a full import clears the index manually and then does a series of 
delta-import batches.


On 10/13/2010 3:51 PM, Tri Nguyen wrote:
> Hi,
>  Can I perform searches against the index while it is being imported?
>  Does importing add 1 document at a time or will solr make a temporary index 
>and
> switch to that index when indexing is done?
>  Thanks,
>  Tri

Re: searching while importing

[Shawn Heisey]

 If you are using the DataImportHandler, you will not be able to search 
new data until the full-import or delta-import is complete and the 
update is committed.  When I do a full reindex, it takes about 5 hours, 
and until it is finished, I cannot search it.


I have not tried to issue a manual commit in the middle of an import to 
see whether that makes data inserted up to that point searchable, but I 
would not expect that to work.


If you need this kind of functionality, you may need to change your 
build system so that a full import clears the index manually and then 
does a series of delta-import batches.



On 10/13/2010 3:51 PM, Tri Nguyen wrote:

Hi,
  
Can I perform searches against the index while it is being imported?
  
Does importing add 1 document at a time or will solr make a temporary index and

switch to that index when indexing is done?
  
Thanks,
  
Tri

searching while importing

[Tri Nguyen]

Hi,
 
Can I perform searches against the index while it is being imported?
 
Does importing add 1 document at a time or will solr make a temporary index and 
switch to that index when indexing is done?
 
Thanks,
 
Tri