recommendation for solr replication throttling

[Alu, Pino [CORP/US]]

Hello,

We are needing a recommendation for solr replication throttling.  What are your 
recommendations for maxWriteMBPerSec value?  Our indexes contain 18 locales and 
size for all indexes is 188GB and growing.
Also will replication throttling work with solr 4.10.3?

Thanks,

Pino Alu | HCL Commerce Administrator :: Emerson.com | Enterprise IT
Emerson | 8000 West Florissant Ave. | St. Louis | MO | 63136 | USA
T +1 314 553 1785
pino@emerson.com<mailto:pino@emerson.com>

Delivering Technology Solutions With Purpose

RE: Solr Replication

[Vadim Ivanov]

Using cdcr with new replica types be aware of 
https://issues.apache.org/jira/browse/SOLR-12057?focusedComm

Parallel indexing to both cluster might be an option as well
-- 
Vadim


> -Original Message-
> From: Bernd Fehling [mailto:bernd.fehl...@uni-bielefeld.de]
> Sent: Monday, January 07, 2019 11:10 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Replication
> 
> In SolrCloud there are Data Centers.
> Your Cluster 1 is DataCenter 1 and your Cluster 2 is Data Center 2.
> You can then use CDCR (Cross Data Center Replication).
> http://lucene.apache.org/solr/guide/7_0/cross-data-center-replication-
> cdcr.html
> 
> Nevertheless I would spend your Cluster 2 another 2 zookeeper instances.
> 
> Regards, Bernd
> 
> Am 07.01.19 um 06:39 schrieb Mannar mannan:
> > Hi All,
> >
> > I would like to configure master slave between two solr cloud clusters (for
> > failover). Below is the scenario
> >
> > Solr version : 7.0
> >
> > Cluster 1:
> > 3 zookeeper instances :   zk1, zk2, zk3
> > 2 solr instances : solr1, solr2
> >
> > Cluster 2:
> > 1 zookeeper instance : bkpzk1,
> > 1 solr instances : bkpsolr1, bkpsolr2
> >
> > Master / Slave :  solr1 / bkpsolr1
> >solr2 / bkpsolr2
> >
> > Is it possible to have master / slave replication configured for solr
> > instances running in cluster1 & cluster2 (for failover). Kindly let me know
> > the possibility.
> >

Re: Solr Replication

[Bernd Fehling]

In SolrCloud there are Data Centers.
Your Cluster 1 is DataCenter 1 and your Cluster 2 is Data Center 2.
You can then use CDCR (Cross Data Center Replication).
http://lucene.apache.org/solr/guide/7_0/cross-data-center-replication-cdcr.html

Nevertheless I would spend your Cluster 2 another 2 zookeeper instances.

Regards, Bernd

Am 07.01.19 um 06:39 schrieb Mannar mannan:

Hi All,

I would like to configure master slave between two solr cloud clusters (for
failover). Below is the scenario

Solr version : 7.0

Cluster 1:
3 zookeeper instances :   zk1, zk2, zk3
2 solr instances : solr1, solr2

Cluster 2:
1 zookeeper instance : bkpzk1,
1 solr instances : bkpsolr1, bkpsolr2

Master / Slave :  solr1 / bkpsolr1
   solr2 / bkpsolr2

Is it possible to have master / slave replication configured for solr
instances running in cluster1 & cluster2 (for failover). Kindly let me know
the possibility.

Solr Replication

[Mannar mannan]

Hi All,

I would like to configure master slave between two solr cloud clusters (for
failover). Below is the scenario

Solr version : 7.0

Cluster 1:
3 zookeeper instances :   zk1, zk2, zk3
2 solr instances : solr1, solr2

Cluster 2:
1 zookeeper instance : bkpzk1,
1 solr instances : bkpsolr1, bkpsolr2

Master / Slave :  solr1 / bkpsolr1
  solr2 / bkpsolr2

Is it possible to have master / slave replication configured for solr
instances running in cluster1 & cluster2 (for failover). Kindly let me know
the possibility.

Re: Solr Replication being flaky (6.2.0)

[Erick Erickson]

bq: That's evidence enough for me to beat on our systems guys

Beat them with a stick. A large stick. If they insist that they can't
up these limits, I always push back hard with "why?". Usually
reluctance to up them is a misplaced efficiency argument

And I should have said you should see exceptions in your Solr logs if
this is really something that happens.

FWIW,
Erick

On Fri, Jan 19, 2018 at 11:07 AM, Pouliot, Scott
 wrote:
> Working on that now to see if it helps us out.  Solr process is NOT dying at 
> all.  Searches are still working as expected, but since we load balance 
> requestsif the master/slave are out of sync the search results vary.
>
> The advice is MUCH appreciated!
>
> -Original Message-
> From: Shawn Heisey [mailto:apa...@elyograg.org]
> Sent: Friday, January 19, 2018 1:49 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Replication being flaky (6.2.0)
>
> On 1/19/2018 11:27 AM, Shawn Heisey wrote:
>> On 1/19/2018 8:54 AM, Pouliot, Scott wrote:
>>> I do have a ticket in with our systems team to up the file handlers
>>> since I am seeing the "Too many files open" error on occasion on our
>>> prod servers.  Is this the setting you're referring to?  Found we
>>> were set to to 1024 using the "Ulimit" command.
>>
>> No, but that often needs increasing too.  I think you need to increase
>> the process limit even if that's not the cause of this particular problem.
>
> Had another thought.  Either of these limits can cause completely 
> unpredictable problems with Solr.  The open file limit could be the reason 
> for these issues, even if you're not actually hitting the process limit.  As 
> I mentioned before, I would expect a process limit to cause Solr to kill 
> itself, and your other messages don't mention problems like that.
>
> The scale of your Solr installation indicates that you should greatly 
> increase both limits on all of your Solr servers.
>
> Thanks,
> Shawn

RE: Solr Replication being flaky (6.2.0)

[Pouliot, Scott]

Working on that now to see if it helps us out.  Solr process is NOT dying at 
all.  Searches are still working as expected, but since we load balance 
requestsif the master/slave are out of sync the search results vary.  

The advice is MUCH appreciated!

-Original Message-
From: Shawn Heisey [mailto:apa...@elyograg.org] 
Sent: Friday, January 19, 2018 1:49 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Replication being flaky (6.2.0)

On 1/19/2018 11:27 AM, Shawn Heisey wrote:
> On 1/19/2018 8:54 AM, Pouliot, Scott wrote:
>> I do have a ticket in with our systems team to up the file handlers 
>> since I am seeing the "Too many files open" error on occasion on our 
>> prod servers.  Is this the setting you're referring to?  Found we 
>> were set to to 1024 using the "Ulimit" command.
> 
> No, but that often needs increasing too.  I think you need to increase 
> the process limit even if that's not the cause of this particular problem.

Had another thought.  Either of these limits can cause completely unpredictable 
problems with Solr.  The open file limit could be the reason for these issues, 
even if you're not actually hitting the process limit.  As I mentioned before, 
I would expect a process limit to cause Solr to kill itself, and your other 
messages don't mention problems like that.

The scale of your Solr installation indicates that you should greatly increase 
both limits on all of your Solr servers.

Thanks,
Shawn

Re: Solr Replication being flaky (6.2.0)

[Shawn Heisey]

On 1/19/2018 11:27 AM, Shawn Heisey wrote:

On 1/19/2018 8:54 AM, Pouliot, Scott wrote:
I do have a ticket in with our systems team to up the file handlers 
since I am seeing the "Too many files open" error on occasion on our 
prod servers.  Is this the setting you're referring to?  Found we were 
set to to 1024 using the "Ulimit" command.


No, but that often needs increasing too.  I think you need to increase 
the process limit even if that's not the cause of this particular problem.


Had another thought.  Either of these limits can cause completely 
unpredictable problems with Solr.  The open file limit could be the 
reason for these issues, even if you're not actually hitting the process 
limit.  As I mentioned before, I would expect a process limit to cause 
Solr to kill itself, and your other messages don't mention problems like 
that.


The scale of your Solr installation indicates that you should greatly 
increase both limits on all of your Solr servers.


Thanks,
Shawn

Re: Solr Replication being flaky (6.2.0)

[Shawn Heisey]

On 1/19/2018 8:54 AM, Pouliot, Scott wrote:

I do have a ticket in with our systems team to up the file handlers since I am seeing the "Too 
many files open" error on occasion on our prod servers.  Is this the setting you're referring 
to?  Found we were set to to 1024 using the "Ulimit" command.


No, but that often needs increasing too.  I think you need to increase 
the process limit even if that's not the cause of this particular problem.


Sounds like you're running on Linux, though ulimit is probably available 
on other platforms too.


If it's Linux, generally you must increase both the number of processes 
and the open file limit in /etc/security/limits.conf.  Trying to use the 
ulimit command generally doesn't work because the kernel has hard limits 
configured that ulimit can't budge.  If it's not Linux, then you'll need 
to consult with an expert in the OS you're running.


Again, assuming Linux, in the output of "ulimit -a" the value I'm 
talking about is the "-u" value -- "max user processes".  The following 
is the additions that I typically make to /etc/security/limits.conf, to 
increase both the open file limit and the process limit for the solr user:


solr    hard    nproc   61440
solr    soft    nproc   40960

solr    hard    nofile  65535
solr    soft    nofile  49151

Are you running into problems where Solr just disappears?  I would 
expect a process limit to generate OutOfMemoryError exceptions. When 
Solr is started with the included shell script, unless it's running with 
the foreground option, OOME will kill the Solr process.  We have issues 
to bring the OOME death option to running in the foreground, as well as 
when running on Windows.


Thanks,
Shawn

RE: Solr Replication being flaky (6.2.0)

[Pouliot, Scott]

That's evidence enough for me to beat on our systems guys to get these file 
handles upped and cross my fingers then!

-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com] 
Sent: Friday, January 19, 2018 1:18 PM
To: solr-user 
Subject: Re: Solr Replication being flaky (6.2.0)

"Could be", certainly. "Definitely is" is iffier ;)...

But the statement "If we restart the Solr service or optimize the core it seems 
to kick back in again.", especially the "optimize" bit (which, by the way you 
should do only if you have the capability of doing it periodically [1]) is some 
evidence that this may be in the vicinity. One of the effects of an optimize is 
to merge your segments files from N to 1. So say you have 10 segments. Each one 
of those may consist of 10-15 individual files, all of which are held open. So 
you'd go from 150 open file handles to 15..

https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flucidworks.com%2F2017%2F10%2F13%2Fsegment-merging-deleted-documents-optimize-may-bad%2F&data=02%7C01%7CScott.Pouliot%40peoplefluent.com%7Cc2912861f58248e3a92808d55f690eb8%7C8b16fb62c78448b6aba889567990e7fe%7C1%7C0%7C636519827178716698&sdata=DxnChrfyTbRDjB7HzqpOE%2BvOJRIxdnrXVCIyfoMjJPU%3D&reserved=0

Best,
Erick

On Fri, Jan 19, 2018 at 9:32 AM, Pouliot, Scott 
 wrote:
> Erick,
>
> Thanks!  Could these settings be toying with replication?  Solr itself seems 
> to be working like a champ, except when things get out of sync.
>
> Scott
>
> -Original Message-
> From: Erick Erickson [mailto:erickerick...@gmail.com]
> Sent: Friday, January 19, 2018 12:27 PM
> To: solr-user 
> Subject: Re: Solr Replication being flaky (6.2.0)
>
> Scott:
>
> We usually recommend setting files and processes very, very high. Like 65K 
> high. Or unlimited if you can.
>
> Plus max user processes should also be bumped very high as well, like 65K as 
> well.
>
> Plus max memory and virtual memory should be unlimited.
>
> We've included warnings at startup for open files and processes, see 
> SOLR-11703
>
> Best,
> Erick
>
> On Fri, Jan 19, 2018 at 7:54 AM, Pouliot, Scott 
>  wrote:
>> I do have a ticket in with our systems team to up the file handlers since I 
>> am seeing the "Too many files open" error on occasion on our prod servers.  
>> Is this the setting you're referring to?  Found we were set to to 1024 using 
>> the "Ulimit" command.
>>
>> -Original Message-
>> From: Shawn Heisey [mailto:apa...@elyograg.org]
>> Sent: Friday, January 19, 2018 10:48 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Solr Replication being flaky (6.2.0)
>>
>> On 1/19/2018 7:50 AM, Pouliot, Scott wrote:
>>> So we're running Solr in a Master/Slave configuration (1 of each) and it 
>>> seems that the replication stalls or stops functioning every now and again. 
>>>  If we restart the Solr service or optimize the core it seems to kick back 
>>> in again.
>>>
>>> Anyone have any idea what might be causing this?  We do have a good amount 
>>> of cores on each server (@150 or so), but I have heard reports of a LOT 
>>> more than that in use.
>>
>> Have you increased the number of processes that the user running Solr is 
>> allowed to start?  Most operating systems limit the number of 
>> threads/processes a user can start to a low value like 1024.  With 150 
>> cores, particularly with background tasks like replication configured, 
>> chances are that Solr is going to need to start a lot of threads.  This is 
>> an OS setting that a lot of Solr admins end up needing to increase.
>>
>> I ran into the process limit on my servers and I don't have anywhere near 
>> 150 cores.
>>
>> The fact that restarting Solr gets it working again (at least
>> temporarily) would fit with a process limit being the problem.  I'm not 
>> guaranteeing that this is the problem, only saying that it fits.
>>
>> Thanks,
>> Shawn

Re: Solr Replication being flaky (6.2.0)

[Erick Erickson]

"Could be", certainly. "Definitely is" is iffier ;)...

But the statement "If we restart the Solr service or optimize the core
it seems to kick back in again.", especially the "optimize" bit
(which, by the way you should do only if you have the capability of
doing it periodically [1]) is some evidence that this may be in the
vicinity. One of the effects of an optimize is to merge your segments
files from N to 1. So say you have 10 segments. Each one of those may
consist of 10-15 individual files, all of which are held open. So
you'd go from 150 open file handles to 15..

https://lucidworks.com/2017/10/13/segment-merging-deleted-documents-optimize-may-bad/

Best,
Erick

On Fri, Jan 19, 2018 at 9:32 AM, Pouliot, Scott
 wrote:
> Erick,
>
> Thanks!  Could these settings be toying with replication?  Solr itself seems 
> to be working like a champ, except when things get out of sync.
>
> Scott
>
> -Original Message-
> From: Erick Erickson [mailto:erickerick...@gmail.com]
> Sent: Friday, January 19, 2018 12:27 PM
> To: solr-user 
> Subject: Re: Solr Replication being flaky (6.2.0)
>
> Scott:
>
> We usually recommend setting files and processes very, very high. Like 65K 
> high. Or unlimited if you can.
>
> Plus max user processes should also be bumped very high as well, like 65K as 
> well.
>
> Plus max memory and virtual memory should be unlimited.
>
> We've included warnings at startup for open files and processes, see 
> SOLR-11703
>
> Best,
> Erick
>
> On Fri, Jan 19, 2018 at 7:54 AM, Pouliot, Scott 
>  wrote:
>> I do have a ticket in with our systems team to up the file handlers since I 
>> am seeing the "Too many files open" error on occasion on our prod servers.  
>> Is this the setting you're referring to?  Found we were set to to 1024 using 
>> the "Ulimit" command.
>>
>> -Original Message-
>> From: Shawn Heisey [mailto:apa...@elyograg.org]
>> Sent: Friday, January 19, 2018 10:48 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Solr Replication being flaky (6.2.0)
>>
>> On 1/19/2018 7:50 AM, Pouliot, Scott wrote:
>>> So we're running Solr in a Master/Slave configuration (1 of each) and it 
>>> seems that the replication stalls or stops functioning every now and again. 
>>>  If we restart the Solr service or optimize the core it seems to kick back 
>>> in again.
>>>
>>> Anyone have any idea what might be causing this?  We do have a good amount 
>>> of cores on each server (@150 or so), but I have heard reports of a LOT 
>>> more than that in use.
>>
>> Have you increased the number of processes that the user running Solr is 
>> allowed to start?  Most operating systems limit the number of 
>> threads/processes a user can start to a low value like 1024.  With 150 
>> cores, particularly with background tasks like replication configured, 
>> chances are that Solr is going to need to start a lot of threads.  This is 
>> an OS setting that a lot of Solr admins end up needing to increase.
>>
>> I ran into the process limit on my servers and I don't have anywhere near 
>> 150 cores.
>>
>> The fact that restarting Solr gets it working again (at least
>> temporarily) would fit with a process limit being the problem.  I'm not 
>> guaranteeing that this is the problem, only saying that it fits.
>>
>> Thanks,
>> Shawn

RE: Solr Replication being flaky (6.2.0)

[Pouliot, Scott]

Erick,

Thanks!  Could these settings be toying with replication?  Solr itself seems to 
be working like a champ, except when things get out of sync.

Scott

-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com] 
Sent: Friday, January 19, 2018 12:27 PM
To: solr-user 
Subject: Re: Solr Replication being flaky (6.2.0)

Scott:

We usually recommend setting files and processes very, very high. Like 65K 
high. Or unlimited if you can.

Plus max user processes should also be bumped very high as well, like 65K as 
well.

Plus max memory and virtual memory should be unlimited.

We've included warnings at startup for open files and processes, see SOLR-11703

Best,
Erick

On Fri, Jan 19, 2018 at 7:54 AM, Pouliot, Scott 
 wrote:
> I do have a ticket in with our systems team to up the file handlers since I 
> am seeing the "Too many files open" error on occasion on our prod servers.  
> Is this the setting you're referring to?  Found we were set to to 1024 using 
> the "Ulimit" command.
>
> -Original Message-
> From: Shawn Heisey [mailto:apa...@elyograg.org]
> Sent: Friday, January 19, 2018 10:48 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Replication being flaky (6.2.0)
>
> On 1/19/2018 7:50 AM, Pouliot, Scott wrote:
>> So we're running Solr in a Master/Slave configuration (1 of each) and it 
>> seems that the replication stalls or stops functioning every now and again.  
>> If we restart the Solr service or optimize the core it seems to kick back in 
>> again.
>>
>> Anyone have any idea what might be causing this?  We do have a good amount 
>> of cores on each server (@150 or so), but I have heard reports of a LOT more 
>> than that in use.
>
> Have you increased the number of processes that the user running Solr is 
> allowed to start?  Most operating systems limit the number of 
> threads/processes a user can start to a low value like 1024.  With 150 cores, 
> particularly with background tasks like replication configured, chances are 
> that Solr is going to need to start a lot of threads.  This is an OS setting 
> that a lot of Solr admins end up needing to increase.
>
> I ran into the process limit on my servers and I don't have anywhere near 150 
> cores.
>
> The fact that restarting Solr gets it working again (at least
> temporarily) would fit with a process limit being the problem.  I'm not 
> guaranteeing that this is the problem, only saying that it fits.
>
> Thanks,
> Shawn

Re: Solr Replication being flaky (6.2.0)

[Erick Erickson]

Scott:

We usually recommend setting files and processes very, very high. Like
65K high. Or unlimited if you can.

Plus max user processes should also be bumped very high as well, like
65K as well.

Plus max memory and virtual memory should be unlimited.

We've included warnings at startup for open files and processes, see SOLR-11703

Best,
Erick

On Fri, Jan 19, 2018 at 7:54 AM, Pouliot, Scott
 wrote:
> I do have a ticket in with our systems team to up the file handlers since I 
> am seeing the "Too many files open" error on occasion on our prod servers.  
> Is this the setting you're referring to?  Found we were set to to 1024 using 
> the "Ulimit" command.
>
> -Original Message-
> From: Shawn Heisey [mailto:apa...@elyograg.org]
> Sent: Friday, January 19, 2018 10:48 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Replication being flaky (6.2.0)
>
> On 1/19/2018 7:50 AM, Pouliot, Scott wrote:
>> So we're running Solr in a Master/Slave configuration (1 of each) and it 
>> seems that the replication stalls or stops functioning every now and again.  
>> If we restart the Solr service or optimize the core it seems to kick back in 
>> again.
>>
>> Anyone have any idea what might be causing this?  We do have a good amount 
>> of cores on each server (@150 or so), but I have heard reports of a LOT more 
>> than that in use.
>
> Have you increased the number of processes that the user running Solr is 
> allowed to start?  Most operating systems limit the number of 
> threads/processes a user can start to a low value like 1024.  With 150 cores, 
> particularly with background tasks like replication configured, chances are 
> that Solr is going to need to start a lot of threads.  This is an OS setting 
> that a lot of Solr admins end up needing to increase.
>
> I ran into the process limit on my servers and I don't have anywhere near 150 
> cores.
>
> The fact that restarting Solr gets it working again (at least
> temporarily) would fit with a process limit being the problem.  I'm not 
> guaranteeing that this is the problem, only saying that it fits.
>
> Thanks,
> Shawn

RE: Solr Replication being flaky (6.2.0)

[Pouliot, Scott]

I do have a ticket in with our systems team to up the file handlers since I am 
seeing the "Too many files open" error on occasion on our prod servers.  Is 
this the setting you're referring to?  Found we were set to to 1024 using the 
"Ulimit" command.

-Original Message-
From: Shawn Heisey [mailto:apa...@elyograg.org] 
Sent: Friday, January 19, 2018 10:48 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr Replication being flaky (6.2.0)

On 1/19/2018 7:50 AM, Pouliot, Scott wrote:
> So we're running Solr in a Master/Slave configuration (1 of each) and it 
> seems that the replication stalls or stops functioning every now and again.  
> If we restart the Solr service or optimize the core it seems to kick back in 
> again.
> 
> Anyone have any idea what might be causing this?  We do have a good amount of 
> cores on each server (@150 or so), but I have heard reports of a LOT more 
> than that in use.

Have you increased the number of processes that the user running Solr is 
allowed to start?  Most operating systems limit the number of threads/processes 
a user can start to a low value like 1024.  With 150 cores, particularly with 
background tasks like replication configured, chances are that Solr is going to 
need to start a lot of threads.  This is an OS setting that a lot of Solr 
admins end up needing to increase.

I ran into the process limit on my servers and I don't have anywhere near 150 
cores.

The fact that restarting Solr gets it working again (at least
temporarily) would fit with a process limit being the problem.  I'm not 
guaranteeing that this is the problem, only saying that it fits.

Thanks,
Shawn

Re: Solr Replication being flaky (6.2.0)

[Shawn Heisey]

On 1/19/2018 7:50 AM, Pouliot, Scott wrote:

So we're running Solr in a Master/Slave configuration (1 of each) and it seems 
that the replication stalls or stops functioning every now and again.  If we 
restart the Solr service or optimize the core it seems to kick back in again.

Anyone have any idea what might be causing this?  We do have a good amount of 
cores on each server (@150 or so), but I have heard reports of a LOT more than 
that in use.


Have you increased the number of processes that the user running Solr is 
allowed to start?  Most operating systems limit the number of 
threads/processes a user can start to a low value like 1024.  With 150 
cores, particularly with background tasks like replication configured, 
chances are that Solr is going to need to start a lot of threads.  This 
is an OS setting that a lot of Solr admins end up needing to increase.


I ran into the process limit on my servers and I don't have anywhere 
near 150 cores.


The fact that restarting Solr gets it working again (at least 
temporarily) would fit with a process limit being the problem.  I'm not 
guaranteeing that this is the problem, only saying that it fits.


Thanks,
Shawn

RE: Solr Replication being flaky (6.2.0)

[Pouliot, Scott]

I'm at the point now where I may end up writing a script to compare 
master/slave nightly...and trigger an optimize or solr restart if there are any 
differences.  Of course I have to check 150+ cores...but it could be done.  I'm 
just hoping I don't need to go that route

-Original Message-
From: David Hastings [mailto:hastings.recurs...@gmail.com] 
Sent: Friday, January 19, 2018 10:35 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr Replication being flaky (6.2.0)

This happens to me quite often as well.  Generally on the replication admin 
screen it will say its downloading a file, but be at 0 or a VERY small kb/sec.  
Then after a restart of the slave its back to downloading at 30 to
100 mg/sec.  Would be curious if there actually is a solution to this aside 
from checking every day if the core replicated.  Im on Solr 5.x by the way -Dave

On Fri, Jan 19, 2018 at 9:50 AM, Pouliot, Scott < 
scott.poul...@peoplefluent.com> wrote:

> So we're running Solr in a Master/Slave configuration (1 of each) and 
> it seems that the replication stalls or stops functioning every now 
> and again.  If we restart the Solr service or optimize the core it 
> seems to kick back in again.
>
> Anyone have any idea what might be causing this?  We do have a good 
> amount of cores on each server (@150 or so), but I have heard reports 
> of a LOT more than that in use.
>
> Here is our master config:
> 
> 
>   
>   startup
>   commit
>
>   
>   00:00:10
> 
> 
> 
> 1
> 
>   
>
> And our slave config:
> 
> 
>
>   
>name="masterUrl">http://server1:8080/solr/${https://na01.safelinks.pro
> tection.outlook.com/?url=solr.core.name&data=02%7C01%7CScott.Pouliot%4
> 0peoplefluent.com%7C8d43918dd95540a3a11708d55f523302%7C8b16fb62c78448b
> 6aba889567990e7fe%7C1%7C1%7C636519729029923349&sdata=Fes6G36gIMRyfahTI
> fftg0eUEVEiVK77B8KpuTr%2FJrA%3D&reserved=0}
> 
>
>   
>   00:00:45
> 
>   
>
>   
> 
>   solr-data-config.xml
> 
>   
>

Re: Solr Replication being flaky (6.2.0)

[David Hastings]

This happens to me quite often as well.  Generally on the replication admin
screen it will say its downloading a file, but be at 0 or a VERY small
kb/sec.  Then after a restart of the slave its back to downloading at 30 to
100 mg/sec.  Would be curious if there actually is a solution to this aside
from checking every day if the core replicated.  Im on Solr 5.x by the way
-Dave

On Fri, Jan 19, 2018 at 9:50 AM, Pouliot, Scott <
scott.poul...@peoplefluent.com> wrote:

> So we're running Solr in a Master/Slave configuration (1 of each) and it
> seems that the replication stalls or stops functioning every now and
> again.  If we restart the Solr service or optimize the core it seems to
> kick back in again.
>
> Anyone have any idea what might be causing this?  We do have a good amount
> of cores on each server (@150 or so), but I have heard reports of a LOT
> more than that in use.
>
> Here is our master config:
> 
> 
>   
>   startup
>   commit
>
>   
>   00:00:10
> 
> 
> 
> 1
> 
>   
>
> And our slave config:
> 
> 
>
>   
>   http://server1:8080/solr/${solr.core.name}
> 
>
>   
>   00:00:45
> 
>   
>
>   
> 
>   solr-data-config.xml
> 
>   
>

Solr Replication being flaky (6.2.0)

[Pouliot, Scott]

So we're running Solr in a Master/Slave configuration (1 of each) and it seems 
that the replication stalls or stops functioning every now and again.  If we 
restart the Solr service or optimize the core it seems to kick back in again.

Anyone have any idea what might be causing this?  We do have a good amount of 
cores on each server (@150 or so), but I have heard reports of a LOT more than 
that in use.

Here is our master config:


  
  startup
  commit

  
  00:00:10



1

  

And our slave config:



  
  http://server1:8080/solr/${solr.core.name}

  
  00:00:45

  

  

  solr-data-config.xml

  

Re: Solr replication

[Satyaprashant Bezwada]

Thanks Eric, fixed the issue. The IT team corrected the solrconfig.xml but 
forgot to execute the zkcli.sh script on solr node. After I executed the script 
its working now.


On 9/20/17, 10:20 AM, "Erick Erickson"  wrote:

WARNING - External email; exercise caution.


Your solrconfig.xml file is mal-formed. The smoking gun is:

 Exception during parsing file: solrconfig.xml

Best,
Erick

On Tue, Sep 19, 2017 at 4:48 PM, Satyaprashant Bezwada
 wrote:
> Need some inputs or help in resolving replication across solr nodes. We 
have installed Solr 6.5 in cloud mode and have 3 ZooKeepers and 2 Solr nodes 
configured. Enabled Solr replication in my Solrj client but the replication 
fails and is unable to create a collection.
>
> The same code works in our different environment where in I have 1 
zookeeper and 3 Solr nodes configured. Here is the exception I see on one of 
the nodes of Solr when I try to create a collection in the environment where it 
fails. I have compared the Solrconfig.xml on both the environments and didn’t 
see any difference.
>
>
>
> 2017-09-19 22:09:35.471 ERROR 
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
> 2017-09-19 22:09:35.475 INFO  
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : 
action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
> 2017-09-19 22:09:35.486 INFO  (qtp401424608-15) [   ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={deleteInstanceDir=true&core=CIUZLW_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2}
 status=0 QTime=6
> 2017-09-19 22:09:36.194 INFO  
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection 
for [CIUZLW]
> 2017-09-19 22:09:38.008 INFO  
(OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_29)
 [   ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-000410 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
> 2017-09-19 22:38:36.634 INFO  (ShutdownMonitor) [   ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=1549725679
> 2017-09-19 22:38:36.644 INFO  (ShutdownMonitor) [   ] o.a.s.c.Overseer 
Overseer (id=170740497916499012-sr01:8983_solr-n_29) closing
> 2017-09-19 22:38:36.645 INFO  
(OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_29) [   ] 
o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
> 2017-09-19 22:38:36.654 INFO  (ShutdownMonitor) [   ] 
o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
> ^CCorp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$
> Corp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$ sudo tail -f 
/var/solr/logs/solr.log
> at java.lang.Thread.run(Thread.java:745)
> 2017-09-19 22:09:35.471 ERROR 
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
> 2017-09-19 22:09:35.475 INFO  
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : 
action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
> 2017-09-19 22:09:35.486 INFO  (qtp401424608-15) [   ] 
o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores 
params={deleteInstanceDir=true&core=CIUZLW_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2}
 status=0 QTime=6
> 2017-09-19 22:09:36.194 INFO  
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection 
for [CIUZLW]
> 2017-09-19 22:09:38.008 INFO  
(OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_29)
 [   ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-000410 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
> 2017-09-19 22:38:36.634 INFO  (ShutdownMonitor) [   ] 
o.a.s.c.CoreContainer Shutting down CoreContainer instance=1549725679
> 2017-09-19 22:38:36.644 INFO  (ShutdownMonitor) [   ] o.a.s.c.Overseer 
Overseer (id=170740497916499012-sr01:8983_solr-n_29) closing
> 2017-09-19 22:38:36.645 INFO  
(OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_29) [   ] 
o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
> 2017-09-19 22:38:36.654 INFO  (ShutdownMonitor) [   ] 
o.a.s.m.SolrMetricManager Closing metric 

Re: Solr replication

[Erick Erickson]

Your solrconfig.xml file is mal-formed. The smoking gun is:

 Exception during parsing file: solrconfig.xml

Best,
Erick

On Tue, Sep 19, 2017 at 4:48 PM, Satyaprashant Bezwada
 wrote:
> Need some inputs or help in resolving replication across solr nodes. We have 
> installed Solr 6.5 in cloud mode and have 3 ZooKeepers and 2 Solr nodes 
> configured. Enabled Solr replication in my Solrj client but the replication 
> fails and is unable to create a collection.
>
> The same code works in our different environment where in I have 1 zookeeper 
> and 3 Solr nodes configured. Here is the exception I see on one of the nodes 
> of Solr when I try to create a collection in the environment where it fails. 
> I have compared the Solrconfig.xml on both the environments and didn’t see 
> any difference.
>
>
>
> 2017-09-19 22:09:35.471 ERROR 
> (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
> o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
> 2017-09-19 22:09:35.475 INFO  
> (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
> o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : 
> action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
> 2017-09-19 22:09:35.486 INFO  (qtp401424608-15) [   ] o.a.s.s.HttpSolrCall 
> [admin] webapp=null path=/admin/cores 
> params={deleteInstanceDir=true&core=CIUZLW_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2}
>  status=0 QTime=6
> 2017-09-19 22:09:36.194 INFO  
> (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
> o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection 
> for [CIUZLW]
> 2017-09-19 22:09:38.008 INFO  
> (OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_29)
>  [   ] o.a.s.c.OverseerTaskQueue Response ZK path: 
> /overseer/collection-queue-work/qnr-000410 doesn't exist.  Requestor may 
> have disconnected from ZooKeeper
> 2017-09-19 22:38:36.634 INFO  (ShutdownMonitor) [   ] o.a.s.c.CoreContainer 
> Shutting down CoreContainer instance=1549725679
> 2017-09-19 22:38:36.644 INFO  (ShutdownMonitor) [   ] o.a.s.c.Overseer 
> Overseer (id=170740497916499012-sr01:8983_solr-n_29) closing
> 2017-09-19 22:38:36.645 INFO  
> (OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_29) [   ] 
> o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
> 2017-09-19 22:38:36.654 INFO  (ShutdownMonitor) [   ] 
> o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
> ^CCorp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$
> Corp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$ sudo tail -f 
> /var/solr/logs/solr.log
> at java.lang.Thread.run(Thread.java:745)
> 2017-09-19 22:09:35.471 ERROR 
> (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
> o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
> 2017-09-19 22:09:35.475 INFO  
> (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
> o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : 
> action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
> 2017-09-19 22:09:35.486 INFO  (qtp401424608-15) [   ] o.a.s.s.HttpSolrCall 
> [admin] webapp=null path=/admin/cores 
> params={deleteInstanceDir=true&core=CIUZLW_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2}
>  status=0 QTime=6
> 2017-09-19 22:09:36.194 INFO  
> (OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
> o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection 
> for [CIUZLW]
> 2017-09-19 22:09:38.008 INFO  
> (OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_29)
>  [   ] o.a.s.c.OverseerTaskQueue Response ZK path: 
> /overseer/collection-queue-work/qnr-000410 doesn't exist.  Requestor may 
> have disconnected from ZooKeeper
> 2017-09-19 22:38:36.634 INFO  (ShutdownMonitor) [   ] o.a.s.c.CoreContainer 
> Shutting down CoreContainer instance=1549725679
> 2017-09-19 22:38:36.644 INFO  (ShutdownMonitor) [   ] o.a.s.c.Overseer 
> Overseer (id=170740497916499012-sr01:8983_solr-n_29) closing
> 2017-09-19 22:38:36.645 INFO  
> (OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_29) [   ] 
> o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
> 2017-09-19 22:38:36.654 INFO  (ShutdownMonitor) [   ] 
> o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
> ^CCorp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$ sudo tail -f 
> /var/solr/logs/solr.log
> 2017-09-19 23:12:22.230 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter Loading 

Solr replication

[Satyaprashant Bezwada]

Need some inputs or help in resolving replication across solr nodes. We have 
installed Solr 6.5 in cloud mode and have 3 ZooKeepers and 2 Solr nodes 
configured. Enabled Solr replication in my Solrj client but the replication 
fails and is unable to create a collection.

The same code works in our different environment where in I have 1 zookeeper 
and 3 Solr nodes configured. Here is the exception I see on one of the nodes of 
Solr when I try to create a collection in the environment where it fails. I 
have compared the Solrconfig.xml on both the environments and didn’t see any 
difference.



2017-09-19 22:09:35.471 ERROR 
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
2017-09-19 22:09:35.475 INFO  
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : 
action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
2017-09-19 22:09:35.486 INFO  (qtp401424608-15) [   ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={deleteInstanceDir=true&core=CIUZLW_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2}
 status=0 QTime=6
2017-09-19 22:09:36.194 INFO  
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection 
for [CIUZLW]
2017-09-19 22:09:38.008 INFO  
(OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_29)
 [   ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-000410 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
2017-09-19 22:38:36.634 INFO  (ShutdownMonitor) [   ] o.a.s.c.CoreContainer 
Shutting down CoreContainer instance=1549725679
2017-09-19 22:38:36.644 INFO  (ShutdownMonitor) [   ] o.a.s.c.Overseer Overseer 
(id=170740497916499012-sr01:8983_solr-n_29) closing
2017-09-19 22:38:36.645 INFO  
(OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_29) [   ] 
o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
2017-09-19 22:38:36.654 INFO  (ShutdownMonitor) [   ] o.a.s.m.SolrMetricManager 
Closing metric reporters for: solr.node
^CCorp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$
Corp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$ sudo tail -f 
/var/solr/logs/solr.log
at java.lang.Thread.run(Thread.java:745)
2017-09-19 22:09:35.471 ERROR 
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
o.a.s.c.OverseerCollectionMessageHandler Cleaning up collection [CIUZLW].
2017-09-19 22:09:35.475 INFO  
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : 
action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
2017-09-19 22:09:35.486 INFO  (qtp401424608-15) [   ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/cores 
params={deleteInstanceDir=true&core=CIUZLW_shard1_replica1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2}
 status=0 QTime=6
2017-09-19 22:09:36.194 INFO  
(OverseerThreadFactory-8-thread-1-processing-n:sr01:8983_solr) [   ] 
o.a.s.c.CreateCollectionCmd Cleaned up artifacts for failed create collection 
for [CIUZLW]
2017-09-19 22:09:38.008 INFO  
(OverseerCollectionConfigSetProcessor-170740497916499012-sr01:8983_solr-n_29)
 [   ] o.a.s.c.OverseerTaskQueue Response ZK path: 
/overseer/collection-queue-work/qnr-000410 doesn't exist.  Requestor may 
have disconnected from ZooKeeper
2017-09-19 22:38:36.634 INFO  (ShutdownMonitor) [   ] o.a.s.c.CoreContainer 
Shutting down CoreContainer instance=1549725679
2017-09-19 22:38:36.644 INFO  (ShutdownMonitor) [   ] o.a.s.c.Overseer Overseer 
(id=170740497916499012-sr01:8983_solr-n_29) closing
2017-09-19 22:38:36.645 INFO  
(OverseerStateUpdate-170740497916499012-sr01:8983_solr-n_29) [   ] 
o.a.s.c.Overseer Overseer Loop exiting : sr01:8983_solr
2017-09-19 22:38:36.654 INFO  (ShutdownMonitor) [   ] o.a.s.m.SolrMetricManager 
Closing metric reporters for: solr.node
^CCorp-QA-West [sbezw...@boardvantage.net@sr1501:1 ~]$ sudo tail -f 
/var/solr/logs/solr.log
2017-09-19 23:12:22.230 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter Loading 
solr.xml from SolrHome (not found in ZooKeeper)
2017-09-19 23:12:22.233 INFO  (main) [   ] o.a.s.c.SolrXmlConfig Loading 
container configuration from /var/solr/data/Node122/solr.xml
2017-09-19 23:12:22.644 INFO  (main) [   ] o.a.s.u.UpdateShardHandler Creating 
UpdateShardHandler HTTP client with params: 
socketTimeout=60&connTimeout=6&retry=true
2017-09-19 23:12:22.650 INFO  (main) [   ] o.a.s.c.ZkContainer Zookeeper 
client=zk01:2181,zk02:2181,zk03:2181
2017-09-19 23:12:22.731 INFO  (main) [   ] o.a.s.c.c.ZkStateReader Updated live 
nodes from Z

Solr replication failure then restart.

[Yunee Lee]

Hi, 

I have a solr replication set up from master to slave in legacy ( It's not from 
the solr cloud).
Somehow the first initial replication doesn't finish and when it reaches 99% 
and got the error as following and then restart from the beginning.
I don't know why it is keep retriggering to start the replication over and over.

ERROR
ReplicationHandler
Index fetch failed :org.apache.solr.common.SolrException: Unable to download 
_55sm.si completely. Downloaded 0!=363

Here is the config in the slave. I wonder if solrconfig has any 
misconfiguration. 



startup
commit

0


 
   http://url.com
 00:05:00
   
  

Anyone had a similar experience, please share how to resolve the issue.
Thanks.

SOLR replication: different behavior for network cut off vs. machine restart

[Grzegorz Huber]

Hi,

We try to set up a SOLR Cloud environment using 1 shard with 2
replicas (1 leader). The replicas are managed by 3 zookeeper
instances.

The setup seems fine when we do the normal work. The data is being
replicated at runtime.

Now we try to simulate erroneous behavior in several cases:

Turn off one of the replicas in two different scenarios: leader and non-leader
Cutting off the network making the non-leader replica down

In both cases the data is being written contentiously to the SOLR Cloud.

CASE 1: The replication process starts after the failed machine gets
boot up again. The complete data set is present in both replicas.
Everything works fine.

CASE 2: Once reconnected to network the non-leader replica starts the
recovery process ,but for some reason the new data from leader is not
being replicated onto the previously failed replica.

>From what I was able to read from logs comparing both cases I don't
understand why SOLR sees

RecoveryStrategy ## currentVersions as present and
RecoveryStrategy ## startupVersions=[[]] (empty)

compared to CASE 1 when RecoveryStrategy ## startupVersions are
filled with objects that are in currentVersions in CASE 2

The general question is... why restarting SOLR results in a successful
migration process, but reconnecting the network does not?

Thanks for any tips / leads!

Cheers,
Greg

Solr Replication error

[Yago Riveiro]

I cached this in my logs. Any reason to this happen?

My Solr version is 5.3.1.

Index fetch failed :org.apache.solr.common.SolrException: Index fetch failed
: 
at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:515)
at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:254)
at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:380)
at
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:162)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:437)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:227)
Caused by: java.nio.file.NoSuchFileException:
/opt/solrcloud-node/collections/collection-2016_shard9_replica2/data/index.20160105005921682/segments_fhj
at 
sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at
sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at 
org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:236)
at 
org.apache.lucene.store.Directory.openChecksumInput(Directory.java:109)
at 
org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:294)
at
org.apache.solr.handler.IndexFetcher.hasUnusedFiles(IndexFetcher.java:568)
at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:397)
... 5 more




-
Best regards
--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-Replication-error-tp4252929.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Replication sometimes coming in log files

[Upayavira]

I bet you have the admin UI open on your second slave. The _=144... is
the give-away. Those requests are the admin UI asking the replication
handler for the status of replication.

Upayavira

On Wed, Sep 9, 2015, at 06:32 AM, Kamal Kishore Aggarwal wrote:
> Hi Team,
> 
> I am currently working with Java-1.7, Solr-4.8.1 with tomcat 7. The solr
> configuration has master & slave ( 2 Slaves) architecture.
> 
> 
> Master & Slave 2 are in same server location (say zone A) , whereas Slave
> 1
> is in another server in different zone (say zone B). There is latency of
> 40
> ms between two zones.
> 
> Now, a days we are facing high load on Slave 1 & we suspect that it is
> due
> to delay in data replication from Master server. These days we are
> finding
> these below mentioned replication information in log files, but such
> lines
> are not in previous files on the Slave 1 server. Also, such information
> is
> not there in any Slave 2 log files (might be due to same zone of master &
> slave 2).
> 
> 
> > INFO: [Core] webapp=/solr path=/replication
> > params={wt=json&command=details&_=1441708786003} status=0 QTime=173
> > INFO: [Core] webapp=/solr path=/replication
> > params={wt=json&command=details&_=1441708787976} status=0 QTime=1807
> > INFO: [Core] webapp=/solr path=/replication
> > params={wt=json&command=details&_=1441708791563} status=0 QTime=7140
> > INFO: [Core] webapp=/solr path=/replication
> > params={wt=json&command=details&_=1441708800450} status=0 QTime=1679
> 
> 
> 
> Please confirm if we our thought that increased replication time (which
> can
> be due to servers connectivity issues) is the reason for high load on
> solr.
> 
> Regards
> Kamal Kishore

Re: Solr Replication sometimes coming in log files

[Mikhail Khludnev]

Hello,

I'd say opposite: high load causes long time to respond. 'command=details'
is rather cheap and fast _I believe_.

On Mon, Sep 14, 2015 at 10:20 AM, Kamal Kishore Aggarwal <
kkroyal@gmail.com> wrote:

> Can anybody suggest me something..
>
> On Wed, Sep 9, 2015 at 11:02 AM, Kamal Kishore Aggarwal <
> kkroyal@gmail.com> wrote:
>
> > Hi Team,
> >
> > I am currently working with Java-1.7, Solr-4.8.1 with tomcat 7. The solr
> > configuration has master & slave ( 2 Slaves) architecture.
> >
> >
> > Master & Slave 2 are in same server location (say zone A) , whereas Slave
> > 1 is in another server in different zone (say zone B). There is latency
> of
> > 40 ms between two zones.
> >
> > Now, a days we are facing high load on Slave 1 & we suspect that it is
> due
> > to delay in data replication from Master server. These days we are
> finding
> > these below mentioned replication information in log files, but such
> lines
> > are not in previous files on the Slave 1 server. Also, such information
> is
> > not there in any Slave 2 log files (might be due to same zone of master &
> > slave 2).
> >
> >
> >> INFO: [Core] webapp=/solr path=/replication
> >> params={wt=json&command=details&_=1441708786003} status=0 QTime=173
> >> INFO: [Core] webapp=/solr path=/replication
> >> params={wt=json&command=details&_=1441708787976} status=0 QTime=1807
> >> INFO: [Core] webapp=/solr path=/replication
> >> params={wt=json&command=details&_=1441708791563} status=0 QTime=7140
> >> INFO: [Core] webapp=/solr path=/replication
> >> params={wt=json&command=details&_=1441708800450} status=0 QTime=1679
> >
> >
> >
> > Please confirm if we our thought that increased replication time (which
> > can be due to servers connectivity issues) is the reason for high load on
> > solr.
> >
> > Regards
> > Kamal Kishore
> >
> >
>



-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

Re: Solr Replication sometimes coming in log files

[Kamal Kishore Aggarwal]

Can anybody suggest me something..

On Wed, Sep 9, 2015 at 11:02 AM, Kamal Kishore Aggarwal <
kkroyal@gmail.com> wrote:

> Hi Team,
>
> I am currently working with Java-1.7, Solr-4.8.1 with tomcat 7. The solr
> configuration has master & slave ( 2 Slaves) architecture.
>
>
> Master & Slave 2 are in same server location (say zone A) , whereas Slave
> 1 is in another server in different zone (say zone B). There is latency of
> 40 ms between two zones.
>
> Now, a days we are facing high load on Slave 1 & we suspect that it is due
> to delay in data replication from Master server. These days we are finding
> these below mentioned replication information in log files, but such lines
> are not in previous files on the Slave 1 server. Also, such information is
> not there in any Slave 2 log files (might be due to same zone of master &
> slave 2).
>
>
>> INFO: [Core] webapp=/solr path=/replication
>> params={wt=json&command=details&_=1441708786003} status=0 QTime=173
>> INFO: [Core] webapp=/solr path=/replication
>> params={wt=json&command=details&_=1441708787976} status=0 QTime=1807
>> INFO: [Core] webapp=/solr path=/replication
>> params={wt=json&command=details&_=1441708791563} status=0 QTime=7140
>> INFO: [Core] webapp=/solr path=/replication
>> params={wt=json&command=details&_=1441708800450} status=0 QTime=1679
>
>
>
> Please confirm if we our thought that increased replication time (which
> can be due to servers connectivity issues) is the reason for high load on
> solr.
>
> Regards
> Kamal Kishore
>
>

Solr Replication sometimes coming in log files

[Kamal Kishore Aggarwal]

Hi Team,

I am currently working with Java-1.7, Solr-4.8.1 with tomcat 7. The solr
configuration has master & slave ( 2 Slaves) architecture.


Master & Slave 2 are in same server location (say zone A) , whereas Slave 1
is in another server in different zone (say zone B). There is latency of 40
ms between two zones.

Now, a days we are facing high load on Slave 1 & we suspect that it is due
to delay in data replication from Master server. These days we are finding
these below mentioned replication information in log files, but such lines
are not in previous files on the Slave 1 server. Also, such information is
not there in any Slave 2 log files (might be due to same zone of master &
slave 2).


> INFO: [Core] webapp=/solr path=/replication
> params={wt=json&command=details&_=1441708786003} status=0 QTime=173
> INFO: [Core] webapp=/solr path=/replication
> params={wt=json&command=details&_=1441708787976} status=0 QTime=1807
> INFO: [Core] webapp=/solr path=/replication
> params={wt=json&command=details&_=1441708791563} status=0 QTime=7140
> INFO: [Core] webapp=/solr path=/replication
> params={wt=json&command=details&_=1441708800450} status=0 QTime=1679



Please confirm if we our thought that increased replication time (which can
be due to servers connectivity issues) is the reason for high load on solr.

Regards
Kamal Kishore

Re: Weird Solr Replication Slave out of sync

[Dmitry Kan]

Hi,
This sounds quite strange. Do you see any error messages either in the solr
admin's replication page or in the master's OR slave's logs?

When we had issues with slave replicating from the master, they related to
slave running out of disk. I'm sure there could be a bunch of other reasons
for failed replication, but those should generally be evident in the logs.

On Tue, Feb 17, 2015 at 7:46 AM, Summer Shire  wrote:

> Hi All,
>
> My master and slave index version and generation is the same
> yet the index is not in sync because when I execute the same query
> on both master and slave I see old docs on slave which should not be there.
>
> I also tried to fetch a specific indexversion on slave using
> command=fetchindex&indexversion=
>
> This is very spooky because I do not get any errors on master or slave.
> Also I see in the logs that the slave is polling the master every 15 mins
> I was able to find this issue only because I was looking at the specific
> old document.
>
> Now I can manually delete the index folder on slave and restart my slave.
> But I really want to find out what could be going on. Because these type
> of issues are going to
> be hard to find especially when there are on errors.
>
> What could be happening. and how can I avoid this from happening ?
>
>
> Thanks,
> Summer
>
>


-- 
Dmitry Kan
Luke Toolbox: http://github.com/DmitryKey/luke
Blog: http://dmitrykan.blogspot.com
Twitter: http://twitter.com/dmitrykan
SemanticAnalyzer: www.semanticanalyzer.info

Weird Solr Replication Slave out of sync

[Summer Shire]

Hi All,

My master and slave index version and generation is the same
yet the index is not in sync because when I execute the same query 
on both master and slave I see old docs on slave which should not be there.

I also tried to fetch a specific indexversion on slave using 
command=fetchindex&indexversion=

This is very spooky because I do not get any errors on master or slave.
Also I see in the logs that the slave is polling the master every 15 mins 
I was able to find this issue only because I was looking at the specific old 
document.

Now I can manually delete the index folder on slave and restart my slave.
But I really want to find out what could be going on. Because these type of 
issues are going to 
be hard to find especially when there are on errors.

What could be happening. and how can I avoid this from happening ?


Thanks,
Summer

Re: solr replication vs. rsync

[Erick Erickson]

bq:  I thought SolrCloud replicas were replication, and you imply parallel
indexing

Absolutely! You couldn't get near-real-time indexing if you relied on
replication a-la
3x. And you also couldn't guarantee consistency.

Say you have 1 shard, a leader and a follower (i.e. 2 replicas). Now you
throw a doc
to be indexed. The sequence is:
leader gets the doc
leader forwards the doc to the follower
leader and follower both add the doc to their local index (and tlog).
follower acks back to leader
leader acks back to client.

So yes, the raw document is forwarded to all replicas before the leader
responds
to the client, the docs all get written to the tlogs, etc. That's the only
way to guarantee
that if the leader goes down, the follower can take over without losing
documents.

Best,
Erick

On Sun, Jan 25, 2015 at 6:15 PM, Dan Davis  wrote:

> @Erick,
>
> Problem space is not constant indexing.   I thought SolrCloud replicas were
> replication, and you imply parallel indexing.  Good to know.
>
> On Sunday, January 25, 2015, Erick Erickson 
> wrote:
>
> > @Shawn: Cool table, thanks!
> >
> > @Dan:
> > Just to throw a different spin on it, if you migrate to SolrCloud, then
> > this question becomes moot as the raw documents are sent to each of the
> > replicas so you very rarely have to copy the full index. Kind of a
> tradeoff
> > between constant load because you're sending the raw documents around
> > whenever you index and peak usage when the index replicates.
> >
> > There are a bunch of other reasons to go to SolrCloud, but you know your
> > problem space best.
> >
> > FWIW,
> > Erick
> >
> > On Sun, Jan 25, 2015 at 9:26 AM, Shawn Heisey  > > wrote:
> >
> > > On 1/24/2015 10:56 PM, Dan Davis wrote:
> > > > When I polled the various projects already using Solr at my
> > > organization, I
> > > > was greatly surprised that none of them were using Solr replication,
> > > > because they had talked about "replicating" the data.
> > > >
> > > > But we are not Pinterest, and do not expect to be taking in changes
> one
> > > > post at a time (at least the engineers don't - just wait until its
> used
> > > for
> > > > a Crud app that wants full-text search on a description field!).
> > > Still,
> > > > rsync can be very, very fast with the right options (-W for gigabit
> > > > ethernet, and maybe -S for sparse files).   I've clocked it at 48
> MB/s
> > > over
> > > > GigE previously.
> > > >
> > > > Does anyone have any numbers for how fast Solr replication goes, and
> > what
> > > > to do to tune it?
> > > >
> > > > I'm not enthusiastic to give-up recently tested cluster stability
> for a
> > > > home grown mess, but I am interested in numbers that are out there.
> > >
> > > Numbers are included on the Solr replication wiki page, both in graph
> > > and numeric form.  Gathering these numbers must have been pretty easy
> --
> > > before the HTTP replication made it into Solr, Solr used to contain an
> > > rsync-based implementation.
> > >
> > > http://wiki.apache.org/solr/SolrReplication#Performance_numbers
> > >
> > > Other data on that wiki page discusses the replication config.  There's
> > > not a lot to tune.
> > >
> > > I run a redundant non-SolrCloud index myself through a different method
> > > -- my indexing program indexes each index copy completely
> independently.
> > >  There is no replication.  This separation allows me to upgrade any
> > > component, or change any part of solrconfig or schema, on either copy
> of
> > > the index without affecting the other copy at all.  With replication,
> if
> > > something is changed on the master or the slave, you might find that
> the
> > > slave no longer works, because it will be handling an index created by
> > > different software or a different config.
> > >
> > > Thanks,
> > > Shawn
> > >
> > >
> >
>

Re: solr replication vs. rsync

[Dan Davis]

@Erick,

Problem space is not constant indexing.   I thought SolrCloud replicas were
replication, and you imply parallel indexing.  Good to know.

On Sunday, January 25, 2015, Erick Erickson  wrote:

> @Shawn: Cool table, thanks!
>
> @Dan:
> Just to throw a different spin on it, if you migrate to SolrCloud, then
> this question becomes moot as the raw documents are sent to each of the
> replicas so you very rarely have to copy the full index. Kind of a tradeoff
> between constant load because you're sending the raw documents around
> whenever you index and peak usage when the index replicates.
>
> There are a bunch of other reasons to go to SolrCloud, but you know your
> problem space best.
>
> FWIW,
> Erick
>
> On Sun, Jan 25, 2015 at 9:26 AM, Shawn Heisey  > wrote:
>
> > On 1/24/2015 10:56 PM, Dan Davis wrote:
> > > When I polled the various projects already using Solr at my
> > organization, I
> > > was greatly surprised that none of them were using Solr replication,
> > > because they had talked about "replicating" the data.
> > >
> > > But we are not Pinterest, and do not expect to be taking in changes one
> > > post at a time (at least the engineers don't - just wait until its used
> > for
> > > a Crud app that wants full-text search on a description field!).
> > Still,
> > > rsync can be very, very fast with the right options (-W for gigabit
> > > ethernet, and maybe -S for sparse files).   I've clocked it at 48 MB/s
> > over
> > > GigE previously.
> > >
> > > Does anyone have any numbers for how fast Solr replication goes, and
> what
> > > to do to tune it?
> > >
> > > I'm not enthusiastic to give-up recently tested cluster stability for a
> > > home grown mess, but I am interested in numbers that are out there.
> >
> > Numbers are included on the Solr replication wiki page, both in graph
> > and numeric form.  Gathering these numbers must have been pretty easy --
> > before the HTTP replication made it into Solr, Solr used to contain an
> > rsync-based implementation.
> >
> > http://wiki.apache.org/solr/SolrReplication#Performance_numbers
> >
> > Other data on that wiki page discusses the replication config.  There's
> > not a lot to tune.
> >
> > I run a redundant non-SolrCloud index myself through a different method
> > -- my indexing program indexes each index copy completely independently.
> >  There is no replication.  This separation allows me to upgrade any
> > component, or change any part of solrconfig or schema, on either copy of
> > the index without affecting the other copy at all.  With replication, if
> > something is changed on the master or the slave, you might find that the
> > slave no longer works, because it will be handling an index created by
> > different software or a different config.
> >
> > Thanks,
> > Shawn
> >
> >
>

Re: solr replication vs. rsync

[Dan Davis]

Thanks!

On Sunday, January 25, 2015, Erick Erickson  wrote:

> @Shawn: Cool table, thanks!
>
> @Dan:
> Just to throw a different spin on it, if you migrate to SolrCloud, then
> this question becomes moot as the raw documents are sent to each of the
> replicas so you very rarely have to copy the full index. Kind of a tradeoff
> between constant load because you're sending the raw documents around
> whenever you index and peak usage when the index replicates.
>
> There are a bunch of other reasons to go to SolrCloud, but you know your
> problem space best.
>
> FWIW,
> Erick
>
> On Sun, Jan 25, 2015 at 9:26 AM, Shawn Heisey  > wrote:
>
> > On 1/24/2015 10:56 PM, Dan Davis wrote:
> > > When I polled the various projects already using Solr at my
> > organization, I
> > > was greatly surprised that none of them were using Solr replication,
> > > because they had talked about "replicating" the data.
> > >
> > > But we are not Pinterest, and do not expect to be taking in changes one
> > > post at a time (at least the engineers don't - just wait until its used
> > for
> > > a Crud app that wants full-text search on a description field!).
> > Still,
> > > rsync can be very, very fast with the right options (-W for gigabit
> > > ethernet, and maybe -S for sparse files).   I've clocked it at 48 MB/s
> > over
> > > GigE previously.
> > >
> > > Does anyone have any numbers for how fast Solr replication goes, and
> what
> > > to do to tune it?
> > >
> > > I'm not enthusiastic to give-up recently tested cluster stability for a
> > > home grown mess, but I am interested in numbers that are out there.
> >
> > Numbers are included on the Solr replication wiki page, both in graph
> > and numeric form.  Gathering these numbers must have been pretty easy --
> > before the HTTP replication made it into Solr, Solr used to contain an
> > rsync-based implementation.
> >
> > http://wiki.apache.org/solr/SolrReplication#Performance_numbers
> >
> > Other data on that wiki page discusses the replication config.  There's
> > not a lot to tune.
> >
> > I run a redundant non-SolrCloud index myself through a different method
> > -- my indexing program indexes each index copy completely independently.
> >  There is no replication.  This separation allows me to upgrade any
> > component, or change any part of solrconfig or schema, on either copy of
> > the index without affecting the other copy at all.  With replication, if
> > something is changed on the master or the slave, you might find that the
> > slave no longer works, because it will be handling an index created by
> > different software or a different config.
> >
> > Thanks,
> > Shawn
> >
> >
>

Re: solr replication vs. rsync

[Erick Erickson]

@Shawn: Cool table, thanks!

@Dan:
Just to throw a different spin on it, if you migrate to SolrCloud, then
this question becomes moot as the raw documents are sent to each of the
replicas so you very rarely have to copy the full index. Kind of a tradeoff
between constant load because you're sending the raw documents around
whenever you index and peak usage when the index replicates.

There are a bunch of other reasons to go to SolrCloud, but you know your
problem space best.

FWIW,
Erick

On Sun, Jan 25, 2015 at 9:26 AM, Shawn Heisey  wrote:

> On 1/24/2015 10:56 PM, Dan Davis wrote:
> > When I polled the various projects already using Solr at my
> organization, I
> > was greatly surprised that none of them were using Solr replication,
> > because they had talked about "replicating" the data.
> >
> > But we are not Pinterest, and do not expect to be taking in changes one
> > post at a time (at least the engineers don't - just wait until its used
> for
> > a Crud app that wants full-text search on a description field!).
> Still,
> > rsync can be very, very fast with the right options (-W for gigabit
> > ethernet, and maybe -S for sparse files).   I've clocked it at 48 MB/s
> over
> > GigE previously.
> >
> > Does anyone have any numbers for how fast Solr replication goes, and what
> > to do to tune it?
> >
> > I'm not enthusiastic to give-up recently tested cluster stability for a
> > home grown mess, but I am interested in numbers that are out there.
>
> Numbers are included on the Solr replication wiki page, both in graph
> and numeric form.  Gathering these numbers must have been pretty easy --
> before the HTTP replication made it into Solr, Solr used to contain an
> rsync-based implementation.
>
> http://wiki.apache.org/solr/SolrReplication#Performance_numbers
>
> Other data on that wiki page discusses the replication config.  There's
> not a lot to tune.
>
> I run a redundant non-SolrCloud index myself through a different method
> -- my indexing program indexes each index copy completely independently.
>  There is no replication.  This separation allows me to upgrade any
> component, or change any part of solrconfig or schema, on either copy of
> the index without affecting the other copy at all.  With replication, if
> something is changed on the master or the slave, you might find that the
> slave no longer works, because it will be handling an index created by
> different software or a different config.
>
> Thanks,
> Shawn
>
>

Re: solr replication vs. rsync

[Shawn Heisey]

On 1/24/2015 10:56 PM, Dan Davis wrote:
> When I polled the various projects already using Solr at my organization, I
> was greatly surprised that none of them were using Solr replication,
> because they had talked about "replicating" the data.
> 
> But we are not Pinterest, and do not expect to be taking in changes one
> post at a time (at least the engineers don't - just wait until its used for
> a Crud app that wants full-text search on a description field!).Still,
> rsync can be very, very fast with the right options (-W for gigabit
> ethernet, and maybe -S for sparse files).   I've clocked it at 48 MB/s over
> GigE previously.
> 
> Does anyone have any numbers for how fast Solr replication goes, and what
> to do to tune it?
> 
> I'm not enthusiastic to give-up recently tested cluster stability for a
> home grown mess, but I am interested in numbers that are out there.

Numbers are included on the Solr replication wiki page, both in graph
and numeric form.  Gathering these numbers must have been pretty easy --
before the HTTP replication made it into Solr, Solr used to contain an
rsync-based implementation.

http://wiki.apache.org/solr/SolrReplication#Performance_numbers

Other data on that wiki page discusses the replication config.  There's
not a lot to tune.

I run a redundant non-SolrCloud index myself through a different method
-- my indexing program indexes each index copy completely independently.
 There is no replication.  This separation allows me to upgrade any
component, or change any part of solrconfig or schema, on either copy of
the index without affecting the other copy at all.  With replication, if
something is changed on the master or the slave, you might find that the
slave no longer works, because it will be handling an index created by
different software or a different config.

Thanks,
Shawn

solr replication vs. rsync

[Dan Davis]

When I polled the various projects already using Solr at my organization, I
was greatly surprised that none of them were using Solr replication,
because they had talked about "replicating" the data.

But we are not Pinterest, and do not expect to be taking in changes one
post at a time (at least the engineers don't - just wait until its used for
a Crud app that wants full-text search on a description field!).Still,
rsync can be very, very fast with the right options (-W for gigabit
ethernet, and maybe -S for sparse files).   I've clocked it at 48 MB/s over
GigE previously.

Does anyone have any numbers for how fast Solr replication goes, and what
to do to tune it?

I'm not enthusiastic to give-up recently tested cluster stability for a
home grown mess, but I am interested in numbers that are out there.

Re: Solr Replication during Tomcat shutdown causes shutdown to hang/fail

[Phil Black-Knight]

I haven't seen any activity regarding this in Jira, just curious if it
would be looked into anytime soon...

On Thu, Oct 2, 2014 at 10:11 AM, Phil Black-Knight <
pblackkni...@globalgiving.org> wrote:

> see the ticket here:
> https://issues.apache.org/jira/browse/SOLR-6579
>
> including a patch to fix it.
>
> On Thu, Oct 2, 2014 at 9:44 AM, Shawn Heisey  wrote:
>
>> On 10/2/2014 7:25 AM, Phil Black-Knight wrote:
>> > I was helping to look into this with Nick & I think we may have figured
>> out
>> > the core of the problem...
>> >
>> > The problem is easily reproducible by starting replication on the slave
>> and
>> > then sending a shutdown command to tomcat (e.g. catalina.sh stop).
>> >
>> > With a debugger attached, it looks like the fsyncService thread is
>> blocking
>> > VM shutdown because it is created as a non-daemon thread.
>>
>> 
>>
>> > I can submit patches if needed... and cross post to the dev mailing
>> list...
>>
>> File a detailed issue in Jira and attach your patch there.  This is our
>> bugtracker.  You need an account on the Apache jira instance to do this:
>>
>> https://issues.apache.org/jira/browse/SOLR
>>
>> Thanks,
>> Shawn
>>
>>
>

Re: Solr Replication during Tomcat shutdown causes shutdown to hang/fail

[Phil Black-Knight]

see the ticket here:
https://issues.apache.org/jira/browse/SOLR-6579

including a patch to fix it.

On Thu, Oct 2, 2014 at 9:44 AM, Shawn Heisey  wrote:

> On 10/2/2014 7:25 AM, Phil Black-Knight wrote:
> > I was helping to look into this with Nick & I think we may have figured
> out
> > the core of the problem...
> >
> > The problem is easily reproducible by starting replication on the slave
> and
> > then sending a shutdown command to tomcat (e.g. catalina.sh stop).
> >
> > With a debugger attached, it looks like the fsyncService thread is
> blocking
> > VM shutdown because it is created as a non-daemon thread.
>
> 
>
> > I can submit patches if needed... and cross post to the dev mailing
> list...
>
> File a detailed issue in Jira and attach your patch there.  This is our
> bugtracker.  You need an account on the Apache jira instance to do this:
>
> https://issues.apache.org/jira/browse/SOLR
>
> Thanks,
> Shawn
>
>

Re: Solr Replication during Tomcat shutdown causes shutdown to hang/fail

[Shawn Heisey]

On 10/2/2014 7:25 AM, Phil Black-Knight wrote:
> I was helping to look into this with Nick & I think we may have figured out
> the core of the problem...
> 
> The problem is easily reproducible by starting replication on the slave and
> then sending a shutdown command to tomcat (e.g. catalina.sh stop).
> 
> With a debugger attached, it looks like the fsyncService thread is blocking
> VM shutdown because it is created as a non-daemon thread.



> I can submit patches if needed... and cross post to the dev mailing list...

File a detailed issue in Jira and attach your patch there.  This is our
bugtracker.  You need an account on the Apache jira instance to do this:

https://issues.apache.org/jira/browse/SOLR

Thanks,
Shawn

RE: Solr Replication during Tomcat shutdown causes shutdown to hang/fail

[Phil Black-Knight]

I was helping to look into this with Nick & I think we may have figured out
the core of the problem...

The problem is easily reproducible by starting replication on the slave and
then sending a shutdown command to tomcat (e.g. catalina.sh stop).

With a debugger attached, it looks like the fsyncService thread is blocking
VM shutdown because it is created as a non-daemon thread.

Essentially what seems to be happening is that the fsyncService thread is
running when 'catalina.sh stop' is executed. This goes in and calls
SnapPuller.destroy() which aborts the current sync. Around line 517 of the
SnapPuller, there is code that is supposed to cleanup the fsyncService
thread, but I don't think it is getting executed because the thread that
called SnapPuller.fetchLatestIndex() is configured as a daemon Thread, so
the JVM ends up shutting that down before it can cleanup the fysncService...

So... it seems like:

if (fsyncService != null)
ExecutorUtil.shutdownNowAndAwaitTermination(fsyncService);
could be added around line 1706 of SnapPuller.java,  or

  puller.setDaemon(*false*);
could be added around line 230 of ReplicationHandler.java, however this
needs some additional work (and I think it might need to be added
regardless) since the cleanup code in SnapPuller(around 517) that shuts
down the fsync thread never gets execute since
logReplicationTimeAndConfFiles() can throw IO exceptions bypassing the rest
of the finally block...So the call to
logReplicationTimeAndConfFiles() around line 512 would need to get wrapped
with a try/catch block to catch the IO exception...

I can submit patches if needed... and cross post to the dev mailing list...

-Phil

Solr Replication during Tomcat shutdown causes shutdown to hang/fail

[Nicholas Violi]

Hello,
I have solr 4.10 running on tomcat 7. I'm doing replication from one master
to about 10 slaves, with standard configuration:

  
   
 ${enable.master:false}
 commit
 startup
 schema.xml,stopwords.txt
 
   
   
 ${enable.slave:false}
 http://master:8080/solr/mycore
 00:00:60
   
  

It appears that if tomcat gets shutdown while solr is replicating, it
prevents tomcat from shutting down fully. Immediately after receiving the
shutdown command, a thread dump is logged into catalina.out (this may have
been turned on by some configuration someone else on my team made). I
removed some threads that didn't look related, mostly about tomcat session
replication, or with names like "http-bio-8080-exec-10".

62252 [http-bio-8080-exec-1] INFO  org.apache.solr.core.SolrCore  –
[mycore] webapp=/solr path=/replication
params={command=details&_=1412014928648&wt=json} status=0 QTime=6
63310 [http-bio-8080-exec-1] INFO  org.apache.solr.core.SolrCore  –
[mycore] webapp=/solr path=/replication
params={command=details&_=1412014929699&wt=json} status=0 QTime=6
2014-09-29 14:22:10
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed
mode):

"fsyncService-12-thread-1" prio=10 tid=0x7f3bd4002000 nid=0x203d
waiting on condition [0x7f3c271f]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0007e1ff4458> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

"explicit-fetchindex-cmd" daemon prio=10 tid=0x7f3c0413e800
nid=0x203c runnable [0x7f3c272f1000]
   java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at
org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:198)
at
org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:174)
at
org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:137)
at
org.apache.solr.common.util.FastInputStream.readWrappedStream(FastInputStream.java:80)
at
org.apache.solr.common.util.FastInputStream.read(FastInputStream.java:114)
at
org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:152)
at
org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchPackets(SnapPuller.java:1239)
at
org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchFile(SnapPuller.java:1187)
at
org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:774)
at
org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:424)
at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:337)
at
org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:227)

"process reaper" daemon prio=10 tid=0x7f3c0409c000 nid=0x203b
waiting on condition [0x7f3c984e9000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x0007dfbfd890> (a
java.util.concurrent.SynchronousQueue$TransferStack)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
at
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

"process reaper" daemon prio=10 tid=0x7f3c0409a000 nid=0x2039
waiting on condition [0x7f3cac04]
   java.lang.Thread.

Re: SolR replication issue

[Mauricio Ferreyra]

The entire stacktrace:

ERROR SolrIndexWriter
Coud not unlock directory after seemingly failed IndexWriter#close()
org.apache.lucene.store.LockReleaseFailedException: Cannot forcefully
unlock a NativeFSLock which is held by another indexer component:
/home/miapp/collection1/data/index.20140901140800014/write.lock
 at
org.apache.lucene.store.NativeFSLock.release(NativeFSLockFactory.java:295)
at org.apache.lucene.index.IndexWriter.unlock(IndexWriter.java:4162)
 at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:156)
at
org.apache.solr.update.DefaultSolrCoreState.newIndexWriter(DefaultSolrCoreState.java:164)
 at
org.apache.solr.update.DirectUpdateHandler2.newIndexWriter(DirectUpdateHandler2.java:624)
at
org.apache.solr.handler.SnapPuller.openNewWriterAndSearcher(SnapPuller.java:622)
 at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:446)
at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:317)
 at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:223)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
 at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
16:39:00
ERROR DefaultSolrCoreState
Error closing old IndexWriter.
core=collection1:java.lang.IllegalArgumentException: Unknown directory:
NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/home/miapp/collection1/data/index.20140901140800014
lockFactory=org.apache.lucene.store.NativeFSLockFactory@77b024a9;
maxCacheMB=48.0 maxMergeSizeMB=4.0)
{NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/home/miapp/collection1/data
lockFactory=org.apache.lucene.store.NativeFSLockFactory@6de47a0f;
maxCacheMB=48.0
maxMergeSizeMB=4.0)=CachedDir<>,?
NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/home/miapp/collection1/data/index.20140901140800016
lockFactory=org.apache.lucene.store.NativeFSLockFactory@791e6360;
maxCacheMB=48.0
maxMergeSizeMB=4.0)=CachedDir<>,?
NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/home/miapp/collection1/data/index.20140901163900022
lockFactory=org.apache.lucene.store.NativeFSLockFactory@6a6de862;
maxCacheMB=48.0
maxMergeSizeMB=4.0)=CachedDir<>,?
NRTCachingDirectory(org.apache.lucene.store.MMapDirectory@/home/miapp/collection1/data/index.20140901140800014
lockFactory=org.apache.lucene.store.NativeFSLockFactory@5863c9e0;
maxCacheMB=48.0
maxMergeSizeMB=4.0)=CachedDir<>}


ERROR SolrIndexWriter SolrIndexWriter was not closed prior to finalize(),​
indicates a bug -- POSSIBLE RESOURCE LEAK!!!






On Mon, Sep 1, 2014 at 4:28 PM, Shawn Heisey  wrote:

> On 9/1/2014 10:31 AM, Mauricio Ferreyra wrote:
> > I'm using Solr 4.3.1 with a master/slave configuration.
> >
> > Configuration:
> >
> > Master:
> > *  *
> > * commit*
> > * startup*
> > * schema.xml,stopwords.txt*
> > *   *
> >
> >
> > Slave:
> >  * *
> > * http://10.xx.xx.xx:9081/solr
> > *
> > * 00:00:60*
> > *   *
> >
> > The replication sometimes fails with the exception
> >
> > *Error closing old IndexWriter.
> > core=collection1:java.lang.IllegalArgumentException: Unknown directory:
> > NRTCachingDirectory...*
> > *ReplicationHandler SnapPull failed
> :org.apache.solr.common.SolrException:
> > Index fetch failed :*
> >
> > This is happening with any index size.
>
> We would need the entire stacktrace from that error message to make any
> real determination, but without that, I offer this:
>
> If it's happening with a tiny index (kilobytes or only a few megabytes),
> then I would suspect a bug in Solr.  4.3.1 is ancient history now --
> Solr's development and release schedule is very aggressive.  There have
> been ten new versions in the 14 months since 4.3.1 was announced, and
> the 4.10 release is imminent.  There have been a number of replication
> bugs fixed in those releases.
>
> If "any index size" means that some of them are in the range of several
> gigabytes, then it may simply be a configuration problem.  The
> commitReserveDuration parameter may need increasing beyond its default
> of ten seconds.  Large updates after optimizing or a major automatic
> merge can take many minutes to transfer, and if that parameter is set
> too low, the old index can disappear before it can finish replicating.
>
> Thanks,
> Shawn
>
>


-- 
*Mauri Ferreyra*
Cordoba - Argentina

Re: SolR replication issue

[Shawn Heisey]

On 9/1/2014 10:31 AM, Mauricio Ferreyra wrote:
> I'm using Solr 4.3.1 with a master/slave configuration.
> 
> Configuration:
> 
> Master:
> *  *
> * commit*
> * startup*
> * schema.xml,stopwords.txt*
> *   *
> 
> 
> Slave:
>  * *
> * http://10.xx.xx.xx:9081/solr
> *
> * 00:00:60*
> *   *
> 
> The replication sometimes fails with the exception
> 
> *Error closing old IndexWriter.
> core=collection1:java.lang.IllegalArgumentException: Unknown directory:
> NRTCachingDirectory...*
> *ReplicationHandler SnapPull failed :org.apache.solr.common.SolrException:
> Index fetch failed :*
> 
> This is happening with any index size.

We would need the entire stacktrace from that error message to make any
real determination, but without that, I offer this:

If it's happening with a tiny index (kilobytes or only a few megabytes),
then I would suspect a bug in Solr.  4.3.1 is ancient history now --
Solr's development and release schedule is very aggressive.  There have
been ten new versions in the 14 months since 4.3.1 was announced, and
the 4.10 release is imminent.  There have been a number of replication
bugs fixed in those releases.

If "any index size" means that some of them are in the range of several
gigabytes, then it may simply be a configuration problem.  The
commitReserveDuration parameter may need increasing beyond its default
of ten seconds.  Large updates after optimizing or a major automatic
merge can take many minutes to transfer, and if that parameter is set
too low, the old index can disappear before it can finish replicating.

Thanks,
Shawn

SolR replication issue

[Mauricio Ferreyra]

Hi folks,
I'm using Solr 4.3.1 with a master/slave configuration.

Configuration:

Master:
*  *
* commit*
* startup*
* schema.xml,stopwords.txt*
*   *


Slave:
 * *
* http://10.xx.xx.xx:9081/solr
*
* 00:00:60*
*   *

The replication sometimes fails with the exception

*Error closing old IndexWriter.
core=collection1:java.lang.IllegalArgumentException: Unknown directory:
NRTCachingDirectory...*
*ReplicationHandler SnapPull failed :org.apache.solr.common.SolrException:
Index fetch failed :*

This is happening with any index size.

Any suggestions would be great

Thanks,

-- 
*Mauri Ferreyra*
Cordoba - Argentina

Inconsistent behavior with Solr replication

[Prasi S]

Hi,
I am using solr 4.4 , replication set with one Master and 1 slave. Master
is set to replicate after startup and commit. It has an internal autocommit
of maxTime:15000.

Slave polls the master every 45sec to check for updates.

I indexed Master with DIH,

First, indexed half million docs in Master and it was replicated to slave.

Second, indexed next half million docs and same was in slave.

Third, while starting the index, by mistake i enalbed clean=true but
immediately gave another import with clean=false. now the Master has 1.5 M
docs but the slave has zero docs. How is this possible.

If i give, replication?command=indexversion in both, the version and
generation numbers are same. but slave has zero.
*Master:*
1402644323820
695
*Slave:*
1402644323820
695

*Admin UI Master REplication page:*

  Index Version Gen Size Master (Searching)
1402644188557
694
63.6 MB
Master (Replicable)
1402647815765
711
-
Settings (Master):

   - replication enable:
   - replicateAfter:commit, startup

*Admin UI Slave replication page:*

 Index Version Gen Size Master (Searching)
1402644188557
694
96.4 MB
Master (Replicable)
1402647831725
712
-
Slave (Searching)
1402647831725
712
67.69 MB
Settings:

   - master url:http:/ip:port/solr/col1
   - polling enable: (interval: 00:00:45)

Settings (Master):

   - replication enable:
   - replicateAfter:commit, startup


Thanks,
Prasi

Re: Solr Replication Issue : Incorrect Base_URL

[Alan Woodward]

Hi Pramod,

You need to set hostContext in your solr.xml.  See 
https://cwiki.apache.org/confluence/display/solr/Format+of+solr.xml

Alan Woodward
www.flax.co.uk


On 13 Jun 2014, at 00:44, pramodEbay wrote:

> Hi,
> I am deploying Solr in a larger web application. The standalone solr
> instance works fine. The path-prefix I use is raptorslrweb. A standalone
> SOLR query to my instance that works is as follows:
> 
> http://hostname:8080/raptorslrweb/solr/reviews/select?q=*%3A*&wt=json&indent=true
> 
> However, when I configure a solr cloud, I get the following error in
> RecoveryStrategy:
> "msg":"org.apache.solr.client.solrj.SolrServerException: Server at
> http://hostname:8080/solr/reviews sent back a redirect (302).",
> 
> The reason is the base_url does not seem to honor the path-prefix.
> clusterstate.json shows the following for the node:
> {"reviews":{
>"shards":{"shard1":{
>"range":null,
>"state":"active",
>"parent":null,
>"replicas":{
>  "core_node1":{
>"state":"down",
>   * "base_url":"http://hostname:8080/solr",*   
>"core":"reviews",
>"node_name":"10.98.63.98:8080_solr"},
> 
> Can someone please tell me where do I tell zookeeper or solr cloud that the
> base url should be hostname:8080/raptorslrweb/solr and not
> hostname:8080/solr.
> 
> Thanks,
> Pramod
> 
> 
> 
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Solr-Replication-Issue-Incorrect-Base-URL-tp4141537.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Solr Replication Issue : Incorrect Base_URL

[pramodEbay]

Hi,
I am deploying Solr in a larger web application. The standalone solr
instance works fine. The path-prefix I use is raptorslrweb. A standalone
SOLR query to my instance that works is as follows:

http://hostname:8080/raptorslrweb/solr/reviews/select?q=*%3A*&wt=json&indent=true

However, when I configure a solr cloud, I get the following error in
RecoveryStrategy:
"msg":"org.apache.solr.client.solrj.SolrServerException: Server at
http://hostname:8080/solr/reviews sent back a redirect (302).",

The reason is the base_url does not seem to honor the path-prefix.
clusterstate.json shows the following for the node:
{"reviews":{
"shards":{"shard1":{
"range":null,
"state":"active",
"parent":null,
"replicas":{
  "core_node1":{
"state":"down",
   * "base_url":"http://hostname:8080/solr",*   
"core":"reviews",
"node_name":"10.98.63.98:8080_solr"},

Can someone please tell me where do I tell zookeeper or solr cloud that the
base url should be hostname:8080/raptorslrweb/solr and not
hostname:8080/solr.

Thanks,
Pramod



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-Replication-Issue-Incorrect-Base-URL-tp4141537.html
Sent from the Solr - User mailing list archive at Nabble.com.

Solr Replication Index directory disappears

[anand chandak]

 Hi,


We are using solr 4.4 version and am facing some issue with replication, 
we are trying to replicate very large index about 105G. Once the 
replication is done, I see the index. directory but the index 
directory is not present ? Why is this happening ?



Also, I see with every iteration the slave is fetching the entire 
content (full replication) from the master, even if nothing is changed 
on the master. What could be reason ?


Thanks,

Anand

Re: SOLR replication question?

[Shawn Heisey]

> I am currently using SOLR 4.4. but not planning to use solrcloud in very
near
> future.
> I have 3 master / 3 slave setup. Each master is linked to its
> corresponding
> slave.. I have disabled auto polling..
> We do both push (using MQ) and pull indexing using SOLRJ indexing
program.
> I have enabled soft commit in slave (to view the changes immediately pushed
> by queue).
> I am thinking of doing the batch indexing in master (optimize and hard
commit) and push indexing in both master / slave.
> I am trying to do more testing with my configuration but thought of getting
> to know some answers before diving very deep...
> Since the queue pushes the docs in master / slave there is a possibility of
> slave having more record compared to master (when master is busy doing
batch
> indexing).. What would happen if the slave has additional segments compared
> to Master. will that be deleted when the replication happens.
> If a message is pushed from a queue to both master and slave during
replication, will there be a latency in seeing that document even if we
use
> softcommit in slave?
> We want to make sure that we are not missing any documents from queue
(since
> its updated via UI and we don't really store that data anywhere except
in
> index)

If you are doing replication, then all updates must go to the master
server. You cannot update the slave directly. The replication happens, the
slave will be identical to the master... Any documents aent to only the
slave will be lost.

Replication will happen according to the interval you have configured, or
since you say you have disabled polling, according to whatever schedule
you manually trigger a replication.

SolrCloud would probably be a better fit for you. With a properly
configured SolrCloud you just index to any host in the cloud and documents
end up exactly where they need to go, and all replicas get updated.

Thanks,
Shawn

SOLR replication question?

[SolrLover]

I am currently using SOLR 4.4. but not planning to use solrcloud in very near
future.

I have 3 master / 3 slave setup. Each master is linked to its corresponding
slave.. I have disabled auto polling.. 

We do both push (using MQ) and pull indexing using SOLRJ indexing program.

I have enabled soft commit in slave (to view the changes immediately pushed
by queue).

I am thinking of doing the batch indexing in master (optimize and hard
commit) and push indexing in both master / slave. 

I am trying to do more testing with my configuration but thought of getting
to know some answers before diving very deep...

Since the queue pushes the docs in master / slave there is a possibility of
slave having more record compared to master (when master is busy doing batch
indexing).. What would happen if the slave has additional segments compared
to Master. will that be deleted when the replication happens.

If a message is pushed from a queue to both master and slave during
replication, will there be a latency in seeing that document even if we use
softcommit in slave?

We want to make sure that we are not missing any documents from queue (since
its updated via UI and we don't really store that data anywhere except in
index).







--
View this message in context: 
http://lucene.472066.n3.nabble.com/SOLR-replication-question-tp4081161.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: problems about solr replication in 4.3

[Erick Erickson]

Well, a full import is going to re-import everything in the database, and the
presumption is that each every document would be replaced (because
presumably you're  is the same). So every document
will be deleted and re-added. So essentially you'll get a completely
new index every time.

In 3.6 are you sure you weren't doing a delta query?

Best
Erick

On Thu, Jul 25, 2013 at 9:57 PM, xiaoqi  wrote:
> thank u for replying very much .
>
> in fact ,we make a process for this problem , we found when master building
> index, it will clean self index when building index . so slave every minute
> to sync index, destroy self index folder.
>
> by the way : we building index using
> dataimport0?command=full-import&clean=false
> ,dataimport1?command=full-import&clean=false,
> dataimport2?command=full-import&clean=false .
>
>  when i using in solr3.6 has no problem ,never delete at first .
>
> does solr 4 need to special config anything ?
>
> thanks a lot .
>
>
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/problems-about-solr-replication-in-4-3-tp4079665p4080480.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: problems about solr replication in 4.3

[xiaoqi]

thank u for replying very much .

in fact ,we make a process for this problem , we found when master building 
index, it will clean self index when building index . so slave every minute
to sync index, destroy self index folder.  

by the way : we building index using
dataimport0?command=full-import&clean=false
,dataimport1?command=full-import&clean=false, 
dataimport2?command=full-import&clean=false .

 when i using in solr3.6 has no problem ,never delete at first . 

does solr 4 need to special config anything ? 

thanks a lot .



--
View this message in context: 
http://lucene.472066.n3.nabble.com/problems-about-solr-replication-in-4-3-tp4079665p4080480.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: problems about solr replication in 4.3

[Erick Erickson]

Are you mixing SolrCloud and old-style master/slave?

There was a bug a while ago (search the JIRA) where
replication was copying the entire index unnecessarily,
but I think that was fixed by 4.3.

Best
Erick

On Tue, Jul 23, 2013 at 6:33 AM, xiaoqi  wrote:
>
> hi,all
>
> i have two solr ,one is master , one is replication , before i use them
> under 3.5 version . it works fine .
>  when i upgrade to 4.3version , i found when replication solr copying index
> from master , it will clean current index and copy new version to self
> folder . slave can't search  during this process !
>
> i am newer to solr 4 , does this normal ? any ideas , thanks !
>
>
>
>
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/problems-about-solr-replication-in-4-3-tp4079665.html
> Sent from the Solr - User mailing list archive at Nabble.com.

problems about solr replication in 4.3

[xiaoqi]

hi,all 

i have two solr ,one is master , one is replication , before i use them
under 3.5 version . it works fine .
 when i upgrade to 4.3version , i found when replication solr copying index
from master , it will clean current index and copy new version to self
folder . slave can't search  during this process !

i am newer to solr 4 , does this normal ? any ideas , thanks !





--
View this message in context: 
http://lucene.472066.n3.nabble.com/problems-about-solr-replication-in-4-3-tp4079665.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr replication is extremely slow(less then 1MB/s)

[William Bell]

I agree. It is even slower when the slave is being pounded.


On Fri, Jun 21, 2013 at 3:35 AM, Ted  wrote:

> Solr replication is extremely slow(less then 1MB/s)
>
> When the replication is runinng,network and disk occupancy rate remained at
> a very low level.
>
> I've tried downloading a piece of the index file with browser.
> like this:
>
> http://master/solr/product/replication?command=filecontent&wt=filestream&indexversion=11&file=
> <
> http://master/solr/product/replication?command=filecontent&wt=filestream&indexversion=11&file=
> >
>
> it can reach more then 10MB/s.
>
> so i'm confused,is there anybody can tell me how to check for this problem?
>
> Environment：
> 1.gigabit network
> 2.both master & slave run on windows
> 3.jdk 1.6u24 i586
> 4.solr 3.5
> 5.tomcat7
>
> Looking forward to your reply,tks!
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-replication-is-extremely-slow-less-then-1MB-s-tp4072096.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Solr replication is extremely slow(less then 1MB/s)

[Ted]

Solr replication is extremely slow(less then 1MB/s)

When the replication is runinng,network and disk occupancy rate remained at
a very low level.

I've tried downloading a piece of the index file with browser.
like this: 
http://master/solr/product/replication?command=filecontent&wt=filestream&indexversion=11&file=
<http://master/solr/product/replication?command=filecontent&wt=filestream&indexversion=11&file=>
  

it can reach more then 10MB/s.

so i'm confused,is there anybody can tell me how to check for this problem?

Environment：
1.gigabit network
2.both master & slave run on windows
3.jdk 1.6u24 i586
4.solr 3.5
5.tomcat7

Looking forward to your reply,tks!



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-replication-is-extremely-slow-less-then-1MB-s-tp4072096.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Troubles with solr replication

[Chris Hostetter]

: Also when I checked the solr log.
: 
: > [org.apache.solr.handler.SnapPuller] Master at:
: > http://192.168.2.204:8080/solr/replication is not available. Index fetch
: > failed. Exception: Connection refused
: 
: 
: BTW, I was able to fetch the replication file with wget directly.

Are you certian that the network setup for your master & slave machines 
alows them to talk to eachother?  you said you could fetch the files from 
the master via wget, but i'm guessing you were running this from your 
local machine -- are you certain that when logged in to 192.168.2.174 you 
can reach port 8080 of 192.168.2.204?


-Hoss

Troubles with solr replication

[Sergii Khomenko]

Hi guys,

I have some problems with Solr replication and can see some
unexpected behavior.
Would be nice to have some answers where am I wrong, or what is the best
way to solve the problem.


I have a replication master-slave. http://192.168.2.204:8080/solr/ is
master and http://192.168.2.174:8080/solr/ is slave. With quite simple
config.




false


commit


startup


schema.xml,boosting.txt




true

http://192.168.2.204:8080/solr/replication
00:00:60

5000


1




The main idea when I started playing around Solr is to replicate some
boosting values.
I wanted to use confFile option for it. An here is my first problem. I
wasn't able to replicate files from master. On slave I was able to see only
schema.xml.

I wanted to check, do I actually have the files and everything correct in
solr config.
So I checked on master the file list and it returns the list of all files
http://192.168.2.204:8080/solr/replication?command=filelist&indexversion=1341328964983


but for slave I can't see anything
http://192.168.2.174:8080/solr/replication?command=filelist&indexversion=1341328964983

returns

invalid indexversion

Seems like we don't have this index version.

After I tried to find what is wrong with that. On slave
http://192.168.2.174:8080/solr/replication?command=indexversion
returns only 0

0
> 0


on master I could see the version of current index

1341328964983
> 3


but slave's
http://192.168.2.174:8080/solr/admin/stats.jsp I can the the right version

indexVersion : 1341328964983
> generation : 3


Also when I checked the solr log.

> [org.apache.solr.handler.SnapPuller] Master at:
> http://192.168.2.204:8080/solr/replication is not available. Index fetch
> failed. Exception: Connection refused


BTW, I was able to fetch the replication file with wget directly.

So my question is: What is wrong with my replication or Solr?
About version, I use some legacy version of Solr: Solr Specification
Version: 3.5.0.2011.11.22.14.54.38, because we have some legacy systems
here.

And another question what is the best way to migrate to the latest version.
I mean to keep alive all the boosting infrastructure based on
ExternalFileField options.

Thank you in advance for your time and help you can provide,
Sergii

Troubles with solr replication

[Sergii Khomenko]

Hi guys,

I have some problems with Solr replication and can see some
unexpected behavior.
Would be nice to have some answers where am I wrong, or what is the best
way to solve the problem.


I have a replication master-slave. http://192.168.2.204:8080/solr/ is
master and http://192.168.2.174:8080/solr/ is slave. With quite simple
config.




false


commit


startup


schema.xml,boosting.txt




true

http://192.168.2.204:8080/solr/replication
00:00:60

5000


1




The main idea when I started playing around Solr is to replicate some
boosting values.
I wanted to use confFile option for it. An here is my first problem. I
wasn't able to replicate files from master. On slave I was able to see only
schema.xml.

I wanted to check, do I actually have the files and everything correct in
solr config.
So I checked on master the file list and it returns the list of all files
http://192.168.2.204:8080/solr/replication?command=filelist&indexversion=1341328964983


but for slave I can't see anything
http://192.168.2.174:8080/solr/replication?command=filelist&indexversion=1341328964983

returns

invalid indexversion

Seems like we don't have this index version.

After I tried to find what is wrong with that. On slave
http://192.168.2.174:8080/solr/replication?command=indexversion
returns only 0

0
> 0


on master I could see the version of current index

1341328964983
> 3


but slave's
http://192.168.2.174:8080/solr/admin/stats.jsp I can the the right version

indexVersion : 1341328964983
> generation : 3


Also when I checked the solr log.

> [org.apache.solr.handler.SnapPuller] Master at:
> http://192.168.2.204:8080/solr/replication is not available. Index fetch
> failed. Exception: Connection refused


BTW, I was able to fetch the replication file with wget directly.

So my question is: What is wrong with my replication or Solr?
About version, I use some legacy version of Solr: Solr Specification
Version: 3.5.0.2011.11.22.14.54.38, because we have some legacy systems
here.

And another question what is the best way to migrate to the latest version.
I mean to keep alive all the boosting infrastructure based on
ExternalFileField options.

Thank you in advance for your time and help you can provide,
Sergii

Re: Problem with Solr replication in solr 4.2

[Mark Miller]

On Mar 21, 2013, at 12:08 PM, Rohit Harchandani  wrote:

> Hey,
> Currently we are using solr 4.0 with a master slave setup. The data gets
> indexed on the master and then we issue a fetchindex command to replicate
> it on the slave. The slave has a postCommit listener which gets kicked off
> when replication finishes and we depend on this listener to know whn
> replication is done. But when I tried to do the same with 4.2, the commit
> does not seem to be happening. Is this a known issue? is there any other
> way to know that replication is done?

Yeah, we stopped doing this commit because it changes some index meta data and 
shouldn't be necessary. I'd open a JIRA issue about being able to listen for 
replication finishing.

> Also, initially when i tried solr 4.2, i noticed with this setup, i noticed
> that with the fetchIndex command, the fields were downloaded to the temp
> folder, but it was never pulled into the index directory on the slave. The
> only file which made it was the lock file. This problem does not happen
> anymore?

I don't know, does it? Can you file a JIRA with instructions to replicate?

4.2.1 is about to go out, if we are quick, perhaps we can address something 
here if there is a problem.

- Mark

> Thanks,
> Rohit

Problem with Solr replication in solr 4.2

[Rohit Harchandani]

Hey,
Currently we are using solr 4.0 with a master slave setup. The data gets
indexed on the master and then we issue a fetchindex command to replicate
it on the slave. The slave has a postCommit listener which gets kicked off
when replication finishes and we depend on this listener to know whn
replication is done. But when I tried to do the same with 4.2, the commit
does not seem to be happening. Is this a known issue? is there any other
way to know that replication is done?
Also, initially when i tried solr 4.2, i noticed with this setup, i noticed
that with the fetchIndex command, the fields were downloaded to the temp
folder, but it was never pulled into the index directory on the slave. The
only file which made it was the lock file. This problem does not happen
anymore?
Thanks,
Rohit

Re: Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?

[Rafał Radecki]

Yes, this is solr 4.1.

2013/3/20 Mark Miller :
> Hmm, I'd have to look, but first to make, this subject says 4.1?
>
> In 4.1 the slave will be ahead because it commits after installing the index. 
> In 4.2 it shouldn't.
>
> Your on?
>
> - Mark
>
> On Mar 20, 2013, at 4:03 AM, Rafał Radecki  wrote:
>
>> Thanks for the info.
>> I understand that the latest "replicateable" version of index may
>> differ from the actual version of index on master/slave.
>> But why when I use
>> /solr/replication?command=indexversion
>>
>> On master:
>>
>> 
>> 
>> 0
>> 0
>> 
>> 1363263304585
>> 4
>> 
>>
>> And on slave:
>>
>> 
>> 
>> 0
>> 1
>> 
>> 1363263600323
>> 5
>> 
>>
>> Why do I get higher version on slave than on master?
>>
>> The page https://wiki.apache.org/solr/SolrReplication states:
>> "Get the latest replicateable index on master:
>> http://master_host:port/solr/replication?command=indexversion"; -> ok,
>> returns latest replicatable index version
>> "Get version number of the index:
>> http://host:port/solr/replication?command=indexversion"; -> for me it
>> means that on slave I should get actual index version and not an index
>> version appropriate for replication (it is a slave, it should not be
>> the source of replication)
>> Yes, it is confusing overall :)
>>
>> My question is: how to get actual index version on master and slave
>> for monitoring purpose? I can use: /solr/replication?command=details
>>
>> On master:
>>
>> 
>> 
>> 0
>> 1
>> 
>> 
>> 22.59 KB
>> /usr/share/solr/data/index/
>> 
>> 
>> 1363263304585
>> 4
>> 
>> _2_Lucene41_0.pos
>> _2.si
>> _2_Lucene41_0.tim
>> _2.fdt
>> _2_Lucene41_0.doc
>> _2_Lucene41_0.tip
>> _2.fdx
>> _2.tvx
>> _2.fnm
>> _2_nrm.cfe
>> _2.tvd
>> _2_Lucene41_0.pay
>> _2_nrm.cfs
>> _2.tvf
>> segments_4
>> 
>> 
>> 
>> true
>> false
>> 1363263304585
>> 4
>> 
>> schema.xml,stopwords.txt
>> 
>> commit
>> startup
>> 
>> true
>> 4
>> 
>> 
>> 
>> This response format is experimental. It is likely to change in the future.
>> 
>> 
>>
>> On slave:
>>
>> 
>> 
>> 0
>> 25
>> 
>> 
>> 22.59 KB
>> /usr/share/solr/data/index/
>> 
>> 
>> 1363263600323
>> 5
>> 
>> _2_Lucene41_0.pos
>> _2.si
>> _2_Lucene41_0.tim
>> _2.fdt
>> _2_Lucene41_0.doc
>> _2_Lucene41_0.tip
>> _2.fdx
>> _2.tvx
>> _2.fnm
>> _2_nrm.cfe
>> _2.tvd
>> _2_Lucene41_0.pay
>> _2_nrm.cfs
>> segments_5
>> _2.tvf
>> 
>> 
>> 
>> false
>> true
>> 1363263304585
>> 4
>> 
>> 
>> 22.59 KB
>> /usr/share/solr/data/index/
>> 
>> 
>> 1363263304585
>> 4
>> 
>> _2_Lucene41_0.pos
>> _2.si
>> _2_Lucene41_0.tim
>> _2.fdt
>> _2_Lucene41_0.doc
>> _2_Lucene41_0.tip
>> _2.fdx
>> _2.tvx
>> _2.fnm
>> _2_nrm.cfe
>> _2.tvd
>> _2_Lucene41_0.pay
>> _2_nrm.cfs
>> _2.tvf
>> segments_4
>> 
>> 
>> 
>> true
>> false
>> 1363263304585
>> 4
>> 
>> schema.xml,stopwords.txt
>> 
>> commit
>> startup
>> 
>> true
>> 4
>> 
>> 
>> http://172.18.19.204:8080/solr
>> 00:00:60
>> Wed Mar 20 08:48:00 CET 2013
>> Thu Mar 14 13:20:00 CET 2013
>> 
>> Thu Mar 14 13:20:00 CET 2013
>> Thu Mar 14 13:19:00 CET 2013
>> Thu Mar 14 12:18:00 CET 2013
>> Thu Mar 14 12:17:00 CET 2013
>> Fri Mar 08 14:55:00 CET 2013
>> Fri Mar 08 14:50:52 CET 2013
>> Fri Mar 08 14:32:00 CET 2013
>> 
>> 7
>> 23212
>> 0
>> Wed Mar 20 08:47:03 CET 2013
>> false
>> false
>> 
>> 
>> 
>> This response format is experimental. It is likely to change in the future.
>> 
>> 
>>
>> Which line with "indexVersion" should I use?
>>
>> Best regards,
>> Rafal Radecki.
>>
>>
>> 2013/3/19 Mark Miller :
>>>
>>> On Mar 15, 2013, at 6:43 AM, Rafał Radecki  wrote:
>>>
>>>> I use http and get /solr/replication?command=indexversion urls to get
>>>> index versions on master and slave. The replication works fine but
>>>> index versions from /solr/replication?command=indexversion differ.
>>>
>>> I think thats normal - it's a little misleading, but the replication index 
>>> version is that of the most recent 'replicatable' commit point. That is 
>>> determined by things like replicate on startup, replicate on commit, etc.
>>>
>>> These are very likely to be different from mater to slave.
>>>
>>> -  Mark
>

Re: Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?

[Mark Miller]

Hmm, I'd have to look, but first to make, this subject says 4.1?

In 4.1 the slave will be ahead because it commits after installing the index. 
In 4.2 it shouldn't.

Your on?

- Mark

On Mar 20, 2013, at 4:03 AM, Rafał Radecki  wrote:

> Thanks for the info.
> I understand that the latest "replicateable" version of index may
> differ from the actual version of index on master/slave.
> But why when I use
> /solr/replication?command=indexversion
> 
> On master:
> 
> 
> 
> 0
> 0
> 
> 1363263304585
> 4
> 
> 
> And on slave:
> 
> 
> 
> 0
> 1
> 
> 1363263600323
> 5
> 
> 
> Why do I get higher version on slave than on master?
> 
> The page https://wiki.apache.org/solr/SolrReplication states:
> "Get the latest replicateable index on master:
> http://master_host:port/solr/replication?command=indexversion"; -> ok,
> returns latest replicatable index version
> "Get version number of the index:
> http://host:port/solr/replication?command=indexversion"; -> for me it
> means that on slave I should get actual index version and not an index
> version appropriate for replication (it is a slave, it should not be
> the source of replication)
> Yes, it is confusing overall :)
> 
> My question is: how to get actual index version on master and slave
> for monitoring purpose? I can use: /solr/replication?command=details
> 
> On master:
> 
> 
> 
> 0
> 1
> 
> 
> 22.59 KB
> /usr/share/solr/data/index/
> 
> 
> 1363263304585
> 4
> 
> _2_Lucene41_0.pos
> _2.si
> _2_Lucene41_0.tim
> _2.fdt
> _2_Lucene41_0.doc
> _2_Lucene41_0.tip
> _2.fdx
> _2.tvx
> _2.fnm
> _2_nrm.cfe
> _2.tvd
> _2_Lucene41_0.pay
> _2_nrm.cfs
> _2.tvf
> segments_4
> 
> 
> 
> true
> false
> 1363263304585
> 4
> 
> schema.xml,stopwords.txt
> 
> commit
> startup
> 
> true
> 4
> 
> 
> 
> This response format is experimental. It is likely to change in the future.
> 
> 
> 
> On slave:
> 
> 
> 
> 0
> 25
> 
> 
> 22.59 KB
> /usr/share/solr/data/index/
> 
> 
> 1363263600323
> 5
> 
> _2_Lucene41_0.pos
> _2.si
> _2_Lucene41_0.tim
> _2.fdt
> _2_Lucene41_0.doc
> _2_Lucene41_0.tip
> _2.fdx
> _2.tvx
> _2.fnm
> _2_nrm.cfe
> _2.tvd
> _2_Lucene41_0.pay
> _2_nrm.cfs
> segments_5
> _2.tvf
> 
> 
> 
> false
> true
> 1363263304585
> 4
> 
> 
> 22.59 KB
> /usr/share/solr/data/index/
> 
> 
> 1363263304585
> 4
> 
> _2_Lucene41_0.pos
> _2.si
> _2_Lucene41_0.tim
> _2.fdt
> _2_Lucene41_0.doc
> _2_Lucene41_0.tip
> _2.fdx
> _2.tvx
> _2.fnm
> _2_nrm.cfe
> _2.tvd
> _2_Lucene41_0.pay
> _2_nrm.cfs
> _2.tvf
> segments_4
> 
> 
> 
> true
> false
> 1363263304585
> 4
> 
> schema.xml,stopwords.txt
> 
> commit
> startup
> 
> true
> 4
> 
> 
> http://172.18.19.204:8080/solr
> 00:00:60
> Wed Mar 20 08:48:00 CET 2013
> Thu Mar 14 13:20:00 CET 2013
> 
> Thu Mar 14 13:20:00 CET 2013
> Thu Mar 14 13:19:00 CET 2013
> Thu Mar 14 12:18:00 CET 2013
> Thu Mar 14 12:17:00 CET 2013
> Fri Mar 08 14:55:00 CET 2013
> Fri Mar 08 14:50:52 CET 2013
> Fri Mar 08 14:32:00 CET 2013
> 
> 7
> 23212
> 0
> Wed Mar 20 08:47:03 CET 2013
> false
> false
> 
> 
> 
> This response format is experimental. It is likely to change in the future.
> 
> 
> 
> Which line with "indexVersion" should I use?
> 
> Best regards,
> Rafal Radecki.
> 
> 
> 2013/3/19 Mark Miller :
>> 
>> On Mar 15, 2013, at 6:43 AM, Rafał Radecki  wrote:
>> 
>>> I use http and get /solr/replication?command=indexversion urls to get
>>> index versions on master and slave. The replication works fine but
>>> index versions from /solr/replication?command=indexversion differ.
>> 
>> I think thats normal - it's a little misleading, but the replication index 
>> version is that of the most recent 'replicatable' commit point. That is 
>> determined by things like replicate on startup, replicate on commit, etc.
>> 
>> These are very likely to be different from mater to slave.
>> 
>> -  Mark

Re: Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?

[Rafał Radecki]

Thanks for the info.
I understand that the latest "replicateable" version of index may
differ from the actual version of index on master/slave.
But why when I use
/solr/replication?command=indexversion

On master:



0
0

1363263304585
4


And on slave:



0
1

1363263600323
5


Why do I get higher version on slave than on master?

The page https://wiki.apache.org/solr/SolrReplication states:
"Get the latest replicateable index on master:
http://master_host:port/solr/replication?command=indexversion"; -> ok,
returns latest replicatable index version
"Get version number of the index:
http://host:port/solr/replication?command=indexversion"; -> for me it
means that on slave I should get actual index version and not an index
version appropriate for replication (it is a slave, it should not be
the source of replication)
Yes, it is confusing overall :)

My question is: how to get actual index version on master and slave
for monitoring purpose? I can use: /solr/replication?command=details

On master:



0
1


22.59 KB
/usr/share/solr/data/index/


1363263304585
4

_2_Lucene41_0.pos
_2.si
_2_Lucene41_0.tim
_2.fdt
_2_Lucene41_0.doc
_2_Lucene41_0.tip
_2.fdx
_2.tvx
_2.fnm
_2_nrm.cfe
_2.tvd
_2_Lucene41_0.pay
_2_nrm.cfs
_2.tvf
segments_4



true
false
1363263304585
4

schema.xml,stopwords.txt

commit
startup

true
4



This response format is experimental. It is likely to change in the future.



On slave:



0
25


22.59 KB
/usr/share/solr/data/index/


1363263600323
5

_2_Lucene41_0.pos
_2.si
_2_Lucene41_0.tim
_2.fdt
_2_Lucene41_0.doc
_2_Lucene41_0.tip
_2.fdx
_2.tvx
_2.fnm
_2_nrm.cfe
_2.tvd
_2_Lucene41_0.pay
_2_nrm.cfs
segments_5
_2.tvf



false
true
1363263304585
4


22.59 KB
/usr/share/solr/data/index/


1363263304585
4

_2_Lucene41_0.pos
_2.si
_2_Lucene41_0.tim
_2.fdt
_2_Lucene41_0.doc
_2_Lucene41_0.tip
_2.fdx
_2.tvx
_2.fnm
_2_nrm.cfe
_2.tvd
_2_Lucene41_0.pay
_2_nrm.cfs
_2.tvf
segments_4



true
false
1363263304585
4

schema.xml,stopwords.txt

commit
startup

true
4


http://172.18.19.204:8080/solr
00:00:60
Wed Mar 20 08:48:00 CET 2013
Thu Mar 14 13:20:00 CET 2013

Thu Mar 14 13:20:00 CET 2013
Thu Mar 14 13:19:00 CET 2013
Thu Mar 14 12:18:00 CET 2013
Thu Mar 14 12:17:00 CET 2013
Fri Mar 08 14:55:00 CET 2013
Fri Mar 08 14:50:52 CET 2013
Fri Mar 08 14:32:00 CET 2013

7
23212
0
Wed Mar 20 08:47:03 CET 2013
false
false



This response format is experimental. It is likely to change in the future.



Which line with "indexVersion" should I use?

Best regards,
Rafal Radecki.


2013/3/19 Mark Miller :
>
> On Mar 15, 2013, at 6:43 AM, Rafał Radecki  wrote:
>
>> I use http and get /solr/replication?command=indexversion urls to get
>> index versions on master and slave. The replication works fine but
>> index versions from /solr/replication?command=indexversion differ.
>
> I think thats normal - it's a little misleading, but the replication index 
> version is that of the most recent 'replicatable' commit point. That is 
> determined by things like replicate on startup, replicate on commit, etc.
>
> These are very likely to be different from mater to slave.
>
> -  Mark

Re: Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?

[Mark Miller]

On Mar 15, 2013, at 6:43 AM, Rafał Radecki  wrote:

> I use http and get /solr/replication?command=indexversion urls to get
> index versions on master and slave. The replication works fine but
> index versions from /solr/replication?command=indexversion differ.

I think thats normal - it's a little misleading, but the replication index 
version is that of the most recent 'replicatable' commit point. That is 
determined by things like replicate on startup, replicate on commit, etc.

These are very likely to be different from mater to slave.

-  Mark

Re: Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?

[Chris Hostetter]

Deja-vu?

http://mail-archives.apache.org/mod_mbox/lucene-general/201303.mbox/%3CCAHd9_iR-HtNDu-3a9A5ekTFdb+5mo1eWVcu4Shp8AD=qtpq...@mail.gmail.com%3E

http://mail-archives.apache.org/mod_mbox/lucene-general/201303.mbox/%3Calpine.DEB.2.02.1303081644040.5502@frisbee%3E

https://issues.apache.org/jira/browse/SOLR-3681


: Date: Fri, 15 Mar 2013 11:43:05 +0100
: From: Rafał Radecki 
: Reply-To: solr-user@lucene.apache.org
: To: solr-user@lucene.apache.org
: Subject: Re: Solr 4.1 monitoring with /solr/replication?command=details -
: indexVersion?
: 
: I use http and get /solr/replication?command=indexversion urls to get
: index versions on master and slave. The replication works fine but
: index versions from /solr/replication?command=indexversion differ.
: 
: Best regards,
: Rafal.
: 
: 2013/3/14 Mark Miller :
: > What calls are you using to get the versions? Or is it the admin UI?
: >
: > Also can you add any details about your setup - if this is a problem, we 
need to duplicate it in one of our unit tests.
: >
: > Also, is it affecting proper replication in any way that you can tell.
: >
: > - Mark
: >
: > On Mar 14, 2013, at 11:12 AM, richardg  wrote:
: >
: >> I believe this is the same issue as described, I'm running 4.2 and as you 
can
: >> see my slave is a couple versions ahead of the master (all three slaves 
show
: >> the same behavior).  This was never the case until I upgraded from 4.0 to
: >> 4.2.
: >>
: >> Master:
: >> 1363272681951
: >> 93
: >> 1,022.31 MB
: >> Slave:
: >> 1363273274085
: >> 95
: >> 1,022.31 MB
: >>
: >>
: >>
: >> --
: >> View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-4-1-monitoring-with-solr-replication-command-details-indexVersion-tp4047329p4047380.html
: >> Sent from the Solr - User mailing list archive at Nabble.com.
: >
: 

-Hoss

Re: Solr Replication

[Erick Erickson]

Why do you think that during a backup any write activity would corrupt your
backup? Solr "backs up" live indexes all the time, that's what's happening,
in effect, when you replicate from a master to a slave. There's no
requirement that the master stop indexing. The replication "backup" command
Ahmet told you about is using the same mechanism.

Solr/Lucene index segments are write-once. backup/replication essentially
copies closed segments around which, since they aren't changing, aren't
affected by other indexing activity.

So the process is this:

1> a backup/replication starts. At this point, a list is made of all the
currently-closed segments and these segments will NOT be deleted until the
backup/replication is complete.
2> all the segments in <1> are copied. This is a complete snapshot of your
index.
3> after all the segments are copied, they are released and may then be
deleted in the normal merge process.

Of course another way to do backups would be set up a machine that just
does a replication (think of it as a "special slave") periodically. Another
would be to have the master keep backups automatically, seesee:
http://wiki.apache.org/solr/SolrReplication,  and look for numberOfBackups.

Best
Erick


On Fri, Mar 15, 2013 at 1:03 AM, vicky desai wrote:

> Hi,
>
> I have a multi core setup and there is continuous updation going on in each
> core. Hence I dont prefer a bckup as it would either cause a downtime or if
> during a backup there is a write activity my backup will be corrupted. Can
> you please suggest if there is a cleaner way to handle this
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-Replication-tp4047266p4047591.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?

[Rafał Radecki]

I use http and get /solr/replication?command=indexversion urls to get
index versions on master and slave. The replication works fine but
index versions from /solr/replication?command=indexversion differ.

Best regards,
Rafal.

2013/3/14 Mark Miller :
> What calls are you using to get the versions? Or is it the admin UI?
>
> Also can you add any details about your setup - if this is a problem, we need 
> to duplicate it in one of our unit tests.
>
> Also, is it affecting proper replication in any way that you can tell.
>
> - Mark
>
> On Mar 14, 2013, at 11:12 AM, richardg  wrote:
>
>> I believe this is the same issue as described, I'm running 4.2 and as you can
>> see my slave is a couple versions ahead of the master (all three slaves show
>> the same behavior).  This was never the case until I upgraded from 4.0 to
>> 4.2.
>>
>> Master:
>> 1363272681951
>> 93
>> 1,022.31 MB
>> Slave:
>> 1363273274085
>> 95
>> 1,022.31 MB
>>
>>
>>
>> --
>> View this message in context: 
>> http://lucene.472066.n3.nabble.com/Solr-4-1-monitoring-with-solr-replication-command-details-indexVersion-tp4047329p4047380.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Solr Replication

[vicky desai]

Hi,

I have a multi core setup and there is continuous updation going on in each
core. Hence I dont prefer a bckup as it would either cause a downtime or if
during a backup there is a write activity my backup will be corrupted. Can
you please suggest if there is a cleaner way to handle this



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-Replication-tp4047266p4047591.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?

[Mark Miller]

What calls are you using to get the versions? Or is it the admin UI?

Also can you add any details about your setup - if this is a problem, we need 
to duplicate it in one of our unit tests.

Also, is it affecting proper replication in any way that you can tell.

- Mark

On Mar 14, 2013, at 11:12 AM, richardg  wrote:

> I believe this is the same issue as described, I'm running 4.2 and as you can
> see my slave is a couple versions ahead of the master (all three slaves show
> the same behavior).  This was never the case until I upgraded from 4.0 to
> 4.2.
> 
> Master:   
> 1363272681951
> 93
> 1,022.31 MB
> Slave:
> 1363273274085
> 95
> 1,022.31 MB
> 
> 
> 
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Solr-4-1-monitoring-with-solr-replication-command-details-indexVersion-tp4047329p4047380.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?

[richardg]

I believe this is the same issue as described, I'm running 4.2 and as you can
see my slave is a couple versions ahead of the master (all three slaves show
the same behavior).  This was never the case until I upgraded from 4.0 to
4.2.

Master: 
1363272681951
93
1,022.31 MB
Slave:  
1363273274085
95
1,022.31 MB



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-4-1-monitoring-with-solr-replication-command-details-indexVersion-tp4047329p4047380.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?

[Mark Miller]

On Mar 14, 2013, at 8:10 AM, Rafał Radecki  wrote:

> Is this a bug?

Yes, 4.1 had some replication issues just as you seem to describe here. It all 
should be fixed in 4.2 which is available now and is a simple upgrade.

- Mark

Re: Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?

[Rafał Radecki]

In the output of:

/solr/replication?command=details

there is indexVersion mentioned many times:



0
3


22.59 KB
/usr/share/solr/data/index/


1363259880360
4

_1.tvx
_1_nrm.cfs
_1_Lucene41_0.doc
_1_Lucene41_0.tim
_1_Lucene41_0.tip
_1.fnm
_1_nrm.cfe
_1.fdx
_1_Lucene41_0.pos
_1.tvf
_1.fdt
_1_Lucene41_0.pay
_1.si
_1.tvd
segments_4



false
true
1363259808632
3


22.59 KB
/usr/share/solr/data/index/


1363263304585
4

_2_Lucene41_0.pos
_2.si
_2_Lucene41_0.tim
_2.fdt
_2_Lucene41_0.doc
_2_Lucene41_0.tip
_2.fdx
_2.tvx
_2.fnm
_2_nrm.cfe
_2.tvd
_2_Lucene41_0.pay
_2_nrm.cfs
_2.tvf
segments_4



true
false
1363263304585
4

schema.xml,stopwords.txt

commit
startup

false
4


http://172.18.19.204:8080/solr
00:00:60
Polling disabled
Thu Mar 14 12:18:00 CET 2013

Thu Mar 14 12:18:00 CET 2013
Thu Mar 14 12:17:00 CET 2013
Fri Mar 08 14:55:00 CET 2013
Fri Mar 08 14:50:52 CET 2013
Fri Mar 08 14:32:00 CET 2013

5
23214
0
Thu Mar 14 13:15:53 CET 2013
true
false



This response format is experimental. It is likely to change in the future.



Which one should be used? Is there any other way to monitor idex
version on master and slave?

Best regards,
Rafał Radecki.

2013/3/14 Rafał Radecki :
> Hi All.
>
> I am monitoring two solr 4.1 solr instances in master-slave setup. On
> both nodes I check url /solr/replication?command=details and parse it
> to get:
> - on master: if replication is enabled -> field replicationEnabled
> - on slave: if replication is enabled -> field replicationEnabled
> - on slave: if polling is disabled -> field isPollingDisabled
> For solr 3.6 I've als used url:
> solr/replication?command=indexversion
> but for 4.1 it gives me different results on master and slave, on
> slave the version is higher despite the fact that replication is
> enabled, polling is enabled and in admin gui
> /solr/#/collection1/replication I have: Index
> Version Gen Size
> Master:
> 1363259808632
> 3
> 22.59 KB
> Slave:
> 1363259808632
> 3
> 22.59 KB
> So as I see it master and slave have the same version of index despite
> the fact that /solr/replication?command=indexversion gives:
> - on master: 1363259808632
> - on slave: 1363259880360 -> higher value
> Is this a bug?
>
> Best regards,
> Rafal Radecki.

Solr 4.1 monitoring with /solr/replication?command=details - indexVersion?

[Rafał Radecki]

Hi All.

I am monitoring two solr 4.1 solr instances in master-slave setup. On
both nodes I check url /solr/replication?command=details and parse it
to get:
- on master: if replication is enabled -> field replicationEnabled
- on slave: if replication is enabled -> field replicationEnabled
- on slave: if polling is disabled -> field isPollingDisabled
For solr 3.6 I've als used url:
solr/replication?command=indexversion
but for 4.1 it gives me different results on master and slave, on
slave the version is higher despite the fact that replication is
enabled, polling is enabled and in admin gui
/solr/#/collection1/replication I have: Index
Version Gen Size
Master: 
1363259808632
3
22.59 KB
Slave:  
1363259808632
3
22.59 KB
So as I see it master and slave have the same version of index despite
the fact that /solr/replication?command=indexversion gives:
- on master: 1363259808632
- on slave: 1363259880360 -> higher value
Is this a bug?

Best regards,
Rafal Radecki.

Re: Solr Replication

[Ahmet Arslan]

Hi Vicky,

May be   startup ?

For backups http://master_host:port/solr/replication?command=backup would be 
more suitable.

or startup


--- On Thu, 3/14/13, vicky desai  wrote:

> From: vicky desai 
> Subject: Solr Replication
> To: solr-user@lucene.apache.org
> Date: Thursday, March 14, 2013, 9:20 AM
> Hi,
> 
> I am using solr 4 setup. For the backup purpose once in a
> day I start one
> additional tomcat server with cores having empty data
> folders and which acts
> as a slave server. However it does not replicate data from
> the master unless
> there is a commit on the master. Is there a possibility to
> pull data from
> master core without firing a commit operation on that core 
> 
> 
> 
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Solr-Replication-tp4047266.html
> Sent from the Solr - User mailing list archive at
> Nabble.com.
> 

Solr Replication

[vicky desai]

Hi,

I am using solr 4 setup. For the backup purpose once in a day I start one
additional tomcat server with cores having empty data folders and which acts
as a slave server. However it does not replicate data from the master unless
there is a commit on the master. Is there a possibility to pull data from
master core without firing a commit operation on that core 



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-Replication-tp4047266.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr replication takes long time

[Victor Ruiz]

While looking at Solr logs, I found a java.lang.OutOfMemoryError: Java heap
space that was happening 2 times per hour 
So I tried to increase the max memory heap assigned to JVM (-Xmx) and since
then  the servers are not crashing, even though the replication takes still
long time to complete. But for now, the 2 slaves can handle with no problems
all the queries.


Regards,
Victor



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388p4046993.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr replication takes long time

[Victor Ruiz]

After upgrading to 4.2, the problem is not yet solved, in this image you can
see, how slow is the transfer speed. At least, after the update the master
is not blocked during replication
<http://lucene.472066.n3.nabble.com/file/n4046951/solr_replication.jpg> 

Any idea?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388p4046951.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr replication takes long time

[Victor Ruiz]

Thanks for your answer Mark. I think I'll try to update to 4.2. I'll keep you
updated.

Anyway, I'd not say that the full index is replicated, I've been monitoring
the replication process in the Solr admin console and there I see that
usually not more than 50-100 files are transferrend, the total size is
rarely greater than 50MB. Is this info trustable?

Victor

Mark Miller-3 wrote
> Okay - yes, 4.0 is a better choice for replication than 4.1.
> 
> It almost sounds like you may be replicating the full index rather than
> just changes or something. 4.0 had a couple issues as well - a couple
> things that were discovered while writing stronger tests for 4.2.
> 
> 4.2 is spreading onto mirrors now.
> 
> - Mark
> 
> On Mar 11, 2013, at 2:00 PM, Victor Ruiz <

> bik1979@

> > wrote:
> 
>> no, Solr 4.0.0, I wanted to update to Solr 4.1 but I read that there was
>> an
>> issue with the replication, so I decided not to try it for now
>> 
>> 
>> Mark Miller-3 wrote
>>> Are you using Solr 4.1?
>>> 
>>> - Mark
>>> 
>>> On Mar 11, 2013, at 1:53 PM, Victor Ruiz <
>> 
>>> bik1979@
>> 
>>> > wrote:
>>> 
>>>> Hi guys,
>>>> 
>>>> I have a problem with Solr replication. I have 2 solr servers (Solr
>>>> 4.0.0) 1
>>>> master and 1 slave (8 processors,16GB RAM ,Ubuntu 11,  ext3,  each). In
>>>> every server, there are 2 independent instances of solr running (I
>>>> tried
>>>> also multicore config, but having independent instances has for me
>>>> better
>>>> performance), every instance having a differente collection. So, we
>>>> have
>>>> 2
>>>> masters in server 1, and 2 slaves in server 2.
>>>> 
>>>> Index size is currently (for the biggest collection) around 17 million
>>>> documents, with a total size near 12 GB. The files transferred every
>>>> replication cycle are typically not more than 100, with a total size
>>>> not
>>>> bigger than 50MB. The other collection is not that big, just around 1
>>>> million docs and not bigger than 2 GB and not a high update ratio. The
>>>> big
>>>> collection has a load around 200 queries per second (MoreLikeThis,
>>>> RealTimeGetHandler , TermVectorComponent mainly), and for the small one
>>>> it
>>>> is below 50 queries per second
>>>> 
>>>> Replication has been working for long time with any problem, but in the
>>>> last
>>>> weeks the replication cycles started to take long and long time for the
>>>> big
>>>> collection, even more than 2 minutes, some times even more. During that
>>>> time, slaves are so overloaded, that many queries are timing out,
>>>> despite
>>>> the timeout in my clients is 30 seconds. 
>>>> 
>>>> The servers are in same LAN, gigabit ethernet, so the broadband should
>>>> not
>>>> be the bottleneck.
>>>> 
>>>> Since the index is receiving frequents updates and deletes (update
>>>> handler
>>>> receives more than 200 request per second for the big collection, but
>>>> not
>>>> more than 5 per second for the small one), I tried to use the
>>>> maxCommitsToKeep attribute, to ensure that no file was deleted during
>>>> replication, but it has no effect. 
>>>> 
>>>> My solrconfig.xml in the big collection is like that:
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
> 
>>>> 
>>>>
>>> 
> 
>>> LUCENE_40
>>> 
> 
>>>> 
>>>>
>>> 
> >
>>> 
>>>  
>>> class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
>>>> 
>>>> 
>>>>
>>> 
> 
>>>>
>>> 
> 
>>> 3
>>> 
> 
>>>> 
>>>>
>>> 
> 
>>>>
>>>>
>>> 
> 
>>> 10
>>> 
> 
>>>>
>>> 
> 
>>> 1
>>> 
> 
>>>>
>>>>
>>> 
> 
>>> 6HOUR
>>> 
> 
>>>> 
>>>>
>>> 
> 
>>>> 
>>>&

Re: Solr replication takes long time

[Mark Miller]

Okay - yes, 4.0 is a better choice for replication than 4.1.

It almost sounds like you may be replicating the full index rather than just 
changes or something. 4.0 had a couple issues as well - a couple things that 
were discovered while writing stronger tests for 4.2.

4.2 is spreading onto mirrors now.

- Mark

On Mar 11, 2013, at 2:00 PM, Victor Ruiz  wrote:

> no, Solr 4.0.0, I wanted to update to Solr 4.1 but I read that there was an
> issue with the replication, so I decided not to try it for now
> 
> 
> Mark Miller-3 wrote
>> Are you using Solr 4.1?
>> 
>> - Mark
>> 
>> On Mar 11, 2013, at 1:53 PM, Victor Ruiz <
> 
>> bik1979@
> 
>> > wrote:
>> 
>>> Hi guys,
>>> 
>>> I have a problem with Solr replication. I have 2 solr servers (Solr
>>> 4.0.0) 1
>>> master and 1 slave (8 processors,16GB RAM ,Ubuntu 11,  ext3,  each). In
>>> every server, there are 2 independent instances of solr running (I tried
>>> also multicore config, but having independent instances has for me better
>>> performance), every instance having a differente collection. So, we have
>>> 2
>>> masters in server 1, and 2 slaves in server 2.
>>> 
>>> Index size is currently (for the biggest collection) around 17 million
>>> documents, with a total size near 12 GB. The files transferred every
>>> replication cycle are typically not more than 100, with a total size not
>>> bigger than 50MB. The other collection is not that big, just around 1
>>> million docs and not bigger than 2 GB and not a high update ratio. The
>>> big
>>> collection has a load around 200 queries per second (MoreLikeThis,
>>> RealTimeGetHandler , TermVectorComponent mainly), and for the small one
>>> it
>>> is below 50 queries per second
>>> 
>>> Replication has been working for long time with any problem, but in the
>>> last
>>> weeks the replication cycles started to take long and long time for the
>>> big
>>> collection, even more than 2 minutes, some times even more. During that
>>> time, slaves are so overloaded, that many queries are timing out, despite
>>> the timeout in my clients is 30 seconds. 
>>> 
>>> The servers are in same LAN, gigabit ethernet, so the broadband should
>>> not
>>> be the bottleneck.
>>> 
>>> Since the index is receiving frequents updates and deletes (update
>>> handler
>>> receives more than 200 request per second for the big collection, but not
>>> more than 5 per second for the small one), I tried to use the
>>> maxCommitsToKeep attribute, to ensure that no file was deleted during
>>> replication, but it has no effect. 
>>> 
>>> My solrconfig.xml in the big collection is like that:
>>> 
>>> 
>>> 
>>> 
>> 
>>> 
>>> 
>> 
>> LUCENE_40
>> 
>>> 
>>> 
>> >> 
>>
>> class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
>>> 
>>> 
>>> 
>> 
>>> 
>> 
>> 3
>> 
>>> 
>>> 
>> 
>>> 
>>> 
>> 
>> 10
>> 
>>> 
>> 
>> 1
>> 
>>> 
>>> 
>> 
>> 6HOUR
>> 
>>> 
>>> 
>> 
>>> 
>>> 
>> 
>>> 
>>> 
>> 
>>> 
>>> 
>> 
>>> 
>>> 
>> 
>>> 
>> 
>> 2000
>> 
>>> 
>> 
>> 3
>> 
>>> 
>> 
>>> 
>>> 
>> 
>>> 
>> 
>> 500
>> 
>>> 
>> 
>>> 
>>> 
>> 
>>> 
>> 
>> ${solr.data.dir:}
>> 
>>> 
>> 
>>> 
>>> 
>> 
>>> 
>>> 
>> 
>>> 
>> 
>> 2048
>> 
>>> 
>>> 
>> >> 
>>  class="solr.FastLRUCache"
>>> size="2048"
>>> initialSize="1024"
>

Re: Solr replication takes long time

[Victor Ruiz]

no, Solr 4.0.0, I wanted to update to Solr 4.1 but I read that there was an
issue with the replication, so I decided not to try it for now


Mark Miller-3 wrote
> Are you using Solr 4.1?
> 
> - Mark
> 
> On Mar 11, 2013, at 1:53 PM, Victor Ruiz <

> bik1979@

> > wrote:
> 
>> Hi guys,
>> 
>> I have a problem with Solr replication. I have 2 solr servers (Solr
>> 4.0.0) 1
>> master and 1 slave (8 processors,16GB RAM ,Ubuntu 11,  ext3,  each). In
>> every server, there are 2 independent instances of solr running (I tried
>> also multicore config, but having independent instances has for me better
>> performance), every instance having a differente collection. So, we have
>> 2
>> masters in server 1, and 2 slaves in server 2.
>> 
>> Index size is currently (for the biggest collection) around 17 million
>> documents, with a total size near 12 GB. The files transferred every
>> replication cycle are typically not more than 100, with a total size not
>> bigger than 50MB. The other collection is not that big, just around 1
>> million docs and not bigger than 2 GB and not a high update ratio. The
>> big
>> collection has a load around 200 queries per second (MoreLikeThis,
>> RealTimeGetHandler , TermVectorComponent mainly), and for the small one
>> it
>> is below 50 queries per second
>> 
>> Replication has been working for long time with any problem, but in the
>> last
>> weeks the replication cycles started to take long and long time for the
>> big
>> collection, even more than 2 minutes, some times even more. During that
>> time, slaves are so overloaded, that many queries are timing out, despite
>> the timeout in my clients is 30 seconds. 
>> 
>> The servers are in same LAN, gigabit ethernet, so the broadband should
>> not
>> be the bottleneck.
>> 
>> Since the index is receiving frequents updates and deletes (update
>> handler
>> receives more than 200 request per second for the big collection, but not
>> more than 5 per second for the small one), I tried to use the
>> maxCommitsToKeep attribute, to ensure that no file was deleted during
>> replication, but it has no effect. 
>> 
>> My solrconfig.xml in the big collection is like that:
>> 
>> 
>> 
>> 
> 
>> 
>>  
> 
> LUCENE_40
> 
>> 
>>  
> >
> 
> class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
>> 
>> 
>>  
> 
>>  
> 
> 3
> 
>> 
>>  
> 
>>  
>>  
> 
> 10
> 
>>  
> 
> 1
> 
>>  
>>  
> 
> 6HOUR
> 
>> 
>>  
> 
>> 
>>  
> 
>> 
>>  
> 
>> 
>>  
> 
>> 
>>  
> 
>>  
> 
> 2000
> 
>>  
> 
> 3
> 
>>  
> 
>> 
>>  
> 
>>  
> 
> 500
> 
>>  
> 
>> 
>>  
> 
>>  
> 
> ${solr.data.dir:}
> 
>>  
> 
>> 
>>  
> 
>> 
>>  
> 
>>  
> 
> 2048
> 
>> 
>>  
> >
>   class="solr.FastLRUCache"
>>  size="2048"
>>  initialSize="1024"
>>  autowarmCount="1024"/>
>> 
>>  
> >
>   class="solr.LRUCache"
>>  size="2048"
>>  initialSize="1024"
>>  autowarmCount="1024"/>
>> 
>>  
>>  
> >
>   class="solr.LRUCache"
>>  size="2048"
>>  initialSize="1024"
>>  autowarmCount="1024"/>
>> 
>>  
> 
> true
> 
>> 
>>  
> 
> 50
> 
>> 
>>  
> 
> 50
> 
>> 
>>  
> 
>>  
> 
>>  
> 
>>  

Re: Solr replication takes long time

[Mark Miller]

Are you using Solr 4.1?

- Mark

On Mar 11, 2013, at 1:53 PM, Victor Ruiz  wrote:

> Hi guys,
> 
> I have a problem with Solr replication. I have 2 solr servers (Solr 4.0.0) 1
> master and 1 slave (8 processors,16GB RAM ,Ubuntu 11,  ext3,  each). In
> every server, there are 2 independent instances of solr running (I tried
> also multicore config, but having independent instances has for me better
> performance), every instance having a differente collection. So, we have 2
> masters in server 1, and 2 slaves in server 2.
> 
> Index size is currently (for the biggest collection) around 17 million
> documents, with a total size near 12 GB. The files transferred every
> replication cycle are typically not more than 100, with a total size not
> bigger than 50MB. The other collection is not that big, just around 1
> million docs and not bigger than 2 GB and not a high update ratio. The big
> collection has a load around 200 queries per second (MoreLikeThis,
> RealTimeGetHandler , TermVectorComponent mainly), and for the small one it
> is below 50 queries per second
> 
> Replication has been working for long time with any problem, but in the last
> weeks the replication cycles started to take long and long time for the big
> collection, even more than 2 minutes, some times even more. During that
> time, slaves are so overloaded, that many queries are timing out, despite
> the timeout in my clients is 30 seconds. 
> 
> The servers are in same LAN, gigabit ethernet, so the broadband should not
> be the bottleneck.
> 
> Since the index is receiving frequents updates and deletes (update handler
> receives more than 200 request per second for the big collection, but not
> more than 5 per second for the small one), I tried to use the
> maxCommitsToKeep attribute, to ensure that no file was deleted during
> replication, but it has no effect. 
> 
> My solrconfig.xml in the big collection is like that:
> 
> 
> 
> 
> 
>   LUCENE_40
> 
>
> class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
> 
> 
>   
>   3
> 
>   
>   
>   10
>   1
>   
>   6HOUR
> 
>   
> 
>   
> 
>   
> 
>   
> 
>   
>   2000
>   3
>   
> 
>   
>   500
>   
> 
>   
>   ${solr.data.dir:}
>   
> 
>   
> 
>   
>   2048
> 
>  class="solr.FastLRUCache"
>   size="2048"
>   initialSize="1024"
>   autowarmCount="1024"/>
> 
>  class="solr.LRUCache"
>   size="2048"
>   initialSize="1024"
>   autowarmCount="1024"/>
> 
>   
>  class="solr.LRUCache"
>   size="2048"
>   initialSize="1024"
>   autowarmCount="1024"/>
> 
>   true
> 
>   50
> 
>   50
> 
>   
>   
>   
>   *:*
>   date:[NOW/DAY-7DAY TO 
> NOW/DAY+1DAY]
>   1000
>   
>   
>   
>class="solr.QuerySenderListener">
>   
>   
>   *:*
>   date:[NOW/DAY-7DAY TO 
> NOW/DAY+1DAY]
>   1000
>   
>   
>   
> 
>   true
> 
>   4
>   
> 
>   
> 
>   
>   
>   ${enable.master:false}
>   commit
>   startup
>   startup
>name="confFiles">schema.xml,solrconfig.xml,stopwords_de.txt,stopwords_en.txt,mapping-FoldToASCII.txt,mapping-FoldToASCI

Solr replication takes long time

[Victor Ruiz]

Hi guys,

I have a problem with Solr replication. I have 2 solr servers (Solr 4.0.0) 1
master and 1 slave (8 processors,16GB RAM ,Ubuntu 11,  ext3,  each). In
every server, there are 2 independent instances of solr running (I tried
also multicore config, but having independent instances has for me better
performance), every instance having a differente collection. So, we have 2
masters in server 1, and 2 slaves in server 2.

Index size is currently (for the biggest collection) around 17 million
documents, with a total size near 12 GB. The files transferred every
replication cycle are typically not more than 100, with a total size not
bigger than 50MB. The other collection is not that big, just around 1
million docs and not bigger than 2 GB and not a high update ratio. The big
collection has a load around 200 queries per second (MoreLikeThis,
RealTimeGetHandler , TermVectorComponent mainly), and for the small one it
is below 50 queries per second

Replication has been working for long time with any problem, but in the last
weeks the replication cycles started to take long and long time for the big
collection, even more than 2 minutes, some times even more. During that
time, slaves are so overloaded, that many queries are timing out, despite
the timeout in my clients is 30 seconds. 

The servers are in same LAN, gigabit ethernet, so the broadband should not
be the bottleneck.

Since the index is receiving frequents updates and deletes (update handler
receives more than 200 request per second for the big collection, but not
more than 5 per second for the small one), I tried to use the
maxCommitsToKeep attribute, to ensure that no file was deleted during
replication, but it has no effect. 

My solrconfig.xml in the big collection is like that:





LUCENE_40





3



10
1

6HOUR










2000
3



500



${solr.data.dir:}





2048








true

50

50




*:*
date:[NOW/DAY-7DAY TO 
NOW/DAY+1DAY]
1000






*:*
date:[NOW/DAY-7DAY TO 
NOW/DAY+1DAY]
1000




true

4






${enable.master:false}
commit
startup
startup
schema.xml,solrconfig.xml,stopwords_de.txt,stopwords_en.txt,mapping-FoldToASCII.txt,mapping-FoldToASCII_de.txt


${enable.slave:false}
http://${MASTER_HOST}:${MASTER_PORT}/solr/${MASTER_CORE}
05:00
${MASTER_HTTP_USER}
${MASTER_HTTP_PWD}




*:*




Poll interval is now set to 5 min, I tried to reduce it to 2 min and to
increase it up to 10, with no effect, the replication is always taking so
long., even  with a poll time of 2 minutes, when there are only a few megas
to replicate.

Any idea suggestion about what could be the problem? 

Thanks in advance,
Victor



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-replication-takes-long-time-tp4046388.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Downloading files from the solr replication Handler

[Eva Lacy]

They are the '\0' character.
what is a marker?

Gettting the following with a wget
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/xml]


On Fri, Nov 30, 2012 at 4:58 PM, Alexandre Rafalovitch
wrote:

> What mime type you get for binary files? Maybe server is misconfigured for
> that extension and sends them as text. Then they could be the markers.
>
> Do they look like markers?
>
> Regards,
> Alex
> On 30 Nov 2012 04:06, "Eva Lacy"  wrote:
>
> > Doesn't make much sense if they are in binary files as well.
> >
> >
> > On Thu, Nov 29, 2012 at 10:16 PM, Lance Norskog 
> wrote:
> >
> > > Maybe these are text encoding markers?
> > >
> > > - Original Message -
> > > | From: "Eva Lacy" 
> > > | To: solr-user@lucene.apache.org
> > > | Sent: Thursday, November 29, 2012 3:53:07 AM
> > > | Subject: Re: Downloading files from the solr replication Handler
> > > |
> > > | I tried downloading them with my browser and also with a c#
> > > | WebRequest.
> > > | If I skip the first and last 4 bytes it seems work fine.
> > > |
> > > |
> > > | On Thu, Nov 29, 2012 at 2:28 AM, Erick Erickson
> > > | wrote:
> > > |
> > > | > How are you downloading them? I suspect the issue is
> > > | > with the download process rather than Solr, but I'm just guessing.
> > > | >
> > > | > Best
> > > | > Erick
> > > | >
> > > | >
> > > | > On Wed, Nov 28, 2012 at 12:19 PM, Eva Lacy  wrote:
> > > | >
> > > | > > Just to add to that, I'm using solr 3.6.1
> > > | > >
> > > | > >
> > > | > > On Wed, Nov 28, 2012 at 5:18 PM, Eva Lacy  wrote:
> > > | > >
> > > | > > > I downloaded some configuration and data files directly from
> > > | > > > solr in an
> > > | > > > attempt to develop a backup solution.
> > > | > > > I noticed there is some characters at the start and end of the
> > > | > > > file
> > > | > that
> > > | > > > aren't in configuration files, I notice the same characters at
> > > | > > > the
> > > | > start
> > > | > > > and end of the data files.
> > > | > > > Anyone with any idea how I can download these files without the
> > > | > > > extra
> > > | > > > characters or predict how many there are going to be so I can
> > > | > > > skip
> > > | > them?
> > > | > > >
> > > | > >
> > > | >
> > > |
> > >
> >
>

Re: Downloading files from the solr replication Handler

[Alexandre Rafalovitch]

What mime type you get for binary files? Maybe server is misconfigured for
that extension and sends them as text. Then they could be the markers.

Do they look like markers?

Regards,
Alex
On 30 Nov 2012 04:06, "Eva Lacy"  wrote:

> Doesn't make much sense if they are in binary files as well.
>
>
> On Thu, Nov 29, 2012 at 10:16 PM, Lance Norskog  wrote:
>
> > Maybe these are text encoding markers?
> >
> > - Original Message -
> > | From: "Eva Lacy" 
> > | To: solr-user@lucene.apache.org
> > | Sent: Thursday, November 29, 2012 3:53:07 AM
> > | Subject: Re: Downloading files from the solr replication Handler
> > |
> > | I tried downloading them with my browser and also with a c#
> > | WebRequest.
> > | If I skip the first and last 4 bytes it seems work fine.
> > |
> > |
> > | On Thu, Nov 29, 2012 at 2:28 AM, Erick Erickson
> > | wrote:
> > |
> > | > How are you downloading them? I suspect the issue is
> > | > with the download process rather than Solr, but I'm just guessing.
> > | >
> > | > Best
> > | > Erick
> > | >
> > | >
> > | > On Wed, Nov 28, 2012 at 12:19 PM, Eva Lacy  wrote:
> > | >
> > | > > Just to add to that, I'm using solr 3.6.1
> > | > >
> > | > >
> > | > > On Wed, Nov 28, 2012 at 5:18 PM, Eva Lacy  wrote:
> > | > >
> > | > > > I downloaded some configuration and data files directly from
> > | > > > solr in an
> > | > > > attempt to develop a backup solution.
> > | > > > I noticed there is some characters at the start and end of the
> > | > > > file
> > | > that
> > | > > > aren't in configuration files, I notice the same characters at
> > | > > > the
> > | > start
> > | > > > and end of the data files.
> > | > > > Anyone with any idea how I can download these files without the
> > | > > > extra
> > | > > > characters or predict how many there are going to be so I can
> > | > > > skip
> > | > them?
> > | > > >
> > | > >
> > | >
> > |
> >
>

Re: Downloading files from the solr replication Handler

[Erick Erickson]

tag me baffled. But these are copied around all the time so I'm guessing an
interaction between your servlet container and your request, which is like
saying "it must be magic". You can tell I'm into places where I'm
clueless

Sorry I can't be more help
Erick


On Fri, Nov 30, 2012 at 4:06 AM, Eva Lacy  wrote:

> Doesn't make much sense if they are in binary files as well.
>
>
> On Thu, Nov 29, 2012 at 10:16 PM, Lance Norskog  wrote:
>
> > Maybe these are text encoding markers?
> >
> > - Original Message -
> > | From: "Eva Lacy" 
> > | To: solr-user@lucene.apache.org
> > | Sent: Thursday, November 29, 2012 3:53:07 AM
> > | Subject: Re: Downloading files from the solr replication Handler
> > |
> > | I tried downloading them with my browser and also with a c#
> > | WebRequest.
> > | If I skip the first and last 4 bytes it seems work fine.
> > |
> > |
> > | On Thu, Nov 29, 2012 at 2:28 AM, Erick Erickson
> > | wrote:
> > |
> > | > How are you downloading them? I suspect the issue is
> > | > with the download process rather than Solr, but I'm just guessing.
> > | >
> > | > Best
> > | > Erick
> > | >
> > | >
> > | > On Wed, Nov 28, 2012 at 12:19 PM, Eva Lacy  wrote:
> > | >
> > | > > Just to add to that, I'm using solr 3.6.1
> > | > >
> > | > >
> > | > > On Wed, Nov 28, 2012 at 5:18 PM, Eva Lacy  wrote:
> > | > >
> > | > > > I downloaded some configuration and data files directly from
> > | > > > solr in an
> > | > > > attempt to develop a backup solution.
> > | > > > I noticed there is some characters at the start and end of the
> > | > > > file
> > | > that
> > | > > > aren't in configuration files, I notice the same characters at
> > | > > > the
> > | > start
> > | > > > and end of the data files.
> > | > > > Anyone with any idea how I can download these files without the
> > | > > > extra
> > | > > > characters or predict how many there are going to be so I can
> > | > > > skip
> > | > them?
> > | > > >
> > | > >
> > | >
> > |
> >
>

Re: Downloading files from the solr replication Handler

[Eva Lacy]

Doesn't make much sense if they are in binary files as well.


On Thu, Nov 29, 2012 at 10:16 PM, Lance Norskog  wrote:

> Maybe these are text encoding markers?
>
> - Original Message -
> | From: "Eva Lacy" 
> | To: solr-user@lucene.apache.org
> | Sent: Thursday, November 29, 2012 3:53:07 AM
> | Subject: Re: Downloading files from the solr replication Handler
> |
> | I tried downloading them with my browser and also with a c#
> | WebRequest.
> | If I skip the first and last 4 bytes it seems work fine.
> |
> |
> | On Thu, Nov 29, 2012 at 2:28 AM, Erick Erickson
> | wrote:
> |
> | > How are you downloading them? I suspect the issue is
> | > with the download process rather than Solr, but I'm just guessing.
> | >
> | > Best
> | > Erick
> | >
> | >
> | > On Wed, Nov 28, 2012 at 12:19 PM, Eva Lacy  wrote:
> | >
> | > > Just to add to that, I'm using solr 3.6.1
> | > >
> | > >
> | > > On Wed, Nov 28, 2012 at 5:18 PM, Eva Lacy  wrote:
> | > >
> | > > > I downloaded some configuration and data files directly from
> | > > > solr in an
> | > > > attempt to develop a backup solution.
> | > > > I noticed there is some characters at the start and end of the
> | > > > file
> | > that
> | > > > aren't in configuration files, I notice the same characters at
> | > > > the
> | > start
> | > > > and end of the data files.
> | > > > Anyone with any idea how I can download these files without the
> | > > > extra
> | > > > characters or predict how many there are going to be so I can
> | > > > skip
> | > them?
> | > > >
> | > >
> | >
> |
>

Re: Downloading files from the solr replication Handler

[Lance Norskog]

Maybe these are text encoding markers?

- Original Message -
| From: "Eva Lacy" 
| To: solr-user@lucene.apache.org
| Sent: Thursday, November 29, 2012 3:53:07 AM
| Subject: Re: Downloading files from the solr replication Handler
| 
| I tried downloading them with my browser and also with a c#
| WebRequest.
| If I skip the first and last 4 bytes it seems work fine.
| 
| 
| On Thu, Nov 29, 2012 at 2:28 AM, Erick Erickson
| wrote:
| 
| > How are you downloading them? I suspect the issue is
| > with the download process rather than Solr, but I'm just guessing.
| >
| > Best
| > Erick
| >
| >
| > On Wed, Nov 28, 2012 at 12:19 PM, Eva Lacy  wrote:
| >
| > > Just to add to that, I'm using solr 3.6.1
| > >
| > >
| > > On Wed, Nov 28, 2012 at 5:18 PM, Eva Lacy  wrote:
| > >
| > > > I downloaded some configuration and data files directly from
| > > > solr in an
| > > > attempt to develop a backup solution.
| > > > I noticed there is some characters at the start and end of the
| > > > file
| > that
| > > > aren't in configuration files, I notice the same characters at
| > > > the
| > start
| > > > and end of the data files.
| > > > Anyone with any idea how I can download these files without the
| > > > extra
| > > > characters or predict how many there are going to be so I can
| > > > skip
| > them?
| > > >
| > >
| >
| 

Re: Downloading files from the solr replication Handler

[Eva Lacy]

I tried downloading them with my browser and also with a c# WebRequest.
If I skip the first and last 4 bytes it seems work fine.


On Thu, Nov 29, 2012 at 2:28 AM, Erick Erickson wrote:

> How are you downloading them? I suspect the issue is
> with the download process rather than Solr, but I'm just guessing.
>
> Best
> Erick
>
>
> On Wed, Nov 28, 2012 at 12:19 PM, Eva Lacy  wrote:
>
> > Just to add to that, I'm using solr 3.6.1
> >
> >
> > On Wed, Nov 28, 2012 at 5:18 PM, Eva Lacy  wrote:
> >
> > > I downloaded some configuration and data files directly from solr in an
> > > attempt to develop a backup solution.
> > > I noticed there is some characters at the start and end of the file
> that
> > > aren't in configuration files, I notice the same characters at the
> start
> > > and end of the data files.
> > > Anyone with any idea how I can download these files without the extra
> > > characters or predict how many there are going to be so I can skip
> them?
> > >
> >
>

Re: Downloading files from the solr replication Handler

[Erick Erickson]

How are you downloading them? I suspect the issue is
with the download process rather than Solr, but I'm just guessing.

Best
Erick


On Wed, Nov 28, 2012 at 12:19 PM, Eva Lacy  wrote:

> Just to add to that, I'm using solr 3.6.1
>
>
> On Wed, Nov 28, 2012 at 5:18 PM, Eva Lacy  wrote:
>
> > I downloaded some configuration and data files directly from solr in an
> > attempt to develop a backup solution.
> > I noticed there is some characters at the start and end of the file that
> > aren't in configuration files, I notice the same characters at the start
> > and end of the data files.
> > Anyone with any idea how I can download these files without the extra
> > characters or predict how many there are going to be so I can skip them?
> >
>

Re: Downloading files from the solr replication Handler

[Eva Lacy]

Just to add to that, I'm using solr 3.6.1


On Wed, Nov 28, 2012 at 5:18 PM, Eva Lacy  wrote:

> I downloaded some configuration and data files directly from solr in an
> attempt to develop a backup solution.
> I noticed there is some characters at the start and end of the file that
> aren't in configuration files, I notice the same characters at the start
> and end of the data files.
> Anyone with any idea how I can download these files without the extra
> characters or predict how many there are going to be so I can skip them?
>

Downloading files from the solr replication Handler

[Eva Lacy]

I downloaded some configuration and data files directly from solr in an
attempt to develop a backup solution.
I noticed there is some characters at the start and end of the file that
aren't in configuration files, I notice the same characters at the start
and end of the data files.
Anyone with any idea how I can download these files without the extra
characters or predict how many there are going to be so I can skip them?

Re: Solr replication

[Erick Erickson]

BTW, when I said "I think to say anything intelligent" I meant "I think for
_me_ to say anything intelligent". Didn't mean to be snarky

Erick


On Mon, Nov 26, 2012 at 8:19 AM, Erick Erickson wrote:

> "both servers are master and slave in the config file". So that means
> they're polling each other? I think to say anything intelligent you're
> going to have to provide more data. Please review:
> http://wiki.apache.org/solr/UsingMailingLists.
>
> The bottom line here is that it sounds like what you're trying to do is
> unusual enough that it's not a typical use case. I further suspect that
> what you're trying to do will cause problems you haven't tested for.
>
> So I wouldn't do this. Simply configuring a proper master/slave setup will
> give you HA out of the box. If a master goes down there's some complexity
> in "promoting" one of your slaves and indexing any stale data.
>
> But I'd go with SolrCloud to "fire and forget" rather than try to force
> Master/Slave to do something it wasn't intended to do.
>
> Best
> Erick
>
>
> On Mon, Nov 26, 2012 at 4:59 AM, jacques.cortes <
> jacques.cor...@laposte.net> wrote:
>
>> Thanks Antoine.
>>
>> In fact, both solr servers are master and slave in config file.
>> They have the same config file and replicate from the same master url with
>> the vip.
>> So, the master is on the server with the vip mounted on.
>> And if heartbeat toggle, the role is toggled too.
>>
>> The question is : what can happend in this configuration?
>>
>> With tests, I can't see nothing bad, but I don't know the intern
>> mecanisms.
>>
>>
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/Solr-replication-tp4021875p4022314.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>
>

Re: Solr replication

[Erick Erickson]

"both servers are master and slave in the config file". So that means
they're polling each other? I think to say anything intelligent you're
going to have to provide more data. Please review:
http://wiki.apache.org/solr/UsingMailingLists.

The bottom line here is that it sounds like what you're trying to do is
unusual enough that it's not a typical use case. I further suspect that
what you're trying to do will cause problems you haven't tested for.

So I wouldn't do this. Simply configuring a proper master/slave setup will
give you HA out of the box. If a master goes down there's some complexity
in "promoting" one of your slaves and indexing any stale data.

But I'd go with SolrCloud to "fire and forget" rather than try to force
Master/Slave to do something it wasn't intended to do.

Best
Erick


On Mon, Nov 26, 2012 at 4:59 AM, jacques.cortes
wrote:

> Thanks Antoine.
>
> In fact, both solr servers are master and slave in config file.
> They have the same config file and replicate from the same master url with
> the vip.
> So, the master is on the server with the vip mounted on.
> And if heartbeat toggle, the role is toggled too.
>
> The question is : what can happend in this configuration?
>
> With tests, I can't see nothing bad, but I don't know the intern mecanisms.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-replication-tp4021875p4022314.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Solr replication

[jacques.cortes]

Thanks Antoine.

In fact, both solr servers are master and slave in config file.
They have the same config file and replicate from the same master url with
the vip.
So, the master is on the server with the vip mounted on.
And if heartbeat toggle, the role is toggled too.

The question is : what can happend in this configuration?

With tests, I can't see nothing bad, but I don't know the intern mecanisms.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-replication-tp4021875p4022314.html
Sent from the Solr - User mailing list archive at Nabble.com.

Solr replication

[jacques.cortes]

I have 2 Solr servers :
- 1 master
- 1 slave

The master server has a mounted VIP by heartbeat and can toggle on the other
server.

What do you think of the idea to put the same config file on the 2 servers
with master's replication handler url pointing on the vip?




--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-replication-tp4021875.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Replication is not Possible on RAMDirectory?

[deniz]

Erik Hatcher-4 wrote
> There's an open issue (with a patch!) that enables this, it seems:
> <https://issues.apache.org/jira/browse/SOLR-3911>;
> 
>   Erik

well patch seems not doing that... i have tried and still getting some error
lines about the dir types




-
Zeki ama calismiyor... Calissa yapar...
--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-Replication-is-not-Possible-on-RAMDirectory-tp4017766p4018670.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Replication is not Possible on RAMDirectory?

[Michael Della Bitta]

Here's some reading:

http://en.wikipedia.org/wiki/Page_cache

Michael Della Bitta


Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271

www.appinions.com

Where Influence Isn’t a Game


On Mon, Nov 5, 2012 at 8:02 PM, deniz  wrote:
> Erik Hatcher-4 wrote
>> There's an open issue (with a patch!) that enables this, it seems:
>> <https://issues.apache.org/jira/browse/SOLR-3911>;
>
>  i will check it for sure, thank you Erik :)
>
>
> Shawn Heisey-4 wrote
>> ... transparently mapping the files on disk to a virtual memory space and
>> using excess RAM to cache that data and make it fast.  If you have
>> enough extra memory (disk cache) to fit the entire index, the OS will
>> never have to read any part of the index from disk more than once
>
> so for disk cache, are there any disks with 1 gigs or more of caches? if am
> not wrong there are mostly 16 or 32mb cache disks around (or i am checking
> the wrong stuff? ) if so, that amount definitely too small...
>
>
>
>
>
> -
> Zeki ama calismiyor... Calissa yapar...
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Solr-Replication-is-not-Possible-on-RAMDirectory-tp4017766p4018396.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Replication is not Possible on RAMDirectory?

[Shawn Heisey]

> Shawn Heisey-4 wrote
>> ... transparently mapping the files on disk to a virtual memory space
>> and
>> using excess RAM to cache that data and make it fast.  If you have
>> enough extra memory (disk cache) to fit the entire index, the OS will
>> never have to read any part of the index from disk more than once
>
> so for disk cache, are there any disks with 1 gigs or more of caches? if
> am
> not wrong there are mostly 16 or 32mb cache disks around (or i am checking
> the wrong stuff? ) if so, that amount definitely too small...


I am not talking about the cache on the actual disk drive, or even cache
on your hard drive controller. I am talking about the operating system
using RAM, specifically RAM not being used by programs, to cache data on
your hard drive. All modern operating systems do it, even the one made in
Redmond that people love to hate.

If you have 16 GB of RAM and all your programs use up 4.5 GB, you can
count on the OS using at least another half GB, so you have about 11 GB
left. The OS is going to put data that it reads and writes to/from your
disk in this space. If you start up another program that wants 2GB, the OS
will simply throw away 2 GB of data in its cache (it's still on the disk,
after all) and give that RAM to the new program.


Solr counts on this OS capability for good performance.

Thanks,
Shawn

Re: Solr Replication is not Possible on RAMDirectory?

[deniz]

Erik Hatcher-4 wrote
> There's an open issue (with a patch!) that enables this, it seems:
> <https://issues.apache.org/jira/browse/SOLR-3911>;

 i will check it for sure, thank you Erik :) 


Shawn Heisey-4 wrote
> ... transparently mapping the files on disk to a virtual memory space and
> using excess RAM to cache that data and make it fast.  If you have
> enough extra memory (disk cache) to fit the entire index, the OS will
> never have to read any part of the index from disk more than once

so for disk cache, are there any disks with 1 gigs or more of caches? if am
not wrong there are mostly 16 or 32mb cache disks around (or i am checking
the wrong stuff? ) if so, that amount definitely too small... 





-
Zeki ama calismiyor... Calissa yapar...
--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-Replication-is-not-Possible-on-RAMDirectory-tp4017766p4018396.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr Replication is not Possible on RAMDirectory?

[Erik Hatcher]

There's an open issue (with a patch!) that enables this, it seems: 
<https://issues.apache.org/jira/browse/SOLR-3911>

Erik

On Nov 5, 2012, at 07:41 , deniz wrote:

> Michael Della Bitta-2 wrote
>> No, RAMDirectory doesn't work for replication. Use MMapDirectory... it
>> ends up storing the index in RAM and more efficiently so, plus it's
>> backed by disk.
>> 
>> Just be sure to not set a big heap because MMapDirectory works outside of
>> heap.
> 
> for my tests, i dont think index is ended up in ram with mmap... i gave
> 4gigs for heap while using mmap and got mapping error while indexing...
> while index should be something around 2 gigs, ram consumption was around
> 300mbs... 
> 
> Can anyone explain why RAMDirectory cant be used for replication? I cant see
> why the master is set for using RAMDirectory and replica is using MMap or
> some other? As far as I understand SolrCloud is some kinda pushing from
> master to replica/slave... so why it is not possible to push from RAM to
> HDD? If my logic is wrong, someone can please explain me all these? 
> 
> 
> 
> -
> Zeki ama calismiyor... Calissa yapar...
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Solr-Replication-is-not-Possible-on-RAMDirectory-tp4017766p4018198.html
> Sent from the Solr - User mailing list archive at Nabble.com.