Re: solr-8983.pid: Permission denied
Solved by setting a SOLR_PID_DIR path in my solr.in.sh. And then giving the solr user ownership of that directory. Thanks On Thu, Oct 15, 2020 at 11:17 AM Alexandre Rafalovitch wrote: > If the .pid file is not at that location, then I would investigate > where that file is instead (after Solr is started). > > If it is in a different location, then you have different environment > expectations, somehow. This, in all honesty, would still be consistent > with my theory that Solr was started somehow differently (perhaps just > this once). > > If it is nowhere, then you may have a permission issue around creating > that file in the first place. > > Basically, I am saying that maybe the issue you have is a symptom of a > deeper discrepancy rather than the actual issue to solve directly. > > Regards, > Alex. > > On Thu, 15 Oct 2020 at 11:03, Ryan W wrote: > > > > The .pid file referenced in the "Permission denied" message does not > exist. > > > > On Thu, Oct 15, 2020 at 11:01 AM Ryan W wrote: > > > > > I have been starting solr like so... > > > > > > service solr start > > > > > > > > > On Thu, Oct 15, 2020 at 10:31 AM Joe Doupnik wrote: > > > > > >> Alex has it right. In my environment I created user "solr" in > group > > >> "users". Then I ensured that "solr:user" owns all of Solr's files. In > > >> addition, I do Solr start/stop with an /etc/init.d script (the Solr > > >> distribution has the basic one which we can embellish) in which there > is > > >> control line RUNAS="solr". The RUNAS variable is used to properly > start > > >> Solr. > > >> Thanks, > > >> Joe D. > > >> > > >> On 15/10/2020 15:02, Alexandre Rafalovitch wrote: > > >> > It sounds like maybe you have started the Solr in a different way > than > > >> > you are restarting it. E.g. maybe you started it manually (bin/solr > > >> > start, probably as a root) but are trying to restart it via service > > >> > script. Who owned the .pid file? I am guessing 'root', while the > > >> > service script probably runs as a different (lower-permission) user. > > >> > > > >> > The practical effect of that assumption is that your environmental > > >> > variables were set differently and various things (e.g. logs) may > not > > >> > be where you expect. > > >> > > > >> > The solution is to be consistent in using the service to > > >> > start/restart/stop your Solr. > > >> > > > >> > Regards, > > >> > Alex. > > >> > > > >> > On Thu, 15 Oct 2020 at 09:51, Ryan W wrote: > > >> >> What is my permissions problem here: > > >> >> > > >> >> [root@faspbsy0002 bin]# service solr restart > > >> >> Sending stop command to Solr running on port 8983 ... waiting up > to 180 > > >> >> seconds to allow Jetty process 38947 to stop gracefully. > > >> >> /opt/solr/bin/solr: line 2125: /opt/solr/bin/solr-8983.pid: > Permission > > >> >> denied > > >> >> > > >> >> What is the practical effect if Solr can't write this solr-8983.pid > > >> file? > > >> >> What user should own the contents of /opt/solr/bin ? > > >> >> > > >> >> Thanks > > >> > > >> >
Re: systemd definition for solr?
I just needed to enable the service with systemctl enable solr.service On Thu, Oct 15, 2020 at 4:03 PM Ryan W wrote: > I didn't realize that to start a systemd service, I need to do... > > systemctl start solr > > ...and not... > > service solr start > > Now the output from the status command looks a bit better, though still > with some problems... > > [root@faspbsy0002 system]# systemctl status solr.service > ? solr.service - LSB: A very fast and reliable search engine. >Loaded: loaded (/etc/rc.d/init.d/solr; bad; vendor preset: disabled) >Active: active (exited) since Thu 2020-10-15 15:58:23 EDT; 19s ago > Docs: man:systemd-sysv-generator(8) > Process: 34100 ExecStop=/etc/rc.d/init.d/solr stop (code=exited, > status=1/FAILURE) > Process: 98871 ExecStart=/etc/rc.d/init.d/solr start (code=exited, > status=0/SUCCESS) > > > > On Thu, Oct 15, 2020 at 3:24 PM Ryan W wrote: > >> Does anyone have a simple systemd definition for a solr service? >> >> The things I am finding on the internet don't work. I am not sure if >> this is the kind of thing where there might be some boilerplate that >> (usually) works? Or do situations vary so much that no boilerplate is >> possible? >> >> Here is what I see when I try to use one of the definitions I found on >> the internet: >> >> [root@faspbsy0002 system]# systemctl status solr.service >> ? solr.service - LSB: A very fast and reliable search engine. >>Loaded: loaded (/etc/rc.d/init.d/solr; bad; vendor preset: disabled) >>Active: failed (Result: exit-code) since Thu 2020-10-15 09:32:02 EDT; >> 5h 50min ago >> Docs: man:systemd-sysv-generator(8) >> Process: 34100 ExecStop=/etc/rc.d/init.d/solr stop (code=exited, >> status=1/FAILURE) >> Process: 1337 ExecStart=/etc/rc.d/init.d/solr start (code=exited, >> status=0/SUCCESS) >> >> Oct 15 09:32:01 faspbsy0002 systemd[1]: Stopping LSB: A very fast and >> reliab >> Oct 15 09:32:01 faspbsy0002 su[34102]: (to solr) root on none >> Oct 15 09:32:02 faspbsy0002 solr[34100]: No process found for Solr node >> runn...3 >> Oct 15 09:32:02 faspbsy0002 systemd[1]: solr.service: control process >> exited...1 >> Oct 15 09:32:02 faspbsy0002 systemd[1]: Stopped LSB: A very fast and >> reliabl >> Oct 15 09:32:02 faspbsy0002 systemd[1]: Unit solr.service entered failed >> state. >> Oct 15 09:32:02 faspbsy0002 systemd[1]: solr.service failed. >> Warning: Journal has been rotated since unit was started. Log output is >> incomplete or unavailable. >> Hint: Some lines were ellipsized, use -l to show in full. >> >
Re: systemd definition for solr?
I didn't realize that to start a systemd service, I need to do... systemctl start solr ...and not... service solr start Now the output from the status command looks a bit better, though still with some problems... [root@faspbsy0002 system]# systemctl status solr.service ? solr.service - LSB: A very fast and reliable search engine. Loaded: loaded (/etc/rc.d/init.d/solr; bad; vendor preset: disabled) Active: active (exited) since Thu 2020-10-15 15:58:23 EDT; 19s ago Docs: man:systemd-sysv-generator(8) Process: 34100 ExecStop=/etc/rc.d/init.d/solr stop (code=exited, status=1/FAILURE) Process: 98871 ExecStart=/etc/rc.d/init.d/solr start (code=exited, status=0/SUCCESS) On Thu, Oct 15, 2020 at 3:24 PM Ryan W wrote: > Does anyone have a simple systemd definition for a solr service? > > The things I am finding on the internet don't work. I am not sure if this > is the kind of thing where there might be some boilerplate that (usually) > works? Or do situations vary so much that no boilerplate is possible? > > Here is what I see when I try to use one of the definitions I found on the > internet: > > [root@faspbsy0002 system]# systemctl status solr.service > ? solr.service - LSB: A very fast and reliable search engine. >Loaded: loaded (/etc/rc.d/init.d/solr; bad; vendor preset: disabled) >Active: failed (Result: exit-code) since Thu 2020-10-15 09:32:02 EDT; > 5h 50min ago > Docs: man:systemd-sysv-generator(8) > Process: 34100 ExecStop=/etc/rc.d/init.d/solr stop (code=exited, > status=1/FAILURE) > Process: 1337 ExecStart=/etc/rc.d/init.d/solr start (code=exited, > status=0/SUCCESS) > > Oct 15 09:32:01 faspbsy0002 systemd[1]: Stopping LSB: A very fast and > reliab > Oct 15 09:32:01 faspbsy0002 su[34102]: (to solr) root on none > Oct 15 09:32:02 faspbsy0002 solr[34100]: No process found for Solr node > runn...3 > Oct 15 09:32:02 faspbsy0002 systemd[1]: solr.service: control process > exited...1 > Oct 15 09:32:02 faspbsy0002 systemd[1]: Stopped LSB: A very fast and > reliabl > Oct 15 09:32:02 faspbsy0002 systemd[1]: Unit solr.service entered failed > state. > Oct 15 09:32:02 faspbsy0002 systemd[1]: solr.service failed. > Warning: Journal has been rotated since unit was started. Log output is > incomplete or unavailable. > Hint: Some lines were ellipsized, use -l to show in full. >
systemd definition for solr?
Does anyone have a simple systemd definition for a solr service? The things I am finding on the internet don't work. I am not sure if this is the kind of thing where there might be some boilerplate that (usually) works? Or do situations vary so much that no boilerplate is possible? Here is what I see when I try to use one of the definitions I found on the internet: [root@faspbsy0002 system]# systemctl status solr.service ? solr.service - LSB: A very fast and reliable search engine. Loaded: loaded (/etc/rc.d/init.d/solr; bad; vendor preset: disabled) Active: failed (Result: exit-code) since Thu 2020-10-15 09:32:02 EDT; 5h 50min ago Docs: man:systemd-sysv-generator(8) Process: 34100 ExecStop=/etc/rc.d/init.d/solr stop (code=exited, status=1/FAILURE) Process: 1337 ExecStart=/etc/rc.d/init.d/solr start (code=exited, status=0/SUCCESS) Oct 15 09:32:01 faspbsy0002 systemd[1]: Stopping LSB: A very fast and reliab Oct 15 09:32:01 faspbsy0002 su[34102]: (to solr) root on none Oct 15 09:32:02 faspbsy0002 solr[34100]: No process found for Solr node runn...3 Oct 15 09:32:02 faspbsy0002 systemd[1]: solr.service: control process exited...1 Oct 15 09:32:02 faspbsy0002 systemd[1]: Stopped LSB: A very fast and reliabl Oct 15 09:32:02 faspbsy0002 systemd[1]: Unit solr.service entered failed state. Oct 15 09:32:02 faspbsy0002 systemd[1]: solr.service failed. Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable. Hint: Some lines were ellipsized, use -l to show in full.
bash: solr: command not found
Hi all, I run solr like this... service solr start However, another user of the server can't run solr this way. They are seeing... bash: solr: command not found How do I enable them to run Solr as a service? Thank you, Ryan
Re: solr-8983.pid: Permission denied
The .pid file referenced in the "Permission denied" message does not exist. On Thu, Oct 15, 2020 at 11:01 AM Ryan W wrote: > I have been starting solr like so... > > service solr start > > > On Thu, Oct 15, 2020 at 10:31 AM Joe Doupnik wrote: > >> Alex has it right. In my environment I created user "solr" in group >> "users". Then I ensured that "solr:user" owns all of Solr's files. In >> addition, I do Solr start/stop with an /etc/init.d script (the Solr >> distribution has the basic one which we can embellish) in which there is >> control line RUNAS="solr". The RUNAS variable is used to properly start >> Solr. >> Thanks, >> Joe D. >> >> On 15/10/2020 15:02, Alexandre Rafalovitch wrote: >> > It sounds like maybe you have started the Solr in a different way than >> > you are restarting it. E.g. maybe you started it manually (bin/solr >> > start, probably as a root) but are trying to restart it via service >> > script. Who owned the .pid file? I am guessing 'root', while the >> > service script probably runs as a different (lower-permission) user. >> > >> > The practical effect of that assumption is that your environmental >> > variables were set differently and various things (e.g. logs) may not >> > be where you expect. >> > >> > The solution is to be consistent in using the service to >> > start/restart/stop your Solr. >> > >> > Regards, >> > Alex. >> > >> > On Thu, 15 Oct 2020 at 09:51, Ryan W wrote: >> >> What is my permissions problem here: >> >> >> >> [root@faspbsy0002 bin]# service solr restart >> >> Sending stop command to Solr running on port 8983 ... waiting up to 180 >> >> seconds to allow Jetty process 38947 to stop gracefully. >> >> /opt/solr/bin/solr: line 2125: /opt/solr/bin/solr-8983.pid: Permission >> >> denied >> >> >> >> What is the practical effect if Solr can't write this solr-8983.pid >> file? >> >> What user should own the contents of /opt/solr/bin ? >> >> >> >> Thanks >> >>
Re: solr-8983.pid: Permission denied
I have been starting solr like so... service solr start On Thu, Oct 15, 2020 at 10:31 AM Joe Doupnik wrote: > Alex has it right. In my environment I created user "solr" in group > "users". Then I ensured that "solr:user" owns all of Solr's files. In > addition, I do Solr start/stop with an /etc/init.d script (the Solr > distribution has the basic one which we can embellish) in which there is > control line RUNAS="solr". The RUNAS variable is used to properly start > Solr. > Thanks, > Joe D. > > On 15/10/2020 15:02, Alexandre Rafalovitch wrote: > > It sounds like maybe you have started the Solr in a different way than > > you are restarting it. E.g. maybe you started it manually (bin/solr > > start, probably as a root) but are trying to restart it via service > > script. Who owned the .pid file? I am guessing 'root', while the > > service script probably runs as a different (lower-permission) user. > > > > The practical effect of that assumption is that your environmental > > variables were set differently and various things (e.g. logs) may not > > be where you expect. > > > > The solution is to be consistent in using the service to > > start/restart/stop your Solr. > > > > Regards, > > Alex. > > > > On Thu, 15 Oct 2020 at 09:51, Ryan W wrote: > >> What is my permissions problem here: > >> > >> [root@faspbsy0002 bin]# service solr restart > >> Sending stop command to Solr running on port 8983 ... waiting up to 180 > >> seconds to allow Jetty process 38947 to stop gracefully. > >> /opt/solr/bin/solr: line 2125: /opt/solr/bin/solr-8983.pid: Permission > >> denied > >> > >> What is the practical effect if Solr can't write this solr-8983.pid > file? > >> What user should own the contents of /opt/solr/bin ? > >> > >> Thanks > >
solr-8983.pid: Permission denied
What is my permissions problem here: [root@faspbsy0002 bin]# service solr restart Sending stop command to Solr running on port 8983 ... waiting up to 180 seconds to allow Jetty process 38947 to stop gracefully. /opt/solr/bin/solr: line 2125: /opt/solr/bin/solr-8983.pid: Permission denied What is the practical effect if Solr can't write this solr-8983.pid file? What user should own the contents of /opt/solr/bin ? Thanks
Re: Memory line in status output
Thanks. The G1 docs say "G1 is designed to provide good overall
performance without the need to specify additional options."
Would that look like this...
GC_TUNE=" \
-XX:+UseG1GC \
"
Is that the most minimal config? Is it typical to use it without options?
On Tue, Oct 13, 2020 at 4:22 PM Walter Underwood
wrote:
> The home page of the Solr admin UI shows all of the options to the JVM.
> That will include the choice of garbage collector.
>
> You can also see the options with “ps -ef | grep solr”.
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/ (my blog)
>
> > On Oct 13, 2020, at 1:19 PM, Ryan W wrote:
> >
> > I think I have it sorted. At this point I'm using GCG1, I take it,
> because
> > most recently I started Solr as a service...
> >
> > service solr start
> >
> > And that is running solr by way of /etc/init.d/solr because I don't have
> > any systemd unit for solr, as explained here...
> >
> https://askubuntu.com/questions/903354/difference-between-systemctl-and-service-commands
> >
> > And I can see in the System V script for solr that /etc/default/
> solr.in.sh
> > is the relevant config file.
> >
> >
> > On Tue, Oct 13, 2020 at 11:23 AM Ryan W wrote:
> >
> >> Or, perhaps if I start solr like so
> >>
> >> service solr start
> >>
> >> ...it will use the solr.in.sh at /etc/default/solr.in.sh ?
> >>
> >>
> >>
> >> On Tue, Oct 13, 2020 at 11:19 AM Ryan W wrote:
> >>
> >>> This is how I start solr:
> >>>
> >>> /opt/solr/bin/solr start
> >>>
> >>> In my /etc/default/solr.in.sh, I have this...
> >>>
> >>> GC_TUNE=" \
> >>> -XX:+UseG1GC \
> >>> -XX:+ParallelRefProcEnabled \
> >>> -XX:G1HeapRegionSize=8m \
> >>> -XX:MaxGCPauseMillis=200 \
> >>> -XX:+UseLargePages \
> >>> -XX:+AggressiveOpts \
> >>> "
> >>>
> >>> But I don't know how to tell if Solr is using that file.
> >>>
> >>> In my /opt/solr/bin there is no solr.in.sh, but there is a
> >>> solr.in.sh.orig -- perhaps I should copy my /etc/default/solr.in.sh to
> >>> /opt/solr/bin ?
> >>>
> >>> I am running Linux (RHEL). The Solr version is 7.7.2. Solr 8.x is not
> >>> compatible with my application.
> >>>
> >>> Thank you.
> >>>
> >>>
> >>> On Mon, Oct 12, 2020 at 9:46 PM Shawn Heisey
> >>> wrote:
> >>>
> >>>> On 10/12/2020 5:11 PM, Ryan W wrote:
> >>>>> Thanks. How do I activate the G1GC collector? Do I do this by
> >>>> editing a
> >>>>> config file, or by adding a parameter when I start solr?
> >>>>>
> >>>>> Oracle's docs are pointing me to a file that supposedly is at
> >>>>> instance-dir/OUD/config/java.properties, but I don't have that path.
> >>>> I am
> >>>>> not sure what is meant by instance-dir here, but perhaps it means my
> >>>> JRE
> >>>>> install, which is at
> >>>>> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre --
> but
> >>>>> there is no "OUD" directory in this location.
> >>>>
> >>>> The collector is chosen by the startup options given to Java, in this
> >>>> case by the start script for Solr. I've never heard of it being set
> by
> >>>> a config in the JRE.
> >>>>
> >>>> In Solr 7, the start script defaults to the CMS collector. We have
> >>>> updated that to G1 in the latest Solr 8.x versions, because CMS has
> been
> >>>> deprecated by Oracle.
> >>>>
> >>>> Adding the following lines to the correct solr.in.sh would change the
> >>>> garbage collector to G1. I got this from the "bin/solr" script in
> Solr
> >>>> 8.5.1:
> >>>>
> >>>> GC_TUNE=('-XX:+UseG1GC' \
> >>>> '-XX:+PerfDisableSharedMem' \
> >>>> '-XX:+ParallelRefProcEnabled' \
> >>>> '-XX:MaxGCPauseMillis=250' \
> >>>> '-XX:+UseLargePages' \
> >>>> '-XX:+AlwaysPreTouch')
> >>>>
> >>>> If you used the service installer script to install Solr, then the
> >>>> correct file to add this to is usually /etc/default/solr.in.sh ...
> but
> >>>> if you did the install manually, it may be in the same bin directory
> >>>> that contains the solr script itself. Your initial message says the
> >>>> solr home is /opt/solr/server/solr so I am assuming it's not running
> on
> >>>> Windows.
> >>>>
> >>>> Thanks,
> >>>> Shawn
> >>>>
> >>>
>
>
Re: Memory line in status output
I think I have it sorted. At this point I'm using GCG1, I take it, because
most recently I started Solr as a service...
service solr start
And that is running solr by way of /etc/init.d/solr because I don't have
any systemd unit for solr, as explained here...
https://askubuntu.com/questions/903354/difference-between-systemctl-and-service-commands
And I can see in the System V script for solr that /etc/default/solr.in.sh
is the relevant config file.
On Tue, Oct 13, 2020 at 11:23 AM Ryan W wrote:
> Or, perhaps if I start solr like so
>
> service solr start
>
> ...it will use the solr.in.sh at /etc/default/solr.in.sh ?
>
>
>
> On Tue, Oct 13, 2020 at 11:19 AM Ryan W wrote:
>
>> This is how I start solr:
>>
>> /opt/solr/bin/solr start
>>
>> In my /etc/default/solr.in.sh, I have this...
>>
>> GC_TUNE=" \
>> -XX:+UseG1GC \
>> -XX:+ParallelRefProcEnabled \
>> -XX:G1HeapRegionSize=8m \
>> -XX:MaxGCPauseMillis=200 \
>> -XX:+UseLargePages \
>> -XX:+AggressiveOpts \
>> "
>>
>> But I don't know how to tell if Solr is using that file.
>>
>> In my /opt/solr/bin there is no solr.in.sh, but there is a
>> solr.in.sh.orig -- perhaps I should copy my /etc/default/solr.in.sh to
>> /opt/solr/bin ?
>>
>> I am running Linux (RHEL). The Solr version is 7.7.2. Solr 8.x is not
>> compatible with my application.
>>
>> Thank you.
>>
>>
>> On Mon, Oct 12, 2020 at 9:46 PM Shawn Heisey
>> wrote:
>>
>>> On 10/12/2020 5:11 PM, Ryan W wrote:
>>> > Thanks. How do I activate the G1GC collector? Do I do this by
>>> editing a
>>> > config file, or by adding a parameter when I start solr?
>>> >
>>> > Oracle's docs are pointing me to a file that supposedly is at
>>> > instance-dir/OUD/config/java.properties, but I don't have that path.
>>> I am
>>> > not sure what is meant by instance-dir here, but perhaps it means my
>>> JRE
>>> > install, which is at
>>> > /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre -- but
>>> > there is no "OUD" directory in this location.
>>>
>>> The collector is chosen by the startup options given to Java, in this
>>> case by the start script for Solr. I've never heard of it being set by
>>> a config in the JRE.
>>>
>>> In Solr 7, the start script defaults to the CMS collector. We have
>>> updated that to G1 in the latest Solr 8.x versions, because CMS has been
>>> deprecated by Oracle.
>>>
>>> Adding the following lines to the correct solr.in.sh would change the
>>> garbage collector to G1. I got this from the "bin/solr" script in Solr
>>> 8.5.1:
>>>
>>>GC_TUNE=('-XX:+UseG1GC' \
>>> '-XX:+PerfDisableSharedMem' \
>>> '-XX:+ParallelRefProcEnabled' \
>>> '-XX:MaxGCPauseMillis=250' \
>>> '-XX:+UseLargePages' \
>>> '-XX:+AlwaysPreTouch')
>>>
>>> If you used the service installer script to install Solr, then the
>>> correct file to add this to is usually /etc/default/solr.in.sh ... but
>>> if you did the install manually, it may be in the same bin directory
>>> that contains the solr script itself. Your initial message says the
>>> solr home is /opt/solr/server/solr so I am assuming it's not running on
>>> Windows.
>>>
>>> Thanks,
>>> Shawn
>>>
>>
Re: Memory line in status output
Or, perhaps if I start solr like so
service solr start
...it will use the solr.in.sh at /etc/default/solr.in.sh ?
On Tue, Oct 13, 2020 at 11:19 AM Ryan W wrote:
> This is how I start solr:
>
> /opt/solr/bin/solr start
>
> In my /etc/default/solr.in.sh, I have this...
>
> GC_TUNE=" \
> -XX:+UseG1GC \
> -XX:+ParallelRefProcEnabled \
> -XX:G1HeapRegionSize=8m \
> -XX:MaxGCPauseMillis=200 \
> -XX:+UseLargePages \
> -XX:+AggressiveOpts \
> "
>
> But I don't know how to tell if Solr is using that file.
>
> In my /opt/solr/bin there is no solr.in.sh, but there is a
> solr.in.sh.orig -- perhaps I should copy my /etc/default/solr.in.sh to
> /opt/solr/bin ?
>
> I am running Linux (RHEL). The Solr version is 7.7.2. Solr 8.x is not
> compatible with my application.
>
> Thank you.
>
>
> On Mon, Oct 12, 2020 at 9:46 PM Shawn Heisey
> wrote:
>
>> On 10/12/2020 5:11 PM, Ryan W wrote:
>> > Thanks. How do I activate the G1GC collector? Do I do this by editing
>> a
>> > config file, or by adding a parameter when I start solr?
>> >
>> > Oracle's docs are pointing me to a file that supposedly is at
>> > instance-dir/OUD/config/java.properties, but I don't have that path. I
>> am
>> > not sure what is meant by instance-dir here, but perhaps it means my JRE
>> > install, which is at
>> > /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre -- but
>> > there is no "OUD" directory in this location.
>>
>> The collector is chosen by the startup options given to Java, in this
>> case by the start script for Solr. I've never heard of it being set by
>> a config in the JRE.
>>
>> In Solr 7, the start script defaults to the CMS collector. We have
>> updated that to G1 in the latest Solr 8.x versions, because CMS has been
>> deprecated by Oracle.
>>
>> Adding the following lines to the correct solr.in.sh would change the
>> garbage collector to G1. I got this from the "bin/solr" script in Solr
>> 8.5.1:
>>
>>GC_TUNE=('-XX:+UseG1GC' \
>> '-XX:+PerfDisableSharedMem' \
>> '-XX:+ParallelRefProcEnabled' \
>> '-XX:MaxGCPauseMillis=250' \
>> '-XX:+UseLargePages' \
>> '-XX:+AlwaysPreTouch')
>>
>> If you used the service installer script to install Solr, then the
>> correct file to add this to is usually /etc/default/solr.in.sh ... but
>> if you did the install manually, it may be in the same bin directory
>> that contains the solr script itself. Your initial message says the
>> solr home is /opt/solr/server/solr so I am assuming it's not running on
>> Windows.
>>
>> Thanks,
>> Shawn
>>
>
Re: Memory line in status output
This is how I start solr:
/opt/solr/bin/solr start
In my /etc/default/solr.in.sh, I have this...
GC_TUNE=" \
-XX:+UseG1GC \
-XX:+ParallelRefProcEnabled \
-XX:G1HeapRegionSize=8m \
-XX:MaxGCPauseMillis=200 \
-XX:+UseLargePages \
-XX:+AggressiveOpts \
"
But I don't know how to tell if Solr is using that file.
In my /opt/solr/bin there is no solr.in.sh, but there is a solr.in.sh.orig
-- perhaps I should copy my /etc/default/solr.in.sh to /opt/solr/bin ?
I am running Linux (RHEL). The Solr version is 7.7.2. Solr 8.x is not
compatible with my application.
Thank you.
On Mon, Oct 12, 2020 at 9:46 PM Shawn Heisey wrote:
> On 10/12/2020 5:11 PM, Ryan W wrote:
> > Thanks. How do I activate the G1GC collector? Do I do this by editing a
> > config file, or by adding a parameter when I start solr?
> >
> > Oracle's docs are pointing me to a file that supposedly is at
> > instance-dir/OUD/config/java.properties, but I don't have that path. I
> am
> > not sure what is meant by instance-dir here, but perhaps it means my JRE
> > install, which is at
> > /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre -- but
> > there is no "OUD" directory in this location.
>
> The collector is chosen by the startup options given to Java, in this
> case by the start script for Solr. I've never heard of it being set by
> a config in the JRE.
>
> In Solr 7, the start script defaults to the CMS collector. We have
> updated that to G1 in the latest Solr 8.x versions, because CMS has been
> deprecated by Oracle.
>
> Adding the following lines to the correct solr.in.sh would change the
> garbage collector to G1. I got this from the "bin/solr" script in Solr
> 8.5.1:
>
>GC_TUNE=('-XX:+UseG1GC' \
> '-XX:+PerfDisableSharedMem' \
> '-XX:+ParallelRefProcEnabled' \
> '-XX:MaxGCPauseMillis=250' \
> '-XX:+UseLargePages' \
> '-XX:+AlwaysPreTouch')
>
> If you used the service installer script to install Solr, then the
> correct file to add this to is usually /etc/default/solr.in.sh ... but
> if you did the install manually, it may be in the same bin directory
> that contains the solr script itself. Your initial message says the
> solr home is /opt/solr/server/solr so I am assuming it's not running on
> Windows.
>
> Thanks,
> Shawn
>
Re: Memory line in status output
Thanks. How do I activate the G1GC collector? Do I do this by editing a
config file, or by adding a parameter when I start solr?
Oracle's docs are pointing me to a file that supposedly is at
instance-dir/OUD/config/java.properties, but I don't have that path. I am
not sure what is meant by instance-dir here, but perhaps it means my JRE
install, which is at
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre -- but
there is no "OUD" directory in this location.
On Mon, Oct 12, 2020 at 11:15 AM Erick Erickson
wrote:
> Solr doesn’t manage this at all, it’s the JVM’s garbage collection
> that occasionally kicks in. In general, memory creeps up until
> the GC threshold is set (which there are about a zillion
> parameters that you can set) and then GC kicks in.
>
> Generally, the recommendation is to use the G1GC collector
> and just leave the default settings as they are.
>
> It’s usually a mistake, BTW, to over-allocate memory. You should shrink the
> heap as far as you can and still maintain a reasonable safety margin. See:
>
> https://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
> What’s a “reasonable safety margin”? Unfortunately you have to experiment.
>
> Best,
> Erick
>
> > On Oct 12, 2020, at 10:33 AM, Ryan W wrote:
> >
> > Hi all,
> >
> > What is the meaning of the "memory" line in the output when I run the
> solr
> > status command? What controls whether that memory gets exhausted? At
> > times if I run "solr status" over and over, that memory number creeps up
> > and up and up. Presumably it is not a good thing if it moves all the way
> > up to my 31GB capacity. What controls whether that happens? How do I
> > prevent that? Or does Solr manage this automatically?
> >
> >
> > $ /opt/solr/bin/solr status
> >
> > Found 1 Solr nodes:
> >
> > Solr process 101530 running on port 8983
> > {
> > "solr_home":"/opt/solr/server/solr",
> > "version":"7.7.2 d4c30fc2856154f2c1fefc589eb7cd070a415b94 - janhoy -
> > 2019-05-28 23:37:48",
> > "startTime":"2020-10-12T12:04:57.379Z",
> > "uptime":"0 days, 1 hours, 46 minutes, 41 seconds",
> > "memory":"3.3 GB (%10.7) of 31 GB"}
>
>
Memory line in status output
Hi all,
What is the meaning of the "memory" line in the output when I run the solr
status command? What controls whether that memory gets exhausted? At
times if I run "solr status" over and over, that memory number creeps up
and up and up. Presumably it is not a good thing if it moves all the way
up to my 31GB capacity. What controls whether that happens? How do I
prevent that? Or does Solr manage this automatically?
$ /opt/solr/bin/solr status
Found 1 Solr nodes:
Solr process 101530 running on port 8983
{
"solr_home":"/opt/solr/server/solr",
"version":"7.7.2 d4c30fc2856154f2c1fefc589eb7cd070a415b94 - janhoy -
2019-05-28 23:37:48",
"startTime":"2020-10-12T12:04:57.379Z",
"uptime":"0 days, 1 hours, 46 minutes, 41 seconds",
"memory":"3.3 GB (%10.7) of 31 GB"}
Re: How to determine why solr stops running?
On Mon, Jun 29, 2020 at 3:13 PM Erick Erickson wrote: > ps aux | grep solr > [solr@faspbsy0002 database-backups]$ ps aux | grep solr solr 72072 1.6 33.4 22847816 10966476 ? Sl 13:35 1:36 java -server -Xms16g -Xmx16g -XX:+UseG1GC -XX:+ParallelRefProcEnabled -XX:G1HeapRegionSize=8m -XX:MaxGCPauseMillis=200 -XX:+UseLargePages -XX:+AggressiveOpts -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/opt/solr/server/logs/solr_gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M -Dsolr.log.dir=/opt/solr/server/logs -Djetty.port=8983 -DSTOP.PORT=7983 -DSTOP.KEY=solrrocks -Duser.timezone=UTC -Djetty.home=/opt/solr/server -Dsolr.solr.home=/opt/solr/server/solr -Dsolr.data.home= -Dsolr.install.dir=/opt/solr -Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf -Xss256k -Dsolr.jetty.https.port=8983 -Dsolr.log.muteconsole -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 /opt/solr/server/logs -jar start.jar --module=http > should show you all the parameters Solr is running with, as would the > admin screen. You should see something like: > > -XX:OnOutOfMemoryError=your_solr_directory/bin/oom_solr.sh > > And there should be some logs laying around if that was the case > similar to: > $SOLR_LOGS_DIR/solr_oom_killer-$SOLR_PORT-$NOW.log > This log is not being written, even though in the oom_solr.sh it does appear a solr_oom_killer-$SOLR_PORT-$NOW.log should be written to the logs directory, but it isn't. There are some log files in /opt/solr/server/logs, and they are indeed being written to. There are fresh entries in the logs, but no sign of any problem. If I grep for oom in the logs directory, the only references I see are benign... just a few entries that list all the flags, and oom_solr.sh is among the settings visible in the entry. And someone did a search for "Mushroom," so there's another instance of oom from that search. As for memory, It Depends (tm). There are configurations > you can make choices about that will affect the heap requirements. > You can’t really draw comparisons between different projects. Your > Drupal + Solr app has how many documents? Indexed how? Searched > how? .vs. this one. > > The usual suspect for configuration settings that are responsible > include: > > - filterCache size too large. Each filterCache entry is bounded by > maxDoc/8 bytes. I’ve seen people set this to over 1M… > > - using non-docValues for fields used for sorting, grouping, function > queries > or faceting. Solr will uninvert the field on the heap, whereas if you have > specified docValues=true, the memory is out in OS memory space rather than > heap. > > - People just putting too many docs in a collection in a single JVM in > aggregate. > All replicas in the same instance are using part of the heap. > > - Having unnecessary options on your fields, although that’s more MMap > space than > heap. > > The problem basically is that all of Solr’s access is essentially random, > so for > performance reasons lots of stuff has to be in memory. > > That said, Solr hasn’t been as careful as it should be about using up > memory, > that’s ongoing. > > If you really want to know what’s using up memory, throw a heap analysis > tool > at it. That’ll give you a clue what’s hogging memory and you can go from > there. > > > On Jun 29, 2020, at 1:48 PM, David Hastings < > hastings.recurs...@gmail.com> wrote: > > > > little nit picky note here, use 31gb, never 32. > > > > On Mon, Jun 29, 2020 at 1:45 PM Ryan W wrote: > > > >> It figures it would happen again a couple hours after I suggested the > issue > >> might be resolved. Just now, Solr stopped running. I cleared the > cache in > >> my app a couple times around the time that it happened, so perhaps that > was > >> somehow too taxing for the server. However, I've never allocated so > much > >> RAM to a website before, so it's odd that I'm getting these failures. > My > >> colleagues were astonished when I said people on the solr-user list were > >> telling me I might need 32GB just for solr. > >> > >> I manage another project that uses Drupal + Solr, and we have a total of > >> 8GB of RAM on that server and Solr never, ever stops. I've been > managing > >> that site for years and never seen a Solr outage. On that project, > >> Drupal + Solr is OK with 8GB, but somehow this other project needs 64 > GB or > >> more? > >> > >> "The thing that’s unsettling about this is that assuming you were > hitting > &
Re: How to determine why solr stops running?
On Mon, Jun 29, 2020 at 1:49 PM David Hastings wrote: > little nit picky note here, use 31gb, never 32. Good to know. Just now I got this output from bin/solr status: "solr_home":"/opt/solr/server/solr", "version":"7.7.2 d4c30fc2856154f2c1fefc589eb7cd070a415b94 - janhoy - 2019-05-28 23:37:48", "startTime":"2020-06-29T17:35:13.966Z", "uptime":"0 days, 1 hours, 32 minutes, 7 seconds", "memory":"9.3 GB (%57.9) of 16 GB"} That's the highest memory use I've seen. Not sure if this indicates 16GB isn't enough. Then I ran it again a couple minutes later and it was down to 598.3 MB. I wonder what accounts for these wide swings. I can't imagine if a few users are doing searches, suddenly it uses 9 GB of RAM. On Mon, Jun 29, 2020 at 1:45 PM Ryan W wrote: > > > It figures it would happen again a couple hours after I suggested the > issue > > might be resolved. Just now, Solr stopped running. I cleared the cache > in > > my app a couple times around the time that it happened, so perhaps that > was > > somehow too taxing for the server. However, I've never allocated so much > > RAM to a website before, so it's odd that I'm getting these failures. My > > colleagues were astonished when I said people on the solr-user list were > > telling me I might need 32GB just for solr. > > > > I manage another project that uses Drupal + Solr, and we have a total of > > 8GB of RAM on that server and Solr never, ever stops. I've been managing > > that site for years and never seen a Solr outage. On that project, > > Drupal + Solr is OK with 8GB, but somehow this other project needs 64 GB > or > > more? > > > > "The thing that’s unsettling about this is that assuming you were hitting > > OOMs, and were running the OOM-killer script, you _should_ have had very > > clear evidence that that was the cause." > > > > How do I know if I'm running the OOM-killer script? > > > > Thank you. > > > > On Mon, Jun 29, 2020 at 12:12 PM Erick Erickson > > > wrote: > > > > > The thing that’s unsettling about this is that assuming you were > hitting > > > OOMs, > > > and were running the OOM-killer script, you _should_ have had very > clear > > > evidence that that was the cause. > > > > > > If you were not running the killer script, the apologies for not asking > > > about that > > > in the first place. Java’s performance is unpredictable when OOMs > happen, > > > which is the point of the killer script: at least Solr stops rather > than > > do > > > something inexplicable. > > > > > > Best, > > > Erick > > > > > > > On Jun 29, 2020, at 11:52 AM, David Hastings < > > > hastings.recurs...@gmail.com> wrote: > > > > > > > > sometimes just throwing money/ram/ssd at the problem is just the best > > > > answer. > > > > > > > > On Mon, Jun 29, 2020 at 11:38 AM Ryan W wrote: > > > > > > > >> Thanks everyone. Just to give an update on this issue, I bumped the > > RAM > > > >> available to Solr up to 16GB a couple weeks ago, and haven’t had any > > > >> problem since. > > > >> > > > >> > > > >> On Tue, Jun 16, 2020 at 1:00 PM David Hastings < > > > >> hastings.recurs...@gmail.com> > > > >> wrote: > > > >> > > > >>> me personally, around 290gb. as much as we could shove into them > > > >>> > > > >>> On Tue, Jun 16, 2020 at 12:44 PM Erick Erickson < > > > erickerick...@gmail.com > > > >>> > > > >>> wrote: > > > >>> > > > >>>> How much physical RAM? A rule of thumb is that you should allocate > > no > > > >>> more > > > >>>> than 25-50 percent of the total physical RAM to Solr. That's > > > >> cumulative, > > > >>>> i.e. the sum of the heap allocations across all your JVMs should > be > > > >> below > > > >>>> that percentage. See Uwe Schindler's mmapdirectiry blog... > > > >>>> > > > >>>> Shot in the dark... > > > >>>> > > > >>>> On Tue, Jun 16, 2020, 11:51 David Hastings < > > > >> hastings.recurs...@gmail.com > > > >>>>
Re: How to determine why solr stops running?
It figures it would happen again a couple hours after I suggested the issue might be resolved. Just now, Solr stopped running. I cleared the cache in my app a couple times around the time that it happened, so perhaps that was somehow too taxing for the server. However, I've never allocated so much RAM to a website before, so it's odd that I'm getting these failures. My colleagues were astonished when I said people on the solr-user list were telling me I might need 32GB just for solr. I manage another project that uses Drupal + Solr, and we have a total of 8GB of RAM on that server and Solr never, ever stops. I've been managing that site for years and never seen a Solr outage. On that project, Drupal + Solr is OK with 8GB, but somehow this other project needs 64 GB or more? "The thing that’s unsettling about this is that assuming you were hitting OOMs, and were running the OOM-killer script, you _should_ have had very clear evidence that that was the cause." How do I know if I'm running the OOM-killer script? Thank you. On Mon, Jun 29, 2020 at 12:12 PM Erick Erickson wrote: > The thing that’s unsettling about this is that assuming you were hitting > OOMs, > and were running the OOM-killer script, you _should_ have had very clear > evidence that that was the cause. > > If you were not running the killer script, the apologies for not asking > about that > in the first place. Java’s performance is unpredictable when OOMs happen, > which is the point of the killer script: at least Solr stops rather than do > something inexplicable. > > Best, > Erick > > > On Jun 29, 2020, at 11:52 AM, David Hastings < > hastings.recurs...@gmail.com> wrote: > > > > sometimes just throwing money/ram/ssd at the problem is just the best > > answer. > > > > On Mon, Jun 29, 2020 at 11:38 AM Ryan W wrote: > > > >> Thanks everyone. Just to give an update on this issue, I bumped the RAM > >> available to Solr up to 16GB a couple weeks ago, and haven’t had any > >> problem since. > >> > >> > >> On Tue, Jun 16, 2020 at 1:00 PM David Hastings < > >> hastings.recurs...@gmail.com> > >> wrote: > >> > >>> me personally, around 290gb. as much as we could shove into them > >>> > >>> On Tue, Jun 16, 2020 at 12:44 PM Erick Erickson < > erickerick...@gmail.com > >>> > >>> wrote: > >>> > >>>> How much physical RAM? A rule of thumb is that you should allocate no > >>> more > >>>> than 25-50 percent of the total physical RAM to Solr. That's > >> cumulative, > >>>> i.e. the sum of the heap allocations across all your JVMs should be > >> below > >>>> that percentage. See Uwe Schindler's mmapdirectiry blog... > >>>> > >>>> Shot in the dark... > >>>> > >>>> On Tue, Jun 16, 2020, 11:51 David Hastings < > >> hastings.recurs...@gmail.com > >>>> > >>>> wrote: > >>>> > >>>>> To add to this, i generally have solr start with this: > >>>>> -Xms31000m-Xmx31000m > >>>>> > >>>>> and the only other thing that runs on them are maria db gallera > >> cluster > >>>>> nodes that are not in use (aside from replication) > >>>>> > >>>>> the 31gb is not an accident either, you dont want 32gb. > >>>>> > >>>>> > >>>>> On Tue, Jun 16, 2020 at 11:26 AM Shawn Heisey > >>>> wrote: > >>>>> > >>>>>> On 6/11/2020 11:52 AM, Ryan W wrote: > >>>>>>>> I will check "dmesg" first, to find out any hardware error > >>> message. > >>>>>> > >>>>>> > >>>>>> > >>>>>>> [1521232.781801] Out of memory: Kill process 117529 (httpd) > >> score 9 > >>>> or > >>>>>>> sacrifice child > >>>>>>> [1521232.782908] Killed process 117529 (httpd), UID 48, > >>>>>> total-vm:675824kB, > >>>>>>> anon-rss:181844kB, file-rss:0kB, shmem-rss:0kB > >>>>>>> > >>>>>>> Is this a relevant "Out of memory" message? Does this suggest an > >>> OOM > >>>>>>> situation is the culprit? > >>>>>> > >>>>>> Because this was in the "dmesg" output, it indicates that it is the > >&
Re: How to determine why solr stops running?
Thanks everyone. Just to give an update on this issue, I bumped the RAM available to Solr up to 16GB a couple weeks ago, and haven’t had any problem since. On Tue, Jun 16, 2020 at 1:00 PM David Hastings wrote: > me personally, around 290gb. as much as we could shove into them > > On Tue, Jun 16, 2020 at 12:44 PM Erick Erickson > wrote: > > > How much physical RAM? A rule of thumb is that you should allocate no > more > > than 25-50 percent of the total physical RAM to Solr. That's cumulative, > > i.e. the sum of the heap allocations across all your JVMs should be below > > that percentage. See Uwe Schindler's mmapdirectiry blog... > > > > Shot in the dark... > > > > On Tue, Jun 16, 2020, 11:51 David Hastings > > > wrote: > > > > > To add to this, i generally have solr start with this: > > > -Xms31000m-Xmx31000m > > > > > > and the only other thing that runs on them are maria db gallera cluster > > > nodes that are not in use (aside from replication) > > > > > > the 31gb is not an accident either, you dont want 32gb. > > > > > > > > > On Tue, Jun 16, 2020 at 11:26 AM Shawn Heisey > > wrote: > > > > > > > On 6/11/2020 11:52 AM, Ryan W wrote: > > > > >> I will check "dmesg" first, to find out any hardware error > message. > > > > > > > > > > > > > > > > > [1521232.781801] Out of memory: Kill process 117529 (httpd) score 9 > > or > > > > > sacrifice child > > > > > [1521232.782908] Killed process 117529 (httpd), UID 48, > > > > total-vm:675824kB, > > > > > anon-rss:181844kB, file-rss:0kB, shmem-rss:0kB > > > > > > > > > > Is this a relevant "Out of memory" message? Does this suggest an > OOM > > > > > situation is the culprit? > > > > > > > > Because this was in the "dmesg" output, it indicates that it is the > > > > operating system killing programs because the *system* doesn't have > any > > > > memory left. It wasn't Java that did this, and it wasn't Solr that > was > > > > killed. It very well could have been Solr that was killed at another > > > > time, though. > > > > > > > > The process that it killed this time is named httpd ... which is most > > > > likely the Apache webserver. Because the UID is 48, this is probably > > an > > > > OS derived from Redhat, where the "apache" user has UID and GID 48 by > > > > default. Apache with its default config can be VERY memory hungry > when > > > > it gets busy. > > > > > > > > > -XX:InitialHeapSize=536870912 -XX:MaxHeapSize=536870912 > > > > > > > > This says that you started Solr with the default 512MB heap. Which > is > > > > VERY VERY small. The default is small so that Solr will start on > > > > virtually any hardware. Almost every user must increase the heap > size. > > > > And because the OS is killing processes, it is likely that the system > > > > does not have enough memory installed for what you have running on > it. > > > > > > > > It is generally not a good idea to share the server hardware between > > > > Solr and other software, unless the system has a lot of spare > > resources, > > > > memory in particular. > > > > > > > > Thanks, > > > > Shawn > > > > > > > > > >
Re: How to determine why solr stops running?
On Mon, Jun 15, 2020 at 2:00 PM Ryan W wrote: > What is the Service definition of Solr in Redhat? > I think maybe you are talking about systemd. Maybe a service definition looks something like this? https://gist.github.com/hammady/3d7b5964c7b0f90997865ebef40bf5e1 I haven't used systemd before. I should probably look into that. It isn't something I am currently using, as far as I know. > >> Thank you. I pasted those settings at the end of my /etc/default/ >> >> solr.in.sh just now and restarted solr. I will see if that fixes it. >> >> Previously, I had no settings at all in solr.in.sh except for >> SOLR_PORT. >> >> >> >> On Thu, Jun 11, 2020 at 1:59 PM Walter Underwood < >> wun...@wunderwood.org> >> >> wrote: >> >> >> >>> 1. You have a tiny heap. 536 Megabytes is not enough. >> >>> 2. I stopped using the CMS GC years ago. >> >>> >> >>> Here is the GC config we use on every one of our 150+ Solr hosts. >> We’re >> >>> still on Java 8, but will be upgrading soon. >> >>> >> >>> SOLR_HEAP=8g >> >>> # Use G1 GC -- wunder 2017-01-23 >> >>> # Settings from https://wiki.apache.org/solr/ShawnHeisey >> >>> GC_TUNE=" \ >> >>> -XX:+UseG1GC \ >> >>> -XX:+ParallelRefProcEnabled \ >> >>> -XX:G1HeapRegionSize=8m \ >> >>> -XX:MaxGCPauseMillis=200 \ >> >>> -XX:+UseLargePages \ >> >>> -XX:+AggressiveOpts \ >> >>> " >> >>> >> >>> wunder >> >>> Walter Underwood >> >>> wun...@wunderwood.org >> >>> http://observer.wunderwood.org/ (my blog) >> >>> >> >>>> On Jun 11, 2020, at 10:52 AM, Ryan W wrote: >> >>>> >> >>>> On Wed, Jun 10, 2020 at 8:35 PM Hup Chen >> wrote: >> >>>> >> >>>>> I will check "dmesg" first, to find out any hardware error message. >> >>>>> >> >>>> >> >>>> Here is what I see toward the end of the output from dmesg: >> >>>> >> >>>> [1521232.781785] [118857]48 118857 108785 677 201 >> >>>> 901 0 httpd >> >>>> [1521232.781787] [118860]48 118860 108785 710 201 >> >>>> 881 0 httpd >> >>>> [1521232.781788] [118862]48 118862 113063 5256 210 >> >>>> 725 0 httpd >> >>>> [1521232.781790] [118864]48 118864 114085 6634 212 >> >>>> 703 0 httpd >> >>>> [1521232.781791] [118871]48 118871 13968732323 262 >> >>>> 620 0 httpd >> >>>> [1521232.781793] [118873]48 118873 108785 821 201 >> >>>> 792 0 httpd >> >>>> [1521232.781795] [118879]48 118879 14026332719 263 >> >>>> 621 0 httpd >> >>>> [1521232.781796] [118903]48 118903 108785 812 201 >> >>>> 771 0 httpd >> >>>> [1521232.781798] [118905]48 118905 113575 5606 211 >> >>>> 660 0 httpd >> >>>> [1521232.781800] [118906]48 118906 113563 5694 211 >> >>>> 626 0 httpd >> >>>> [1521232.781801] Out of memory: Kill process 117529 (httpd) score 9 >> or >> >>>> sacrifice child >> >>>> [1521232.782908] Killed process 117529 (httpd), UID 48, >> >>> total-vm:675824kB, >> >>>> anon-rss:181844kB, file-rss:0kB, shmem-rss:0kB >> >>>> >> >>>> Is this a relevant "Out of memory" message? Does this suggest an OOM >> >>>> situation is the culprit? >> >>>> >> >>>> When I grep in the solr logs for oom, I see some entries like this... >> >>>> >> >>>> ./solr_gc.log.4.current:CommandLine flags: -XX:CICompilerCount=4 >> >>>> -XX:CMSInitiatingOccupancyFraction=50 >> >>> -XX:CMSMaxAbortablePrecleanTime=6000 >> >>>> -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark >> >>>> -XX:ConcGCThreads=4 -XX:GCLogFileSize=20971520 >> >>>> -XX:InitialHeapSize=536870912 -XX:MaxHeapSize=536870912 >> >>>> -XX:MaxNe
Re: How to determine why solr stops running?
On Mon, Jun 15, 2020 at 1:56 PM Jörn Franke wrote: > What is the Service definition of Solr in Redhat? > I am not sure what that means. What is a service definition? I am using Solr in conjunction with Drupal's Search API Solr module: https://www.drupal.org/project/search_api_solr > > > Am 15.06.2020 um 19:46 schrieb Ryan W : > > > > It happened again today. Again, no other apparent problems on the > server. > > Nothing else is stopping. Nothing in the logs that strikes me as useful. > > I'm using Red Hat Linux 7.8 and Solr 7.7.2. > > > > Solr is stopping a couple times per week and I don't know how to > determine > > why. > > > >> On Sun, Jun 14, 2020 at 9:41 AM Ryan W wrote: > >> > >> Thank you. I pasted those settings at the end of my /etc/default/ > >> solr.in.sh just now and restarted solr. I will see if that fixes it. > >> Previously, I had no settings at all in solr.in.sh except for > SOLR_PORT. > >> > >> On Thu, Jun 11, 2020 at 1:59 PM Walter Underwood > > >> wrote: > >> > >>> 1. You have a tiny heap. 536 Megabytes is not enough. > >>> 2. I stopped using the CMS GC years ago. > >>> > >>> Here is the GC config we use on every one of our 150+ Solr hosts. We’re > >>> still on Java 8, but will be upgrading soon. > >>> > >>> SOLR_HEAP=8g > >>> # Use G1 GC -- wunder 2017-01-23 > >>> # Settings from https://wiki.apache.org/solr/ShawnHeisey > >>> GC_TUNE=" \ > >>> -XX:+UseG1GC \ > >>> -XX:+ParallelRefProcEnabled \ > >>> -XX:G1HeapRegionSize=8m \ > >>> -XX:MaxGCPauseMillis=200 \ > >>> -XX:+UseLargePages \ > >>> -XX:+AggressiveOpts \ > >>> " > >>> > >>> wunder > >>> Walter Underwood > >>> wun...@wunderwood.org > >>> http://observer.wunderwood.org/ (my blog) > >>> > >>>> On Jun 11, 2020, at 10:52 AM, Ryan W wrote: > >>>> > >>>> On Wed, Jun 10, 2020 at 8:35 PM Hup Chen wrote: > >>>> > >>>>> I will check "dmesg" first, to find out any hardware error message. > >>>>> > >>>> > >>>> Here is what I see toward the end of the output from dmesg: > >>>> > >>>> [1521232.781785] [118857]48 118857 108785 677 201 > >>>> 901 0 httpd > >>>> [1521232.781787] [118860]48 118860 108785 710 201 > >>>> 881 0 httpd > >>>> [1521232.781788] [118862]48 118862 113063 5256 210 > >>>> 725 0 httpd > >>>> [1521232.781790] [118864]48 118864 114085 6634 212 > >>>> 703 0 httpd > >>>> [1521232.781791] [118871]48 118871 13968732323 262 > >>>> 620 0 httpd > >>>> [1521232.781793] [118873]48 118873 108785 821 201 > >>>> 792 0 httpd > >>>> [1521232.781795] [118879]48 118879 14026332719 263 > >>>> 621 0 httpd > >>>> [1521232.781796] [118903]48 118903 108785 812 201 > >>>> 771 0 httpd > >>>> [1521232.781798] [118905]48 118905 113575 5606 211 > >>>> 660 0 httpd > >>>> [1521232.781800] [118906]48 118906 113563 5694 211 > >>>> 626 0 httpd > >>>> [1521232.781801] Out of memory: Kill process 117529 (httpd) score 9 or > >>>> sacrifice child > >>>> [1521232.782908] Killed process 117529 (httpd), UID 48, > >>> total-vm:675824kB, > >>>> anon-rss:181844kB, file-rss:0kB, shmem-rss:0kB > >>>> > >>>> Is this a relevant "Out of memory" message? Does this suggest an OOM > >>>> situation is the culprit? > >>>> > >>>> When I grep in the solr logs for oom, I see some entries like this... > >>>> > >>>> ./solr_gc.log.4.current:CommandLine flags: -XX:CICompilerCount=4 > >>>> -XX:CMSInitiatingOccupancyFraction=50 > >>> -XX:CMSMaxAbortablePrecleanTime=6000 > >>>> -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark > >>>> -XX:ConcGCThreads=4 -XX:GCLogFileSize=20971520 > >>>> -XX:InitialHeapSize=536870912 -XX:MaxHeapSize=536870912 > >&
Re: How to determine why solr stops running?
It happened again today. Again, no other apparent problems on the server. Nothing else is stopping. Nothing in the logs that strikes me as useful. I'm using Red Hat Linux 7.8 and Solr 7.7.2. Solr is stopping a couple times per week and I don't know how to determine why. On Sun, Jun 14, 2020 at 9:41 AM Ryan W wrote: > Thank you. I pasted those settings at the end of my /etc/default/ > solr.in.sh just now and restarted solr. I will see if that fixes it. > Previously, I had no settings at all in solr.in.sh except for SOLR_PORT. > > On Thu, Jun 11, 2020 at 1:59 PM Walter Underwood > wrote: > >> 1. You have a tiny heap. 536 Megabytes is not enough. >> 2. I stopped using the CMS GC years ago. >> >> Here is the GC config we use on every one of our 150+ Solr hosts. We’re >> still on Java 8, but will be upgrading soon. >> >> SOLR_HEAP=8g >> # Use G1 GC -- wunder 2017-01-23 >> # Settings from https://wiki.apache.org/solr/ShawnHeisey >> GC_TUNE=" \ >> -XX:+UseG1GC \ >> -XX:+ParallelRefProcEnabled \ >> -XX:G1HeapRegionSize=8m \ >> -XX:MaxGCPauseMillis=200 \ >> -XX:+UseLargePages \ >> -XX:+AggressiveOpts \ >> " >> >> wunder >> Walter Underwood >> wun...@wunderwood.org >> http://observer.wunderwood.org/ (my blog) >> >> > On Jun 11, 2020, at 10:52 AM, Ryan W wrote: >> > >> > On Wed, Jun 10, 2020 at 8:35 PM Hup Chen wrote: >> > >> >> I will check "dmesg" first, to find out any hardware error message. >> >> >> > >> > Here is what I see toward the end of the output from dmesg: >> > >> > [1521232.781785] [118857]48 118857 108785 677 201 >> > 901 0 httpd >> > [1521232.781787] [118860]48 118860 108785 710 201 >> > 881 0 httpd >> > [1521232.781788] [118862]48 118862 113063 5256 210 >> > 725 0 httpd >> > [1521232.781790] [118864]48 118864 114085 6634 212 >> > 703 0 httpd >> > [1521232.781791] [118871]48 118871 13968732323 262 >> > 620 0 httpd >> > [1521232.781793] [118873]48 118873 108785 821 201 >> > 792 0 httpd >> > [1521232.781795] [118879]48 118879 14026332719 263 >> > 621 0 httpd >> > [1521232.781796] [118903]48 118903 108785 812 201 >> > 771 0 httpd >> > [1521232.781798] [118905]48 118905 113575 5606 211 >> > 660 0 httpd >> > [1521232.781800] [118906]48 118906 113563 5694 211 >> > 626 0 httpd >> > [1521232.781801] Out of memory: Kill process 117529 (httpd) score 9 or >> > sacrifice child >> > [1521232.782908] Killed process 117529 (httpd), UID 48, >> total-vm:675824kB, >> > anon-rss:181844kB, file-rss:0kB, shmem-rss:0kB >> > >> > Is this a relevant "Out of memory" message? Does this suggest an OOM >> > situation is the culprit? >> > >> > When I grep in the solr logs for oom, I see some entries like this... >> > >> > ./solr_gc.log.4.current:CommandLine flags: -XX:CICompilerCount=4 >> > -XX:CMSInitiatingOccupancyFraction=50 >> -XX:CMSMaxAbortablePrecleanTime=6000 >> > -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark >> > -XX:ConcGCThreads=4 -XX:GCLogFileSize=20971520 >> > -XX:InitialHeapSize=536870912 -XX:MaxHeapSize=536870912 >> > -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=8 >> > -XX:MinHeapDeltaBytes=196608 -XX:NewRatio=3 -XX:NewSize=134217728 >> > -XX:NumberOfGCLogFiles=9 -XX:OldPLABSize=16 -XX:OldSize=402653184 >> > -XX:-OmitStackTraceInFastThrow >> > -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 >> /opt/solr/server/logs >> > -XX:ParallelGCThreads=4 -XX:+ParallelRefProcEnabled >> > -XX:PretenureSizeThreshold=67108864 -XX:+PrintGC >> > -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps >> > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC >> > -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4 >> > -XX:TargetSurvivorRatio=90 -XX:ThreadStackSize=256 >> > -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers >> > -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation >> > -XX:+UseParNewGC >> > >> > Buried in there I see "OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh". >> But I >> > think this is just a
Re: How to determine why solr stops running?
Thank you. I pasted those settings at the end of my /etc/default/solr.in.sh just now and restarted solr. I will see if that fixes it. Previously, I had no settings at all in solr.in.sh except for SOLR_PORT. On Thu, Jun 11, 2020 at 1:59 PM Walter Underwood wrote: > 1. You have a tiny heap. 536 Megabytes is not enough. > 2. I stopped using the CMS GC years ago. > > Here is the GC config we use on every one of our 150+ Solr hosts. We’re > still on Java 8, but will be upgrading soon. > > SOLR_HEAP=8g > # Use G1 GC -- wunder 2017-01-23 > # Settings from https://wiki.apache.org/solr/ShawnHeisey > GC_TUNE=" \ > -XX:+UseG1GC \ > -XX:+ParallelRefProcEnabled \ > -XX:G1HeapRegionSize=8m \ > -XX:MaxGCPauseMillis=200 \ > -XX:+UseLargePages \ > -XX:+AggressiveOpts \ > " > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > > On Jun 11, 2020, at 10:52 AM, Ryan W wrote: > > > > On Wed, Jun 10, 2020 at 8:35 PM Hup Chen wrote: > > > >> I will check "dmesg" first, to find out any hardware error message. > >> > > > > Here is what I see toward the end of the output from dmesg: > > > > [1521232.781785] [118857]48 118857 108785 677 201 > > 901 0 httpd > > [1521232.781787] [118860]48 118860 108785 710 201 > > 881 0 httpd > > [1521232.781788] [118862]48 118862 113063 5256 210 > > 725 0 httpd > > [1521232.781790] [118864]48 118864 114085 6634 212 > > 703 0 httpd > > [1521232.781791] [118871]48 118871 13968732323 262 > > 620 0 httpd > > [1521232.781793] [118873]48 118873 108785 821 201 > > 792 0 httpd > > [1521232.781795] [118879]48 118879 14026332719 263 > > 621 0 httpd > > [1521232.781796] [118903]48 118903 108785 812 201 > > 771 0 httpd > > [1521232.781798] [118905]48 118905 113575 5606 211 > > 660 0 httpd > > [1521232.781800] [118906]48 118906 113563 5694 211 > > 626 0 httpd > > [1521232.781801] Out of memory: Kill process 117529 (httpd) score 9 or > > sacrifice child > > [1521232.782908] Killed process 117529 (httpd), UID 48, > total-vm:675824kB, > > anon-rss:181844kB, file-rss:0kB, shmem-rss:0kB > > > > Is this a relevant "Out of memory" message? Does this suggest an OOM > > situation is the culprit? > > > > When I grep in the solr logs for oom, I see some entries like this... > > > > ./solr_gc.log.4.current:CommandLine flags: -XX:CICompilerCount=4 > > -XX:CMSInitiatingOccupancyFraction=50 > -XX:CMSMaxAbortablePrecleanTime=6000 > > -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark > > -XX:ConcGCThreads=4 -XX:GCLogFileSize=20971520 > > -XX:InitialHeapSize=536870912 -XX:MaxHeapSize=536870912 > > -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=8 > > -XX:MinHeapDeltaBytes=196608 -XX:NewRatio=3 -XX:NewSize=134217728 > > -XX:NumberOfGCLogFiles=9 -XX:OldPLABSize=16 -XX:OldSize=402653184 > > -XX:-OmitStackTraceInFastThrow > > -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 > /opt/solr/server/logs > > -XX:ParallelGCThreads=4 -XX:+ParallelRefProcEnabled > > -XX:PretenureSizeThreshold=67108864 -XX:+PrintGC > > -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps > > -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC > > -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4 > > -XX:TargetSurvivorRatio=90 -XX:ThreadStackSize=256 > > -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers > > -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation > > -XX:+UseParNewGC > > > > Buried in there I see "OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh". > But I > > think this is just a setting that indicates what to do in case of an OOM. > > And if I look in that oom_solr.sh file, I see it would write an entry to > a > > solr_oom_kill log. And there is no such log in the logs directory. > > > > Many thanks. > > > > > > > > > >> Then use some system admin tools to monitor that server, > >> for instance, top, vmstat, lsof, iostat ... or simply install some nice > >> free monitoring tool into this system, like monit, monitorix, nagios. > >> Good luck! > >> > >> > >> From: Ryan W > >> Sent: Thursday, June 11, 2020 2:13 AM > >>
Re: How to determine why solr stops running?
On Wed, Jun 10, 2020 at 8:35 PM Hup Chen wrote: > I will check "dmesg" first, to find out any hardware error message. > Here is what I see toward the end of the output from dmesg: [1521232.781785] [118857]48 118857 108785 677 201 901 0 httpd [1521232.781787] [118860]48 118860 108785 710 201 881 0 httpd [1521232.781788] [118862]48 118862 113063 5256 210 725 0 httpd [1521232.781790] [118864]48 118864 114085 6634 212 703 0 httpd [1521232.781791] [118871]48 118871 13968732323 262 620 0 httpd [1521232.781793] [118873]48 118873 108785 821 201 792 0 httpd [1521232.781795] [118879]48 118879 14026332719 263 621 0 httpd [1521232.781796] [118903]48 118903 108785 812 201 771 0 httpd [1521232.781798] [118905]48 118905 113575 5606 211 660 0 httpd [1521232.781800] [118906]48 118906 113563 5694 211 626 0 httpd [1521232.781801] Out of memory: Kill process 117529 (httpd) score 9 or sacrifice child [1521232.782908] Killed process 117529 (httpd), UID 48, total-vm:675824kB, anon-rss:181844kB, file-rss:0kB, shmem-rss:0kB Is this a relevant "Out of memory" message? Does this suggest an OOM situation is the culprit? When I grep in the solr logs for oom, I see some entries like this... ./solr_gc.log.4.current:CommandLine flags: -XX:CICompilerCount=4 -XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000 -XX:+CMSParallelRemarkEnabled -XX:+CMSScavengeBeforeRemark -XX:ConcGCThreads=4 -XX:GCLogFileSize=20971520 -XX:InitialHeapSize=536870912 -XX:MaxHeapSize=536870912 -XX:MaxNewSize=134217728 -XX:MaxTenuringThreshold=8 -XX:MinHeapDeltaBytes=196608 -XX:NewRatio=3 -XX:NewSize=134217728 -XX:NumberOfGCLogFiles=9 -XX:OldPLABSize=16 -XX:OldSize=402653184 -XX:-OmitStackTraceInFastThrow -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 /opt/solr/server/logs -XX:ParallelGCThreads=4 -XX:+ParallelRefProcEnabled -XX:PretenureSizeThreshold=67108864 -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -XX:+PrintTenuringDistribution -XX:SurvivorRatio=4 -XX:TargetSurvivorRatio=90 -XX:ThreadStackSize=256 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation -XX:+UseParNewGC Buried in there I see "OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh". But I think this is just a setting that indicates what to do in case of an OOM. And if I look in that oom_solr.sh file, I see it would write an entry to a solr_oom_kill log. And there is no such log in the logs directory. Many thanks. > Then use some system admin tools to monitor that server, > for instance, top, vmstat, lsof, iostat ... or simply install some nice > free monitoring tool into this system, like monit, monitorix, nagios. > Good luck! > > > From: Ryan W > Sent: Thursday, June 11, 2020 2:13 AM > To: solr-user@lucene.apache.org > Subject: Re: How to determine why solr stops running? > > Hi all, > > People keep suggesting I check the logs for errors. What do those errors > look like? Does anyone have examples of the text of a Solr oom error? Or > the text of any other errors I should be looking for the next time solr > fails? Are there phrases I should grep for in the logs? Should I be > looking in the Solr logs for an OOM error, or in the Apache logs? > > There is nothing failing on the server except for solr -- at least not that > I can see. There is no apparent problem with the hardware or anything else > on the server. The OS is Red Hat Enterprise Linux. The server has 16 GB of > RAM and hosts one website that does not get a huge amount of traffic. > > When the start command is given to solr, does it first check to see if solr > is running, or does it always start solr whether it is already running or > not? > > Many thanks! > Ryan > > > On Tue, Jun 9, 2020 at 7:58 AM Erick Erickson > wrote: > > > To add to what Dave said, if you have a particular machine that’s prone > to > > suddenly stopping, that’s usually a red flag that you should seriously > > think about hardware issues. > > > > If the problem strikes different machines, then I agree with Shawn that > > the first thing I’d be suspicious of is OOM errors. > > > > FWIW, > > Erick > > > > > On Jun 9, 2020, at 6:05 AM, Dave wrote: > > > > > > I’ll add that whenever I’ve had a solr instance shut down, for me it’s > > been a hardware failure. Either the ram or the disk got a “glitch” and > both > > of
Re: How to determine why solr stops running?
Hi all, People keep suggesting I check the logs for errors. What do those errors look like? Does anyone have examples of the text of a Solr oom error? Or the text of any other errors I should be looking for the next time solr fails? Are there phrases I should grep for in the logs? Should I be looking in the Solr logs for an OOM error, or in the Apache logs? There is nothing failing on the server except for solr -- at least not that I can see. There is no apparent problem with the hardware or anything else on the server. The OS is Red Hat Enterprise Linux. The server has 16 GB of RAM and hosts one website that does not get a huge amount of traffic. When the start command is given to solr, does it first check to see if solr is running, or does it always start solr whether it is already running or not? Many thanks! Ryan On Tue, Jun 9, 2020 at 7:58 AM Erick Erickson wrote: > To add to what Dave said, if you have a particular machine that’s prone to > suddenly stopping, that’s usually a red flag that you should seriously > think about hardware issues. > > If the problem strikes different machines, then I agree with Shawn that > the first thing I’d be suspicious of is OOM errors. > > FWIW, > Erick > > > On Jun 9, 2020, at 6:05 AM, Dave wrote: > > > > I’ll add that whenever I’ve had a solr instance shut down, for me it’s > been a hardware failure. Either the ram or the disk got a “glitch” and both > of these are relatively fragile and wear and tear type parts of the > machine, and should be expected to fail and be replaced from time to time. > Solr is pretty aggressive with its logging so there are a lot of writes > always happening and of course reads, if the disk has any issues or the > memory it can lock it up and bring her down, more so if you have any > spellcheck dictionaries or suggesters being built on start up. > > > > Just my experience with this, could be wrong (most likely wrong) but we > always have extra drives and memory around the server room for this > reason. At least once or twice a year we will have a disk failure in the > raid and need to swap in a new one. > > > > Good luck though, also solr should be logging it’s failures so it would > be good to look there too > > > >> On Jun 9, 2020, at 2:35 AM, Shawn Heisey wrote: > >> > >> On 5/14/2020 7:22 AM, Ryan W wrote: > >>> I manage a site where solr has stopped running a couple times in the > past > >>> week. The server hasn't been rebooted, so that's not the reason. What > else > >>> causes solr to stop running? How can I investigate why this is > happening? > >> > >> Any situation where Solr stops running and nobody requested the stop is > a result of a serious problem that must be thoroughly investigated. I > think it's a bad idea for Solr to automatically restart when it stops > unexpectedly. Chances are that whatever caused the crash is going to > simply make the crash happen again until the problem is solved. > Automatically restarting could hide problems from the system administrator. > >> > >> The only way a Solr auto-restart would be acceptable to me is if it > sends a high priority alert to the sysadmin EVERY time it executes an > auto-restart. It really is that bad of a problem. > >> > >> The causes of Solr crashes (that I can think of) include the following. > I believe I have listed these four options from most likely to least likely: > >> > >> * Java OutOfMemoryError exceptions. On non-windows systems, the > "bin/solr" script starts Solr with an option that results in Solr's death > anytime one of these exceptions occurs. We do this because program > operation is indeterminate and completely unpredictable when OOME occurs, > so it's far safer to stop running. That exception can be caused by several > things, some of which actually do not involve memory at all. If you're > running on Windows via the bin\solr.cmd command, then this will not happen > ... but OOME could still cause a crash, because as I already mentioned, > program operation is unpredictable when OOME occurs. > >> > >> * The OS kills Solr because system memory is completely exhausted and > Solr is the process using the most memory. Linux calls this the > "oom-killer" ... I am pretty sure something like it exists on most > operating systems. > >> > >> * Corruption somewhere in the system. Could be in Java, the OS, Solr, > or data used by any of those. > >> > >> * A very serious bug in Solr's code that we haven't discovered yet. > >> > >> I included that last one simply for completeness. A bug that
Re: Script to check if solr is running
"A simple cronjob with /bin/solr status and /bin/solr start should do the trick." I don't know what that would look like. Wouldn't the job have to check the status and only give the start command if solr isn't running? I don't think it's possible to put logic in a cron job. I think it would have to be in a script, with a cron job to run the script. I've never had cause to write such a script, though, so I don't know how it's done. On Fri, Jun 5, 2020 at 11:19 AM Dunigan, Craig A. < craig.duni...@landsend.com> wrote: > A simple cronjob with /bin/solr status and directory>/bin/solr start should do the trick. There must be a Windows > equivalent if that’s what you’re using. > > From: Ryan W > Sent: Thursday, June 4, 2020 11:39 AM > To: solr-user@lucene.apache.org > Subject: Re: Script to check if solr is running > > WARNING: This email originated outside of Lands’ End. Please be on the > lookout for phishing scams and do not open attachments or click links from > people you do not know.. > > Or is it not much overhead to give the command to start solr if it is > already running? Maybe it's not necessary to check if it's running? Is > there any downside to giving the start command every 15 minutes or so > whether it is running or not? > > Thanks. > > On Thu, Jun 4, 2020 at 12:36 PM Ryan W rya...@gmail.com>> wrote: > > > Does anyone have a script that checks if solr is running and then starts > > it if it isn't running? Occasionally my solr stops running even if there > > has been no Apache restart. I haven't been able to determine the root > > cause, so the next best thing might be to check every 15 minutes or so if > > it's running and run it if it has stopped. > > > > Thanks. > > >
Re: How to determine why solr stops running?
"If Solr auto-restarts" It doesn't auto-restart. Is there some auto-restart functionality? I'm not aware of that. On Mon, Jun 8, 2020 at 7:10 AM Radu Gheorghe wrote: > Hi Ryan, > > If Solr auto-restarts, I suppose it's systemd doing that. When it restarts > the Solr service, systemd should log this (maybe somethibg like: journalctl > --no-pager | grep -i solr). > > Then you can go in your Solr logs and check what happened right before that > time. Also, check system logs for what happened before Solr was restarted. > > Best regards, > Radu > > https://sematext.com/ > > joi, 4 iun. 2020, 19:24 Ryan W a scris: > > > Happened again today. Solr stopped running. Apache hasn't stopped in 10 > > days, so this is not due to a server reboot. > > > > Solr is not being run with the oom-killer. And when I grep for ERROR in > > the logs, there is nothing from today. > > > > On Mon, May 18, 2020 at 3:15 PM James Greene < > ja...@jamesaustingreene.com> > > wrote: > > > > > I usually do a combination of grepping for ERROR in solr logs and > > checking > > > journalctl to see if an external program may have killed the process. > > > > > > > > > Cheers, > > > > > > / > > > * James Austin Greene > > > * www.jamesaustingreene.com > > > * 336-lol-nerd > > > / > > > > > > > > > On Mon, May 18, 2020 at 1:39 PM Erick Erickson < > erickerick...@gmail.com> > > > wrote: > > > > > > > ps aux | grep solr > > > > > > > > on a *.nix system will show you all the runtime parameters. > > > > > > > > > On May 18, 2020, at 12:46 PM, Ryan W wrote: > > > > > > > > > > Is there a config file containing the start params? I run solr > > like... > > > > > > > > > > bin/solr start > > > > > > > > > > I have not seen anything in the logs that seems informative. When I > > > grep > > > > in > > > > > the logs directory for 'memory', I see nothing besides a couple > > entries > > > > > like... > > > > > > > > > > 2020-05-14 13:05:56.155 INFO (main) [ ] > > > > o.a.s.h.a.MetricsHistoryHandler > > > > > No .system collection, keeping metrics history in memory. > > > > > > > > > > I don't know what that entry means, though the date does roughly > > > coincide > > > > > with the last time solr stopped running. > > > > > > > > > > Thank you. > > > > > > > > > > > > > > > On Mon, May 18, 2020 at 12:00 PM Erick Erickson < > > > erickerick...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > >> Probably, but check that you are running with the oom-killer, > it'll > > be > > > > in > > > > >> your start params. > > > > >> > > > > >> But absent that, something external will be the culprit, Solr > > doesn't > > > > stop > > > > >> by itself. Do look at the Solr log once things stop, it should > show > > if > > > > >> someone or something stopped it. > > > > >> > > > > >> On Mon, May 18, 2020, 10:43 Ryan W wrote: > > > > >> > > > > >>> I don't see any log file with "oom" in the file name. Does that > > mean > > > > >> there > > > > >>> hasn't been an out-of-memory issue? Thanks. > > > > >>> > > > > >>> On Thu, May 14, 2020 at 10:05 AM James Greene < > > > > >> ja...@jamesaustingreene.com > > > > >>>> > > > > >>> wrote: > > > > >>> > > > > >>>> Check the log for for an OOM crash. Fatal exceptions will be in > > the > > > > >> main > > > > >>>> solr log and out of memory errors will be in their own -oom log. > > > > >>>> > > > > >>>> I've encountered quite a few solr crashes and usually it's when > > > > >> there's a > > > > >>>> threshold of concurrent users and/or indexing happening. > > > > >>>> > > > > >>>> > > > > >>>> > > > > >>>> On Thu, May 14, 2020, 9:23 AM Ryan W wrote: > > > > >>>> > > > > >>>>> Hi all, > > > > >>>>> > > > > >>>>> I manage a site where solr has stopped running a couple times > in > > > the > > > > >>> past > > > > >>>>> week. The server hasn't been rebooted, so that's not the > reason. > > > > >> What > > > > >>>> else > > > > >>>>> causes solr to stop running? How can I investigate why this is > > > > >>>> happening? > > > > >>>>> > > > > >>>>> Thank you, > > > > >>>>> Ryan > > > > >>>>> > > > > >>>> > > > > >>> > > > > >> > > > > > > > > > > > > > >
Re: Script to check if solr is running
Or is it not much overhead to give the command to start solr if it is already running? Maybe it's not necessary to check if it's running? Is there any downside to giving the start command every 15 minutes or so whether it is running or not? Thanks. On Thu, Jun 4, 2020 at 12:36 PM Ryan W wrote: > Does anyone have a script that checks if solr is running and then starts > it if it isn't running? Occasionally my solr stops running even if there > has been no Apache restart. I haven't been able to determine the root > cause, so the next best thing might be to check every 15 minutes or so if > it's running and run it if it has stopped. > > Thanks. >
Script to check if solr is running
Does anyone have a script that checks if solr is running and then starts it if it isn't running? Occasionally my solr stops running even if there has been no Apache restart. I haven't been able to determine the root cause, so the next best thing might be to check every 15 minutes or so if it's running and run it if it has stopped. Thanks.
Re: How to determine why solr stops running?
Happened again today. Solr stopped running. Apache hasn't stopped in 10 days, so this is not due to a server reboot. Solr is not being run with the oom-killer. And when I grep for ERROR in the logs, there is nothing from today. On Mon, May 18, 2020 at 3:15 PM James Greene wrote: > I usually do a combination of grepping for ERROR in solr logs and checking > journalctl to see if an external program may have killed the process. > > > Cheers, > > / > * James Austin Greene > * www.jamesaustingreene.com > * 336-lol-nerd > / > > > On Mon, May 18, 2020 at 1:39 PM Erick Erickson > wrote: > > > ps aux | grep solr > > > > on a *.nix system will show you all the runtime parameters. > > > > > On May 18, 2020, at 12:46 PM, Ryan W wrote: > > > > > > Is there a config file containing the start params? I run solr like... > > > > > > bin/solr start > > > > > > I have not seen anything in the logs that seems informative. When I > grep > > in > > > the logs directory for 'memory', I see nothing besides a couple entries > > > like... > > > > > > 2020-05-14 13:05:56.155 INFO (main) [ ] > > o.a.s.h.a.MetricsHistoryHandler > > > No .system collection, keeping metrics history in memory. > > > > > > I don't know what that entry means, though the date does roughly > coincide > > > with the last time solr stopped running. > > > > > > Thank you. > > > > > > > > > On Mon, May 18, 2020 at 12:00 PM Erick Erickson < > erickerick...@gmail.com > > > > > > wrote: > > > > > >> Probably, but check that you are running with the oom-killer, it'll be > > in > > >> your start params. > > >> > > >> But absent that, something external will be the culprit, Solr doesn't > > stop > > >> by itself. Do look at the Solr log once things stop, it should show if > > >> someone or something stopped it. > > >> > > >> On Mon, May 18, 2020, 10:43 Ryan W wrote: > > >> > > >>> I don't see any log file with "oom" in the file name. Does that mean > > >> there > > >>> hasn't been an out-of-memory issue? Thanks. > > >>> > > >>> On Thu, May 14, 2020 at 10:05 AM James Greene < > > >> ja...@jamesaustingreene.com > > >>>> > > >>> wrote: > > >>> > > >>>> Check the log for for an OOM crash. Fatal exceptions will be in the > > >> main > > >>>> solr log and out of memory errors will be in their own -oom log. > > >>>> > > >>>> I've encountered quite a few solr crashes and usually it's when > > >> there's a > > >>>> threshold of concurrent users and/or indexing happening. > > >>>> > > >>>> > > >>>> > > >>>> On Thu, May 14, 2020, 9:23 AM Ryan W wrote: > > >>>> > > >>>>> Hi all, > > >>>>> > > >>>>> I manage a site where solr has stopped running a couple times in > the > > >>> past > > >>>>> week. The server hasn't been rebooted, so that's not the reason. > > >> What > > >>>> else > > >>>>> causes solr to stop running? How can I investigate why this is > > >>>> happening? > > >>>>> > > >>>>> Thank you, > > >>>>> Ryan > > >>>>> > > >>>> > > >>> > > >> > > > > >
Re: How to determine why solr stops running?
Is there a config file containing the start params? I run solr like... bin/solr start I have not seen anything in the logs that seems informative. When I grep in the logs directory for 'memory', I see nothing besides a couple entries like... 2020-05-14 13:05:56.155 INFO (main) [ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory. I don't know what that entry means, though the date does roughly coincide with the last time solr stopped running. Thank you. On Mon, May 18, 2020 at 12:00 PM Erick Erickson wrote: > Probably, but check that you are running with the oom-killer, it'll be in > your start params. > > But absent that, something external will be the culprit, Solr doesn't stop > by itself. Do look at the Solr log once things stop, it should show if > someone or something stopped it. > > On Mon, May 18, 2020, 10:43 Ryan W wrote: > > > I don't see any log file with "oom" in the file name. Does that mean > there > > hasn't been an out-of-memory issue? Thanks. > > > > On Thu, May 14, 2020 at 10:05 AM James Greene < > ja...@jamesaustingreene.com > > > > > wrote: > > > > > Check the log for for an OOM crash. Fatal exceptions will be in the > main > > > solr log and out of memory errors will be in their own -oom log. > > > > > > I've encountered quite a few solr crashes and usually it's when > there's a > > > threshold of concurrent users and/or indexing happening. > > > > > > > > > > > > On Thu, May 14, 2020, 9:23 AM Ryan W wrote: > > > > > > > Hi all, > > > > > > > > I manage a site where solr has stopped running a couple times in the > > past > > > > week. The server hasn't been rebooted, so that's not the reason. > What > > > else > > > > causes solr to stop running? How can I investigate why this is > > > happening? > > > > > > > > Thank you, > > > > Ryan > > > > > > > > > >
Re: How to determine why solr stops running?
I don't see any log file with "oom" in the file name. Does that mean there hasn't been an out-of-memory issue? Thanks. On Thu, May 14, 2020 at 10:05 AM James Greene wrote: > Check the log for for an OOM crash. Fatal exceptions will be in the main > solr log and out of memory errors will be in their own -oom log. > > I've encountered quite a few solr crashes and usually it's when there's a > threshold of concurrent users and/or indexing happening. > > > > On Thu, May 14, 2020, 9:23 AM Ryan W wrote: > > > Hi all, > > > > I manage a site where solr has stopped running a couple times in the past > > week. The server hasn't been rebooted, so that's not the reason. What > else > > causes solr to stop running? How can I investigate why this is > happening? > > > > Thank you, > > Ryan > > >
How to determine why solr stops running?
Hi all, I manage a site where solr has stopped running a couple times in the past week. The server hasn't been rebooted, so that's not the reason. What else causes solr to stop running? How can I investigate why this is happening? Thank you, Ryan
Re: How do *you* restrict access to Solr?
On Tue, Mar 17, 2020 at 6:05 AM Jan Høydahl wrote: > You can consider upgrading to Solr 8.5 which is to be released in a couple > of days, which makes it easy to whitelist IP addresses in solr.in.sh: > Thanks. That is good news, though it won't help me this time around. My application framework (Drupal) doesn't support Solr 8. I may try Solr 6 again, or take another stab at getting the Basic Authentication plugin to work in Solr 7. My Solr install isn't web-accessible, so the only threats would come from inside the network. > > # Allow IPv4/IPv6 localhost, the 192.168.0.x IPv4 network, and > 2000:123:4:5:: IPv6 network. > SOLR_IP_WHITELIST="127.0.0.1, [::1], 192.168.0.0/24, [2000:123:4:5::]/64" > > > https://lucene.apache.org/solr/guide/8_5/securing-solr.html#enable-ip-access-control > > But please please do not expose Solr, even if secured, to untrusted > networks and never to the public internet. > > Jan > > > 16. mar. 2020 kl. 16:46 skrev Ryan W : > > > > On Mon, Mar 16, 2020 at 10:51 AM Susheel Kumar > > wrote: > > > >> Basic auth should help you to start > >> > >> > https://lucene.apache.org/solr/guide/8_1/basic-authentication-plugin.html > > > > > > > > Thanks. I think I will give up on the plugin system. I haven't been > able > > to get the plugin system to work, and it creates too many opportunities > for > > human error. Even if I can get it working this week, what about 6 months > > from now or a year from now when something goes wrong and I have to debug > > it. It seems like far too much overhead to provide the desired security > > benefit, except perhaps in situations where an organization has Solr > > specialists who can maintain the system. > >
Re: How do *you* restrict access to Solr?
On Mon, Mar 16, 2020 at 10:51 AM Susheel Kumar wrote: > Basic auth should help you to start > > https://lucene.apache.org/solr/guide/8_1/basic-authentication-plugin.html Thanks. I think I will give up on the plugin system. I haven't been able to get the plugin system to work, and it creates too many opportunities for human error. Even if I can get it working this week, what about 6 months from now or a year from now when something goes wrong and I have to debug it. It seems like far too much overhead to provide the desired security benefit, except perhaps in situations where an organization has Solr specialists who can maintain the system.
Re: How do *you* restrict access to Solr?
On Mon, Mar 16, 2020 at 11:40 AM Walter Underwood wrote: > Also, even if you prevent access to the admin UI, a request to /update can > delete > all the content. It is really easy. This Gist shows how. > > https://gist.github.com/nz/673027/313f70681daa985ea13ba33a385753aef951a0f3 This seems important. In other words, my work isn't necessarily done if I've secured the graphical UI. I can't just visit the admin UI page to see if my efforts are successful. > > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > > On Mar 16, 2020, at 8:20 AM, David Hastings < > hastings.recurs...@gmail.com> wrote: > > > > master slave is the idea that you have an indexing server you do all > > indexing to and a search server that replicates the index, to deliver the > > results etc. if you keep the indexer separate you can tune it > differently > > as well as protect the data. also means you can remove the delete/update > > request handlers from the slave/searcher > > > > yes security by obscurity isnt ideal, but the over head of adding > > authentication to requests i find unnecessary, > > > > On Mon, Mar 16, 2020 at 11:16 AM Ryan W wrote: > > > >> On Mon, Mar 16, 2020 at 11:09 AM Walter Underwood < > wun...@wunderwood.org> > >> wrote: > >> > >>> What access do you want to prevent? How do you prefer to authenticate? > >>> How do you manage users or roles? Master/slave or Solr Cloud? > >>> > >> > >> I want to prevent access to the admin UI. > >> > >> I don't want to manage users or roles, preferably. I have only one > user: > >> staff. I want to prevent the public from accessing the admin UI. I'd > be > >> happy if I could set an IP address whitelist... especially if I don't > have > >> to learn a new framework (which I will never use for any other purpose) > to > >> do it. > >> > >> I don't know what master/slave is. These are new concepts that weren't > >> required to secure Solr prior to 7x, and this is my first project using > a > >> version after 6x. > >> > >> Thanks! > >> > >> > >> > >>> > >>> wunder > >>> Walter Underwood > >>> wun...@wunderwood.org > >>> http://observer.wunderwood.org/ (my blog) > >>> > >>>> On Mar 16, 2020, at 7:44 AM, Ryan W wrote: > >>>> > >>>> How do you, personally, do it? Do you use IPTables? Basic > >>> Authentication > >>>> Plugin? Something else? > >>>> > >>>> I'm asking in part so I'l have something to search for. I don't know > >>> where > >>>> I should begin, so I figured I would ask how others do it. > >>>> > >>>> I haven't been able to find anything that works, so if you can tell me > >>> what > >>>> works for you, I can at least narrow it down a bit and do some Google > >>>> searches. Do I need to learn Solr's plugin system? Am I starting in > >> the > >>>> right place if I follow this document: > >>>> > >>> > >> > https://lucene.apache.org/solr/guide/7_0/rule-based-authorization-plugin.html#rule-based-authorization-plugin > >>>> > >>>> Initially, the above document seems far too comprehensive for my > needs. > >>> I > >>>> just want to block access to the Solr admin UI, and the list of > >>> predefined > >>>> permissions in that document don't seem to be relevant. Also, it > seems > >>>> unlikely this plugin system is necessary just to control access to the > >>>> admin UI... or maybe it necessary? > >>>> > >>>> In any case, what is your approach? > >>>> > >>>> I'm using version 7.7.2 of Solr. > >>>> > >>>> Thanks! > >>> > >>> > >> > >
Re: How do *you* restrict access to Solr?
On Mon, Mar 16, 2020 at 11:32 AM Dunigan, Craig A. < craig.duni...@landsend.com> wrote: > Here are my suggestions. If you’re okay with IP restrictions only, then > iptables. Thanks! Just knowing this is an option helps. I took a stab at it but it didn't work initially, but at least now I know there's a reason to keep trying it. > If you don’t have *nix or root access, an Apache proxy server with Allow > from . I do have root access and can edit the Apache config. Can I restrict access in the Apache config? If so, that would be a great solution. My situation is fairly typical. I have a LAMP environment with Red Hat linux. I'm not quite sure how to make my Apache directives specific to the Solr install. Again, just knowing this is an option would be helpful. The Solr docs don't mention this possibility, I don't think. > If you want really, really secure, an stunnel front-end that requires > client certs that you install in your browsers. For us, we have a load > balancer with VIPs that restrict access to the internal IP range of the > building that houses IT, but not everyone has the luxury of hardware > solutions. > > From: Ryan W > Sent: Monday, March 16, 2020 10:20 AM > To: solr-user@lucene.apache.org > Subject: Re: How do *you* restrict access to Solr? > > WARNING: This email originated outside of Lands’ End. Please be on the > lookout for phishing scams and do not open attachments or click links from > people you do not know.. > > On Mon, Mar 16, 2020 at 10:50 AM David Hastings < > hastings.recurs...@gmail.com<mailto:hastings.recurs...@gmail.com>> wrote: > > > Honestly? I know this isnt what youre going to want to hear, but security > > through obscurity. no one else knows what port the servers on, and its > not > > accessible from anything outside of the internal network. > > > That doesn't sound like security through obscurity, as long as you are > confident that access to the internal network is limited... to whatever > degree you require. I'd certainly be happy if I could restrict access > based on IP. > > > > > if your solr > > install can be accessed from an external IP you have much larger issues. > > > > On Mon, Mar 16, 2020 at 10:44 AM Ryan W rya...@gmail.com>> wrote: > > > > > How do you, personally, do it? Do you use IPTables? Basic > > Authentication > > > Plugin? Something else? > > > > > > I'm asking in part so I'l have something to search for. I don't know > > where > > > I should begin, so I figured I would ask how others do it. > > > > > > I haven't been able to find anything that works, so if you can tell me > > what > > > works for you, I can at least narrow it down a bit and do some Google > > > searches. Do I need to learn Solr's plugin system? Am I starting in the > > > right place if I follow this document: > > > > > > > > > https://lucene.apache.org/solr/guide/7_0/rule-based-authorization-plugin.html#rule-based-authorization-plugin > < > https://lucene.apache.org/solr/guide/7_0/rule-based-authorization-plugin.html#rule-based-authorization-plugin > > > > > > > > Initially, the above document seems far too comprehensive for my needs. > > I > > > just want to block access to the Solr admin UI, and the list of > > predefined > > > permissions in that document don't seem to be relevant. Also, it seems > > > unlikely this plugin system is necessary just to control access to the > > > admin UI... or maybe it necessary? > > > > > > In any case, what is your approach? > > > > > > I'm using version 7.7.2 of Solr. > > > > > > Thanks! > > > > > >
Re: How do *you* restrict access to Solr?
On Mon, Mar 16, 2020 at 10:50 AM David Hastings < hastings.recurs...@gmail.com> wrote: > Honestly? I know this isnt what youre going to want to hear, but security > through obscurity. no one else knows what port the servers on, and its not > accessible from anything outside of the internal network. That doesn't sound like security through obscurity, as long as you are confident that access to the internal network is limited... to whatever degree you require. I'd certainly be happy if I could restrict access based on IP. > if your solr > install can be accessed from an external IP you have much larger issues. > On Mon, Mar 16, 2020 at 10:44 AM Ryan W wrote: > > > How do you, personally, do it? Do you use IPTables? Basic > Authentication > > Plugin? Something else? > > > > I'm asking in part so I'l have something to search for. I don't know > where > > I should begin, so I figured I would ask how others do it. > > > > I haven't been able to find anything that works, so if you can tell me > what > > works for you, I can at least narrow it down a bit and do some Google > > searches. Do I need to learn Solr's plugin system? Am I starting in the > > right place if I follow this document: > > > > > https://lucene.apache.org/solr/guide/7_0/rule-based-authorization-plugin.html#rule-based-authorization-plugin > > > > Initially, the above document seems far too comprehensive for my needs. > I > > just want to block access to the Solr admin UI, and the list of > predefined > > permissions in that document don't seem to be relevant. Also, it seems > > unlikely this plugin system is necessary just to control access to the > > admin UI... or maybe it necessary? > > > > In any case, what is your approach? > > > > I'm using version 7.7.2 of Solr. > > > > Thanks! > > >
Re: How do *you* restrict access to Solr?
On Mon, Mar 16, 2020 at 11:09 AM Walter Underwood wrote: > What access do you want to prevent? How do you prefer to authenticate? > How do you manage users or roles? Master/slave or Solr Cloud? > I want to prevent access to the admin UI. I don't want to manage users or roles, preferably. I have only one user: staff. I want to prevent the public from accessing the admin UI. I'd be happy if I could set an IP address whitelist... especially if I don't have to learn a new framework (which I will never use for any other purpose) to do it. I don't know what master/slave is. These are new concepts that weren't required to secure Solr prior to 7x, and this is my first project using a version after 6x. Thanks! > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > > On Mar 16, 2020, at 7:44 AM, Ryan W wrote: > > > > How do you, personally, do it? Do you use IPTables? Basic > Authentication > > Plugin? Something else? > > > > I'm asking in part so I'l have something to search for. I don't know > where > > I should begin, so I figured I would ask how others do it. > > > > I haven't been able to find anything that works, so if you can tell me > what > > works for you, I can at least narrow it down a bit and do some Google > > searches. Do I need to learn Solr's plugin system? Am I starting in the > > right place if I follow this document: > > > https://lucene.apache.org/solr/guide/7_0/rule-based-authorization-plugin.html#rule-based-authorization-plugin > > > > Initially, the above document seems far too comprehensive for my needs. > I > > just want to block access to the Solr admin UI, and the list of > predefined > > permissions in that document don't seem to be relevant. Also, it seems > > unlikely this plugin system is necessary just to control access to the > > admin UI... or maybe it necessary? > > > > In any case, what is your approach? > > > > I'm using version 7.7.2 of Solr. > > > > Thanks! > >
Re: How do *you* restrict access to Solr?
Thanks Jorn, though this all seems unrealistic. Because the technical skill required to secure Solr far exceeds the technical skill required to install it, I suspect there are probably a lot of insecure installs out there. In many cases this will not apply: "if you work with people that know a bit about those topics in your enterprise." Solr is used in many situations where the developer does not have access to a large enterprise with highly specialized assistance. On Mon, Mar 16, 2020 at 11:00 AM Jörn Franke wrote: > Solr should not be accessible to end users directly - only through a > dedicated application in between. > > Then in an enterprise setting it is mostly Kerberos auth. and https (do > not forget about zookeeper when using Solr cloud here you can also have > Kerberos auth and in recent version also SSL). It is not that difficult to > configure if you work with people that know a bit about those topics in > your enterprise. > > In a Cloud based scenario jwt token can make sense. > > Do not do security by obscurity. You owe it to the users that potentially > also have private data on Solr. > > > Am 16.03.2020 um 15:44 schrieb Ryan W : > > > > How do you, personally, do it? Do you use IPTables? Basic > Authentication > > Plugin? Something else? > > > > I'm asking in part so I'l have something to search for. I don't know > where > > I should begin, so I figured I would ask how others do it. > > > > I haven't been able to find anything that works, so if you can tell me > what > > works for you, I can at least narrow it down a bit and do some Google > > searches. Do I need to learn Solr's plugin system? Am I starting in the > > right place if I follow this document: > > > https://lucene.apache.org/solr/guide/7_0/rule-based-authorization-plugin.html#rule-based-authorization-plugin > > > > Initially, the above document seems far too comprehensive for my needs. > I > > just want to block access to the Solr admin UI, and the list of > predefined > > permissions in that document don't seem to be relevant. Also, it seems > > unlikely this plugin system is necessary just to control access to the > > admin UI... or maybe it necessary? > > > > In any case, what is your approach? > > > > I'm using version 7.7.2 of Solr. > > > > Thanks! >
How do *you* restrict access to Solr?
How do you, personally, do it? Do you use IPTables? Basic Authentication Plugin? Something else? I'm asking in part so I'l have something to search for. I don't know where I should begin, so I figured I would ask how others do it. I haven't been able to find anything that works, so if you can tell me what works for you, I can at least narrow it down a bit and do some Google searches. Do I need to learn Solr's plugin system? Am I starting in the right place if I follow this document: https://lucene.apache.org/solr/guide/7_0/rule-based-authorization-plugin.html#rule-based-authorization-plugin Initially, the above document seems far too comprehensive for my needs. I just want to block access to the Solr admin UI, and the list of predefined permissions in that document don't seem to be relevant. Also, it seems unlikely this plugin system is necessary just to control access to the admin UI... or maybe it necessary? In any case, what is your approach? I'm using version 7.7.2 of Solr. Thanks!
How to use InetAccessHandler?
Does anyone have an example of how InetAccessHandler is implemented? My jetty version is 9.4.14. Will I be able to whitelist IP addresses just by dropping a jetty-inetaccess.xml file in my server/etc directory? Are there other steps required? If this is the right way to do it, what is the proper syntax for whitelisting an IP address in a file like this: https://fossies.org/linux/jetty-distribution/modules/inetaccess/jetty-inetaccess.xml Thanks, Ryan
Re: Restrict access to admin interface by IP address in Solr 7.7.2
It looks like IPAccessHandler is deprecated in my version of jetty (9.4.14) in favor of InetAccessHandler. However, I can find no information or examples of how to implement InetAccessHandler. I see some disembodied code examples here and there, but no indication of where and how to implement. On Wed, Mar 11, 2020 at 6:25 PM Jan Høydahl wrote: > That should still work if you follow Jetty’s procedures for the version of > Jetty you’re at. > In 8.5 this will be much simpler: > https://issues.apache.org/jira/browse/SOLR-14136 > > Jan > > > 11. mar. 2020 kl. 20:39 skrev Ryan W : > > > > What options do I have for restricting access to the admin UI by IP > > address? In Solr 6.x, this could be done with an IPAccessHandler in > > jetty.xml, but this doesn't seem to work in Solr 7.x. Is there some > other > > way to do it? > > > > Thanks! > >
Restrict access to admin interface by IP address in Solr 7.7.2
What options do I have for restricting access to the admin UI by IP address? In Solr 6.x, this could be done with an IPAccessHandler in jetty.xml, but this doesn't seem to work in Solr 7.x. Is there some other way to do it? Thanks!
Re: "SolrCore Initialization Failures" in the Solr's current UI, but not in the original UI
I upgraded to v7.7.2 and am not having this issue anymore.
On Mon, Mar 9, 2020 at 10:04 AM Ryan W wrote:
> I am using v6.6.6, which is the most recent release in the v6 branch, and
> is the branch commonly used with my app. I have tried on Chrome, Firefox
> and Internet Explorer, and I see the same problem in all three. Albeit it
> seems to be a minor problem.
>
> On Thu, Mar 5, 2020 at 4:34 PM Jan Høydahl wrote:
>
>> v6.6.0 is from 2017 and not supported anymore. You are really encouraged
>> to upgrade!
>>
>> Are you by any chance using Internet Explorer? (See
>> https://stackoverflow.com/questions/56262704/solr-solrcore-initialization-failures-core-error
>> )
>>
>> Jan
>>
>> > 5. mar. 2020 kl. 22:03 skrev Ryan W :
>> >
>> > Hi all,
>> >
>> > On my dev server, in the Solr admin UI, I see...
>> >
>> >SolrCore Initialization Failures
>> >{{core}}: {{error}}
>> >Please check your logs for more information
>> >{{exception.msg}}
>> >
>> > These appear to be template variables, but they are never populated.
>> They
>> > just dump to the screen. There is nothing useful in the log. If I
>> click
>> > over to the "original UI" via the link in the upper right corner of the
>> > screen, there is no error displayed and all seems to be well.
>> Meanwhile,
>> > Apache Solr seems to be functioning OK. So this does not appear to be a
>> > significant problem, but still a little annoying.
>> >
>> > I am trying to get the dev server to perform the same way as the staging
>> > server, but at least in the "current UI," I get these errors in my dev
>> > server. I have tried to compared the two installs very carefully, but
>> > can't find any discrepancy between staging and dev.
>> >
>> > Both servers run...
>> > Solr 6.6.6
>> > RHEL 7
>> >
>> > Thank you,
>> > Ryan
>>
>>
Re: "SolrCore Initialization Failures" in the Solr's current UI, but not in the original UI
I am using v6.6.6, which is the most recent release in the v6 branch, and
is the branch commonly used with my app. I have tried on Chrome, Firefox
and Internet Explorer, and I see the same problem in all three. Albeit it
seems to be a minor problem.
On Thu, Mar 5, 2020 at 4:34 PM Jan Høydahl wrote:
> v6.6.0 is from 2017 and not supported anymore. You are really encouraged
> to upgrade!
>
> Are you by any chance using Internet Explorer? (See
> https://stackoverflow.com/questions/56262704/solr-solrcore-initialization-failures-core-error
> )
>
> Jan
>
> > 5. mar. 2020 kl. 22:03 skrev Ryan W :
> >
> > Hi all,
> >
> > On my dev server, in the Solr admin UI, I see...
> >
> >SolrCore Initialization Failures
> >{{core}}: {{error}}
> >Please check your logs for more information
> >{{exception.msg}}
> >
> > These appear to be template variables, but they are never populated.
> They
> > just dump to the screen. There is nothing useful in the log. If I click
> > over to the "original UI" via the link in the upper right corner of the
> > screen, there is no error displayed and all seems to be well. Meanwhile,
> > Apache Solr seems to be functioning OK. So this does not appear to be a
> > significant problem, but still a little annoying.
> >
> > I am trying to get the dev server to perform the same way as the staging
> > server, but at least in the "current UI," I get these errors in my dev
> > server. I have tried to compared the two installs very carefully, but
> > can't find any discrepancy between staging and dev.
> >
> > Both servers run...
> > Solr 6.6.6
> > RHEL 7
> >
> > Thank you,
> > Ryan
>
>
"SolrCore Initialization Failures" in the Solr's current UI, but not in the original UI
Hi all,
On my dev server, in the Solr admin UI, I see...
SolrCore Initialization Failures
{{core}}: {{error}}
Please check your logs for more information
{{exception.msg}}
These appear to be template variables, but they are never populated. They
just dump to the screen. There is nothing useful in the log. If I click
over to the "original UI" via the link in the upper right corner of the
screen, there is no error displayed and all seems to be well. Meanwhile,
Apache Solr seems to be functioning OK. So this does not appear to be a
significant problem, but still a little annoying.
I am trying to get the dev server to perform the same way as the staging
server, but at least in the "current UI," I get these errors in my dev
server. I have tried to compared the two installs very carefully, but
can't find any discrepancy between staging and dev.
Both servers run...
Solr 6.6.6
RHEL 7
Thank you,
Ryan
