Re: How to determine why solr stops running?

[Otis Gospodnetić]

Hi,

Maybe https://github.com/sematext/solr-diagnostics can be of use?

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



On Mon, Jun 29, 2020 at 3:46 PM Erick Erickson 
wrote:

> Really look at your cache size settings.
>
> This is to eliminate this scenario:
> - your cache sizes are very large
> - when you looked and the memory was 9G, you also had a lot of cache
> entries
> - there was a commit, which threw out the old cache and reduced your cache
> size
>
> This is frankly kind of unlikely, but worth checking.
>
> The other option is that you haven’t been hitting OOMs at all and that’s a
> complete
> red herring. Let’s say in actuality, you only need an 8G heap or even
> smaller. By
> overallocating memory garbage will simply accumulate for a long time and
> when it
> is eventually collected, _lots_ of memory will be collected.
>
> Another rather unlikely scenario, but again worth checking.
>
> Best,
> Erick
>
> > On Jun 29, 2020, at 3:27 PM, Ryan W  wrote:
> >
> > On Mon, Jun 29, 2020 at 3:13 PM Erick Erickson 
> > wrote:
> >
> >> ps aux | grep solr
> >>
> >
> > [solr@faspbsy0002 database-backups]$ ps aux | grep solr
> > solr  72072  1.6 33.4 22847816 10966476 ?   Sl   13:35   1:36 java
> > -server -Xms16g -Xmx16g -XX:+UseG1GC -XX:+ParallelRefProcEnabled
> > -XX:G1HeapRegionSize=8m -XX:MaxGCPauseMillis=200 -XX:+UseLargePages
> > -XX:+AggressiveOpts -verbose:gc -XX:+PrintHeapAtGC -XX:+PrintGCDetails
> > -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
> > -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
> > -Xloggc:/opt/solr/server/logs/solr_gc.log -XX:+UseGCLogFileRotation
> > -XX:NumberOfGCLogFiles=9 -XX:GCLogFileSize=20M
> > -Dsolr.log.dir=/opt/solr/server/logs -Djetty.port=8983 -DSTOP.PORT=7983
> > -DSTOP.KEY=solrrocks -Duser.timezone=UTC -Djetty.home=/opt/solr/server
> > -Dsolr.solr.home=/opt/solr/server/solr -Dsolr.data.home=
> > -Dsolr.install.dir=/opt/solr
> > -Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf
> > -Xss256k -Dsolr.jetty.https.port=8983 -Dsolr.log.muteconsole
> > -XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983
> /opt/solr/server/logs
> > -jar start.jar --module=http
> >
> >
> >
> >> should show you all the parameters Solr is running with, as would the
> >> admin screen. You should see something like:
> >>
> >> -XX:OnOutOfMemoryError=your_solr_directory/bin/oom_solr.sh
> >>
> >> And there should be some logs laying around if that was the case
> >> similar to:
> >> $SOLR_LOGS_DIR/solr_oom_killer-$SOLR_PORT-$NOW.log
> >>
> >
> > This log is not being written, even though in the oom_solr.sh it does
> > appear a solr_oom_killer-$SOLR_PORT-$NOW.log should be written to the
> logs
> > directory, but it isn't. There are some log files in
> /opt/solr/server/logs,
> > and they are indeed being written to.  There are fresh entries in the
> logs,
> > but no sign of any problem.  If I grep for oom in the logs directory, the
> > only references I see are benign... just a few entries that list all the
> > flags, and oom_solr.sh is among the settings visible in the entry.  And
> > someone did a search for "Mushroom," so there's another instance of oom
> > from that search.
> >
> >
> > As for memory, It Depends (tm). There are configurations
> >> you can make choices about that will affect the heap requirements.
> >> You can’t really draw comparisons between different projects. Your
> >> Drupal + Solr app has how many documents? Indexed how? Searched
> >> how? .vs. this one.
> >>
> >> The usual suspect for configuration settings that are responsible
> >> include:
> >>
> >> - filterCache size too large. Each filterCache entry is bounded by
> >> maxDoc/8 bytes. I’ve seen people set this to over 1M…
> >>
> >> - using non-docValues for fields used for sorting, grouping, function
> >> queries
> >> or faceting. Solr will uninvert the field on the heap, whereas if you
> have
> >> specified docValues=true, the memory is out in OS memory space rather
> than
> >> heap.
> >>
> >> - People just putting too many docs in a collection in a single JVM in
> >> aggregate.
> >> All replicas in the same instance are using part of the heap.
> >>
> >> - Having unnecessary options on your fields, although that’s more MMap
> >> space than
> >> heap.
> >>
> >> The problem basically is that all of Solr’s access is essentially
> random,
> >> so for
> >> performance reasons lots of stuff has to be in memory.
> >>
> >> That said, Solr hasn’t been as careful as it should be about using up
> >> memory,
> >> that’s ongoing.
> >>
> >> If you really want to know what’s using up memory, throw a heap analysis
> >> tool
> >> at it. That’ll give you a clue what’s hogging memory and you can go from
> >> there.
> >>
> >>> On Jun 29, 2020, at 1:48 PM, David Hastings <
> >> hastings.recurs...@gmail.com> wrote:
> >>>
> >>> little nit picky note here, use 31gb, never 32.
> >>>
> >>> On Mon, Jun 29, 2020 at 

Re: Is there a recommended open source GUI tool for monitoring 'zookeeper'?

[Otis Gospodnetić]

Hi,

Sematext's monitoring agent with a ZooKeeper integration is open-source:
https://github.com/sematext/sematext-agent-java

The ZK integration is at
https://github.com/sematext/sematext-agent-integrations/tree/master/zookeeper
(Solr and SolrCloud integrations are in the same repo)

If you can't use Sematext Cloud  for
monitoring, you can use the above open-source agent and ship
ZooKeeper/Solr/SolrCloud metrics to InfluxDB (open-source) and view it with
something like Grafana (also open-source).

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



On Wed, Jan 9, 2019 at 7:55 PM 유정인  wrote:

> Hi
>
> Is there a recommended open source GUI tool for monitoring 'zookeeper'?
>
>

Re: jetty access logs for monitoring

[Otis Gospodnetić]

Hi,

I think Logagent  has support for Solr
logs.  Aha, yes, it does:
https://github.com/sematext/logagent-js/blob/master/patterns.yml (search
for Solr).

But if you are looking for Solr metrics monitoring, see Solr monitoring
 in Sematext Cloud
(disclosure: see signature :)).

HTH

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Mon, Oct 30, 2017 at 3:56 PM, j.s.  wrote:

> hi
>
> my current problem is monitoring my solr instance. it would be best if i
> could get a jetty access log to appear. i think i can do this with the log4
> interface, but i have not yet figured out the appropriate configuration
> changes.
>
> can someone direct me toward some information on this? i'm running solr
> 6.6. i've looked in the ref guide and around the wiki and have not found
> anything that answers my question. the stuff i've found is for solr 4.X ...
>
> any help is welcome, thx.
>

Re: Disadvantages of having many cores

[Otis Gospodnetić]

Hi,

Core per day is not too bad.  I assume you'll want to keep 7 days or maybe
30 or 60 or 180 days worth of logs.  That won't result in too many cores,
given adequate hardware.

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Fri, Jul 28, 2017 at 9:04 AM, Chellasamy G 
wrote:

> Hi,
>
>
>
> I am working on a log management tool and considering to use solr to
> index/search the logs.
>
> I have few doubts about how to organize or create the cores.
>
>
>
> The tool  should process 200 million events per day with each event
> containing 40 to 50 fields. Currently I have planned to create a core per
> day pushing all the data to the day's core. This may lead to the creation
> of many cores. Is this a good design? If not please suggest a good
> design.(Also, if multiple cores are used, will it slowdown the solr
> process' uptime)
>
>
>
>
>
> Thanks,
>
> Satyan
>
>
>
>

Re: Poll: Master-Slave or SolrCloud?

[Otis Gospodnetić]

Hi Erick,

Could one run *only* embedded ZK on some SolrCloud nodes, sans any data?
It would be equivalent of dedicated Elasticsearch nodes, which is the
current ES best practice/recommendation.  I've never heard of anyone being
scared of running 3 dedicated master ES nodes, so if SolrCloud offered the
same, perhaps even completely hiding ZK from users, that would present the
same level of complexity (err, simplicity) ES users love about ES.  Don't
want to talk about SolrCloud vs. ES here at all, just trying to share
observations since we work a lot with both Elasticsearch and Solr(Cloud) at
Sematext.

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Tue, Apr 25, 2017 at 4:03 PM, Erick Erickson <erickerick...@gmail.com>
wrote:

> bq: I read somewhere that you should run your own ZK externally, and
> turn off SolrCloud
>
> this is a bit confused. "turn off SolrCloud" has nothing to do with
> running ZK internally or externally. SolrCloud requires ZK, whether
> internal or external is irrelevant to the term SolrCloud.
>
> On to running an external ZK ensemble. Mostly, that's administratively
> by far the safest. If you're running the embedded ZK, then the ZK
> instances are tied to your Solr instance. Now if, for any reason, your
> Solr nodes hosting ZK go down, you lose ZK quorum, can't index.
> etc
>
> Now consider a cluster with, say, 100 Solr nodes. Not talking replicas
> in a collection here, I'm talking 100 physical machines. BTW, this is
> not even close to the largest ones I'm aware of. Which three (for
> example) are running ZK? If I want to upgrade Solr I better make
> really sure not to upgrade to of the Solr instances running ZK at once
> if I want my cluster to keep going
>
> And, ZK is sensitive to system resources. So putting ZK on a Solr node
> then hosing, say, updates to my Solr cluster can cause ZK to be
> starved for resources.
>
> This is one of those deals where _functionally_, it's OK to run
> embedded ZK, but administratively it's suspect.
>
> Best,
> Erick
>
> On Tue, Apr 25, 2017 at 10:49 AM, Rick Leir <rl...@leirtech.com> wrote:
> > All,
> > I read somewhere that you should run your own ZK externally, and turn
> off SolrCloud. Comments please!
> > Rick
> >
> > On April 25, 2017 1:33:31 PM EDT, "Otis Gospodnetić" <
> otis.gospodne...@gmail.com> wrote:
> >>This is interesting - that ZK is seen as adding so much complexity that
> >>it
> >>turns people off!
> >>
> >>If you think about it, Elasticsearch users have no choice -- except
> >>their
> >>"ZK" is built-in, hidden, so one doesn't have to think about it, at
> >>least
> >>not initially.
> >>
> >>I think I saw mentions (maybe on user or dev MLs or JIRA) about
> >>potentially, in the future, there only being SolrCloud mode (and
> >>dropping
> >>SolrCloud name in favour of Solr).  If the above comment from Charlie
> >>about
> >>complexity is really true for Solr users, and if that's the reason why
> >>we
> >>see so few people running SolrCloud today, perhaps that's a good signal
> >>for
> >>Solr development/priorities in terms of ZK
> >>hiding/automating/embedding/something...
> >>
> >>Otis
> >>--
> >>Monitoring - Log Management - Alerting - Anomaly Detection
> >>Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> >>
> >>
> >>On Tue, Apr 25, 2017 at 4:50 AM, Charlie Hull <char...@flax.co.uk>
> >>wrote:
> >>
> >>> On 24/04/2017 15:58, Otis Gospodnetić wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I'm really really surprised here.  Back in 2013 we did a poll to see
> >>how
> >>>> people were running Master-Slave (4.x back then) and SolrCloud was a
> >>bit
> >>>> more popular than Master-Slave:
> >>>> https://sematext.com/blog/2013/02/25/poll-solr-cloud-or-not/
> >>>>
> >>>> Here is a fresh new poll with pretty much the same question - How do
> >>you
> >>>> run your Solr?
> >><https://twitter.com/sematext/status/854927627748036608> -
> >>>> and guess what?  SolrCloud is *not* at all a lot more prevalent than
> >>>> Master-Slave.
> >>>>
> >>>> We definitely see a lot more SolrCloud used by Sematext Solr
> >>>> consulting/support customers, so I'm a bit surprised by the results
> >

Re: Poll: Master-Slave or SolrCloud?

[Otis Gospodnetić]

This is interesting - that ZK is seen as adding so much complexity that it
turns people off!

If you think about it, Elasticsearch users have no choice -- except their
"ZK" is built-in, hidden, so one doesn't have to think about it, at least
not initially.

I think I saw mentions (maybe on user or dev MLs or JIRA) about
potentially, in the future, there only being SolrCloud mode (and dropping
SolrCloud name in favour of Solr).  If the above comment from Charlie about
complexity is really true for Solr users, and if that's the reason why we
see so few people running SolrCloud today, perhaps that's a good signal for
Solr development/priorities in terms of ZK
hiding/automating/embedding/something...

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Tue, Apr 25, 2017 at 4:50 AM, Charlie Hull <char...@flax.co.uk> wrote:

> On 24/04/2017 15:58, Otis Gospodnetić wrote:
>
>> Hi,
>>
>> I'm really really surprised here.  Back in 2013 we did a poll to see how
>> people were running Master-Slave (4.x back then) and SolrCloud was a bit
>> more popular than Master-Slave:
>> https://sematext.com/blog/2013/02/25/poll-solr-cloud-or-not/
>>
>> Here is a fresh new poll with pretty much the same question - How do you
>> run your Solr? <https://twitter.com/sematext/status/854927627748036608> -
>> and guess what?  SolrCloud is *not* at all a lot more prevalent than
>> Master-Slave.
>>
>> We definitely see a lot more SolrCloud used by Sematext Solr
>> consulting/support customers, so I'm a bit surprised by the results of
>> this
>> poll so far.
>>
>
> I'm not particularly surprised. We regularly see clients either with
> single nodes or elderly versions of Solr (or even Lucene). Zookeeper is
> still seen as a bit of a black art. Once you move from 'how do I run a
> search engine' to 'how do I manage a cluster of servers with scaling for
> performance/resilience/failover' you're looking at a completely new set
> of skills and challenges, which I think puts many people off.
>
> Charlie
>
>>
>> Is anyone else surprised by this?  See https://twitter.com/sematext/
>> status/854927627748036608
>>
>> Thanks,
>> Otis
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection
>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>>
>>
>> ---
>> This email has been checked for viruses by AVG.
>> http://www.avg.com
>>
>>
>
> --
> Charlie Hull
> Flax - Open Source Enterprise Search
>
> tel/fax: +44 (0)8700 118334
> mobile:  +44 (0)7767 825828
> web: www.flax.co.uk
>

Re: Poll: Master-Slave or SolrCloud?

[Otis Gospodnetić]

Hi,

I think it's roughly the same profile of people.  The poll from 2013 was on
Sematext blog and the new one is on Sematext Twitter account.  But it
doesn't really matter so much whether people are the same or not.  What
amazes me that in 2017 we don't see a lot more SolrCloud users!

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Mon, Apr 24, 2017 at 8:04 PM, Erick Erickson <erickerick...@gmail.com>
wrote:

> Yeah, this is kind of counter to my expectations too. I guess my
> question is whether the same people are responding to the new survey
> as the old one. "If it ain't broke" and all that.
>
> Erick
>
> On Mon, Apr 24, 2017 at 7:58 AM, Otis Gospodnetić
> <otis.gospodne...@gmail.com> wrote:
> > Hi,
> >
> > I'm really really surprised here.  Back in 2013 we did a poll to see how
> > people were running Master-Slave (4.x back then) and SolrCloud was a bit
> > more popular than Master-Slave:
> > https://sematext.com/blog/2013/02/25/poll-solr-cloud-or-not/
> >
> > Here is a fresh new poll with pretty much the same question - How do you
> > run your Solr? <https://twitter.com/sematext/status/854927627748036608>
> -
> > and guess what?  SolrCloud is *not* at all a lot more prevalent than
> > Master-Slave.
> >
> > We definitely see a lot more SolrCloud used by Sematext Solr
> > consulting/support customers, so I'm a bit surprised by the results of
> this
> > poll so far.
> >
> > Is anyone else surprised by this?  See https://twitter.com/sematext/
> > status/854927627748036608
> >
> > Thanks,
> > Otis
> > --
> > Monitoring - Log Management - Alerting - Anomaly Detection
> > Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>

Poll: Master-Slave or SolrCloud?

[Otis Gospodnetić]

Hi,

I'm really really surprised here.  Back in 2013 we did a poll to see how
people were running Master-Slave (4.x back then) and SolrCloud was a bit
more popular than Master-Slave:
https://sematext.com/blog/2013/02/25/poll-solr-cloud-or-not/

Here is a fresh new poll with pretty much the same question - How do you
run your Solr?  -
and guess what?  SolrCloud is *not* at all a lot more prevalent than
Master-Slave.

We definitely see a lot more SolrCloud used by Sematext Solr
consulting/support customers, so I'm a bit surprised by the results of this
poll so far.

Is anyone else surprised by this?  See https://twitter.com/sematext/
status/854927627748036608

Thanks,
Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/

Re: What is the bottleneck for an optimise operation?

[Otis Gospodnetić]

Hi,

It's simply expensive.  You are rewriting your whole index.

Why are you running optimize?  Are you seeing performance problems you are
trying to fix with optimize?

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Thu, Mar 2, 2017 at 10:39 AM, Caruana, Matthew  wrote:

> Thank you. The question remains however, if this is such a hefty operation
> then why is it walking to the destination instead of running, so to speak?
>
> Is the process throttled in some way?
>
> > On 2 Mar 2017, at 16:20, David Hastings 
> wrote:
> >
> > Agreed, and since it takes three times the space is part of the reason it
> > takes so long, so that 190gb index ends up writing another 380 gb until
> it
> > compresses down and deletes the two left over files.  its a pretty hefty
> > operation
> >
> > On Thu, Mar 2, 2017 at 10:13 AM, Alexandre Rafalovitch <
> arafa...@gmail.com>
> > wrote:
> >
> >> Optimize operation is no longer recommended for Solr, as the
> >> background merges got a lot smarter.
> >>
> >> It is an extremely expensive operation that can require up to 3-times
> >> amount of disk during the processing.
> >>
> >> This is not to say yours is a valid question, which I am leaving to
> >> others to respond.
> >>
> >> Regards,
> >>   Alex.
> >> 
> >> http://www.solr-start.com/ - Resources for Solr users, new and
> experienced
> >>
> >>
> >> On 2 March 2017 at 10:04, Caruana, Matthew  wrote:
> >>> I’m currently performing an optimise operation on a ~190GB index with
> >> about 4 million documents. The process has been running for hours.
> >>>
> >>> This is surprising, because the machine is an EC2 r4.xlarge with four
> >> cores and 30GB of RAM, 24GB of which is allocated to the JVM.
> >>>
> >>> The load average has been steady at about 1.3. Memory usage is 25% or
> >> less the whole time. iostat reports ~6% util.
> >>>
> >>> What gives?
> >>>
> >>> Running Solr 6.4.1.
> >>
>
>

Re: What is the bottleneck for an optimise operation?

[Otis Gospodnetić]

Hi Matthew,

I'm guessing it's the EBS.  With EBS we've seen:
* cpu.system going up in some kernels
* low read/write speeds and maxed out IO at times

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Thu, Mar 2, 2017 at 10:04 AM, Caruana, Matthew  wrote:

> I’m currently performing an optimise operation on a ~190GB index with
> about 4 million documents. The process has been running for hours.
>
> This is surprising, because the machine is an EC2 r4.xlarge with four
> cores and 30GB of RAM, 24GB of which is allocated to the JVM.
>
> The load average has been steady at about 1.3. Memory usage is 25% or less
> the whole time. iostat reports ~6% util.
>
> What gives?
>
> Running Solr 6.4.1.

Re: [ANN] InvisibleQueriesRequestHandler

[Otis Gospodnetić]

Nice.

Here is something similar: https://github.com/sematext/solr-researcher -
hope others find it useful, too.

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Mon, Dec 5, 2016 at 4:18 AM, Andrea Gazzarini  wrote:

> Hi guys,
> I developed this handler [1] while doing some work on a Magento ->  Solr
> project.
>
> If someone is interested (this is a post [2] where I briefly explain the
> goal), or wants to contribute with some idea / improvement, feel free to
> give me a shout or a feedback.
>
> Best,
> Andrea
>
> [1] https://github.com/agazzarini/invisible-queries-request-handler
> [2]
> https://andreagazzarini.blogspot.it/2016/12/composing-
> and-reusing-request-handlers.html
>

Re: Writing Solr Custom Components

[Otis Gospodnetić]

John, if it helps, here are a few examples of custom Solr SearchComponents:

https://github.com/sematext/query-segmenter
https://github.com/sematext/solr-researcher
https://github.com/sematext/solr-autocomplete

I hope this helps.

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Wed, Oct 5, 2016 at 9:34 AM, John Bickerstaff 
wrote:

> Thank you both!
>
> On Oct 5, 2016 2:32 AM, "Charlie Hull"  wrote:
>
> > On 04/10/2016 17:23, John Bickerstaff wrote:
> >
> >> All,
> >>
> >> I'm looking for information on writing custom Solr components.  A quick
> >> search showed nothing really recent and before I dig deeper, I thought
> I'd
> >> ask the community for anything you are aware of.
> >>
> >> Thanks
> >>
> >> We wrote a few for the BioSolr project: https://github.com/flaxsearch/
> > BioSolr - the ontology one might be useful
> https://github.com/flaxsearch/
> > BioSolr/tree/master/ontology Count yourself lucky, you could be doing it
> > for Elasticsearch :) http://www.flax.co.uk/blog/201
> > 6/01/27/fun-frustration-writing-plugin-elasticsearch-ontology-indexing/
> >
> > Charlie
> >
> > --
> > Charlie Hull
> > Flax - Open Source Enterprise Search
> >
> > tel/fax: +44 (0)8700 118334
> > mobile:  +44 (0)7767 825828
> > web: www.flax.co.uk
> >
>

Re: Reset JMX counters for monitoring without restarting

[Otis Gospodnetić]

Hi Jeff,

With info that Solr provides in JMX you have to keep track of things
yourself, do subtractions and counting yourself.
If you don't feel like reinventing that, see
https://sematext.com/spm/integrations/solr-monitoring/

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Sat, Apr 2, 2016 at 1:06 PM, Jeff Courtade 
wrote:

> Hi,
>
> I am putting together some montitors for various things.
>
> The counters seem to be  ... from the beginning of the process.
>
> This makes many of them not so useful for long term monitoring and
> alerting.
>
> Is there a way to rest the counters without restarting solr or reloading a
> core?
>
> For instance: these seem to be from the time the process started.
>
> java -jar /opt/scripts/pkg/cmdline-jmxclient-0.10.3.jar - localhost:9010
>
> solr/collection1:id=org.apache.solr.handler.component.SearchHandler,type=/select
> $NAME
>
> avgTimePerRequest.value  363.66010984870064
> medianRequestTime.value  1240.416114498
> 75thPcRequestTime.value  1614.2324915
> 95thPcRequestTime.value  3048.37888109
> 99thPcRequestTime.value  5930.183086690001
>
>
>
> --
> Thanks,
>
> Jeff Courtade
> M: 240.507.6116
>

Re: High Cpu sys usage

[Otis Gospodnetić]

Hi,

On Wed, Mar 16, 2016 at 10:59 AM, Patrick Plaatje 
wrote:

> Hi,
>
> From the sar output you supplied, it looks like you might have a memory
> issue on your hosts. The memory usage just before your crash seems to be
> *very* close to 100%. Even the slightest increase (Solr itself, or possibly
> by a system service) could caused the system crash. What are the
> specifications of your hosts and how much memory are you allocating?


That's normal actually - http://www.linuxatemyram.com/

You *want* Linux to be using all your memory - you paid for it :)

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/




>


>
>
> On 16/03/2016, 14:52, "YouPeng Yang"  wrote:
>
> >Hi
> > It happened again,and worse thing is that my system went to crash.we can
> >even not connect to it with ssh.
> > I use the sar command to capture the statistics information about it.Here
> >are my details:
> >
> >
> >[1]cpu(by using sar -u),we have to restart our system just as the red font
> >LINUX RESTART in the logs.
>
> >--
> >03:00:01 PM all  7.61  0.00  0.92  0.07  0.00
> >91.40
> >03:10:01 PM all  7.71  0.00  1.29  0.06  0.00
> >90.94
> >03:20:01 PM all  7.62  0.00  1.98  0.06  0.00
> >90.34
> >03:30:35 PM all  5.65  0.00 31.08  0.04  0.00
> >63.23
> >03:42:40 PM all 47.58  0.00 52.25  0.00  0.00
> > 0.16
> >Average:all  8.21  0.00  1.57  0.05  0.00
> >90.17
> >
> >04:42:04 PM   LINUX RESTART
> >
> >04:50:01 PM CPU %user %nice   %system   %iowait%steal
> >%idle
> >05:00:01 PM all  3.49  0.00  0.62  0.15  0.00
> >95.75
> >05:10:01 PM all  9.03  0.00  0.92  0.28  0.00
> >89.77
> >05:20:01 PM all  7.06  0.00  0.78  0.05  0.00
> >92.11
> >05:30:01 PM all  6.67  0.00  0.79  0.06  0.00
> >92.48
> >05:40:01 PM all  6.26  0.00  0.76  0.05  0.00
> >92.93
> >05:50:01 PM all  5.49  0.00  0.71  0.05  0.00
> >93.75
>
> >--
> >
> >[2]mem(by using sar -r)
>
> >--
> >03:00:01 PM   1519272 196633272 99.23361112  76364340 143574212
> >47.77
> >03:10:01 PM   1451764 196700780 99.27361196  76336340 143581608
> >47.77
> >03:20:01 PM   1453400 196699144 99.27361448  76248584 143551128
> >47.76
> >03:30:35 PM   1513844 196638700 99.24361648  76022016 143828244
> >47.85
> >03:42:40 PM   1481108 196671436 99.25361676  75718320 144478784
> >48.07
> >Average:  5051607 193100937 97.45362421  81775777 142758861
> >47.50
> >
> >04:42:04 PM   LINUX RESTART
> >
> >04:50:01 PM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit
> >%commit
> >05:00:01 PM 154357132  43795412 22.10 92012  18648644 134950460
> >44.90
> >05:10:01 PM 136468244  61684300 31.13219572  31709216 134966548
> >44.91
> >05:20:01 PM 135092452  63060092 31.82221488  32162324 134949788
> >44.90
> >05:30:01 PM 133410464  64742080 32.67233848  32793848 134976828
> >44.91
> >05:40:01 PM 132022052  66130492 33.37235812  33278908 135007268
> >44.92
> >05:50:01 PM 130630408  67522136 34.08237140  33900912 135099764
> >44.95
> >Average:136996792  61155752 30.86206645  30415642 134991776
> >44.91
>
> >--
> >
> >
> >As the blue font parts show that my hardware crash from 03:30:35.It is
> hung
> >up until I restart it manually at 04:42:04
> >ALl the above information just snapshot the performance when it crashed
> >while there is nothing cover the reason.I have also
> >check the /var/log/messages and find nothing useful.
> >
> >Note that I run the command- sar -v .It shows something abnormal:
>
> >
> >02:50:01 PM  11542262  9216 76446   258
> >03:00:01 PM  11645526  9536 76421   258
> >03:10:01 PM  11748690  9216 76451   258
> >03:20:01 PM  11850191  9152 76331   258
> >03:30:35 PM  11972313 10112132625   258
> >03:42:40 PM  12177319 13760340227   258
> >Average:  8293601  8950 68187   161
> >
> >04:42:04 PM   LINUX RESTART
> >
> >04:50:01 PM dentunusd   file-nr  inode-nrpty-nr
> >05:00:01 PM 35410  7616 35223 4
> >05:10:01 PM137320  7296 42632   

Re: High Cpu sys usage

[Otis Gospodnetić]

Hi,

I looked at those metrics outputs, but nothing jumps out at me as
problematic.

How full are your JVM heap memory pools?  If you are using SPM to monitor
your Solr/Tomcat/Jetty/... look for a chart that looks like this:
https://apps.sematext.com/spm-reports/s/zB3JcdZyRn

If some of these lines are close to 100% and stay close or at 100%, that's
typically a bad sign.
Next, look at your Garbage Collection times and counts.  If you look at
your GC metrics for e.g. a month and see a recent increase in GC times or
counts then, yes, you have an issue with your memory/heap and that is what
is increasing your CPU usage.

If it looks like heap/GC are not the issue and it's really something inside
Solr, you could profile it with either one of the standard profilers or
something like
https://sematext.com/blog/2016/03/17/on-demand-java-profiling/ .  If there
is something in Solr chewing on the CPU, this should show it.

I hope this helps.

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Wed, Mar 16, 2016 at 10:52 AM, YouPeng Yang 
wrote:

> Hi
>  It happened again,and worse thing is that my system went to crash.we can
> even not connect to it with ssh.
>  I use the sar command to capture the statistics information about it.Here
> are my details:
>
>
> [1]cpu(by using sar -u),we have to restart our system just as the red font
> LINUX RESTART in the logs.
>
> --
> 03:00:01 PM all  7.61  0.00  0.92  0.07  0.00
> 91.40
> 03:10:01 PM all  7.71  0.00  1.29  0.06  0.00
> 90.94
> 03:20:01 PM all  7.62  0.00  1.98  0.06  0.00
> 90.34
> 03:30:35 PM all  5.65  0.00 31.08  0.04  0.00
> 63.23
> 03:42:40 PM all 47.58  0.00 52.25  0.00  0.00
>  0.16
> Average:all  8.21  0.00  1.57  0.05  0.00
> 90.17
>
> 04:42:04 PM   LINUX RESTART
>
> 04:50:01 PM CPU %user %nice   %system   %iowait%steal
> %idle
> 05:00:01 PM all  3.49  0.00  0.62  0.15  0.00
> 95.75
> 05:10:01 PM all  9.03  0.00  0.92  0.28  0.00
> 89.77
> 05:20:01 PM all  7.06  0.00  0.78  0.05  0.00
> 92.11
> 05:30:01 PM all  6.67  0.00  0.79  0.06  0.00
> 92.48
> 05:40:01 PM all  6.26  0.00  0.76  0.05  0.00
> 92.93
> 05:50:01 PM all  5.49  0.00  0.71  0.05  0.00
> 93.75
>
> --
>
> [2]mem(by using sar -r)
>
> --
> 03:00:01 PM   1519272 196633272 99.23361112  76364340 143574212
> 47.77
> 03:10:01 PM   1451764 196700780 99.27361196  76336340 143581608
> 47.77
> 03:20:01 PM   1453400 196699144 99.27361448  76248584 143551128
> 47.76
> 03:30:35 PM   1513844 196638700 99.24361648  76022016 143828244
> 47.85
> 03:42:40 PM   1481108 196671436 99.25361676  75718320 144478784
> 48.07
> Average:  5051607 193100937 97.45362421  81775777 142758861
> 47.50
>
> 04:42:04 PM   LINUX RESTART
>
> 04:50:01 PM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit
> %commit
> 05:00:01 PM 154357132  43795412 22.10 92012  18648644 134950460
> 44.90
> 05:10:01 PM 136468244  61684300 31.13219572  31709216 134966548
> 44.91
> 05:20:01 PM 135092452  63060092 31.82221488  32162324 134949788
> 44.90
> 05:30:01 PM 133410464  64742080 32.67233848  32793848 134976828
> 44.91
> 05:40:01 PM 132022052  66130492 33.37235812  33278908 135007268
> 44.92
> 05:50:01 PM 130630408  67522136 34.08237140  33900912 135099764
> 44.95
> Average:136996792  61155752 30.86206645  30415642 134991776
> 44.91
>
> --
>
>
> As the blue font parts show that my hardware crash from 03:30:35.It is hung
> up until I restart it manually at 04:42:04
> ALl the above information just snapshot the performance when it crashed
> while there is nothing cover the reason.I have also
> check the /var/log/messages and find nothing useful.
>
> Note that I run the command- sar -v .It shows something abnormal:
>
> 
> 02:50:01 PM  11542262  9216 76446   258
> 03:00:01 PM  11645526  9536 76421   258
> 03:10:01 PM  11748690  9216 76451   258
> 03:20:01 PM  11850191  9152 76331   258
> 03:30:35 PM  11972313 10112132625   258
> 03:42:40 PM  12177319 13760340227   

Re: Solr memory usage

[Otis Gospodnetić]

Hi Steve,

Fluctuation is OK.  100% utilization for more than a moment is not :)

Not sure what tool(s) you use for monitoring your Solr servers, but look
under "JVM Pool Utilization" in SPM if you're using SPM.
Or this live demo of a Solr system:
* click on https://apps.sematext.com/demo to get into the demo account
* look at "JVM Pool Utilization" on
https://apps.sematext.com/spm-reports/mainPage.do?selectedApplication=1704=poolReportPage=1449865787801=false

And on that JVM Pool Size chart on top of the page you will see giant saw
pattern which is a healthy sign :)

HTH
Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Wed, Dec 9, 2015 at 9:56 AM, Steven White  wrote:

> Thanks Erick!!  Your summary and the blog by Uwe (thank you too Uwe) are
> very helpful.
>
> A follow up question.  I also noticed the "JVM-Memory" report off Solr's
> home page is fluctuating.  I expect some fluctuation, but it kinda worries
> me when it fluctuates up / down in a range of 4 GB and maybe more.  I.e.:
> at times it is at 5 GB and other times it is at 10 GB (this is while I'm
> running my search tests).  What does such high fluctuation means?
>
> If it helps, Solr's "JVM-Memory" report states 2.5 GB usage when Solr is
> first started and before I run any search on it.  I'm taking this as my
> base startup memory usage.
>
> Steve
>
> On Tue, Dec 8, 2015 at 3:17 PM, Erick Erickson 
> wrote:
>
> > You're doing nothing wrong, that particular bit of advice has
> > always needed a bit of explanation.
> >
> > Solr (well, actually Lucene) uses MMapDirectory for much of
> > the index structure which uses the OS memory rather than
> > the JVM heap. See Uwe's excellent:
> >
> > http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
> >
> > Plus, the size on disk includes the stored data, which is in the *.fdt
> > files in data/index. Very little of the stored data is kept in the JVM
> > so that's another reason your Java heap may be smaller than
> > your raw index size on disk.
> >
> > The advice about fitting your entire index into memory really has
> > the following caveats (at least).
> > 1> "memory" includes the OS memory available to the process
> > 2> The size of the index on disk is misleading, the *.fdt files
> >  should be subtracted in order to get a truer picture.
> > 3> Both Solr and Lucene create structures in the Java JVM
> >  that are _not_ reflected in the size on disk.
> >
> > <1> and <2> mean the JVM memory necessary is smaller
> > than the size on disk.
> >
> > <3> means the JVM memory will be larger than.
> >
> > So you're doing the right thing, testing and seeing what you
> > _really_ need. I'd pretty much take your test, add some
> > padding and consider it good. You're _not_ doing the
> > really bad thing of using the same query over and over
> > again and hoping .
> >
> > Best,
> > Erick
> >
> >
> > On Tue, Dec 8, 2015 at 11:54 AM, Steven White 
> > wrote:
> > > Hi folks,
> > >
> > > My index size on disk (optimized) is 20 GB (single core, single index).
> > I
> > > have a system with 64 GB of RAM.  I start Solr with 24 GB of RAM.
> > >
> > > I have run load tests (up to 100 concurrent users) for hours where each
> > > user issuing unique searches (the same search is never executed again
> for
> > > at least 30 minute since it was last executed).  In all tests I run,
> > Solr's
> > > JVM memory never goes over 10 GB (monitoring http://localhost:8983/).
> > >
> > > I read over and over, for optimal performance, Solr should be given
> > enough
> > > RAM to hold the index in memory.  Well, I have done that and some but
> > yet I
> > > don't see Solr using up that whole RAM.  What am I doing wrong?  Is my
> > test
> > > at fault?  I doubled the test load (number of users) and didn't see
> much
> > of
> > > a difference with RAM usage but yet my search performance went down
> > (takes
> > > about 40% longer now).  I run my tests again but this time with only 12
> > GB
> > > of RAM given to Solr.  Test result didn't differ much from the 24 GB
> run
> > > and Solr never used more than 10 GB of RAM.
> > >
> > > Can someone help me understand this?  I don't want to give Solr RAM
> that
> > it
> > > won't use.
> > >
> > > PS: This is simply search tests, there is no update to the index at
> all.
> > >
> > > Thanks in advanced.
> > >
> > > Steve
> >
>

Re: Solr Log Analysis

[Otis Gospodnetić]

Hi Magesh,

Here are 2 more solutions you could use:

1) Site Search Analytics  -- this
basically integrates into your search results via JS like Google Analytics
and automatically captures a bunch of search and click data and gives you a
number of search-focused reports/charts (at aggregate level) out of the box.
2) Logsene  - this will take any events and
let you search then, filter them, graph them, aggregate on them, alert on
them, put them on custom dashboards, correlate with app/server logs, with
Solr metrics (via SPM for Solr
), etc. etc.  If
you happen to like/know Kibana or Banana or Spunk this will look very
familiar.  As a matter of fact, Kibana 4 is integrated into Logsene.

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Wed, Sep 23, 2015 at 8:16 PM, Tarala, Magesh  wrote:

> I'm using Solr 4.10.4 in a 3 node cloud setup. I have 3 shards and 3
> replicas for the collection.
>
> I want to analyze the logs to extract the queries and query times. Is
> there a tool or script someone has created already for this?
>
> Thanks,
> Magesh
>

Re: Solr health check monitor for multiple cores

[Otis Gospodnetić]

Hi Daniel,

Lots of organizations use our SPM to monitor Solr / SolrCloud (see
http://sematext.com/spm/integrations/solr-monitoring.html ) .  We're
working on "Top Queries" type of reporting for Solr, which I'm very excited
about. :)

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Fri, Aug 21, 2015 at 5:28 PM, Davis, Daniel (NIH/NLM) [C] <
daniel.da...@nih.gov> wrote:

> I'm wondering what different folks do out there for a health monitor for
> Solr.   I'm running Solr 5.2.1, so far without Solr Cloud, and I anticipate
> having multiple cores.
>
> For now, I can make use solr/corename/admin/ping, but how can I have Solr
> ping all cores?
>
> Dan Davis, Systems/Applications Architect (Contractor),
> Office of Computer and Communications Systems,
> National Library of Medicine, NIH
>
>

Re: solr training

[Otis Gospodnetić]

Hi Tim,

A slightly delayed reply ;)
We are running Solr training in NYC next month -
http://sematext.com/training/solr-training.html - 2nd seat is 50% off.

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Fri, May 1, 2015 at 2:18 PM, Tim Dunphy  wrote:

> Hey guys,
>
>  My company has a training budget that it wants me to use. So what I'd like
> to find out is if there is any instructor lead courses in the NY/NJ area,
> or courses online that are instructor lead that you could recommend?
>
> Thanks,
> Tim
>
> --
> GPG me!!
>
> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
>