Re: Indexed Data Size

[Greg Harris]

Brett, it’s probably because you hit the 5g default segment size limit on
Solr and in order to merge segments a huge number of the docs within the
segment must be marked as deleted. So even if large amounts of docs are
deleted docs within the segment, the segment is still there, happily taking
up space. That could theoretically be a reason for a optimize, but you’d
want to specify maxsegments with the goal of not merging to a single
segment for the entire index. Ideally you should just keep as many of the
logs as you actually use (which is hopefully more limited than what you are
keeping). Since the segments will be somewhat time based they would
eventually disappear/merge through time, hopefully negating any reason to
consider having to optimize

Greg

On Tue, Aug 13, 2019 at 3:31 PM Moyer, Brett  wrote:

> Turns out this is due to a job that indexes logs. We were able to clear
> some with another job. We are working through the value of these indexed
> logs. Thanks for all your help!
>
> Brett Moyer
> Manager, Sr. Technical Lead | TFS Technology
>   Public Production Support
>   Digital Search & Discovery
>
> 8625 Andrew Carnegie Blvd | 4th floor
> Charlotte, NC 28263
> Tel: 704.988.4508
> Fax: 704.988.4907
> bmo...@tiaa.org
>
> -Original Message-
> From: Shawn Heisey 
> Sent: Friday, August 9, 2019 2:25 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Indexed Data Size
>
> On 8/9/2019 12:17 PM, Moyer, Brett wrote:
> > The biggest is /data/solr/system_logs_shard1_replica_n1/data/index,
> files with the extensions I stated previously. Each is 5gb and there are a
> few hundred. Dated by to last 3 months. I don’t understand why there are so
> many files with such small indexes. Not sure how to clean them up.
>
> Can you get a screenshot of the core overview for that particular core?
> Solr should correctly calculate the size on the overview based on what
> files are actually in the index directory.
>
> Thanks,
> Shawn
> *
> This e-mail may contain confidential or privileged information.
> If you are not the intended recipient, please notify the sender
> immediately and then delete it.
>
> TIAA
> *
>

Re: Solr Heap Usage

[Greg Harris]

+1 for eclipse mat. Yourkit is another option. Heap dumps are invaluable
but a pain. If you’re just interested in overall heap and gc analysis I use
gc-viewer, which is usually all you need to know. I do heap dumps when
there are for large deviations from expectations and it is non obvious why

Greg

On Fri, Jun 7, 2019 at 11:30 AM John Davis 
wrote:

> What would be the best way to understand where heap is being used?
>
> On Tue, Jun 4, 2019 at 9:31 PM Greg Harris  wrote:
>
> > Just a couple of points I’d make here. I did some testing a while back in
> > which if no commit is made, (hard or soft) there are internal memory
> > structures holding tlogs and it will continue to get worse the more docs
> > that come in. I don’t know if that’s changed in further versions. I’d
> > recommend doing commits with some amount of frequency in indexing heavy
> > apps, otherwise you are likely to have heap issues. I personally would
> > advocate for some of the points already made. There are too many
> variables
> > going on here and ways to modify stuff to make sizing decisions and think
> > you’re doing anything other than a pure guess if you don’t test and
> > monitor. I’d advocate for a process in which testing is done regularly to
> > figure out questions like number of shards/replicas, heap size, memory
> etc.
> > Hard data, good process and regular testing will trump guesswork every
> time
> >
> > Greg
> >
> > On Tue, Jun 4, 2019 at 9:22 AM John Davis 
> > wrote:
> >
> > > You might want to test with softcommit of hours vs 5m for heavy
> indexing
> > +
> > > light query -- even though there is internal memory structure overhead
> > for
> > > no soft commits, in our testing a 5m soft commit (via commitWithin) has
> > > resulted in a very very large heap usage which I suspect is because of
> > > other overhead associated with it.
> > >
> > > On Tue, Jun 4, 2019 at 8:03 AM Erick Erickson  >
> > > wrote:
> > >
> > > > I need to update that, didn’t understand the bits about retaining
> > > internal
> > > > memory structures at the time.
> > > >
> > > > > On Jun 4, 2019, at 2:10 AM, John Davis 
> > > > wrote:
> > > > >
> > > > > Erick - These conflict, what's changed?
> > > > >
> > > > > So if I were going to recommend settings, they’d be something like
> > > this:
> > > > > Do a hard commit with openSearcher=false every 60 seconds.
> > > > > Do a soft commit every 5 minutes.
> > > > >
> > > > > vs
> > > > >
> > > > > Index-heavy, Query-light
> > > > > Set your soft commit interval quite long, up to the maximum latency
> > you
> > > > can
> > > > > stand for documents to be visible. This could be just a couple of
> > > minutes
> > > > > or much longer. Maybe even hours with the capability of issuing a
> > hard
> > > > > commit (openSearcher=true) or soft commit on demand.
> > > > >
> > > >
> > >
> >
> https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Sun, Jun 2, 2019 at 8:58 PM Erick Erickson <
> > erickerick...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > >>> I've looked through SolrJ, DIH and others -- is the bottomline
> > > > >>> across all of them to "batch updates" and not commit as long as
> > > > possible?
> > > > >>
> > > > >> Of course it’s more complicated than that ;)….
> > > > >>
> > > > >> But to start, yes, I urge you to batch. Here’s some stats:
> > > > >> https://lucidworks.com/2015/10/05/really-batch-updates-solr-2/
> > > > >>
> > > > >> Note that at about 100 docs/batch you hit diminishing returns.
> > > > _However_,
> > > > >> that test was run on a single shard collection, so if you have 10
> > > shards
> > > > >> you’d
> > > > >> have to send 1,000 docs/batch. I wouldn’t sweat that number much,
> > just
> > > > >> don’t
> > > > >> send one at a time. And there are the usual gotchas if your
> > documents
> > > > are
> > > > >> 1M .vs. 1K

Re: Solr Heap Usage

[Greg Harris]

Just a couple of points I’d make here. I did some testing a while back in
which if no commit is made, (hard or soft) there are internal memory
structures holding tlogs and it will continue to get worse the more docs
that come in. I don’t know if that’s changed in further versions. I’d
recommend doing commits with some amount of frequency in indexing heavy
apps, otherwise you are likely to have heap issues. I personally would
advocate for some of the points already made. There are too many variables
going on here and ways to modify stuff to make sizing decisions and think
you’re doing anything other than a pure guess if you don’t test and
monitor. I’d advocate for a process in which testing is done regularly to
figure out questions like number of shards/replicas, heap size, memory etc.
Hard data, good process and regular testing will trump guesswork every time

Greg

On Tue, Jun 4, 2019 at 9:22 AM John Davis  wrote:

> You might want to test with softcommit of hours vs 5m for heavy indexing +
> light query -- even though there is internal memory structure overhead for
> no soft commits, in our testing a 5m soft commit (via commitWithin) has
> resulted in a very very large heap usage which I suspect is because of
> other overhead associated with it.
>
> On Tue, Jun 4, 2019 at 8:03 AM Erick Erickson 
> wrote:
>
> > I need to update that, didn’t understand the bits about retaining
> internal
> > memory structures at the time.
> >
> > > On Jun 4, 2019, at 2:10 AM, John Davis 
> > wrote:
> > >
> > > Erick - These conflict, what's changed?
> > >
> > > So if I were going to recommend settings, they’d be something like
> this:
> > > Do a hard commit with openSearcher=false every 60 seconds.
> > > Do a soft commit every 5 minutes.
> > >
> > > vs
> > >
> > > Index-heavy, Query-light
> > > Set your soft commit interval quite long, up to the maximum latency you
> > can
> > > stand for documents to be visible. This could be just a couple of
> minutes
> > > or much longer. Maybe even hours with the capability of issuing a hard
> > > commit (openSearcher=true) or soft commit on demand.
> > >
> >
> https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
> > >
> > >
> > >
> > >
> > > On Sun, Jun 2, 2019 at 8:58 PM Erick Erickson  >
> > > wrote:
> > >
> > >>> I've looked through SolrJ, DIH and others -- is the bottomline
> > >>> across all of them to "batch updates" and not commit as long as
> > possible?
> > >>
> > >> Of course it’s more complicated than that ;)….
> > >>
> > >> But to start, yes, I urge you to batch. Here’s some stats:
> > >> https://lucidworks.com/2015/10/05/really-batch-updates-solr-2/
> > >>
> > >> Note that at about 100 docs/batch you hit diminishing returns.
> > _However_,
> > >> that test was run on a single shard collection, so if you have 10
> shards
> > >> you’d
> > >> have to send 1,000 docs/batch. I wouldn’t sweat that number much, just
> > >> don’t
> > >> send one at a time. And there are the usual gotchas if your documents
> > are
> > >> 1M .vs. 1K.
> > >>
> > >> About committing. No, don’t hold off as long as possible. When you
> > commit,
> > >> segments are merged. _However_, the default 100M internal buffer size
> > means
> > >> that segments are written anyway even if you don’t hit a commit point
> > when
> > >> you have 100M of index data, and merges happen anyway. So you won’t
> save
> > >> anything on merging by holding off commits.
> > >> And you’ll incur penalties. Here’s more than you want to know about
> > >> commits:
> > >>
> > >>
> >
> https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
> > >>
> > >> But some key take-aways… If for some reason Solr abnormally
> > >> terminates, the accumulated documents since the last hard
> > >> commit are replayed. So say you don’t commit for an hour of
> > >> furious indexing and someone does a “kill -9”. When you restart
> > >> Solr it’ll try to re-index all the docs for the last hour. Hard
> commits
> > >> with openSearcher=false aren’t all that expensive. I usually set mine
> > >> for a minute and forget about it.
> > >>
> > >> Transaction logs hold a window, _not_ the entire set of operations
> > >> since time began. When you do a hard commit, the current tlog is
> > >> closed and a new one opened and ones that are “too old” are deleted.
> If
> > >> you never commit you have a huge transaction log to no good purpose.
> > >>
> > >> Also, while indexing, in order to accommodate “Real Time Get”, all
> > >> the docs indexed since the last searcher was opened have a pointer
> > >> kept in memory. So if you _never_ open a new searcher, that internal
> > >> structure can get quite large. So in bulk-indexing operations, I
> > >> suggest you open a searcher every so often.
> > >>
> > >> Opening a new searcher isn’t terribly expensive if you have no
> > autowarming
> > >> going on. Autowarming as defined in solrconfig.xml in filterCache,
> > >> 

Re: Latest advice on G1 collector?

[Greg Harris]

Just to chime in a few quick thoughts

I think my experience to this point is G1 (barring unknown lucene bug risk)
is actually a lower risk easier collector to use. However that doesn't
necessarily mean better. You don't have set the space sizes or any number
of all sorts of various parameters you seem to have to set with cms. It can
control pause variability much more than cms does. CMS also has the dubious
distinction of working well when things are fine and being a single
threaded full GC disaster on failures. Some of the settings that Solr uses
are really more matters of opinion than one size fits all. A 50 percent
initiating ratio can consume 4 CPUs permanently at the default settings if
you haven't set your size correctly. 90 percent target survivor ratio can
actually cause very long minor gcs if the survivor space fills creating
much more variable behavior than most people realize although for the most
part don't notice. And CMS goes on and on with all these settings that
really requires more thorough analysis and learning about each setting to
an almost absurd level. G1 has a very small number of easily understandable
settings to it that controls pauses and variability really well. It does
come at a risk of throughput, but for Solr pause goals are far more
important to me than throughout. All that said, I've still typically used
and seen CMS in most circumstances because I have way more experience with
it. And I think a well functioning CMS is more likely to have lower pauses
and better throughput. Its just riskier that it might work much worse. I
think I also don't feel like I know all the warts of G1 yet, so that has
also kept me reticent to use it more.

On Jan 24, 2017 5:46 PM, "Shawn Heisey"  wrote:

> On 1/23/2017 1:00 PM, Walter Underwood wrote:
> > We have a workload with very long queries, and that can drive the CMS
> collector into using about 20% of the CPU time. So I’m ready to try G1 on a
> couple of replicas and see what happens. I’ve already upgraded to Java 8
> update 121.
> >
> > I’ve read these pages:
> >
> > https://wiki.apache.org/solr/ShawnHeisey#G1_.28Garbage_
> First.29_Collector  ShawnHeisey#G1_.28Garbage_First.29_Collector>
> > https://gist.github.com/rockagen/e6d28244e1d540c05144370d6a64ba66 <
> https://gist.github.com/rockagen/e6d28244e1d540c05144370d6a64ba66>
>
> I would really like a concrete reason as to why the Lucene wiki
> recommends NEVER using G1.
>
> https://wiki.apache.org/lucene-java/JavaBugs#Oracle_
> Java_.2F_Sun_Java_.2F_OpenJDK_Bugs
>
> If that recommendation can be backed up with demonstrable problems, then
> it would make sense.  I took a look through some of the email history
> sent by Jenkins, which runs automated testing of Lucene and Solr using
> various configurations and Java versions.  Problems that were detected
> on tests run with the G1 collector *also* happen on test runs using
> other collectors.  The number of new messages from tests using G1 are a
> very minor portion of the total number of new messages.  If G1 were a
> root cause of big problems, I would expect the number of new failures
> using G1 to be somewhere near half of the total, possibly more.
>
> As many of you know, Solr's essential functionality comes from Lucene,
> so this does matter for Solr.
>
> I myself have never had an issue running Solr with the G1 collector.  I
> haven't found any open and substantiated bugs on Lucene or Solr that
> document real problems with G1 on a 64-bit JVM.  There is one bug that
> happens on a 32-bit JVM ... but most users are NOT limited to 32-bit.
> For those that are limited that way, CMS is probably plenty fast because
> the heap can't go beyond 2GB.
>
> For my production and dev systems, the 4.x versions are running the G1
> collector.  Most of the 5.x and later installs are using the GC tuning
> that Solr contains by default, which is CMS.
>
> Thanks,
> Shawn
>
>

Re: Cannot talk to ZooKeeper - Updates are disabled (Solr 6.3.0)

[Greg Harris]

You can lose access to zk from either the solr side or the zk side. You
need to determine which is which. No hard and fast rules. If you're
restarting solr and everything comes back online, my bet is zk is fine,
which in the grand scheme of things is usually but not always the case

On Dec 29, 2016 9:50 AM, "Yago Riveiro"  wrote:

If I lost quorum on Zookeeper, this is a “fault” in the Zookeeper cluster,
therefore I should see something in the logs right?

The question here is, why I need to restart the node again?, if Zookeeper
recover its quorum, the Solr node should be in read-write mode again …

Any ideas how can test if I’m lost the Zookeeper quorum?

--

/Yago Riveiro

On 29 Dec 2016 16:07 +, Susheel Kumar , wrote:
> I believe this comes when Zookeeper quorum is not maintained. Do not see
> any way around except bringing the quorum back?
>
> Thanks,
> Susheel
>
> On Thu, Dec 29, 2016 at 9:27 AM, Yago Riveiro  wrote:
>
> > There is any way to recover from a exception
> > "org.apache.solr.common.SolrException: Cannot talk to ZooKeeper -
Updates
> > are disabled" without restart the affected node node?
> >
> > Regards,
> > /Yago
> >
> >
> >
> > -
> > Best regards
> >
> > /Yago
> > --
> > View this message in context: http://lucene.472066.n3.
> > nabble.com/Cannot-talk-to-ZooKeeper-Updates-are-
> > disabled-Solr-6-3-0-tp4311582.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >

Re: Very long young generation stop the world GC pause

[Greg Harris]

Your gun (not quite smoking yet, we still need the fingerprints) is this:

[Times: user=0.00 sys=94.28, real=97.19 secs]

Normal GC pauses are generally almost entirely user CPU, very short and
multiprocessor. Something else is sometimes happening with either the JVM
or OS which is causing this process to be single threaded, blocked and
being executed by system CPU -- ie: lower level processing unrelated to a
normal GC of Solr. This is non normal GC behavior and probably not going to
be fixed by any JVM parameter changes (but you can always try). Any unusual
disk or network IO? You need to understand what else is going on with your
system at the same moment in time this happens. You might see if you can do
an strace to see what its trying to do.

On Thu, Dec 8, 2016 at 12:06 AM, forest_soup  wrote:

> As you can see in the gc log, the long GC pause is not a full GC. It's a
> young generation GC instead.
> In our case, full gc is fast and young gc got some long stw pause.
> Do you have any comments on that, as we usually believe full gc may cause
> longer pause, but young generation should be ok?
>
> 2016-11-22T20:43:16.463+: 2942054.509: Total time for which application
> threads were stopped: 0.0029195 seconds, Stopping threads took: 0.804
> seconds
> {Heap before GC invocations=2246 (full 0):
>  garbage-first heap   total 26673152K, used 4683965K [0x7f0c1000,
> 0x7f0c108065c0, 0x7f141000)
>   region size 8192K, 162 young (1327104K), 17 survivors (139264K)
>  Metaspace   used 56487K, capacity 57092K, committed 58368K, reserved
> 59392K
> 2016-11-22T20:43:16.555+: 2942054.602: [GC pause (G1 Evacuation Pause)
> (young)
> Desired survivor size 88080384 bytes, new threshold 15 (max 15)
> - age   1:   28176280 bytes,   28176280 total
> - age   2:5632480 bytes,   33808760 total
> - age   3:9719072 bytes,   43527832 total
> - age   4:6219408 bytes,   49747240 total
> - age   5:4465544 bytes,   54212784 total
> - age   6:3417168 bytes,   57629952 total
> - age   7:5343072 bytes,   62973024 total
> - age   8:2784808 bytes,   65757832 total
> - age   9:6538056 bytes,   72295888 total
> - age  10:6368016 bytes,   78663904 total
> - age  11: 695216 bytes,   79359120 total
> , 97.2044320 secs]
>[Parallel Time: 19.8 ms, GC Workers: 18]
>   [GC Worker Start (ms): Min: 2942054602.1, Avg: 2942054604.6, Max:
> 2942054612.7, Diff: 10.6]
>   [Ext Root Scanning (ms): Min: 0.0, Avg: 2.4, Max: 6.7, Diff: 6.7,
> Sum:
> 43.5]
>   [Update RS (ms): Min: 0.0, Avg: 3.0, Max: 15.9, Diff: 15.9, Sum:
> 54.0]
>  [Processed Buffers: Min: 0, Avg: 10.7, Max: 39, Diff: 39, Sum:
> 192]
>   [Scan RS (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.6]
>   [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.0, Diff: 0.0,
> Sum: 0.0]
>   [Object Copy (ms): Min: 0.1, Avg: 9.2, Max: 13.4, Diff: 13.3, Sum:
> 165.9]
>   [Termination (ms): Min: 0.0, Avg: 2.5, Max: 2.7, Diff: 2.7, Sum:
> 44.1]
>  [Termination Attempts: Min: 1, Avg: 1.5, Max: 3, Diff: 2, Sum: 27]
>   [GC Worker Other (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.0, Sum:
> 0.6]
>   [GC Worker Total (ms): Min: 9.0, Avg: 17.1, Max: 19.7, Diff: 10.6,
> Sum: 308.7]
>   [GC Worker End (ms): Min: 2942054621.8, Avg: 2942054621.8, Max:
> 2942054621.8, Diff: 0.0]
>[Code Root Fixup: 0.1 ms]
>[Code Root Purge: 0.0 ms]
>[Clear CT: 0.2 ms]
>[Other: 97184.3 ms]
>   [Choose CSet: 0.0 ms]
>   [Ref Proc: 8.5 ms]
>   [Ref Enq: 0.2 ms]
>   [Redirty Cards: 0.2 ms]
>   [Humongous Register: 0.1 ms]
>   [Humongous Reclaim: 0.1 ms]
>   [Free CSet: 0.4 ms]
>[Eden: 1160.0M(1160.0M)->0.0B(1200.0M) Survivors: 136.0M->168.0M Heap:
> 4574.2M(25.4G)->3450.8M(26.8G)]
> Heap after GC invocations=2247 (full 0):
>  garbage-first heap   total 28049408K, used 3533601K [0x7f0c1000,
> 0x7f0c10806b00, 0x7f141000)
>   region size 8192K, 21 young (172032K), 21 survivors (172032K)
>  Metaspace   used 56487K, capacity 57092K, committed 58368K, reserved
> 59392K
> }
>  [Times: user=0.00 sys=94.28, real=97.19 secs]
> 2016-11-22T20:44:53.760+: 2942151.806: Total time for which application
> threads were stopped: 97.2053747 seconds, Stopping threads took: 0.0001373
> seconds
>
>
>
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/Very-long-young-generation-stop-the-world-GC-
> pause-tp4308911p4308912.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Re: Solr Issue

[Greg Harris]

Hi Monti,

As pointed out there is a huge gap of no information. There are two primary
possibilities. One is that something about your resources is depleted. As
Shawn has pointed out... watch them as you start up. Two, Solr is somehow
locked or waiting on something. Since there is no information at the INFO
level you can either throw your logs into DEBUG level and pray to god you
see something relevant in the noise. Or, you can run a thread dump after
you've hit your gap in time and see what the server is doing at that
moment. My recommendation is to do the latter. I'd also wonder about that 0
cores found entry as well.

Greg



On Dec 7, 2016 3:02 AM, "Monti Chandra"  wrote:

> Hello team,
> I am working on solr version to 6.2.1. It was working so nice for the first
> 20 days and now the server is restarting very slow(15-20 min).
> Please get the hardware specs of my system below:
> Linux version 3.10.0-327.el7.x86_64 (buil...@kbuilder.dev.centos.org) (gcc
> version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) )
> kernel-3.10.0-327.el7.x86_64
> It is working fine when i am taking solr directory to another server of
> same configuartion.
>
> As I am using 6.2.1 basically,  So i have downloaded the fresh
> solr-6.2.1(without any collection) and try to start the same but the server
> is not  able to start effectively. please get the log below:
>
> 0INFO  (main) [   ] o.e.j.u.log Logging initialized @257ms
> 131  INFO  (main) [   ] o.e.j.s.Server jetty-9.3.8.v20160314
> 146  INFO  (main) [   ] o.e.j.d.p.ScanningAppProvider Deployment monitor
> [file:///root/temp/solr-6.2.1/server/contexts/] at interval 0
> 311  INFO  (main) [   ] o.e.j.w.StandardDescriptorProcessor NO JSP Support
> for /solr, did not find org.apache.jasper.servlet.JspServlet
> 320  WARN  (main) [   ] o.e.j.s.SecurityHandler
> ServletContext@o.e.j.w.WebAppContext@5383967b{/solr,
> file:///root/temp/solr-6.2.1/server/solr-webapp/webapp/,
> STARTING}{/root/temp/solr-6.2.1/server/solr-webapp/webapp}
> has uncovered http methods for path: /
> 327  INFO  (main) [   ] o.a.s.s.SolrDispatchFilter
> SolrDispatchFilter.init(): WebAppClassLoader=1465085305@57536d79
> 340  INFO  (main) [   ] o.a.s.c.SolrResourceLoader JNDI not configured for
> solr (NoInitialContextEx)
> 340  INFO  (main) [   ] o.a.s.c.SolrResourceLoader using system property
> solr.solr.home: /root/temp/solr-6.2.1/server/solr
> 340  INFO  (main) [   ] o.a.s.c.SolrResourceLoader new SolrResourceLoader
> for directory: '/root/temp/solr-6.2.1/server/solr'
> 340  INFO  (main) [   ] o.a.s.c.SolrResourceLoader JNDI not configured for
> solr (NoInitialContextEx)
> 340  INFO  (main) [   ] o.a.s.c.SolrResourceLoader using system property
> solr.solr.home: /root/temp/solr-6.2.1/server/solr
> 345  INFO  (main) [   ] o.a.s.c.SolrXmlConfig Loading container
> configuration from /root/temp/solr-6.2.1/server/solr/solr.xml
> 402  INFO  (main) [   ] o.a.s.c.CorePropertiesLocator Config-defined core
> root directory: /root/temp/solr-6.2.1/server/solr
> 424  INFO  (main) [   ] o.a.s.c.CoreContainer New CoreContainer 1663619914
> 425  INFO  (main) [   ] o.a.s.c.CoreContainer Loading cores into
> CoreContainer [instanceDir=/root/temp/solr-6.2.1/server/solr]
> 425  WARN  (main) [   ] o.a.s.c.CoreContainer Couldn't add files from
> /root/temp/solr-6.2.1/server/solr/lib to classpath:
> /root/temp/solr-6.2.1/server/solr/lib
> 434  INFO  (main) [   ] o.a.s.h.c.HttpShardHandlerFactory created with
> socketTimeout : 60,connTimeout : 6,maxConnectionsPerHost :
> 20,maxConnections : 1,corePoolSize : 0,maximumPoolSize :
> 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy :
> false,useRetries : false,connectionsEvictorSleepDelay :
> 5000,maxConnectionIdleTime : 4,
> 592  INFO  (main) [   ] o.a.s.u.UpdateShardHandler Creating
> UpdateShardHandler HTTP client with params:
> socketTimeout=60=6=true
> 594  INFO  (main) [   ] o.a.s.l.LogWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
> 594  INFO  (main) [   ] o.a.s.l.LogWatcher Registering Log Listener [Log4j
> (org.slf4j.impl.Log4jLoggerFactory)]
> 596  INFO  (main) [   ] o.a.s.c.CoreContainer Security conf doesn't exist.
> Skipping setup for authorization module.
> 596  INFO  (main) [   ] o.a.s.c.CoreContainer No authentication plugin
> used.
> 127902 INFO  (main) [   ] o.a.s.c.CorePropertiesLocator Looking for core
> definitions underneath /root/temp/solr-6.2.1/server/solr
> 127906 INFO  (main) [   ] o.a.s.c.CorePropertiesLocator Found 0 core
> definitions
> 127909 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter
> user.dir=/root/temp/solr-6.2.1/server
> 127909 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter
> SolrDispatchFilter.init() done
> 127918 INFO  (main) [   ] o.e.j.s.h.ContextHandler Started
> o.e.j.w.WebAppContext@5383967b
> {/solr,file:///root/temp/solr-6.2.1/server/solr-webapp/
> webapp/,AVAILABLE}{/root/temp/solr-6.2.1/server/solr-webapp/webapp}
> 127926 INFO  (main) [   ] 

Re: Memory leak in Solr

[Greg Harris]

Hi,

All your stats show is large memory requirements to Solr. There is no
direct mapping of number of documents and queries to memory reqts as
requested in that article. Different Solr projects can yield extremely,
extremely different requirements. If you want to understand your memory
usage better, you need to do a heap dump and to analyze it with something
like Eclipse MemoryAnalyzer or YourKit. Its STW, so you will have a little
bit of downtime. In 4.10 I'd almost already guess that your culprit is not
using docValues for things being faceted, grouped, sorted on leaving you
with a large fieldCache and yielding large memory requirements which will
not be cleaned upon a gc as they are still "live objects". While I couldn't
say that's true for sure without more analysis, its IME, pretty common.

Greg


On Dec 2, 2016 11:01 AM, "S G"  wrote:

Hi,

This post shows some stats on Solr which indicate that there might be a
memory leak in there.

http://stackoverflow.com/questions/40939166/is-this-a-memory-leak-in-solr

Can someone please help to debug this?
It might be a very good step in making Solr stable if we can fix this.

Thanks
SG

Re: Solr CPU Usage

[Greg Harris]

Here is a quick way you can identify which thread is taking up all your CPU.

1) Look at top (or htop) sorted by CPU Usage and with threads toggled on -
hit capital 'H'
2) Get the native process ids of the threads taking up a lot of CPU
3) Convert that number to hex using a converter:
http://www.mathsisfun.com/binary-decimal-hexadecimal-converter.html
4) Use the hex number to identify the problematic threads on the thread
dump via the nid= value. So for example:
nid=0x549 would equate to the native thread id of 1353 on top.

Take a thread dump and identify any problematic threads so you can see the
stack trace.
However, Chris has pointed out that there is as of yet no evidence your
outage is related to CPU overload.

Greg

On Thu, Aug 28, 2014 at 6:45 PM, rulinma ruli...@gmail.com wrote:

 I think that is configs not tuned well.
 Can use jmx to monitor what is doing?



 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/Solr-CPU-Usage-tp4155370p4155747.html
 Sent from the Solr - User mailing list archive at Nabble.com.

RE: Solr query performance tool

[Greg Harris]

You have to be careful looking at the QTime's. They do not include garbage 
collection. I've run into issues where QTime is short (cause it was), it just 
happened that the query came in during a long garbage collection where 
everything was paused. So you can get into situations where once the 15 second 
GC is done everything performs as expected! I'd make sure and have an external 
querying tool and you can monitor GC times as well via JMX.



From: bbarani [bbar...@gmail.com]
Sent: Monday, June 03, 2013 8:58 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr query performance tool

You can use this tool to analyze the logs..

https://github.com/dfdeshom/solr-loganalyzer

We use solrmeter to test the performance / Stress testing.

https://code.google.com/p/solrmeter/





--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-query-performance-tool-tp4066900p4067869.html
Sent from the Solr - User mailing list archive at Nabble.com.