Thanks.
We find little evidence that page/disk cache is causing this issue. We use sar
to collect statistics. Here is the statistics on a node where the query took
maximum time. (out of 5 shards, one with most data takes long time) However,
we're reducing heap size and testing in QA.
CPU %user %nice %system %iowait %steal
%idle
17:00:01 all 2.11 0.00 0.04 0.00 0.00
97.85
Average: all 7.52 0.00 0.16 0.02 0.00
92.31
tps rtps wtps bread/s bwrtn/s
17:00:01 10.63 0.00 10.63 0.00 140.56
Average: 73.90 2.65 71.24 314.24 1507.93
pgpgin/s pgpgout/s fault/s majflt/s
17:00:01 0.00 23.42 367.95 0.00
Average: 52.37 251.32 586.79 0.82
Our current JVM is 30G and usage is ~26G. If we reduce JVM to 25G, we're afraid
of hitting OOM error in Java.
-----Original Message-----
From: Michael Della Bitta [mailto:michael.della.bi...@appinions.com]
Sent: Thursday, February 27, 2014 3:45 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr4 performance
You would get more room for disk cache by reducing your large heap.
Otherwise, you'd have to add more RAM to your systems or shard your index
to more nodes to gain more RAM that way.
The Linux VM subsystem actually has a number of tuning parameters (like
vm.bdflush, vm.swappiness and vm.pagecache), but I don't know if there's
any definitive information about how to set them appropriately for Solr.
Michael Della Bitta
Applications Developer
o: +1 646 532 3062
appinions inc.
"The Science of Influence Marketing"
18 East 41st Street
New York, NY 10017
t: @appinions <https://twitter.com/Appinions> | g+:
plus.google.com/appinions<https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts>
w: appinions.com <http://www.appinions.com/>
On Thu, Feb 27, 2014 at 3:09 PM, Joshi, Shital <shital.jo...@gs.com> wrote:
> Hi Michael,
>
> If page cache is the issue, what is the solution?
>
> Thanks!
>
> -----Original Message-----
> From: Michael Della Bitta [mailto:michael.della.bi...@appinions.com]
> Sent: Monday, February 24, 2014 9:54 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr4 performance
>
> I'm not sure how you're measuring free RAM. Maybe this will help:
>
> http://www.linuxatemyram.com/play.html
>
> Michael Della Bitta
>
> Applications Developer
>
> o: +1 646 532 3062
>
> appinions inc.
>
> "The Science of Influence Marketing"
>
> 18 East 41st Street
>
> New York, NY 10017
>
> t: @appinions <https://twitter.com/Appinions> | g+:
> plus.google.com/appinions<
> https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts
> >
> w: appinions.com <http://www.appinions.com/>
>
>
> On Mon, Feb 24, 2014 at 5:35 PM, Joshi, Shital <shital.jo...@gs.com>
> wrote:
>
> > Thanks.
> >
> > We found some evidence that this could be the issue. We're monitoring
> > closely to confirm this.
> >
> > One question though: none of our nodes show more that 50% of physical
> > memory used. So there is enough memory available for memory mapped files.
> > Can this kind of pause still happen?
> >
> >
> > -----Original Message-----
> > From: Michael Della Bitta [mailto:michael.della.bi...@appinions.com]
> > Sent: Friday, February 21, 2014 5:28 PM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Solr4 performance
> >
> > It could be that your query is churning the page cache on that node
> > sometimes, so Solr pauses so the OS can drag those pages off of disk.
> Have
> > you tried profiling your iowait in top or iostat during these pauses?
> > (assuming you're using linux).
> >
> > Michael Della Bitta
> >
> > Applications Developer
> >
> > o: +1 646 532 3062
> >
> > appinions inc.
> >
> > "The Science of Influence Marketing"
> >
> > 18 East 41st Street
> >
> > New York, NY 10017
> >
> > t: @appinions <https://twitter.com/Appinions> | g+:
> > plus.google.com/appinions<
> >
> https://plus.google.com/u/0/b/112002776285509593336/112002776285509593336/posts
> > >
> > w: appinions.com <http://www.appinions.com/>
> >
> >
> > On Fri, Feb 21, 2014 at 5:20 PM, Joshi, Shital <shital.jo...@gs.com>
> > wrote:
> >
> > > Thanks for your answer.
> > >
> > > We confirmed that it is not GC issue.
> > >
> > > The auto warming query looks good too and queries before and after the
> > > long running query comes back really quick. The only thing stands out
> is
> > > shard on which query takes long time has couple million more documents
> > than
> > > other shards.
> > >
> > > -----Original Message-----
> > > From: Michael Della Bitta [mailto:michael.della.bi...@appinions.com]
> > > Sent: Thursday, February 20, 2014 5:26 PM
> > > To: solr-user@lucene.apache.org
> > > Subject: RE: Solr4 performance
> > >
> > > Hi,
> > >
> > > As for your first question, setting openSearcher to true means you will
> > see
> > > the new docs after every hard commit. Soft and hard commits only become
> > > isolated from one another with that set to false.
> > >
> > > Your second problem might be explained by your large heap and garbage
> > > collection. Walking a heap that large can take an appreciable amount of
> > > time. You might consider turning on the JVM options for logging GC and
> > > seeing if you can correlate your slow responses to times when your JVM
> is
> > > garbage collecting.
> > >
> > > Hope that helps,
> > > On Feb 20, 2014 4:52 PM, "Joshi, Shital" <shital.jo...@gs.com> wrote:
> > >
> > > > Hi!
> > > >
> > > > I have few other questions regarding Solr4 performance issue we're
> > > facing.
> > > >
> > > > We're committing data to Solr4 every ~30 seconds (up to 20K rows). We
> > use
> > > > commit=false in update URL. We have only hard commit setting in Solr4
> > > > config.
> > > >
> > > > <autoCommit>
> > > > <maxTime>${solr.autoCommit.maxTime:600000}</maxTime>
> > > > <maxDocs>100000</maxDocs>
> > > > <openSearcher>true</openSearcher>
> > > > </autoCommit>
> > > >
> > > >
> > > > Since we're not using Soft commit at all (commit=false), the caches
> > will
> > > > not get reloaded for every commit and recently added documents will
> not
> > > be
> > > > visible, correct?
> > > >
> > > > What we see is queries which usually take few milli seconds, takes
> ~40
> > > > seconds once in a while. Can high IO during hard commit cause queries
> > to
> > > > slow down?
> > > >
> > > > For some shards we see 98% full physical memory. We have 60GB machine
> > (30
> > > > GB JVM, 28 GB free RAM, ~35 GB of index). We're ruling out that high
> > > > physical memory would cause queries to slow down. We're in process of
> > > > reducing JVM size anyways.
> > > >
> > > > We have never run optimization till now. QA optimization didn't yield
> > in
> > > > performance gain.
> > > >
> > > > Thanks much for all help.
> > > >
> > > > -----Original Message-----
> > > > From: Shawn Heisey [mailto:s...@elyograg.org]
> > > > Sent: Tuesday, February 18, 2014 4:55 PM
> > > > To: solr-user@lucene.apache.org
> > > > Subject: Re: Solr4 performance
> > > >
> > > > On 2/18/2014 2:14 PM, Joshi, Shital wrote:
> > > > > Thanks much for all suggestions. We're looking into reducing
> > allocated
> > > > heap size of Solr4 JVM.
> > > > >
> > > > > We're using NRTCachingDirectoryFactory. Does it use MMapDirectory
> > > > internally? Can someone please confirm?
> > > >
> > > > In Solr, NRTCachingDirectory does indeed use MMapDirectory as its
> > > > default delegate. That's probably also the case with Lucene -- these
> > > > are Lucene classes, after all.
> > > >
> > > > MMapDirectory is almost always the most efficient way to handle
> on-disk
> > > > indexes.
> > > >
> > > > Thanks,
> > > > Shawn
> > > >
> > > >
> > >
> >
>
