On 11/22/2013 2:17 PM, Dave Seltzer wrote:
So I made a few changes, but I still seem to be dealing with this pesky
periodic slowness.
Changes:
1) I'm now only forcing commits every 5 minutes. This was done by
specifying commitWithin=300000 when doing document adds.
2) I'm specifying an -Xmx12g to force the java heap to take more memory
3) I'm using the GC configuration parameters from the wiki (
http://wiki.apache.org/solr/ShawnHeisey#GC_Tuning)
<snip>
I'm still seeing the same periodic slowness about every 3.5 minutes. This
slowness occurs whether or not I'm indexing content, so it appears to be
unrelated to my commit schedule.
It sounds like your heap isn't too small. Try reducing it to 5GB, then
to 4GB after some testing, so more memory gets used by the OS disk
cache. I would also recommend trying perhaps 100 threads on your test
app rather than 200. Work your way up until you find the point where it
just can't handle the load.
See the most recent graph here:
http://farm4.staticflickr.com/3819/10999523464_328814e358_o.png
To keep things consistent I'm still testing with 200 threads. When I test
with 10 threads everything is much faster, but I still get the same
periodic slowness.
One thing I've noticed is that while Java is aware of the 12 gig heap, Solr
doesn't seem to be using much of it. The system panel of the Web UI shows
11.5GB of JVM-Memory available, but only 2.11GB in use.
The memory usage in the admin UI is an instantaneous snapshot. If you
use jvisualvm or jconsole (included in the Java JDK) to get a graph of
memory usage, you'll see it change over time. As Java allocates
objects, memory usage increases until it's using all the heap. Some
amount of that allocation will be objects that are no longer in use --
garbage. Then garbage collection will kick in and memory usage will
drop down to however much is actually in use in the particular memory
pool that's being collected. This is what people often refer to as the
sawtooth pattern.
Here's a couple of screenshots. The jconsole program is running on
Windows 7, Solr is running on Linux. One screenshot is the graph, the
other is the VM summary where you can see that Solr has been running for
nearly 8 days. This is one of my production Solr servers, so some of
the parameters are slightly different than what's on my wiki:
https://dl.dropboxusercontent.com/u/97770508/solr-jconsole.png
https://dl.dropboxusercontent.com/u/97770508/solr-jconsole-summary.png
If you do not have a GUI installed on the actual Solr machine, you'll
need to use remote JMX to connect jconsole. In the init script on my
wiki page, you can see JMX options. With those, you can tell a remote
jconsole to use server.example.com:8686 instead of a local PID. You can
use any port you want that's not already in use instead of 8686.
Running jconsole with -interval=1 will make the graph update once a
second, I think it's every 5 seconds by default.
You can also hit reload on the dashboard page to see how memory usage
changes over time, but it's not as useful as a graph. Memory usage will
not change by much if you are not actively querying or indexing.
Thanks,
Shawn
