On 10/22/2012 3:11 PM, Dotan Cohen wrote:
On Mon, Oct 22, 2012 at 10:01 PM, Walter Underwood
<wun...@wunderwood.org> wrote:
First, stop optimizing. You do not need to manually force merges. The system
does a great job. Forcing merges (optimize) uses a lot of CPU and disk IO and
might be the cause of your problem.
Thanks. Looking at the index statistics, I see that within minutes
after running optimize that the stats say the index needs to be
reoptimized. Though, the index still reads and writes fine even in
that state.
As soon as you make any change at all to an index, it's no longer
"optimized." Delete one document, add one document, anything. Most of
the time you will not see a performance increase from optimizing an
index that consists of one large segment and a bunch of very tiny
segments or deleted documents.
Second, the OS will use the "extra" memory for file buffers, which really helps
performance, so you might not need to do anything. This will work better after you stop
forcing merges. A forced merge replaces every file, so the OS needs to reload everything
into file buffers.
I don't see that the memory is being used:
$ free -g
total used free shared buffers cached
Mem: 14 2 12 0 0 1
-/+ buffers/cache: 0 14
Swap: 0 0 0
How big is your index, and did you run this right after a reboot? If
you did, then the cache will be fairly empty, and Solr has only read
enough from the index files to open the searcher.The number is probably
too small to show up on a gigabyte scale. As you issue queries, the
cached amount will get bigger. If your index is small enough to fit in
the 14GB of free RAM that you have, you can manually populate the disk
cache by going to your index directory and doing 'cat * > /dev/null'
from the commandline or a script. The first time you do it, it may go
slowly, but if you immediately do it again, it will complete VERY fast
-- the data will all be in RAM.
The 'free -m' command in your first email shows cache usage of 1243MB,
which suggests that maybe your index is considerably smaller than your
available RAM. Having loads of free RAM is a good thing for just about
any workload, but especially for Solr.Try running the free command
without the -g so you can see those numbers in kilobytes.
I have seen a tendency towards creating huge caches in Solr because
people have lots of memory. It's important to realize that the OS is
far better at the overall job of caching the index files than Solr
itself is. Solr caches are meant to cache result sets from queries and
filters, not large sections of the actual index contents. Make the
caches big enough that you see some benefit, but not big enough to suck
up all your RAM.
If you are having warm time problems, make the autowarm counts low. I
have run into problems with warming on my filter cache, because we have
filters that are extremely hairy and slow to run. I had to reduce my
autowarm count on the filter cache to FOUR, with a cache size of 512.
When it is 8 or higher, it can take over a minute to autowarm.
Thanks,
Shawn
