Thanks again.
I’m learning more about Solr in this thread than in my previous months
reading about it!
Moving to Solr Cloud is a possibility we’ve discussed and I guess it
will eventually happen, as the index will grow no matter what.
I’ve already lowered filterCache from 512 to 64 and I’m looking forward
to seeing what happens in the next few days. Our filter cache hit ratio
was 0.99, so I would expect this to go down but if we can have a more
efficiente memory usage I think e.g. an extra second for each search is
still acceptable.
Regarding the startup scripts we’re using the ones included with Solr.
As for the use of filters we’re always using the same four filters,
IIRC. In any case we’ll review the code to ensure that that’s the case.
I’m aware of the need to reindex when the schema changes, but thanks for
the reminder. We’ll add docValues because I think that’ll make a
significant difference in our case. We’ll also try to leave space for
the disk cache as we’re using spinning disk storage.
Thanks again to everybody for the useful and insightful replies.
Alfonso
On 12/12/2016 14:12, Shawn Heisey wrote:
On 12/12/2016 3:13 AM, Alfonso Muñoz-Pomer Fuentes wrote:
I’m writing because in our web application we’re using Solr 5.1.0 and
currently we’re hosting it on a VM with 32 GB of RAM (of which 30 are
dedicated to Solr and nothing else is running there). We have four
cores, that are this size:
- 25.56 GB, Num Docs = 57,860,845
- 12.09 GB, Num Docs = 173,491,631
(The other two cores are about 10 MB, 20k docs)
An OOM indicates that a Java application is requesting more memory than
it has been told it can use. There are only two remedies for OOM errors:
Increase the heap, or make the program use less memory. In this email,
I have concentrated on ways to reduce the memory requirements.
These index sizes and document counts are relatively small to Solr -- as
long as you have enough memory and are smart about how it's used.
Solr 5.1.0 comes with GC tuning built into the startup scripts, using
some well-tested CMS settings. If you are using those startup scripts,
then the parallel collector will NOT be default. No matter what
collector is in use, it cannot fix OOM problems. It may change when and
how frequently they occur, but it can't do anything about them.
We aren’t indexing on this machine, and we’re getting OOM relatively
quickly (after about 14 hours of regular use). Right now we have a
Cron job that restarts Solr every 12 hours, so it’s not pretty. We use
faceting quite heavily and mostly as a document storage server (we
want full data sets instead of the n most relevant results).
Like Toke, I suspect two things: a very large filterCache, and the heavy
facet usage, maybe both. Enabling docValues on the fields you're using
for faceting and reindexing will make the latter more memory efficient,
and likely faster. Reducing the filterCache size would help the
former. Note that if you have a completely static index, then it is
more likely that you will fill up the filterCache over time.
I don’t know if what we’re experiencing is usual given the index size
and memory constraint of the VM, or something looks like it’s wildly
misconfigured. What do you think? Any useful pointers for some tuning
we could do to improve the service? Would upgrading to Solr 6 make sense?
As I already mentioned, the first thing I'd check is the size of the
filterCache. Reduce it, possibly so it's VERY small. Do everything you
can to assure that you are re-using filters, not sending many unique
filters. One of the most common things that leads to low filter re-use
is using the bare NOW keyword in date filters and queries. Use NOW/HOUR
or NOW/DAY instead -- NOW changes once a millisecond, so it is typically
unique for every query. FilterCache entries are huge, as you were told
in another reply.
Unless you use docValues, or utilize the facet.method parameter VERY
carefully, each field you facet on will tie up a large section of memory
containing the value for that field in EVERY document in the index.
With the document counts you've got, this is a LOT of memory.
It is strongly recommended to have docValues enabled on every field
you're using for faceting. If you change the schema in this manner, a
full reindex will be required before you can use that field again.
There is another problem lurking here that Toke already touched on:
Leaving only 2GB of RAM for the OS to handle disk caching will result in
terrible performance.
What you've been told by me and and in other replies is discussed here:
https://wiki.apache.org/solr/SolrPerformanceProblems
Thanks,
Shawn
--
Alfonso Muñoz-Pomer Fuentes
Software Engineer @ Expression Atlas Team
European Bioinformatics Institute (EMBL-EBI)
European Molecular Biology Laboratory
Tel:+ 44 (0) 1223 49 2633
Skype: amunozpomer
