[
https://issues.apache.org/jira/browse/SOLR-236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12679649#action_12679649
]
Mark Miller commented on SOLR-236:
----------------------------------
bq. we have already ordered another 16 GB for the machine to try and stave off
the problem. We should have it in next week.
Great. You've got a lot going on here, and 4 GB is on the extremely low end of
what I'd suggest.
bq.I restrict commits severely -
Good news again.
bq. In response to this problem I have already dramatically reduced the
following options:
Dropping the merge factor is not likely to help much. It will increase the time
it takes to add docs (merges occur much more often) for the benefit of
maintaining an almost optimized index at all times (hence the faster search
speed). Not a big RAM factor though.
Also, dropping the max buffered docs is also probably not a huge saver, and
will only affect RAM usage during indexing. Going from 1000 to 100 will likely
hurt indexing performance and not save that much RAM in the larger scheme of
things.
And dropping the maxFieldLength will hide parts of the document that are over
that length - perhaps youll end up with a handful fewer index terms, but again,
not likely a big savings here and may do more harm than good.
My suggestion of lowering your cache sizes was just a thought to eek out some
more RAM for you. Its not really suggested though if you can get more RAM. For
best performance, those caches should be set correctly. If you are using the
fieldcache method for faceting, you want the size of the filter cache to be the
same as the number of unique terms you are faceting on. The other caches are
not so large that I would suggest trimming them.
The reality is, you've got 4 million docs, sorting (uses field caches),
faceting (likely uses field caches), and this resource intensive field collapse
patch. More RAM is probably your best bet. Every document you add potentially
adds to the RAM usage of each of these things. That doesn't mean you don't have
a different problem (it does seem weird it ballooned all of a sudden), but your
running some RAM hungry stuff here, and it wouldn't blow my mind that 3 gig is
not enough to handle it. It could be that only recently the right searches
started coming in at the right times to fire up all your needs at once. Much of
this may be lazy loaded or loaded on the fly depending on if and how you have
configured your warming searches.
> Field collapsing
> ----------------
>
> Key: SOLR-236
> URL: https://issues.apache.org/jira/browse/SOLR-236
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 1.3
> Reporter: Emmanuel Keller
> Fix For: 1.5
>
> Attachments: collapsing-patch-to-1.3.0-dieter.patch,
> collapsing-patch-to-1.3.0-ivan.patch, collapsing-patch-to-1.3.0-ivan_2.patch,
> collapsing-patch-to-1.3.0-ivan_3.patch,
> field-collapsing-extended-592129.patch, field_collapsing_1.1.0.patch,
> field_collapsing_1.3.patch, field_collapsing_dsteigerwald.diff,
> field_collapsing_dsteigerwald.diff, field_collapsing_dsteigerwald.diff,
> SOLR-236-FieldCollapsing.patch, SOLR-236-FieldCollapsing.patch,
> SOLR-236-FieldCollapsing.patch, solr-236.patch
>
>
> This patch include a new feature called "Field collapsing".
> "Used in order to collapse a group of results with similar value for a given
> field to a single entry in the result set. Site collapsing is a special case
> of this, where all results for a given web site is collapsed into one or two
> entries in the result set, typically with an associated "more documents from
> this site" link. See also Duplicate detection."
> http://www.fastsearch.com/glossary.aspx?m=48&amid=299
> The implementation add 3 new query parameters (SolrParams):
> "collapse.field" to choose the field used to group results
> "collapse.type" normal (default value) or adjacent
> "collapse.max" to select how many continuous results are allowed before
> collapsing
> TODO (in progress):
> - More documentation (on source code)
> - Test cases
> Two patches:
> - "field_collapsing.patch" for current development version
> - "field_collapsing_1.1.0.patch" for Solr-1.1.0
> P.S.: Feedback and misspelling correction are welcome ;-)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
