[ https://issues.apache.org/jira/browse/SOLR-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16911237#comment-16911237 ]
Dr Oleg Savrasov edited comment on SOLR-6930 at 8/20/19 11:35 AM: ------------------------------------------------------------------ Our customers very often experience OOM because of caching too large UnInvertedField items. This happens when they try to count facets or sort by not docvalues field. We don't want to fully disable UnInvertedFields usage because in some cases they work better than docvalues. For example, if the field is sparse and defined just in a few documents out of the millions, it seems better to create and cache relatively small UnInvertedField rather than traverse full docvalues structure. So to prevent OOM we need a kind of circuit breaker which would count memory utilized by UnInvertedField items and forbid caching them, when some configured threshold is exceeded. As you can see, with such an approach we are not trying to predict memory consumption, we're just restricting UnInvertedField cache by consumed memory. Since such a breakers try to control memory usage, ideally they should be part of some Global Resource Manager, see https://issues.apache.org/jira/browse/SOLR-13578. But we don't have such a manager so far, so for the time being we've tried to rely on existing Metric framework. Memory consumption is controlled by custom MetricReporter, so to enable circuit breaker it's necessary to configure appropriate unInvertedField metric reporter in solr.xml, for example: {{{{<solr>}}}} {{ {{ <metrics>}}}} {{ {{ <reporter name="unInvertedField" group="node" class="org.apache.solr.metrics.breaker.MemoryCircuitBreaker">}}}} {{ {{ <str name="limit">20%</str>}}}} {{ </reporter> }} {{ {{ </metrics>}}}} {{ {{</solr>}}}} was (Author: osavrasov): Our customers very often experience OOM because of caching too large UnInvertedField items. This happens when they try to count facets or sort by not docvalues field. We don't want to fully disable UnInvertedFields usage because in some cases they work better than docvalues. For example, if the field is sparse and defined just in a few documents out of the millions, it seems better to create and cache relatively small UnInvertedField rather than traverse full docvalues structure. So to prevent OOM we need a kind of circuit breaker which would count memory utilized by UnInvertedField items and forbid caching them, when some configured threshold is exceeded. As you can see, with such an approach we are not trying to predict memory consumption, we're just restricting UnInvertedField cache by consumed memory. Since such a breakers try to control memory usage, ideally they should be part of some Global Resource Manager, see https://issues.apache.org/jira/browse/SOLR-13578. But we don't have such a manager so far, so for the time being we've tried to rely on existing Metric framework. Memory consumption is controlled by custom MetricReporter, so to enable circuit breaker it's necessary to configure appropriate unInvertedField metric reporter in solr.xml, for example: {{<solr>}} {{ <metrics>}} {{ <reporter name="unInvertedField" group="node" class="org.apache.solr.metrics.breaker.MemoryCircuitBreaker">}} {{ <str name="limit">20%</str>}} {{ </reporter> }} {{ </metrics>}} {{</solr>}} > Provide "Circuit Breakers" For Expensive Solr Queries > ----------------------------------------------------- > > Key: SOLR-6930 > URL: https://issues.apache.org/jira/browse/SOLR-6930 > Project: Solr > Issue Type: Improvement > Components: search > Reporter: Mike Drob > Priority: Major > Attachments: SOLR-6930.patch > > > Ref: > http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html > ES currently allows operators to configure "circuit breakers" to preemptively > fail queries that are estimated too large rather than allowing an OOM > Exception to happen. We might be able to do the same thing. -- This message was sent by Atlassian Jira (v8.3.2#803003) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org