RE: OutofMemory on Highlightling
Is it possible to read only maxAnalyzedChars from the stored field
instead of reading the complete field in the memory? For instance, in my
case, is it possible to read only first 50K characters instead of
complete 1 MB stored text? That will help minimizing the memory usage
(Though, it will still take 50K * 500 * 2 = 50 MB for 500 results).
I would really appreciate some feedback on this issue...
Thanks,
Siddharth
-Original Message-
From: Gargate, Siddharth [mailto:sgarg...@ptc.com]
Sent: Friday, April 24, 2009 10:46 AM
To: solr-user@lucene.apache.org
Subject: RE: OutofMemory on Highlightling
I am not sure whether lazy loading should help solve this problem. I
have set enableLazyFieldLoading to true but it is not helping.
I went through the code and observed that
DefaultSolrHighlighter.doHighlighting is reading all the documents and
the fields for highlighting (In my case, 1 MB stored field is read for
all documents).
Also I am confused over the following code in SolrIndexSearcher.doc()
method
if(!enableLazyFieldLoading || fields == null) {
d = searcher.getIndexReader().document(i);
} else {
d = searcher.getIndexReader().document(i,
new SetNonLazyFieldSelector(fields));
}
Are we setting the fields as NonLazy even if lazy loading is enabled?
Thanks,
Siddharth
-Original Message-
From: Gargate, Siddharth [mailto:sgarg...@ptc.com]
Sent: Wednesday, April 22, 2009 11:12 AM
To: solr-user@lucene.apache.org
Subject: RE: OutofMemory on Highlightling
Here is the stack trace
SEVERE: java.lang.OutOfMemoryError: Java heap space
at
java.lang.StringCoding$StringDecoder.decode(StringCoding.java:133)
at java.lang.StringCoding.decode(StringCoding.java:173)
at java.lang.String.init(String.java:444)
at
org.apache.lucene.store.IndexInput.readString(IndexInput.java:125)
at
org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:390)
at
org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:230)
at
org.apache.lucene.index.SegmentReader.document(SegmentReader.java:892)
at
org.apache.lucene.index.MultiSegmentReader.document(MultiSegmentReader.j
ava:277)
at
org.apache.solr.search.SolrIndexReader.document(SolrIndexReader.java:176
)
at
org.apache.solr.search.SolrIndexSearcher.doc(SolrIndexSearcher.java:457)
at
org.apache.solr.search.SolrIndexSearcher.readDocs(SolrIndexSearcher.java
:482)
at
org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultS
olrHighlighter.java:253)
at
org.apache.solr.handler.component.HighlightComponent.process(HighlightCo
mponent.java:84)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(Search
Handler.java:195)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerB
ase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1333)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.ja
va:303)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.j
ava:232)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv
e.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv
e.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java
:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java
:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.
java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:2
86)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:84
5)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(
Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
-Original Message-
From: Gargate, Siddharth [mailto:sgarg...@ptc.com]
Sent: Wednesday, April 22, 2009 9:29 AM
To: solr-user@lucene.apache.org
Subject: RE: OutofMemory on Highlightling
I tried disabling the documentCache but still the same issue.
documentCache
class=solr.LRUCache
size=0
initialSize=0
autowarmCount=0/
-Original Message-
From: Koji Sekiguchi [mailto:k...@r.email.ne.jp]
Sent: Monday, April 20, 2009 4:38 PM
To: solr-user@lucene.apache.org
Subject: Re: OutofMemory on Highlightling
Gargate, Siddharth wrote:
Anybody facing the same issue? Following is my configuration
...
field name=content type=text indexed=true stored=false
multiValued=true/
field name=teaser type=text indexed=false stored=true
RE: OutofMemory on Highlightling
I am not sure whether lazy loading should help solve this problem. I
have set enableLazyFieldLoading to true but it is not helping.
I went through the code and observed that
DefaultSolrHighlighter.doHighlighting is reading all the documents and
the fields for highlighting (In my case, 1 MB stored field is read for
all documents).
Also I am confused over the following code in SolrIndexSearcher.doc()
method
if(!enableLazyFieldLoading || fields == null) {
d = searcher.getIndexReader().document(i);
} else {
d = searcher.getIndexReader().document(i,
new SetNonLazyFieldSelector(fields));
}
Are we setting the fields as NonLazy even if lazy loading is enabled?
Thanks,
Siddharth
-Original Message-
From: Gargate, Siddharth [mailto:sgarg...@ptc.com]
Sent: Wednesday, April 22, 2009 11:12 AM
To: solr-user@lucene.apache.org
Subject: RE: OutofMemory on Highlightling
Here is the stack trace
SEVERE: java.lang.OutOfMemoryError: Java heap space
at
java.lang.StringCoding$StringDecoder.decode(StringCoding.java:133)
at java.lang.StringCoding.decode(StringCoding.java:173)
at java.lang.String.init(String.java:444)
at
org.apache.lucene.store.IndexInput.readString(IndexInput.java:125)
at
org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:390)
at
org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:230)
at
org.apache.lucene.index.SegmentReader.document(SegmentReader.java:892)
at
org.apache.lucene.index.MultiSegmentReader.document(MultiSegmentReader.j
ava:277)
at
org.apache.solr.search.SolrIndexReader.document(SolrIndexReader.java:176
)
at
org.apache.solr.search.SolrIndexSearcher.doc(SolrIndexSearcher.java:457)
at
org.apache.solr.search.SolrIndexSearcher.readDocs(SolrIndexSearcher.java
:482)
at
org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultS
olrHighlighter.java:253)
at
org.apache.solr.handler.component.HighlightComponent.process(HighlightCo
mponent.java:84)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(Search
Handler.java:195)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerB
ase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1333)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.ja
va:303)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.j
ava:232)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv
e.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv
e.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java
:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java
:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.
java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:2
86)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:84
5)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(
Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
-Original Message-
From: Gargate, Siddharth [mailto:sgarg...@ptc.com]
Sent: Wednesday, April 22, 2009 9:29 AM
To: solr-user@lucene.apache.org
Subject: RE: OutofMemory on Highlightling
I tried disabling the documentCache but still the same issue.
documentCache
class=solr.LRUCache
size=0
initialSize=0
autowarmCount=0/
-Original Message-
From: Koji Sekiguchi [mailto:k...@r.email.ne.jp]
Sent: Monday, April 20, 2009 4:38 PM
To: solr-user@lucene.apache.org
Subject: Re: OutofMemory on Highlightling
Gargate, Siddharth wrote:
Anybody facing the same issue? Following is my configuration
...
field name=content type=text indexed=true stored=false
multiValued=true/
field name=teaser type=text indexed=false stored=true/
copyField source=content dest=teaser maxChars=100 /
...
...
requestHandler name=standard class=solr.SearchHandler
default=true
lst name=defaults
str name=echoParamsexplicit/str
int name=rows500/int
str name=hltrue/str
str name=flid,score/str
str name=hl.flteaser/str
str name=hl.alternateFieldteaser/str
int name=hl.fragsize200/int
int name=hl.maxAlternateFieldLength200/int
int name=hl.maxAnalyzedChars500/int
/lst
/requestHandler
...
Search works fine if I disable
RE: OutofMemory on Highlightling
I tried disabling the documentCache but still the same issue. documentCache class=solr.LRUCache size=0 initialSize=0 autowarmCount=0/ -Original Message- From: Koji Sekiguchi [mailto:k...@r.email.ne.jp] Sent: Monday, April 20, 2009 4:38 PM To: solr-user@lucene.apache.org Subject: Re: OutofMemory on Highlightling Gargate, Siddharth wrote: Anybody facing the same issue? Following is my configuration ... field name=content type=text indexed=true stored=false multiValued=true/ field name=teaser type=text indexed=false stored=true/ copyField source=content dest=teaser maxChars=100 / ... ... requestHandler name=standard class=solr.SearchHandler default=true lst name=defaults str name=echoParamsexplicit/str int name=rows500/int str name=hltrue/str str name=flid,score/str str name=hl.flteaser/str str name=hl.alternateFieldteaser/str int name=hl.fragsize200/int int name=hl.maxAlternateFieldLength200/int int name=hl.maxAnalyzedChars500/int /lst /requestHandler ... Search works fine if I disable highlighting and it brings 500 results. But if I enable hightlighting and set the no. of rows to just 20 I get OOME. How about switching documentCache off? Koji
RE: OutofMemory on Highlightling
Here is the stack trace SEVERE: java.lang.OutOfMemoryError: Java heap space at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:133) at java.lang.StringCoding.decode(StringCoding.java:173) at java.lang.String.init(String.java:444) at org.apache.lucene.store.IndexInput.readString(IndexInput.java:125) at org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:390) at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:230) at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:892) at org.apache.lucene.index.MultiSegmentReader.document(MultiSegmentReader.j ava:277) at org.apache.solr.search.SolrIndexReader.document(SolrIndexReader.java:176 ) at org.apache.solr.search.SolrIndexSearcher.doc(SolrIndexSearcher.java:457) at org.apache.solr.search.SolrIndexSearcher.readDocs(SolrIndexSearcher.java :482) at org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultS olrHighlighter.java:253) at org.apache.solr.handler.component.HighlightComponent.process(HighlightCo mponent.java:84) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(Search Handler.java:195) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerB ase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1333) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.ja va:303) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.j ava:232) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica tionFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt erChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv e.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv e.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java :128) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java :102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve. java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:2 86) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:84 5) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process( Http11Protocol.java:583) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) at java.lang.Thread.run(Thread.java:619) -Original Message- From: Gargate, Siddharth [mailto:sgarg...@ptc.com] Sent: Wednesday, April 22, 2009 9:29 AM To: solr-user@lucene.apache.org Subject: RE: OutofMemory on Highlightling I tried disabling the documentCache but still the same issue. documentCache class=solr.LRUCache size=0 initialSize=0 autowarmCount=0/ -Original Message- From: Koji Sekiguchi [mailto:k...@r.email.ne.jp] Sent: Monday, April 20, 2009 4:38 PM To: solr-user@lucene.apache.org Subject: Re: OutofMemory on Highlightling Gargate, Siddharth wrote: Anybody facing the same issue? Following is my configuration ... field name=content type=text indexed=true stored=false multiValued=true/ field name=teaser type=text indexed=false stored=true/ copyField source=content dest=teaser maxChars=100 / ... ... requestHandler name=standard class=solr.SearchHandler default=true lst name=defaults str name=echoParamsexplicit/str int name=rows500/int str name=hltrue/str str name=flid,score/str str name=hl.flteaser/str str name=hl.alternateFieldteaser/str int name=hl.fragsize200/int int name=hl.maxAlternateFieldLength200/int int name=hl.maxAnalyzedChars500/int /lst /requestHandler ... Search works fine if I disable highlighting and it brings 500 results. But if I enable hightlighting and set the no. of rows to just 20 I get OOME. How about switching documentCache off? Koji
RE: OutofMemory on Highlightling
Anybody facing the same issue? Following is my configuration ... field name=content type=text indexed=true stored=false multiValued=true/ field name=teaser type=text indexed=false stored=true/ copyField source=content dest=teaser maxChars=100 / ... ... requestHandler name=standard class=solr.SearchHandler default=true lst name=defaults str name=echoParamsexplicit/str int name=rows500/int str name=hltrue/str str name=flid,score/str str name=hl.flteaser/str str name=hl.alternateFieldteaser/str int name=hl.fragsize200/int int name=hl.maxAlternateFieldLength200/int int name=hl.maxAnalyzedChars500/int /lst /requestHandler ... Search works fine if I disable highlighting and it brings 500 results. But if I enable hightlighting and set the no. of rows to just 20 I get OOME. -Original Message- From: Gargate, Siddharth [mailto:sgarg...@ptc.com] Sent: Friday, April 17, 2009 11:32 AM To: solr-user@lucene.apache.org Subject: RE: OutofMemory on Highlightling I tried hl.maxAnalyzedChars=500 but still the same issue. I get OOM for row size 20 only. -Original Message- From: Otis Gospodnetic [mailto:otis_gospodne...@yahoo.com] Sent: Thursday, April 16, 2009 9:56 PM To: solr-user@lucene.apache.org Subject: Re: OutofMemory on Highlightling Hi, Have you tried: http://wiki.apache.org/solr/HighlightingParameters#head-2ca22f63cb8d1b2b a3ff0cfc05e85b94898c59cf Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Gargate, Siddharth sgarg...@ptc.com To: solr-user@lucene.apache.org Sent: Thursday, April 16, 2009 6:33:46 AM Subject: OutofMemory on Highlightling Hi, I am analyzing the memory usage for my Solr setup. I am testing with 500 text documents of 2 MB each. I have defined a field for displaying the teasers and storing 1 MB of text in it. I am testing with just 128 MB maxHeap(I know I should be increasing it but just testing the worst case scenario). If I search for all 500 documents with row size as 500 and highlighting disabled, it works fine. But if I enable highlighting I get OutofMemoryError. Looks like stored field for all the matched results are read into the memory. How to avoid this memory consumption? Thanks, Siddharth
Re: OutofMemory on Highlightling
Gargate, Siddharth wrote: Anybody facing the same issue? Following is my configuration ... field name=content type=text indexed=true stored=false multiValued=true/ field name=teaser type=text indexed=false stored=true/ copyField source=content dest=teaser maxChars=100 / ... ... requestHandler name=standard class=solr.SearchHandler default=true lst name=defaults str name=echoParamsexplicit/str int name=rows500/int str name=hltrue/str str name=flid,score/str str name=hl.flteaser/str str name=hl.alternateFieldteaser/str int name=hl.fragsize200/int int name=hl.maxAlternateFieldLength200/int int name=hl.maxAnalyzedChars500/int /lst /requestHandler ... Search works fine if I disable highlighting and it brings 500 results. But if I enable hightlighting and set the no. of rows to just 20 I get OOME. How about switching documentCache off? Koji
RE: OutofMemory on Highlightling
I tried hl.maxAnalyzedChars=500 but still the same issue. I get OOM for row size 20 only. -Original Message- From: Otis Gospodnetic [mailto:otis_gospodne...@yahoo.com] Sent: Thursday, April 16, 2009 9:56 PM To: solr-user@lucene.apache.org Subject: Re: OutofMemory on Highlightling Hi, Have you tried: http://wiki.apache.org/solr/HighlightingParameters#head-2ca22f63cb8d1b2b a3ff0cfc05e85b94898c59cf Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Gargate, Siddharth sgarg...@ptc.com To: solr-user@lucene.apache.org Sent: Thursday, April 16, 2009 6:33:46 AM Subject: OutofMemory on Highlightling Hi, I am analyzing the memory usage for my Solr setup. I am testing with 500 text documents of 2 MB each. I have defined a field for displaying the teasers and storing 1 MB of text in it. I am testing with just 128 MB maxHeap(I know I should be increasing it but just testing the worst case scenario). If I search for all 500 documents with row size as 500 and highlighting disabled, it works fine. But if I enable highlighting I get OutofMemoryError. Looks like stored field for all the matched results are read into the memory. How to avoid this memory consumption? Thanks, Siddharth
OutofMemory on Highlightling
Hi, I am analyzing the memory usage for my Solr setup. I am testing with 500 text documents of 2 MB each. I have defined a field for displaying the teasers and storing 1 MB of text in it. I am testing with just 128 MB maxHeap(I know I should be increasing it but just testing the worst case scenario). If I search for all 500 documents with row size as 500 and highlighting disabled, it works fine. But if I enable highlighting I get OutofMemoryError. Looks like stored field for all the matched results are read into the memory. How to avoid this memory consumption? Thanks, Siddharth
Re: OutofMemory on Highlightling
Hi, Have you tried: http://wiki.apache.org/solr/HighlightingParameters#head-2ca22f63cb8d1b2ba3ff0cfc05e85b94898c59cf Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: Gargate, Siddharth sgarg...@ptc.com To: solr-user@lucene.apache.org Sent: Thursday, April 16, 2009 6:33:46 AM Subject: OutofMemory on Highlightling Hi, I am analyzing the memory usage for my Solr setup. I am testing with 500 text documents of 2 MB each. I have defined a field for displaying the teasers and storing 1 MB of text in it. I am testing with just 128 MB maxHeap(I know I should be increasing it but just testing the worst case scenario). If I search for all 500 documents with row size as 500 and highlighting disabled, it works fine. But if I enable highlighting I get OutofMemoryError. Looks like stored field for all the matched results are read into the memory. How to avoid this memory consumption? Thanks, Siddharth
