RE: OutofMemory on Highlightling
1) please don't cross post ... this is a user question, not development
discussion .. the user list is the appropriate place to have the
discussion.
2) ...
: Is it possible to read only maxAnalyzedChars from the stored field
: instead of reading the complete field in the memory? For instance, in my
nope ... the underlying Lucene call lets you read in all fields, or some
fields, but not parts of some fields.
if you are willing to only use the first N chars when doing highlighting,
then you can use the to ensure that only
thoe N chars get stored in your destination field ... then highlight using
that field.
: Also I am confused over the following code in SolrIndexSearcher.doc()
: method
:
: if(!enableLazyFieldLoading || fields == null) {
: d = searcher.getIndexReader().document(i);
: } else {
: d = searcher.getIndexReader().document(i,
: new SetNonLazyFieldSelector(fields));
: }
It says that if enableLazyFieldLoading is false or if no field names have
been specified: go ahead and load the whole document; otherwise use a
FieldSelector that loads the specified fields, but leaves the other fields
to be loaded lazily if needed.
-Hoss
RE: OutofMemory on Highlightling
Is it possible to read only maxAnalyzedChars from the stored field
instead of reading the complete field in the memory? For instance, in my
case, is it possible to read only first 50K characters instead of
complete 1 MB stored text? That will help minimizing the memory usage
(Though, it will still take 50K * 500 * 2 = 50 MB for 500 results).
I would really appreciate some feedback on this issue...
Thanks,
Siddharth
-Original Message-
From: Gargate, Siddharth [mailto:sgarg...@ptc.com]
Sent: Friday, April 24, 2009 10:46 AM
To: solr-user@lucene.apache.org
Subject: RE: OutofMemory on Highlightling
I am not sure whether lazy loading should help solve this problem. I
have set enableLazyFieldLoading to true but it is not helping.
I went through the code and observed that
DefaultSolrHighlighter.doHighlighting is reading all the documents and
the fields for highlighting (In my case, 1 MB stored field is read for
all documents).
Also I am confused over the following code in SolrIndexSearcher.doc()
method
if(!enableLazyFieldLoading || fields == null) {
d = searcher.getIndexReader().document(i);
} else {
d = searcher.getIndexReader().document(i,
new SetNonLazyFieldSelector(fields));
}
Are we setting the fields as NonLazy even if lazy loading is enabled?
Thanks,
Siddharth
-Original Message-
From: Gargate, Siddharth [mailto:sgarg...@ptc.com]
Sent: Wednesday, April 22, 2009 11:12 AM
To: solr-user@lucene.apache.org
Subject: RE: OutofMemory on Highlightling
Here is the stack trace
SEVERE: java.lang.OutOfMemoryError: Java heap space
at
java.lang.StringCoding$StringDecoder.decode(StringCoding.java:133)
at java.lang.StringCoding.decode(StringCoding.java:173)
at java.lang.String.(String.java:444)
at
org.apache.lucene.store.IndexInput.readString(IndexInput.java:125)
at
org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:390)
at
org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:230)
at
org.apache.lucene.index.SegmentReader.document(SegmentReader.java:892)
at
org.apache.lucene.index.MultiSegmentReader.document(MultiSegmentReader.j
ava:277)
at
org.apache.solr.search.SolrIndexReader.document(SolrIndexReader.java:176
)
at
org.apache.solr.search.SolrIndexSearcher.doc(SolrIndexSearcher.java:457)
at
org.apache.solr.search.SolrIndexSearcher.readDocs(SolrIndexSearcher.java
:482)
at
org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultS
olrHighlighter.java:253)
at
org.apache.solr.handler.component.HighlightComponent.process(HighlightCo
mponent.java:84)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(Search
Handler.java:195)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerB
ase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1333)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.ja
va:303)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.j
ava:232)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv
e.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv
e.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java
:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java
:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.
java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:2
86)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:84
5)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(
Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
-Original Message-
From: Gargate, Siddharth [mailto:sgarg...@ptc.com]
Sent: Wednesday, April 22, 2009 9:29 AM
To: solr-user@lucene.apache.org
Subject: RE: OutofMemory on Highlightling
I tried disabling the documentCache but still the same issue.
-Original Message-
From: Koji Sekiguchi [mailto:k...@r.email.ne.jp]
Sent: Monday, April 20, 2009 4:38 PM
To: solr-user@lucene.apache.org
Subject: Re: OutofMemory on Highlightling
Gargate, Siddharth wrote:
> Anybody facing the same issue? Following is my configuration
> ...
> multiValued="true"/>
>
>
> ...
>
> ...
> default="true">
>
>explicit
>
>500
> true
>
RE: OutofMemory on Highlightling
I am not sure whether lazy loading should help solve this problem. I
have set enableLazyFieldLoading to true but it is not helping.
I went through the code and observed that
DefaultSolrHighlighter.doHighlighting is reading all the documents and
the fields for highlighting (In my case, 1 MB stored field is read for
all documents).
Also I am confused over the following code in SolrIndexSearcher.doc()
method
if(!enableLazyFieldLoading || fields == null) {
d = searcher.getIndexReader().document(i);
} else {
d = searcher.getIndexReader().document(i,
new SetNonLazyFieldSelector(fields));
}
Are we setting the fields as NonLazy even if lazy loading is enabled?
Thanks,
Siddharth
-Original Message-
From: Gargate, Siddharth [mailto:sgarg...@ptc.com]
Sent: Wednesday, April 22, 2009 11:12 AM
To: solr-user@lucene.apache.org
Subject: RE: OutofMemory on Highlightling
Here is the stack trace
SEVERE: java.lang.OutOfMemoryError: Java heap space
at
java.lang.StringCoding$StringDecoder.decode(StringCoding.java:133)
at java.lang.StringCoding.decode(StringCoding.java:173)
at java.lang.String.(String.java:444)
at
org.apache.lucene.store.IndexInput.readString(IndexInput.java:125)
at
org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:390)
at
org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:230)
at
org.apache.lucene.index.SegmentReader.document(SegmentReader.java:892)
at
org.apache.lucene.index.MultiSegmentReader.document(MultiSegmentReader.j
ava:277)
at
org.apache.solr.search.SolrIndexReader.document(SolrIndexReader.java:176
)
at
org.apache.solr.search.SolrIndexSearcher.doc(SolrIndexSearcher.java:457)
at
org.apache.solr.search.SolrIndexSearcher.readDocs(SolrIndexSearcher.java
:482)
at
org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultS
olrHighlighter.java:253)
at
org.apache.solr.handler.component.HighlightComponent.process(HighlightCo
mponent.java:84)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(Search
Handler.java:195)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerB
ase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1333)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.ja
va:303)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.j
ava:232)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica
tionFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt
erChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv
e.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv
e.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java
:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java
:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.
java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:2
86)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:84
5)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(
Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
-Original Message-
From: Gargate, Siddharth [mailto:sgarg...@ptc.com]
Sent: Wednesday, April 22, 2009 9:29 AM
To: solr-user@lucene.apache.org
Subject: RE: OutofMemory on Highlightling
I tried disabling the documentCache but still the same issue.
-Original Message-
From: Koji Sekiguchi [mailto:k...@r.email.ne.jp]
Sent: Monday, April 20, 2009 4:38 PM
To: solr-user@lucene.apache.org
Subject: Re: OutofMemory on Highlightling
Gargate, Siddharth wrote:
> Anybody facing the same issue? Following is my configuration
> ...
> multiValued="true"/>
>
>
> ...
>
> ...
> default="true">
>
>explicit
>
>500
> true
> id,score
> teaser
> teaser
> 200
> 200
> 500
>
>
> ...
>
> Search works fine if I disable highlighting and it brings 500 results.
> But if I enable hightlighting and set the no. of rows to just 20 I get
> OOME.
>
>
How about switching documentCache off?
Koji
RE: OutofMemory on Highlightling
Here is the stack trace SEVERE: java.lang.OutOfMemoryError: Java heap space at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:133) at java.lang.StringCoding.decode(StringCoding.java:173) at java.lang.String.(String.java:444) at org.apache.lucene.store.IndexInput.readString(IndexInput.java:125) at org.apache.lucene.index.FieldsReader.addField(FieldsReader.java:390) at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:230) at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:892) at org.apache.lucene.index.MultiSegmentReader.document(MultiSegmentReader.j ava:277) at org.apache.solr.search.SolrIndexReader.document(SolrIndexReader.java:176 ) at org.apache.solr.search.SolrIndexSearcher.doc(SolrIndexSearcher.java:457) at org.apache.solr.search.SolrIndexSearcher.readDocs(SolrIndexSearcher.java :482) at org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultS olrHighlighter.java:253) at org.apache.solr.handler.component.HighlightComponent.process(HighlightCo mponent.java:84) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(Search Handler.java:195) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerB ase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1333) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.ja va:303) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.j ava:232) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applica tionFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilt erChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValv e.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValv e.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java :128) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java :102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve. java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:2 86) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:84 5) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process( Http11Protocol.java:583) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) at java.lang.Thread.run(Thread.java:619) -Original Message- From: Gargate, Siddharth [mailto:sgarg...@ptc.com] Sent: Wednesday, April 22, 2009 9:29 AM To: solr-user@lucene.apache.org Subject: RE: OutofMemory on Highlightling I tried disabling the documentCache but still the same issue. -Original Message- From: Koji Sekiguchi [mailto:k...@r.email.ne.jp] Sent: Monday, April 20, 2009 4:38 PM To: solr-user@lucene.apache.org Subject: Re: OutofMemory on Highlightling Gargate, Siddharth wrote: > Anybody facing the same issue? Following is my configuration > ... > multiValued="true"/> > > > ... > > ... > default="true"> > >explicit > >500 > true > id,score > teaser > teaser > 200 > 200 > 500 > > > ... > > Search works fine if I disable highlighting and it brings 500 results. > But if I enable hightlighting and set the no. of rows to just 20 I get > OOME. > > How about switching documentCache off? Koji
RE: OutofMemory on Highlightling
I tried disabling the documentCache but still the same issue. -Original Message- From: Koji Sekiguchi [mailto:k...@r.email.ne.jp] Sent: Monday, April 20, 2009 4:38 PM To: solr-user@lucene.apache.org Subject: Re: OutofMemory on Highlightling Gargate, Siddharth wrote: > Anybody facing the same issue? Following is my configuration > ... > multiValued="true"/> > > > ... > > ... > default="true"> > >explicit > >500 > true > id,score > teaser > teaser > 200 > 200 > 500 > > > ... > > Search works fine if I disable highlighting and it brings 500 results. > But if I enable hightlighting and set the no. of rows to just 20 I get > OOME. > > How about switching documentCache off? Koji
Re: OutofMemory on Highlightling
Gargate, Siddharth wrote: Anybody facing the same issue? Following is my configuration ... ... ... explicit 500 true id,score teaser teaser 200 200 500 ... Search works fine if I disable highlighting and it brings 500 results. But if I enable hightlighting and set the no. of rows to just 20 I get OOME. How about switching documentCache off? Koji
RE: OutofMemory on Highlightling
Anybody facing the same issue? Following is my configuration ... ... ... explicit 500 true id,score teaser teaser 200 200 500 ... Search works fine if I disable highlighting and it brings 500 results. But if I enable hightlighting and set the no. of rows to just 20 I get OOME. -Original Message- From: Gargate, Siddharth [mailto:sgarg...@ptc.com] Sent: Friday, April 17, 2009 11:32 AM To: solr-user@lucene.apache.org Subject: RE: OutofMemory on Highlightling I tried hl.maxAnalyzedChars=500 but still the same issue. I get OOM for row size 20 only. -Original Message- From: Otis Gospodnetic [mailto:otis_gospodne...@yahoo.com] Sent: Thursday, April 16, 2009 9:56 PM To: solr-user@lucene.apache.org Subject: Re: OutofMemory on Highlightling Hi, Have you tried: http://wiki.apache.org/solr/HighlightingParameters#head-2ca22f63cb8d1b2b a3ff0cfc05e85b94898c59cf Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From: "Gargate, Siddharth" > To: solr-user@lucene.apache.org > Sent: Thursday, April 16, 2009 6:33:46 AM > Subject: OutofMemory on Highlightling > > Hi, > > I am analyzing the memory usage for my Solr setup. I am > testing with 500 text documents of 2 MB each. > > I have defined a field for displaying the teasers and storing 1 MB of > text in it. I am testing with just 128 MB maxHeap(I know I should be > increasing it but just testing the worst case scenario). > > If I search for all 500 documents with row size as 500 and highlighting > disabled, it works fine. But if I enable highlighting I get > OutofMemoryError. > > Looks like stored field for all the matched results are read into the > memory. How to avoid this memory consumption? > > > > Thanks, > > Siddharth
RE: OutofMemory on Highlightling
I tried hl.maxAnalyzedChars=500 but still the same issue. I get OOM for row size 20 only. -Original Message- From: Otis Gospodnetic [mailto:otis_gospodne...@yahoo.com] Sent: Thursday, April 16, 2009 9:56 PM To: solr-user@lucene.apache.org Subject: Re: OutofMemory on Highlightling Hi, Have you tried: http://wiki.apache.org/solr/HighlightingParameters#head-2ca22f63cb8d1b2b a3ff0cfc05e85b94898c59cf Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From: "Gargate, Siddharth" > To: solr-user@lucene.apache.org > Sent: Thursday, April 16, 2009 6:33:46 AM > Subject: OutofMemory on Highlightling > > Hi, > > I am analyzing the memory usage for my Solr setup. I am > testing with 500 text documents of 2 MB each. > > I have defined a field for displaying the teasers and storing 1 MB of > text in it. I am testing with just 128 MB maxHeap(I know I should be > increasing it but just testing the worst case scenario). > > If I search for all 500 documents with row size as 500 and highlighting > disabled, it works fine. But if I enable highlighting I get > OutofMemoryError. > > Looks like stored field for all the matched results are read into the > memory. How to avoid this memory consumption? > > > > Thanks, > > Siddharth
Re: OutofMemory on Highlightling
Hi, Have you tried: http://wiki.apache.org/solr/HighlightingParameters#head-2ca22f63cb8d1b2ba3ff0cfc05e85b94898c59cf Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From: "Gargate, Siddharth" > To: solr-user@lucene.apache.org > Sent: Thursday, April 16, 2009 6:33:46 AM > Subject: OutofMemory on Highlightling > > Hi, > > I am analyzing the memory usage for my Solr setup. I am > testing with 500 text documents of 2 MB each. > > I have defined a field for displaying the teasers and storing 1 MB of > text in it. I am testing with just 128 MB maxHeap(I know I should be > increasing it but just testing the worst case scenario). > > If I search for all 500 documents with row size as 500 and highlighting > disabled, it works fine. But if I enable highlighting I get > OutofMemoryError. > > Looks like stored field for all the matched results are read into the > memory. How to avoid this memory consumption? > > > > Thanks, > > Siddharth
