RE: Sql entity processor sortedmapbackedcache out of memory issue

[Srinivas Kashyap]

Hi Shawn,

Continuing with the older thread, I have implemented WHERE clause on the inner 
child entity. When the import is run, whether it brings only the records 
matched with WHERE condition to JVM memory or will it bring entire SQL with 
joined tables on to JVM and does the WHERE filter in memory?

Also, I have written custom java code, 'onImportEnd' event listener. Can I call 
destroy() method of SortedMapBackedCache class to remove the cached entities in 
this event listener. This is required since for every import, there would be 
some entities which would be new and wouldn't be present in previous run of dih 
cache. My assumption is, when I call destroy method it would free up the JVM 
memory and wouldn't cause OOM.


Also Is there a way I can specify Garbage collection to run on DIHCache every 
time when an import is finished on a core.

P.S: Ours is a standalone Solr server with 18 cores in it. Each core is in sync 
by running full-import on SortedMapBackedCache entities with WHERE clause based 
on timestamp(last index time) on child entities.

-Original Message-
From: Shawn Heisey 
Sent: 09 April 2019 13:27
To: solr-user@lucene.apache.org
Subject: Re: Sql entity processor sortedmapbackedcache out of memory issue

On 4/8/2019 11:47 PM, Srinivas Kashyap wrote:
> I'm using DIH to index the data and the structure of the DIH is like below 
> for solr core:
>
> 
> 16 child entities
> 
>
> During indexing, since the number of requests being made to database was 
> high(to process one document 17 queries) and was utilizing most of 
> connections of database thereby blocking our web application.

If you have 17 entities, then one document will indeed take 17 queries.
That's the nature of multiple DIH entities.

> To tackle it, we implemented SORTEDMAPBACKEDCACHE with cacheImpl parameter to 
> reduce the number of requests to database.

When you use SortedMapBackedCache on an entity, you are asking Solr to store 
the results of the entire query in memory, even if you don't need all of the 
results.  If the database has a lot of rows, that's going to take a lot of 
memory.

In your excerpt from the config, your inner entity doesn't have a WHERE clause. 
 Which means that it's going to retrieve all of the rows of the ABC table for 
*EVERY* single entry in the DEF table.  That's going to be exceptionally slow.  
Normally the SQL query on inner entities will have some kind of WHERE clause 
that limits the results to rows that match the entry from the outer entity.

You may need to write a custom indexing program that runs separately from Solr, 
possibly on an entirely different server.  That might be a lot more efficient 
than DIH.

Thanks,
Shawn

DISCLAIMER:
E-mails and attachments from Bamboo Rose, LLC are confidential.
If you are not the intended recipient, please notify the sender immediately by 
replying to the e-mail, and then delete it without making copies or using it in 
any way.
No representation is made that this email or any attachments are free of 
viruses. Virus scanning is recommended and is the responsibility of the 
recipient.

Disclaimer

The information contained in this communication from the sender is 
confidential. It is intended solely for use by the recipient and others 
authorized to receive it. If you are not the recipient, you are hereby notified 
that any disclosure, copying, distribution or taking action in relation of the 
contents of this information is strictly prohibited and may be unlawful.

This email has been scanned for viruses and malware, and may have been 
automatically archived by Mimecast Ltd, an innovator in Software as a Service 
(SaaS) for business. Providing a safer and more useful place for your human 
generated data. Specializing in; Security, archiving and compliance. To find 
out more visit the Mimecast website.

Re: solr suggester.rebuild takes forever and eventually runs out of memory on production

[Anthony Groves]

Hi Sebastian,
I saw similar undesirable results with the standard SuggestComponent when
we upgraded multiple Solr versions. For our index of 2.8M docs, it takes
6-7 minutes to build a suggestion index using SuggestComponent out of the
box, so I wouldn't be surprised if your 7.2M doc index takes a good while.

Do you have any custom Solr plugins? Suggestions are an area where
customization really seems to come in handy. We handle that "suggester not
built" exception here, by just returning an empty suggest results so the
clients do not blow up:
https://github.com/oreillymedia/ifpress-solr-plugin/blob/bf3b07c5be32fbcfa7b6fdfd439d511ef60dab68/src/main/java/com/ifactory/press/db/solr/spelling/suggest/SafariInfixSuggester.java#L141
I was also able to speed up the build slightly by excluding specific
suggest contexts at build time rather than query time (a specific type of
documents we didn't care to include in suggestions).

We did way more sophisticated things with the older SpellCheckComponent
(see MultiSuggester in that same repo) like progress logging and returning
the existing suggest index while a new one is building. If you do choose to
write a custom plugin, feel free to re-use any code there that may help
(Apache License 2.0).


*Anthony Groves*  | Technical Lead, Search
O'Reilly Media, Inc.  | 724.255.7323 <724-255-7323>  |  oreilly.com
<http://oreilly.com/search>


On Fri, Jul 24, 2020 at 4:52 AM Sebastian Riemer 
wrote:

> Oh, I am sorry, I totally forgot to mention our solr version, it's 7.7.3.
>
> -Ursprüngliche Nachricht-
> Von: Sebastian Riemer [mailto:s.rie...@littera.eu]
> Gesendet: Freitag, 24. Juli 2020 09:53
> An: solr-user@lucene.apache.org
> Betreff: solr suggester.rebuild takes forever and eventually runs out of
> memory on production
>
> Dear mailing list community,
>
> we have troubles when starting the Suggester-Build on one of our
> production servers.
>
>
> 1.   We execute the required query with the suggest.build parameter
>
> 2.   It seems solr is taking up the task to recreate the suggester
> index (we see that the CPU rises significantly)
>
> 3.   It takes forever to build (and seems to never finish!)
>
> 4.   Sometimes  the linux OOM killer strikes and usually picks the
> solr process and kills it
> 5.   During the rebuild calling the suggester results in "suggester
> not built" exception
> 6.   Restarting the solr-Service has no effect. It just continues the
> rebuild
>
> How long should it take for that task, given that our index currently
> holds approximately 7,2 Mio  documents in a parent/child structure?
> Is it possible, to query the progress of the suggester.build task after it
> was started?
> How can we be sure, whether the suggester.build task is still running or
> whether it is finished?
>
> Which factors have the most significant impact on the duration of the
> rebuild process, given that we use the config below? (Let me now, if you
> need additional information) Can we speed up the process somehow?
>
> Best regards,
> Sebastian
> Solrconfig.xml
> 
>
> name="name">infixSuggester
> name="lookupImpl">AnalyzingInfixLookupFactory
> name="indexPath">infix_suggestions
>
> name="dictionaryImpl">DocumentDictionaryFactory
> name="field">SUGGEST
> name="suggestAnalyzerFieldType">textSuggest
> name="buildOnStartup">false
> name="buildOnCommit">false
>
> 
> 
>
> name="suggest">true
> name="suggest.dictionary">infixSuggester
> name="suggest.onlyMorePopular">true
> name="suggest.count">500
> name="suggest.collate">true
>
>
>suggest
>
> 
>
>
> Mit freundlichen Grüßen
> Sebastian Riemer, BSc
>

AW: solr suggester.rebuild takes forever and eventually runs out of memory on production

[Sebastian Riemer]

Oh, I am sorry, I totally forgot to mention our solr version, it's 7.7.3.

-Ursprüngliche Nachricht-
Von: Sebastian Riemer [mailto:s.rie...@littera.eu] 
Gesendet: Freitag, 24. Juli 2020 09:53
An: solr-user@lucene.apache.org
Betreff: solr suggester.rebuild takes forever and eventually runs out of memory 
on production

Dear mailing list community,

we have troubles when starting the Suggester-Build on one of our production 
servers.


1.   We execute the required query with the suggest.build parameter

2.   It seems solr is taking up the task to recreate the suggester index 
(we see that the CPU rises significantly)

3.   It takes forever to build (and seems to never finish!)

4.   Sometimes  the linux OOM killer strikes and usually picks the solr 
process and kills it
5.   During the rebuild calling the suggester results in "suggester not 
built" exception
6.   Restarting the solr-Service has no effect. It just continues the 
rebuild

How long should it take for that task, given that our index currently holds 
approximately 7,2 Mio  documents in a parent/child structure?
Is it possible, to query the progress of the suggester.build task after it was 
started?
How can we be sure, whether the suggester.build task is still running or 
whether it is finished?

Which factors have the most significant impact on the duration of the rebuild 
process, given that we use the config below? (Let me now, if you need 
additional information) Can we speed up the process somehow?

Best regards,
Sebastian
Solrconfig.xml

   
   infixSuggester
   AnalyzingInfixLookupFactory
   infix_suggestions
   
   DocumentDictionaryFactory
   SUGGEST
   textSuggest
   false
   false
   


   
   true
   infixSuggester
   true
   500
   true
   
   
   suggest
   



Mit freundlichen Grüßen
Sebastian Riemer, BSc

solr suggester.rebuild takes forever and eventually runs out of memory on production

[Sebastian Riemer]

Dear mailing list community,

we have troubles when starting the Suggester-Build on one of our production 
servers.


1.   We execute the required query with the suggest.build parameter

2.   It seems solr is taking up the task to recreate the suggester index 
(we see that the CPU rises significantly)

3.   It takes forever to build (and seems to never finish!)

4.   Sometimes  the linux OOM killer strikes and usually picks the solr 
process and kills it
5.   During the rebuild calling the suggester results in "suggester not 
built" exception
6.   Restarting the solr-Service has no effect. It just continues the 
rebuild

How long should it take for that task, given that our index currently holds 
approximately 7,2 Mio  documents in a parent/child structure?
Is it possible, to query the progress of the suggester.build task after it was 
started?
How can we be sure, whether the suggester.build task is still running or 
whether it is finished?

Which factors have the most significant impact on the duration of the rebuild 
process, given that we use the config below? (Let me now, if you need 
additional information)
Can we speed up the process somehow?

Best regards,
Sebastian
Solrconfig.xml

   
   infixSuggester
   AnalyzingInfixLookupFactory
   infix_suggestions
   
   DocumentDictionaryFactory
   SUGGEST
   textSuggest
   false
   false
   


   
   true
   infixSuggester
   true
   500
   true
   
   
   suggest
   



Mit freundlichen Grüßen
Sebastian Riemer, BSc

Re: Out of memory errors with Spatial indexing

[David Smiley]

I believe you are experiencing this bug: LUCENE-5056

The fix would probably be adjusting code in here
org.apache.lucene.spatial.query.SpatialArgs#calcDistanceFromErrPct

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Mon, Jul 6, 2020 at 5:18 AM Sunil Varma  wrote:

> Hi David
> Thanks for your response. Yes, I noticed that all the data causing issue
> were at the poles. I tried the "RptWithGeometrySpatialField" field type
> definition but get a "Spatial context does not support S2 spatial
> index"error. Setting "spatialContextFactory="Geo3D" I still see the
> original OOM error .
>
> On Sat, 4 Jul 2020 at 05:49, David Smiley  wrote:
>
> > Hi Sunil,
> >
> > Your shape is at a pole, and I'm aware of a bug causing an exponential
> > explosion of needed grid squares when you have polygons super-close to
> the
> > pole.  Might you try S2PrefixTree instead?  I forget if this would fix it
> > or not by itself.  For indexing non-point data, I recommend
> > class="solr.RptWithGeometrySpatialField" which internally is based off a
> > combination of a course grid and storing the original vector geometry for
> > accurate verification:
> >  > class="solr.RptWithGeometrySpatialField"
> >   prefixTree="s2" />
> > The internally coarser grid will lessen the impact of that pole bug.
> >
> > ~ David Smiley
> > Apache Lucene/Solr Search Developer
> > http://www.linkedin.com/in/davidwsmiley
> >
> >
> > On Fri, Jul 3, 2020 at 7:48 AM Sunil Varma 
> > wrote:
> >
> > > We are seeing OOM errors  when trying to index some spatial data. I
> > believe
> > > the data itself might not be valid but it shouldn't cause the Server to
> > > crash. We see this on both Solr 7.6 and Solr 8. Below is the input that
> > is
> > > causing the error.
> > >
> > > {
> > > "id": "bad_data_1",
> > > "spatialwkt_srpt": "LINESTRING (-126.86037681029909 -90.0
> > > 1.000150474662E30, 73.58164711175415 -90.0 1.000150474662E30,
> > > 74.52836551959528 -90.0 1.000150474662E30, 74.97006811540834 -90.0
> > > 1.000150474662E30)"
> > > }
> > >
> > > Above dynamic field is mapped to field type "location_rpt" (
> > > solr.SpatialRecursivePrefixTreeFieldType).
> > >
> > >   Any pointers to get around this issue would be highly appreciated.
> > >
> > > Thanks!
> > >
> >
>

Re: Out of memory errors with Spatial indexing

[Sunil Varma]

Hi David
Thanks for your response. Yes, I noticed that all the data causing issue
were at the poles. I tried the "RptWithGeometrySpatialField" field type
definition but get a "Spatial context does not support S2 spatial
index"error. Setting "spatialContextFactory="Geo3D" I still see the
original OOM error .

On Sat, 4 Jul 2020 at 05:49, David Smiley  wrote:

> Hi Sunil,
>
> Your shape is at a pole, and I'm aware of a bug causing an exponential
> explosion of needed grid squares when you have polygons super-close to the
> pole.  Might you try S2PrefixTree instead?  I forget if this would fix it
> or not by itself.  For indexing non-point data, I recommend
> class="solr.RptWithGeometrySpatialField" which internally is based off a
> combination of a course grid and storing the original vector geometry for
> accurate verification:
>  class="solr.RptWithGeometrySpatialField"
>   prefixTree="s2" />
> The internally coarser grid will lessen the impact of that pole bug.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Fri, Jul 3, 2020 at 7:48 AM Sunil Varma 
> wrote:
>
> > We are seeing OOM errors  when trying to index some spatial data. I
> believe
> > the data itself might not be valid but it shouldn't cause the Server to
> > crash. We see this on both Solr 7.6 and Solr 8. Below is the input that
> is
> > causing the error.
> >
> > {
> > "id": "bad_data_1",
> > "spatialwkt_srpt": "LINESTRING (-126.86037681029909 -90.0
> > 1.000150474662E30, 73.58164711175415 -90.0 1.000150474662E30,
> > 74.52836551959528 -90.0 1.000150474662E30, 74.97006811540834 -90.0
> > 1.000150474662E30)"
> > }
> >
> > Above dynamic field is mapped to field type "location_rpt" (
> > solr.SpatialRecursivePrefixTreeFieldType).
> >
> >   Any pointers to get around this issue would be highly appreciated.
> >
> > Thanks!
> >
>

Re: Out of memory errors with Spatial indexing

[David Smiley]

Hi Sunil,

Your shape is at a pole, and I'm aware of a bug causing an exponential
explosion of needed grid squares when you have polygons super-close to the
pole.  Might you try S2PrefixTree instead?  I forget if this would fix it
or not by itself.  For indexing non-point data, I recommend
class="solr.RptWithGeometrySpatialField" which internally is based off a
combination of a course grid and storing the original vector geometry for
accurate verification:

The internally coarser grid will lessen the impact of that pole bug.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Fri, Jul 3, 2020 at 7:48 AM Sunil Varma  wrote:

> We are seeing OOM errors  when trying to index some spatial data. I believe
> the data itself might not be valid but it shouldn't cause the Server to
> crash. We see this on both Solr 7.6 and Solr 8. Below is the input that is
> causing the error.
>
> {
> "id": "bad_data_1",
> "spatialwkt_srpt": "LINESTRING (-126.86037681029909 -90.0
> 1.000150474662E30, 73.58164711175415 -90.0 1.000150474662E30,
> 74.52836551959528 -90.0 1.000150474662E30, 74.97006811540834 -90.0
> 1.000150474662E30)"
> }
>
> Above dynamic field is mapped to field type "location_rpt" (
> solr.SpatialRecursivePrefixTreeFieldType).
>
>   Any pointers to get around this issue would be highly appreciated.
>
> Thanks!
>

Out of memory errors with Spatial indexing

[Sunil Varma]

We are seeing OOM errors  when trying to index some spatial data. I believe
the data itself might not be valid but it shouldn't cause the Server to
crash. We see this on both Solr 7.6 and Solr 8. Below is the input that is
causing the error.

{
"id": "bad_data_1",
"spatialwkt_srpt": "LINESTRING (-126.86037681029909 -90.0
1.000150474662E30, 73.58164711175415 -90.0 1.000150474662E30,
74.52836551959528 -90.0 1.000150474662E30, 74.97006811540834 -90.0
1.000150474662E30)"
}

Above dynamic field is mapped to field type "location_rpt" (
solr.SpatialRecursivePrefixTreeFieldType).

  Any pointers to get around this issue would be highly appreciated.

Thanks!

Re: FuzzyQuery causing Out of Memory Errors in 8.5.x

[Colvin Cowie]

https://issues.apache.org/jira/browse/SOLR-14428



On Thu, 23 Apr 2020 at 08:45, Colvin Cowie 
wrote:

> I created a little test that fires off fuzzy queries from random UUID
> strings for 5 minutes
> *FIELD_NAME + ":" + UUID.randomUUID().toString().replace("-", "") + "~2"*
>
> The change in heap usage is really severe.
>
> On 8.5.1 Solr went OOM almost immediately on a 512mb heap, and with a 4GB
> heap it only just stayed alive.
> On 8.3.1 it was completely happy.
>
> I'm guessing that the memory might be being leaked if the FuzzyQuery
> objects are referenced from the cache, while the FuzzyTermsEnum would not
> have been.
>
> I'm going to raise an issue
>
>
> On Wed, 22 Apr 2020 at 19:44, Colvin Cowie 
> wrote:
>
>> Hello,
>>
>> I'm moving our product from 8.3.1 to 8.5.1 in dev and we've got tests
>> failing because Solr is getting OOMEs with a 512mb heap where it was
>> previously fine.
>>
>> I ran our tests on both versions with jconsole to track the heap usage.
>> Here's a little comparison. 8.5.1 dies part way through
>> https://drive.google.com/open?id=113Ujts-lzv9ZBJOUB78LA2Qw5PsIsajO
>>
>> We have our own query parser as an extension to Solr, and we do various
>> things with user queries, including generating FuzzyQuery-s. Our
>> implementation of org.apache.solr.search.QParser.parse() isn't stateful and
>> parses the qstr and returns new Query objects each time it's called.
>> With JProfiler on I can see that the majority of the heap is being
>> allocated through FuzzyQuery's constructor.
>> https://issues.apache.org/jira/browse/LUCENE-9068 moved construction of
>> the automata from the FuzzyTermsEnum to the FuzzyQuery's constructor.
>>
>> When profiling on 8.3.1 we still have a fairly large number of
>> FuzzyTermEnums created at times, but that's accounting for about ~40mb of
>> the heap for a few seconds rather than the 100mb to 300mb of continual
>> allocation for FuzzyQuery I'm seeing in 8.5.
>>
>> It's definitely possible that we're doing something wrong in our
>> extension (which I can't share the source of) but it seems like the memory
>> cost of FuzzyQuery now is totally disproportionate to what it was before.
>> We've not had issues like this with our extension before (which doesn't
>> mean that our parser is flawless, but it's not been causing noticeable
>> problems for the last 4 years).
>>
>>
>> So I suppose the question is, are we misusing FuzzyQuery in some way
>> (hard for you to say without seeing the source), or are the recent changes
>> using more memory than they should?
>>
>> I will investigate further into what we're doing. But I could maybe use
>> some help to create a stress test for Lucene itself that compares the
>> memory consumption of the old FuzzyQuery vs the new, to see whether it's
>> fundamentally bad for memory or if it's just how we're using it.
>>
>> Regards,
>> Colvin
>>
>>
>>
>>

Re: FuzzyQuery causing Out of Memory Errors in 8.5.x

[Colvin Cowie]

I created a little test that fires off fuzzy queries from random UUID
strings for 5 minutes
*FIELD_NAME + ":" + UUID.randomUUID().toString().replace("-", "") + "~2"*

The change in heap usage is really severe.

On 8.5.1 Solr went OOM almost immediately on a 512mb heap, and with a 4GB
heap it only just stayed alive.
On 8.3.1 it was completely happy.

I'm guessing that the memory might be being leaked if the FuzzyQuery
objects are referenced from the cache, while the FuzzyTermsEnum would not
have been.

I'm going to raise an issue


On Wed, 22 Apr 2020 at 19:44, Colvin Cowie 
wrote:

> Hello,
>
> I'm moving our product from 8.3.1 to 8.5.1 in dev and we've got tests
> failing because Solr is getting OOMEs with a 512mb heap where it was
> previously fine.
>
> I ran our tests on both versions with jconsole to track the heap usage.
> Here's a little comparison. 8.5.1 dies part way through
> https://drive.google.com/open?id=113Ujts-lzv9ZBJOUB78LA2Qw5PsIsajO
>
> We have our own query parser as an extension to Solr, and we do various
> things with user queries, including generating FuzzyQuery-s. Our
> implementation of org.apache.solr.search.QParser.parse() isn't stateful and
> parses the qstr and returns new Query objects each time it's called.
> With JProfiler on I can see that the majority of the heap is being
> allocated through FuzzyQuery's constructor.
> https://issues.apache.org/jira/browse/LUCENE-9068 moved construction of
> the automata from the FuzzyTermsEnum to the FuzzyQuery's constructor.
>
> When profiling on 8.3.1 we still have a fairly large number of
> FuzzyTermEnums created at times, but that's accounting for about ~40mb of
> the heap for a few seconds rather than the 100mb to 300mb of continual
> allocation for FuzzyQuery I'm seeing in 8.5.
>
> It's definitely possible that we're doing something wrong in our extension
> (which I can't share the source of) but it seems like the memory cost of
> FuzzyQuery now is totally disproportionate to what it was before. We've not
> had issues like this with our extension before (which doesn't mean that our
> parser is flawless, but it's not been causing noticeable problems for the
> last 4 years).
>
>
> So I suppose the question is, are we misusing FuzzyQuery in some way (hard
> for you to say without seeing the source), or are the recent changes using
> more memory than they should?
>
> I will investigate further into what we're doing. But I could maybe use
> some help to create a stress test for Lucene itself that compares the
> memory consumption of the old FuzzyQuery vs the new, to see whether it's
> fundamentally bad for memory or if it's just how we're using it.
>
> Regards,
> Colvin
>
>
>
>

FuzzyQuery causing Out of Memory Errors in 8.5.x

[Colvin Cowie]

Hello,

I'm moving our product from 8.3.1 to 8.5.1 in dev and we've got tests
failing because Solr is getting OOMEs with a 512mb heap where it was
previously fine.

I ran our tests on both versions with jconsole to track the heap usage.
Here's a little comparison. 8.5.1 dies part way through
https://drive.google.com/open?id=113Ujts-lzv9ZBJOUB78LA2Qw5PsIsajO

We have our own query parser as an extension to Solr, and we do various
things with user queries, including generating FuzzyQuery-s. Our
implementation of org.apache.solr.search.QParser.parse() isn't stateful and
parses the qstr and returns new Query objects each time it's called.
With JProfiler on I can see that the majority of the heap is being
allocated through FuzzyQuery's constructor.
https://issues.apache.org/jira/browse/LUCENE-9068 moved construction of the
automata from the FuzzyTermsEnum to the FuzzyQuery's constructor.

When profiling on 8.3.1 we still have a fairly large number of
FuzzyTermEnums created at times, but that's accounting for about ~40mb of
the heap for a few seconds rather than the 100mb to 300mb of continual
allocation for FuzzyQuery I'm seeing in 8.5.

It's definitely possible that we're doing something wrong in our extension
(which I can't share the source of) but it seems like the memory cost of
FuzzyQuery now is totally disproportionate to what it was before. We've not
had issues like this with our extension before (which doesn't mean that our
parser is flawless, but it's not been causing noticeable problems for the
last 4 years).


So I suppose the question is, are we misusing FuzzyQuery in some way (hard
for you to say without seeing the source), or are the recent changes using
more memory than they should?

I will investigate further into what we're doing. But I could maybe use
some help to create a stress test for Lucene itself that compares the
memory consumption of the old FuzzyQuery vs the new, to see whether it's
fundamentally bad for memory or if it's just how we're using it.

Regards,
Colvin

RE: Sql entity processor sortedmapbackedcache out of memory issue

[Srinivas Kashyap]

Hi Shawn, Mikhail

Any suggestions/pointers for using zipper algorithm. I'm facing below error.

Thanks and Regards,
Srinivas Kashyap
**

From: Srinivas Kashyap  
Sent: 12 April 2019 03:10 PM
To: solr-user@lucene.apache.org
Subject: RE: Sql entity processor sortedmapbackedcache out of memory issue

Hi Shawn/Mikhail Khludnev,

I was going through Jira  https://issues.apache.org/jira/browse/SOLR-4799 and 
see, I can do my intended activity by specifying zipper.

I tried doing it, however I'm getting error as below:

Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: 
java.lang.IllegalArgumentException: expect increasing foreign keys for Relation 
CHILD_KEY=PARENT.PARENT_KEY got: QA-HQ008880,HQ011782 at 
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:62)
at 
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:246)
at 
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
at 
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:514)
at 
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
... 5 more
Caused by: java.lang.IllegalArgumentException: expect increasing foreign keys 
for Relation CHILD_KEY=PARENT.PARENT_KEY got: QA-HQ008880,HQ011782 at 
org.apache.solr.handler.dataimport.Zipper.supplyNextChild(Zipper.java:70)
at 
org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:126)
at 
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:74)
at 
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)


Below is my dih config:















Thanks and Regards,
Srinivas Kashyap

-Original Message-
From: Shawn Heisey 
Sent: 09 April 2019 01:27 PM
To: solr-user@lucene.apache.org
Subject: Re: Sql entity processor sortedmapbackedcache out of memory issue

On 4/8/2019 11:47 PM, Srinivas Kashyap wrote:
> I'm using DIH to index the data and the structure of the DIH is like below 
> for solr core:
>
> 
> 16 child entities
> 
>
> During indexing, since the number of requests being made to database was 
> high(to process one document 17 queries) and was utilizing most of 
> connections of database thereby blocking our web application.

If you have 17 entities, then one document will indeed take 17 queries.
That's the nature of multiple DIH entities.

> To tackle it, we implemented SORTEDMAPBACKEDCACHE with cacheImpl parameter to 
> reduce the number of requests to database.

When you use SortedMapBackedCache on an entity, you are asking Solr to store 
the results of the entire query in memory, even if you don't need all of the 
results.  If the database has a lot of rows, that's going to take a lot of 
memory.

In your excerpt from the config, your inner entity doesn't have a WHERE clause. 
 Which means that it's going to retrieve all of the rows of the ABC table for 
*EVERY* single entry in the DEF table.  That's going to be exceptionally slow.  
Normally the SQL query on inner entities will have some kind of WHERE clause 
that limits the results to rows that match the entry from the outer entity.

You may need to write a custom indexing program that runs separately from Solr, 
possibly on an entirely different server.  That might be a lot more efficient 
than DIH.

Thanks,
Shawn

DISCLAIMER:
E-mails and attachments from Bamboo Rose, LLC are confidential.
If you are not the intended recipient, please notify the sender immediately by 
replying to the e-mail, and then delete it without making copies or using it in 
any way.
No representation is made that this email or any attachments are free of 
viruses. Virus scanning is recommended and is the responsibility of the 
recipient.

Re: Sql entity processor sortedmapbackedcache out of memory issue

[Nitin Kumar]

Is caching works with other entity processor like SolrEntityprocessor?

On Fri 12 Apr, 2019, 3:10 PM Srinivas Kashyap, 
wrote:

> Hi Shawn/Mikhail Khludnev,
>
> I was going through Jira  https://issues.apache.org/jira/browse/SOLR-4799
> and see, I can do my intended activity by specifying zipper.
>
> I tried doing it, however I'm getting error as below:
>
> Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
> java.lang.IllegalArgumentException: expect increasing foreign keys for
> Relation CHILD_KEY=PARENT.PARENT_KEY got: QA-HQ008880,HQ011782
> at
> org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:62)
> at
> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:246)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:514)
> at
> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
> ... 5 more
> Caused by: java.lang.IllegalArgumentException: expect increasing foreign
> keys for Relation CHILD_KEY=PARENT.PARENT_KEY got: QA-HQ008880,HQ011782
> at
> org.apache.solr.handler.dataimport.Zipper.supplyNextChild(Zipper.java:70)
> at
> org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:126)
> at
> org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:74)
> at
> org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)
>
>
> Below is my dih config:
>
>
>  query="SELECT
> PQRS,PARENT_KEY,L,M,N,O FROM DEF order by PARENT_KEY DESC"
> >
>
>  />
>  />
>  />
>
>  name="childentity1" pk="PQRS"
>
> query="SELECT A,B,C,D,E,F,CHILD_KEY,MODIFY_TS FROM ABC ORDER BY CHILD_KEY
> DESC"
>
> processor="SqlEntityProcessor" join="zipper" where="CHILD_KEY=
> PARENT.PARENT_KEY"
> >
>
>  name="A" column="A" />
>  name="B" column="B" />
> 
>
>
> Thanks and Regards,
> Srinivas Kashyap
>
> -Original Message-
> From: Shawn Heisey 
> Sent: 09 April 2019 01:27 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Sql entity processor sortedmapbackedcache out of memory issue
>
> On 4/8/2019 11:47 PM, Srinivas Kashyap wrote:
> > I'm using DIH to index the data and the structure of the DIH is like
> below for solr core:
> >
> > 
> > 16 child entities
> > 
> >
> > During indexing, since the number of requests being made to database was
> high(to process one document 17 queries) and was utilizing most of
> connections of database thereby blocking our web application.
>
> If you have 17 entities, then one document will indeed take 17 queries.
> That's the nature of multiple DIH entities.
>
> > To tackle it, we implemented SORTEDMAPBACKEDCACHE with cacheImpl
> parameter to reduce the number of requests to database.
>
> When you use SortedMapBackedCache on an entity, you are asking Solr to
> store the results of the entire query in memory, even if you don't need all
> of the results.  If the database has a lot of rows, that's going to take a
> lot of memory.
>
> In your excerpt from the config, your inner entity doesn't have a WHERE
> clause.  Which means that it's going to retrieve all of the rows of the ABC
> table for *EVERY* single entry in the DEF table.  That's going to be
> exceptionally slow.  Normally the SQL query on inner entities will have
> some kind of WHERE clause that limits the results to rows that match the
> entry from the outer entity.
>
> You may need to write a custom indexing program that runs separately from
> Solr, possibly on an entirely different server.  That might be a lot more
> efficient than DIH.
>
> Thanks,
> Shawn
> 
> DISCLAIMER:
> E-mails and attachments from Bamboo Rose, LLC are confidential.
> If you are not the intended recipient, please notify the sender
> immediately by replying to the e-mail, and then delete it without making
> copies or using it in any way.
> No representation is made that this email or any attachments are free of
> viruses. Virus scanning is recommended and is the responsibility of the
> recipient.
>

RE: Sql entity processor sortedmapbackedcache out of memory issue

[Srinivas Kashyap]

Hi Shawn/Mikhail Khludnev,

I was going through Jira  https://issues.apache.org/jira/browse/SOLR-4799 and 
see, I can do my intended activity by specifying zipper.

I tried doing it, however I'm getting error as below:

Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: 
java.lang.IllegalArgumentException: expect increasing foreign keys for Relation 
CHILD_KEY=PARENT.PARENT_KEY got: QA-HQ008880,HQ011782
at 
org.apache.solr.handler.dataimport.DataImportHandlerException.wrapAndThrow(DataImportHandlerException.java:62)
at 
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:246)
at 
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:475)
at 
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:514)
at 
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
... 5 more
Caused by: java.lang.IllegalArgumentException: expect increasing foreign keys 
for Relation CHILD_KEY=PARENT.PARENT_KEY got: QA-HQ008880,HQ011782
at org.apache.solr.handler.dataimport.Zipper.supplyNextChild(Zipper.java:70)
at 
org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:126)
at 
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:74)
at 
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProcessorWrapper.java:243)


Below is my dih config:















Thanks and Regards,
Srinivas Kashyap

-Original Message-
From: Shawn Heisey 
Sent: 09 April 2019 01:27 PM
To: solr-user@lucene.apache.org
Subject: Re: Sql entity processor sortedmapbackedcache out of memory issue

On 4/8/2019 11:47 PM, Srinivas Kashyap wrote:
> I'm using DIH to index the data and the structure of the DIH is like below 
> for solr core:
>
> 
> 16 child entities
> 
>
> During indexing, since the number of requests being made to database was 
> high(to process one document 17 queries) and was utilizing most of 
> connections of database thereby blocking our web application.

If you have 17 entities, then one document will indeed take 17 queries.
That's the nature of multiple DIH entities.

> To tackle it, we implemented SORTEDMAPBACKEDCACHE with cacheImpl parameter to 
> reduce the number of requests to database.

When you use SortedMapBackedCache on an entity, you are asking Solr to store 
the results of the entire query in memory, even if you don't need all of the 
results.  If the database has a lot of rows, that's going to take a lot of 
memory.

In your excerpt from the config, your inner entity doesn't have a WHERE clause. 
 Which means that it's going to retrieve all of the rows of the ABC table for 
*EVERY* single entry in the DEF table.  That's going to be exceptionally slow.  
Normally the SQL query on inner entities will have some kind of WHERE clause 
that limits the results to rows that match the entry from the outer entity.

You may need to write a custom indexing program that runs separately from Solr, 
possibly on an entirely different server.  That might be a lot more efficient 
than DIH.

Thanks,
Shawn

DISCLAIMER:
E-mails and attachments from Bamboo Rose, LLC are confidential.
If you are not the intended recipient, please notify the sender immediately by 
replying to the e-mail, and then delete it without making copies or using it in 
any way.
No representation is made that this email or any attachments are free of 
viruses. Virus scanning is recommended and is the responsibility of the 
recipient.

Re: Sql entity processor sortedmapbackedcache out of memory issue

[Shawn Heisey]

On 4/8/2019 11:47 PM, Srinivas Kashyap wrote:

I'm using DIH to index the data and the structure of the DIH is like below for 
solr core:


16 child entities


During indexing, since the number of requests being made to database was 
high(to process one document 17 queries) and was utilizing most of connections 
of database thereby blocking our web application.


If you have 17 entities, then one document will indeed take 17 queries. 
That's the nature of multiple DIH entities.



To tackle it, we implemented SORTEDMAPBACKEDCACHE with cacheImpl parameter to 
reduce the number of requests to database.


When you use SortedMapBackedCache on an entity, you are asking Solr to 
store the results of the entire query in memory, even if you don't need 
all of the results.  If the database has a lot of rows, that's going to 
take a lot of memory.


In your excerpt from the config, your inner entity doesn't have a WHERE 
clause.  Which means that it's going to retrieve all of the rows of the 
ABC table for *EVERY* single entry in the DEF table.  That's going to be 
exceptionally slow.  Normally the SQL query on inner entities will have 
some kind of WHERE clause that limits the results to rows that match the 
entry from the outer entity.


You may need to write a custom indexing program that runs separately 
from Solr, possibly on an entirely different server.  That might be a 
lot more efficient than DIH.


Thanks,
Shawn

Sql entity processor sortedmapbackedcache out of memory issue

[Srinivas Kashyap]

Hello,

I'm using DIH to index the data and the structure of the DIH is like below for 
solr core:


16 child entities


During indexing, since the number of requests being made to database was 
high(to process one document 17 queries) and was utilizing most of connections 
of database thereby blocking our web application.

To tackle it, we implemented SORTEDMAPBACKEDCACHE with cacheImpl parameter to 
reduce the number of requests to database.












.
.
.
.
.
.
.


We have 8GB Physical memory system(RAM) with 5GB of it allocated to JVM and 
when we do full-import, only 17 requests are made to database. However, it is 
shooting up memory consumption and is making the JVM out of memory. Out of 
memory is happening depending on the number of records each entity is bringing 
in to the memory. For Dev and QA environments, the above memory config is 
sufficient. When we move to production, we have to increase the memory to 
around 16GB of RAM and 12 GB of JVM.

Is there any logic/configurations to limit the memory usage?

Thanks and Regards,
Srinivas Kashyap


DISCLAIMER:
E-mails and attachments from Bamboo Rose, LLC are confidential.
If you are not the intended recipient, please notify the sender immediately by 
replying to the e-mail, and then delete it without making copies or using it in 
any way.
No representation is made that this email or any attachments are free of 
viruses. Virus scanning is recommended and is the responsibility of the 
recipient.

Re: unable to create new threads: out-of-memory issues

[Erick Erickson]

Absolutely increase the file limit before going down other avenues. I
recommend 65K. This is because I've spent way more time than I
want to think about finding out that this is the problem as it can pop
out in unexpected ways, ways that are totally _not_ obvious.

It's one of those things that you can do in 5 minutes and if it's
_not_ the problem no harm done, and if it _is_ the problem it'll be
good for your blood pressure ;)

Best,
Erick

On Tue, Feb 12, 2019 at 8:38 AM Walter Underwood  wrote:
>
> Create one instance of HttpSolrClient and reuse it. It is thread-safe. It 
> also keeps a connection pool, so reusing the same one will be faster.
>
> Do you really need atomic updates? Those are much slower because they have to 
> read the document before updating.
>
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
>
> > On Feb 12, 2019, at 6:58 AM, Martin Frank Hansen (MHQ)  wrote:
> >
> > Hi Mikhail,
> >
> > Thanks for your help. I will try it.
> >
> > -Original Message-
> > From: Mikhail Khludnev 
> > Sent: 12. februar 2019 15:54
> > To: solr-user 
> > Subject: Re: unable to create new threads: out-of-memory issues
> >
> > 1. you can jstack  to find it out.
> > 2. It might create a thread, I don't know.
> > 3. SolrClient is definitely a subject for heavy reuse.
> >
> > On Tue, Feb 12, 2019 at 5:16 PM Martin Frank Hansen (MHQ) 
> > wrote:
> >
> >> Hi Mikhail,
> >>
> >> I am using Solrj but think I might have found the problem.
> >>
> >> I am doing a atomicUpdate on existing documents, and found out that I
> >> create a new SolrClient for each document. I guess this is where all
> >> the threads are coming from. Is it correct that when creating a
> >> SolrClient, I also create a new thread?
> >>
> >> SolrClient solr = new HttpSolrClient.Builder(urlString).build();
> >>
> >> Thanks
> >>
> >> -Original Message-
> >> From: Mikhail Khludnev 
> >> Sent: 12. februar 2019 15:09
> >> To: solr-user 
> >> Subject: Re: unable to create new threads: out-of-memory issues
> >>
> >> Hello, Martin.
> >> How do you index? Where did you get this error?
> >> Usually it occurs in custom code with many new Thread() calls and
> >> usually healed with thread poling.
> >>
> >> On Tue, Feb 12, 2019 at 3:25 PM Martin Frank Hansen (MHQ) 
> >> wrote:
> >>
> >>> Hi,
> >>>
> >>> I am trying to create an index on a small Linux server running
> >>> Solr-7.5.0, but keep running into problems.
> >>>
> >>> When I try to index a file-folder of roughly 18 GB (18000 files) I
> >>> get the following error from the server:
> >>>
> >>> java.lang.OutOfMemoryError: unable to create new native thread.
> >>>
> >>> From the server I can see the following limits:
> >>>
> >>> User$ ulimit -a
> >>> core file size (blocks, -c) 0
> >>> data seg size (kbytes, -d) unlimited
> >>> scheduling priority (-e) 0
> >>> file size   (blocks, -f)
> >> unlimited
> >>> pending signals  (-i) 257568
> >>> max locked memory (kbytes, -l) 64
> >>> max memory size  (kbytes, -m) unlimited
> >>> open files(-n) 1024
> >>> pipe size   (512 bytes, -p) 8
> >>> POSIX message queues(bytes, -q) 819200
> >>> real-time priority  (-r) 0
> >>> stack size  (kbytes, -s) 8192
> >>> cpu time   (seconds, -t) unlimited
> >>> max user processes  (-u) 257568
> >>> virtual memory  (kbytes, -v) unlimited
> >>> file locks  (-x) unlimited
> >>>
> >>> I do not see any limits on threads only on open files.
> >>>
> >>> I have added a autoCommit of a maximum of 1000 documents, but that
> >>> did not help. How can I increase the thread limit, or is there
> >>> another way of solving this issue? Any help is appreciated.
> >>>
> >>> Best regards
> >>

Re: unable to create new threads: out-of-memory issues

[Walter Underwood]

Create one instance of HttpSolrClient and reuse it. It is thread-safe. It also 
keeps a connection pool, so reusing the same one will be faster.

Do you really need atomic updates? Those are much slower because they have to 
read the document before updating.

wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/  (my blog)

> On Feb 12, 2019, at 6:58 AM, Martin Frank Hansen (MHQ)  wrote:
> 
> Hi Mikhail, 
> 
> Thanks for your help. I will try it. 
> 
> -Original Message-
> From: Mikhail Khludnev  
> Sent: 12. februar 2019 15:54
> To: solr-user 
> Subject: Re: unable to create new threads: out-of-memory issues
> 
> 1. you can jstack  to find it out.
> 2. It might create a thread, I don't know.
> 3. SolrClient is definitely a subject for heavy reuse.
> 
> On Tue, Feb 12, 2019 at 5:16 PM Martin Frank Hansen (MHQ) 
> wrote:
> 
>> Hi Mikhail,
>> 
>> I am using Solrj but think I might have found the problem.
>> 
>> I am doing a atomicUpdate on existing documents, and found out that I 
>> create a new SolrClient for each document. I guess this is where all 
>> the threads are coming from. Is it correct that when creating a 
>> SolrClient, I also create a new thread?
>> 
>> SolrClient solr = new HttpSolrClient.Builder(urlString).build();
>> 
>> Thanks
>> 
>> -Original Message-
>> From: Mikhail Khludnev 
>> Sent: 12. februar 2019 15:09
>> To: solr-user 
>> Subject: Re: unable to create new threads: out-of-memory issues
>> 
>> Hello, Martin.
>> How do you index? Where did you get this error?
>> Usually it occurs in custom code with many new Thread() calls and 
>> usually healed with thread poling.
>> 
>> On Tue, Feb 12, 2019 at 3:25 PM Martin Frank Hansen (MHQ) 
>> wrote:
>> 
>>> Hi,
>>> 
>>> I am trying to create an index on a small Linux server running 
>>> Solr-7.5.0, but keep running into problems.
>>> 
>>> When I try to index a file-folder of roughly 18 GB (18000 files) I 
>>> get the following error from the server:
>>> 
>>> java.lang.OutOfMemoryError: unable to create new native thread.
>>> 
>>> From the server I can see the following limits:
>>> 
>>> User$ ulimit -a
>>> core file size (blocks, -c) 0
>>> data seg size (kbytes, -d) unlimited
>>> scheduling priority (-e) 0
>>> file size   (blocks, -f)
>> unlimited
>>> pending signals  (-i) 257568
>>> max locked memory (kbytes, -l) 64
>>> max memory size  (kbytes, -m) unlimited
>>> open files(-n) 1024
>>> pipe size   (512 bytes, -p) 8
>>> POSIX message queues(bytes, -q) 819200
>>> real-time priority  (-r) 0
>>> stack size  (kbytes, -s) 8192
>>> cpu time   (seconds, -t) unlimited
>>> max user processes  (-u) 257568
>>> virtual memory  (kbytes, -v) unlimited
>>> file locks  (-x) unlimited
>>> 
>>> I do not see any limits on threads only on open files.
>>> 
>>> I have added a autoCommit of a maximum of 1000 documents, but that 
>>> did not help. How can I increase the thread limit, or is there 
>>> another way of solving this issue? Any help is appreciated.
>>> 
>>> Best regards
>>> 
>>> Martin
>>> 
>>> Beskyttelse af dine personlige oplysninger er vigtig for os. Her 
>>> finder du KMD’s 
>>> Privatlivspolitik<http://www.kmd.dk/Privatlivspolitik>, der 
>>> fortæller,
>> hvordan vi behandler oplysninger om dig.
>>> 
>>> Protection of your personal data is important to us. Here you can 
>>> read KMD’s Privacy Policy<http://www.kmd.net/Privacy-Policy> 
>>> outlining how we process your personal data.
>>> 
>>> Vi gør opmærksom på, at denne e-mail kan indeholde fortrolig information.
>>> Hvis du ved en fejltagelse modtager e-mailen, beder vi dig venligst 
>>> informere afsender om fejlen ved at bruge svarfunktionen. Samtidig 
>>> beder vi dig slette e-mailen i dit system uden at videresende eller
>> kopiere den.
>>> Selvom e-mailen og ethvert vedhæftet bilag efter vores over

RE: unable to create new threads: out-of-memory issues

[Vadim Ivanov]

Hi!
I had the same issue and found that actual problem with the file limit (in 
spite of the error message)
To increase file limit:

On Linux, you can increase the limits by running the following command as root:
sysctl -w vm.max_map_count=262144

To set this value permanently, update the vm.max_map_count setting in 
/etc/sysctl.conf. 
To verify after rebooting, run sysctl vm.max_map_count.

Hope, it'll help
-- 
Vadim


> -Original Message-
> From: Martin Frank Hansen (MHQ) [mailto:m...@kmd.dk]
> Sent: Tuesday, February 12, 2019 3:25 PM
> To: solr-user@lucene.apache.org
> Subject: unable to create new threads: out-of-memory issues
> 
> Hi,
> 
> I am trying to create an index on a small Linux server running Solr-7.5.0, but
> keep running into problems.
> 
> When I try to index a file-folder of roughly 18 GB (18000 files) I get the
> following error from the server:
> 
> java.lang.OutOfMemoryError: unable to create new native thread.
> 
> From the server I can see the following limits:
> 
> User$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size   (blocks, -f) unlimited
> pending signals  (-i) 257568
> max locked memory (kbytes, -l) 64
> max memory size  (kbytes, -m) unlimited
> open files(-n) 1024
> pipe size   (512 bytes, -p) 8
> POSIX message queues(bytes, -q) 819200
> real-time priority  (-r) 0
> stack size  (kbytes, -s) 8192
> cpu time   (seconds, -t) unlimited
> max user processes  (-u) 257568
> virtual memory  (kbytes, -v) unlimited
> file locks  (-x) unlimited
> 
> I do not see any limits on threads only on open files.
> 
> I have added a autoCommit of a maximum of 1000 documents, but that did
> not help. How can I increase the thread limit, or is there another way of
> solving this issue? Any help is appreciated.
> 
> Best regards
> 
> Martin
> 
> Beskyttelse af dine personlige oplysninger er vigtig for os. Her finder du 
> KMD’s
> Privatlivspolitik<http://www.kmd.dk/Privatlivspolitik>, der fortæller, hvordan
> vi behandler oplysninger om dig.
> 
> Protection of your personal data is important to us. Here you can read KMD’s
> Privacy Policy<http://www.kmd.net/Privacy-Policy> outlining how we process
> your personal data.
> 
> Vi gør opmærksom på, at denne e-mail kan indeholde fortrolig information.
> Hvis du ved en fejltagelse modtager e-mailen, beder vi dig venligst informere
> afsender om fejlen ved at bruge svarfunktionen. Samtidig beder vi dig slette 
> e-
> mailen i dit system uden at videresende eller kopiere den. Selvom e-mailen og
> ethvert vedhæftet bilag efter vores overbevisning er fri for virus og andre 
> fejl,
> som kan påvirke computeren eller it-systemet, hvori den modtages og læses,
> åbnes den på modtagerens eget ansvar. Vi påtager os ikke noget ansvar for
> tab og skade, som er opstået i forbindelse med at modtage og bruge e-mailen.
> 
> Please note that this message may contain confidential information. If you
> have received this message by mistake, please inform the sender of the
> mistake by sending a reply, then delete the message from your system
> without making, distributing or retaining any copies of it. Although we 
> believe
> that the message and any attachments are free from viruses and other errors
> that might affect the computer or it-system where it is received and read, the
> recipient opens the message at his or her own risk. We assume no
> responsibility for any loss or damage arising from the receipt or use of this
> message.

RE: unable to create new threads: out-of-memory issues

[Martin Frank Hansen (MHQ)]

Hi Mikhail, 

Thanks for your help. I will try it. 

-Original Message-
From: Mikhail Khludnev  
Sent: 12. februar 2019 15:54
To: solr-user 
Subject: Re: unable to create new threads: out-of-memory issues

1. you can jstack  to find it out.
2. It might create a thread, I don't know.
3. SolrClient is definitely a subject for heavy reuse.

On Tue, Feb 12, 2019 at 5:16 PM Martin Frank Hansen (MHQ) 
wrote:

> Hi Mikhail,
>
> I am using Solrj but think I might have found the problem.
>
> I am doing a atomicUpdate on existing documents, and found out that I 
> create a new SolrClient for each document. I guess this is where all 
> the threads are coming from. Is it correct that when creating a 
> SolrClient, I also create a new thread?
>
> SolrClient solr = new HttpSolrClient.Builder(urlString).build();
>
> Thanks
>
> -Original Message-
> From: Mikhail Khludnev 
> Sent: 12. februar 2019 15:09
> To: solr-user 
> Subject: Re: unable to create new threads: out-of-memory issues
>
> Hello, Martin.
> How do you index? Where did you get this error?
>  Usually it occurs in custom code with many new Thread() calls and 
> usually healed with thread poling.
>
> On Tue, Feb 12, 2019 at 3:25 PM Martin Frank Hansen (MHQ) 
> wrote:
>
> > Hi,
> >
> > I am trying to create an index on a small Linux server running 
> > Solr-7.5.0, but keep running into problems.
> >
> > When I try to index a file-folder of roughly 18 GB (18000 files) I 
> > get the following error from the server:
> >
> > java.lang.OutOfMemoryError: unable to create new native thread.
> >
> > From the server I can see the following limits:
> >
> > User$ ulimit -a
> > core file size (blocks, -c) 0
> > data seg size (kbytes, -d) unlimited
> > scheduling priority (-e) 0
> > file size   (blocks, -f)
> unlimited
> > pending signals  (-i) 257568
> > max locked memory (kbytes, -l) 64
> > max memory size  (kbytes, -m) unlimited
> > open files(-n) 1024
> > pipe size   (512 bytes, -p) 8
> > POSIX message queues(bytes, -q) 819200
> > real-time priority  (-r) 0
> > stack size  (kbytes, -s) 8192
> > cpu time   (seconds, -t) unlimited
> > max user processes  (-u) 257568
> > virtual memory  (kbytes, -v) unlimited
> > file locks  (-x) unlimited
> >
> > I do not see any limits on threads only on open files.
> >
> > I have added a autoCommit of a maximum of 1000 documents, but that 
> > did not help. How can I increase the thread limit, or is there 
> > another way of solving this issue? Any help is appreciated.
> >
> > Best regards
> >
> > Martin
> >
> > Beskyttelse af dine personlige oplysninger er vigtig for os. Her 
> > finder du KMD’s 
> > Privatlivspolitik<http://www.kmd.dk/Privatlivspolitik>, der 
> > fortæller,
> hvordan vi behandler oplysninger om dig.
> >
> > Protection of your personal data is important to us. Here you can 
> > read KMD’s Privacy Policy<http://www.kmd.net/Privacy-Policy> 
> > outlining how we process your personal data.
> >
> > Vi gør opmærksom på, at denne e-mail kan indeholde fortrolig information.
> > Hvis du ved en fejltagelse modtager e-mailen, beder vi dig venligst 
> > informere afsender om fejlen ved at bruge svarfunktionen. Samtidig 
> > beder vi dig slette e-mailen i dit system uden at videresende eller
> kopiere den.
> > Selvom e-mailen og ethvert vedhæftet bilag efter vores overbevisning 
> > er fri for virus og andre fejl, som kan påvirke computeren eller 
> > it-systemet, hvori den modtages og læses, åbnes den på modtagerens 
> > eget ansvar. Vi påtager os ikke noget ansvar for tab og skade, som 
> > er opstået i forbindelse med at modtage og bruge e-mailen.
> >
> > Please note that this message may contain confidential information. 
> > If you have received this message by mistake, please inform the 
> > sender of the mistake by sending a reply, then delete the message 
> > from your system without making, distributing or retaining any copies of it.
> > Although we believe that the message and any attachments are free 
> > from viruses and other errors that might affect the computer or 
> > it-system where it is received and read, the recipient opens the 
> > message at his or
> her own risk.
> > We assume no responsibility for any loss or damage arising from the 
> > receipt or use of this message.
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>


--
Sincerely yours
Mikhail Khludnev

Re: unable to create new threads: out-of-memory issues

[Mikhail Khludnev]

1. you can jstack  to find it out.
2. It might create a thread, I don't know.
3. SolrClient is definitely a subject for heavy reuse.

On Tue, Feb 12, 2019 at 5:16 PM Martin Frank Hansen (MHQ) 
wrote:

> Hi Mikhail,
>
> I am using Solrj but think I might have found the problem.
>
> I am doing a atomicUpdate on existing documents, and found out that I
> create a new SolrClient for each document. I guess this is where all the
> threads are coming from. Is it correct that when creating a SolrClient, I
> also create a new thread?
>
> SolrClient solr = new HttpSolrClient.Builder(urlString).build();
>
> Thanks
>
> -Original Message-
> From: Mikhail Khludnev 
> Sent: 12. februar 2019 15:09
> To: solr-user 
> Subject: Re: unable to create new threads: out-of-memory issues
>
> Hello, Martin.
> How do you index? Where did you get this error?
>  Usually it occurs in custom code with many new Thread() calls and usually
> healed with thread poling.
>
> On Tue, Feb 12, 2019 at 3:25 PM Martin Frank Hansen (MHQ) 
> wrote:
>
> > Hi,
> >
> > I am trying to create an index on a small Linux server running
> > Solr-7.5.0, but keep running into problems.
> >
> > When I try to index a file-folder of roughly 18 GB (18000 files) I get
> > the following error from the server:
> >
> > java.lang.OutOfMemoryError: unable to create new native thread.
> >
> > From the server I can see the following limits:
> >
> > User$ ulimit -a
> > core file size (blocks, -c) 0
> > data seg size (kbytes, -d) unlimited
> > scheduling priority (-e) 0
> > file size   (blocks, -f)
> unlimited
> > pending signals  (-i) 257568
> > max locked memory (kbytes, -l) 64
> > max memory size  (kbytes, -m) unlimited
> > open files(-n) 1024
> > pipe size   (512 bytes, -p) 8
> > POSIX message queues(bytes, -q) 819200
> > real-time priority  (-r) 0
> > stack size  (kbytes, -s) 8192
> > cpu time   (seconds, -t) unlimited
> > max user processes  (-u) 257568
> > virtual memory  (kbytes, -v) unlimited
> > file locks  (-x) unlimited
> >
> > I do not see any limits on threads only on open files.
> >
> > I have added a autoCommit of a maximum of 1000 documents, but that did
> > not help. How can I increase the thread limit, or is there another way
> > of solving this issue? Any help is appreciated.
> >
> > Best regards
> >
> > Martin
> >
> > Beskyttelse af dine personlige oplysninger er vigtig for os. Her
> > finder du KMD’s
> > Privatlivspolitik<http://www.kmd.dk/Privatlivspolitik>, der fortæller,
> hvordan vi behandler oplysninger om dig.
> >
> > Protection of your personal data is important to us. Here you can read
> > KMD’s Privacy Policy<http://www.kmd.net/Privacy-Policy> outlining how
> > we process your personal data.
> >
> > Vi gør opmærksom på, at denne e-mail kan indeholde fortrolig information.
> > Hvis du ved en fejltagelse modtager e-mailen, beder vi dig venligst
> > informere afsender om fejlen ved at bruge svarfunktionen. Samtidig
> > beder vi dig slette e-mailen i dit system uden at videresende eller
> kopiere den.
> > Selvom e-mailen og ethvert vedhæftet bilag efter vores overbevisning
> > er fri for virus og andre fejl, som kan påvirke computeren eller
> > it-systemet, hvori den modtages og læses, åbnes den på modtagerens
> > eget ansvar. Vi påtager os ikke noget ansvar for tab og skade, som er
> > opstået i forbindelse med at modtage og bruge e-mailen.
> >
> > Please note that this message may contain confidential information. If
> > you have received this message by mistake, please inform the sender of
> > the mistake by sending a reply, then delete the message from your
> > system without making, distributing or retaining any copies of it.
> > Although we believe that the message and any attachments are free from
> > viruses and other errors that might affect the computer or it-system
> > where it is received and read, the recipient opens the message at his or
> her own risk.
> > We assume no responsibility for any loss or damage arising from the
> > receipt or use of this message.
> >
>
>
> --
> Sincerely yours
> Mikhail Khludnev
>


-- 
Sincerely yours
Mikhail Khludnev

RE: unable to create new threads: out-of-memory issues

[Martin Frank Hansen (MHQ)]

Hi Mikhail, 

I am using Solrj but think I might have found the problem. 

I am doing a atomicUpdate on existing documents, and found out that I create a 
new SolrClient for each document. I guess this is where all the threads are 
coming from. Is it correct that when creating a SolrClient, I also create a new 
thread? 

SolrClient solr = new HttpSolrClient.Builder(urlString).build();

Thanks 

-Original Message-
From: Mikhail Khludnev  
Sent: 12. februar 2019 15:09
To: solr-user 
Subject: Re: unable to create new threads: out-of-memory issues

Hello, Martin.
How do you index? Where did you get this error?
 Usually it occurs in custom code with many new Thread() calls and usually 
healed with thread poling.

On Tue, Feb 12, 2019 at 3:25 PM Martin Frank Hansen (MHQ) 
wrote:

> Hi,
>
> I am trying to create an index on a small Linux server running 
> Solr-7.5.0, but keep running into problems.
>
> When I try to index a file-folder of roughly 18 GB (18000 files) I get 
> the following error from the server:
>
> java.lang.OutOfMemoryError: unable to create new native thread.
>
> From the server I can see the following limits:
>
> User$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size   (blocks, -f) unlimited
> pending signals  (-i) 257568
> max locked memory (kbytes, -l) 64
> max memory size  (kbytes, -m) unlimited
> open files(-n) 1024
> pipe size   (512 bytes, -p) 8
> POSIX message queues(bytes, -q) 819200
> real-time priority  (-r) 0
> stack size  (kbytes, -s) 8192
> cpu time   (seconds, -t) unlimited
> max user processes  (-u) 257568
> virtual memory  (kbytes, -v) unlimited
> file locks  (-x) unlimited
>
> I do not see any limits on threads only on open files.
>
> I have added a autoCommit of a maximum of 1000 documents, but that did 
> not help. How can I increase the thread limit, or is there another way 
> of solving this issue? Any help is appreciated.
>
> Best regards
>
> Martin
>
> Beskyttelse af dine personlige oplysninger er vigtig for os. Her 
> finder du KMD’s 
> Privatlivspolitik<http://www.kmd.dk/Privatlivspolitik>, der fortæller, 
> hvordan vi behandler oplysninger om dig.
>
> Protection of your personal data is important to us. Here you can read 
> KMD’s Privacy Policy<http://www.kmd.net/Privacy-Policy> outlining how 
> we process your personal data.
>
> Vi gør opmærksom på, at denne e-mail kan indeholde fortrolig information.
> Hvis du ved en fejltagelse modtager e-mailen, beder vi dig venligst 
> informere afsender om fejlen ved at bruge svarfunktionen. Samtidig 
> beder vi dig slette e-mailen i dit system uden at videresende eller kopiere 
> den.
> Selvom e-mailen og ethvert vedhæftet bilag efter vores overbevisning 
> er fri for virus og andre fejl, som kan påvirke computeren eller 
> it-systemet, hvori den modtages og læses, åbnes den på modtagerens 
> eget ansvar. Vi påtager os ikke noget ansvar for tab og skade, som er 
> opstået i forbindelse med at modtage og bruge e-mailen.
>
> Please note that this message may contain confidential information. If 
> you have received this message by mistake, please inform the sender of 
> the mistake by sending a reply, then delete the message from your 
> system without making, distributing or retaining any copies of it. 
> Although we believe that the message and any attachments are free from 
> viruses and other errors that might affect the computer or it-system 
> where it is received and read, the recipient opens the message at his or her 
> own risk.
> We assume no responsibility for any loss or damage arising from the 
> receipt or use of this message.
>


--
Sincerely yours
Mikhail Khludnev

Re: unable to create new threads: out-of-memory issues

[Mikhail Khludnev]

Hello, Martin.
How do you index? Where did you get this error?
 Usually it occurs in custom code with many new Thread() calls and usually
healed with thread poling.

On Tue, Feb 12, 2019 at 3:25 PM Martin Frank Hansen (MHQ) 
wrote:

> Hi,
>
> I am trying to create an index on a small Linux server running Solr-7.5.0,
> but keep running into problems.
>
> When I try to index a file-folder of roughly 18 GB (18000 files) I get the
> following error from the server:
>
> java.lang.OutOfMemoryError: unable to create new native thread.
>
> From the server I can see the following limits:
>
> User$ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size   (blocks, -f) unlimited
> pending signals  (-i) 257568
> max locked memory (kbytes, -l) 64
> max memory size  (kbytes, -m) unlimited
> open files(-n) 1024
> pipe size   (512 bytes, -p) 8
> POSIX message queues(bytes, -q) 819200
> real-time priority  (-r) 0
> stack size  (kbytes, -s) 8192
> cpu time   (seconds, -t) unlimited
> max user processes  (-u) 257568
> virtual memory  (kbytes, -v) unlimited
> file locks  (-x) unlimited
>
> I do not see any limits on threads only on open files.
>
> I have added a autoCommit of a maximum of 1000 documents, but that did not
> help. How can I increase the thread limit, or is there another way of
> solving this issue? Any help is appreciated.
>
> Best regards
>
> Martin
>
> Beskyttelse af dine personlige oplysninger er vigtig for os. Her finder du
> KMD’s Privatlivspolitik, der
> fortæller, hvordan vi behandler oplysninger om dig.
>
> Protection of your personal data is important to us. Here you can read
> KMD’s Privacy Policy outlining how we
> process your personal data.
>
> Vi gør opmærksom på, at denne e-mail kan indeholde fortrolig information.
> Hvis du ved en fejltagelse modtager e-mailen, beder vi dig venligst
> informere afsender om fejlen ved at bruge svarfunktionen. Samtidig beder vi
> dig slette e-mailen i dit system uden at videresende eller kopiere den.
> Selvom e-mailen og ethvert vedhæftet bilag efter vores overbevisning er fri
> for virus og andre fejl, som kan påvirke computeren eller it-systemet,
> hvori den modtages og læses, åbnes den på modtagerens eget ansvar. Vi
> påtager os ikke noget ansvar for tab og skade, som er opstået i forbindelse
> med at modtage og bruge e-mailen.
>
> Please note that this message may contain confidential information. If you
> have received this message by mistake, please inform the sender of the
> mistake by sending a reply, then delete the message from your system
> without making, distributing or retaining any copies of it. Although we
> believe that the message and any attachments are free from viruses and
> other errors that might affect the computer or it-system where it is
> received and read, the recipient opens the message at his or her own risk.
> We assume no responsibility for any loss or damage arising from the receipt
> or use of this message.
>


-- 
Sincerely yours
Mikhail Khludnev

unable to create new threads: out-of-memory issues

[Martin Frank Hansen (MHQ)]

Hi,

I am trying to create an index on a small Linux server running Solr-7.5.0, but 
keep running into problems.

When I try to index a file-folder of roughly 18 GB (18000 files) I get the 
following error from the server:

java.lang.OutOfMemoryError: unable to create new native thread.

>From the server I can see the following limits:

User$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals  (-i) 257568
max locked memory (kbytes, -l) 64
max memory size  (kbytes, -m) unlimited
open files(-n) 1024
pipe size   (512 bytes, -p) 8
POSIX message queues(bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 8192
cpu time   (seconds, -t) unlimited
max user processes  (-u) 257568
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited

I do not see any limits on threads only on open files.

I have added a autoCommit of a maximum of 1000 documents, but that did not 
help. How can I increase the thread limit, or is there another way of solving 
this issue? Any help is appreciated.

Best regards

Martin

Beskyttelse af dine personlige oplysninger er vigtig for os. Her finder du 
KMD’s Privatlivspolitik, der fortæller, 
hvordan vi behandler oplysninger om dig.

Protection of your personal data is important to us. Here you can read KMD’s 
Privacy Policy outlining how we process your 
personal data.

Vi gør opmærksom på, at denne e-mail kan indeholde fortrolig information. Hvis 
du ved en fejltagelse modtager e-mailen, beder vi dig venligst informere 
afsender om fejlen ved at bruge svarfunktionen. Samtidig beder vi dig slette 
e-mailen i dit system uden at videresende eller kopiere den. Selvom e-mailen og 
ethvert vedhæftet bilag efter vores overbevisning er fri for virus og andre 
fejl, som kan påvirke computeren eller it-systemet, hvori den modtages og 
læses, åbnes den på modtagerens eget ansvar. Vi påtager os ikke noget ansvar 
for tab og skade, som er opstået i forbindelse med at modtage og bruge e-mailen.

Please note that this message may contain confidential information. If you have 
received this message by mistake, please inform the sender of the mistake by 
sending a reply, then delete the message from your system without making, 
distributing or retaining any copies of it. Although we believe that the 
message and any attachments are free from viruses and other errors that might 
affect the computer or it-system where it is received and read, the recipient 
opens the message at his or her own risk. We assume no responsibility for any 
loss or damage arising from the receipt or use of this message.

Re: SOLR 7.0 DIH out of memory issue with sqlserver

[Erick Erickson]

Tanya:

Good to hear. You probably want to configure hard  as
well, and in your case perhaps with openSearcher=true

Indexing is only half the problem. It's quite possible that what's
happening is your index is just growing and that's pushing the
boundaries of Java heap. What I'm thinking is that DIH may be
importing small batches, but as more docs are added the underlying
index grows by enough to hit an OOM with batch N+1.

You could just move the indexing off to a client which would give you
a way to see which it is (index growth or jdbc config). Here's a
sample:
https://lucidworks.com/2012/02/14/indexing-with-solrj/

Best,
Erick
On Wed, Sep 19, 2018 at 1:22 PM Tanya Bompi  wrote:
>
> Hi Erick,
>   Thank you for the follow-up. I have resolved the issue with the increase
> in heapSize and I am able to set the SOLR VM to initialize with a  3G heap
> size and the subset of 1 mil records was fetched successfully. Although it
> fails with the entire 3 mil records. So something is off with the adaptive
> buffering setting as I see  it is not helping. And I also set the
> autosoftcommit param. I might have to increase the heap size further to see
> if it helps. I will keep posted if my issue doesnt resolve.
>
> Thanks,
> Tanya
>
> On Wed, Sep 19, 2018 at 8:22 AM Erick Erickson 
> wrote:
>
> > Has this ever worked? IOW, is this something that's changed or has
> > just never worked?
> >
> > The obvious first step is to start Solr with more than 1G of memory.
> > Solr _likes_ memory and a 1G heap is quite small. But you say:
> > "Increasing the heap size further doesnt start SOLR instance itself.".
> > How much RAM do you have on your machine? What other programs are
> > running? You should be able to increase the heap and start Solr if you
> > have the RAM on your machine so I'd figure out what's behind that
> > issue first. I regularly start Solr with 16 or 32G of memory on my
> > local machines, I know of installations running Solr with 60G heaps so
> > this points to something really odd about your environment.
> >
> > When you "increase it further", exactly _how_ does Solr fail to start?
> > What appears in the Solr logs? etc. Really, troubleshoot that issue
> > first I'd recommend.
> >
> > If DIH still needs a ridiculous amount of memory, it's usually the
> > JDBC driver trying to read all the rows into memory at once and you'll
> > have to explore the jdbc driver settings in detail.
> >
> > Best,
> > Erick
> > On Tue, Sep 18, 2018 at 11:16 PM Tanya Bompi 
> > wrote:
> > >
> > > Hi,
> > >   I am using the Microsoft Jdbc driver 6.4 version in Solr 7.4.0 . I have
> > > tried removing the selectMethod=Cursor and still it runs out of heap
> > space.
> > > Do we have anyone who has faced similar issue.
> > >
> > > Thanks
> > > Tanya
> > >
> > >
> > > On Tue, Sep 18, 2018 at 6:38 PM Shawn Heisey 
> > wrote:
> > >
> > > > On 9/18/2018 4:48 PM, Tanya Bompi wrote:
> > > > >I have the SOLR 7.0 setup with the DataImportHandler connecting
> > to the
> > > > > sql server db. I keep getting OutOfMemory: Java Heap Space when
> > doing a
> > > > > full import. The size of the records is around 3 million so not very
> > > > huge.
> > > > > I tried the following steps and nothing helped thus far.
> > > >
> > > > See this wiki page:
> > > >
> > > > https://wiki.apache.org/solr/DataImportHandlerFaq
> > > >
> > > > You already have the suggested fix -- setting responseBuffering to
> > > > adaptive.  You might try upgrading the driver.  If that doesn't work,
> > > > you're probably going to need to talk to Microsoft about what you need
> > > > to do differently on the JDBC url.
> > > >
> > > > I did find this page:
> > > >
> > > >
> > > >
> > https://docs.microsoft.com/en-us/sql/connect/jdbc/using-adaptive-buffering?view=sql-server-2017
> > > >
> > > > This says that when using adaptive buffering, you should avoid using
> > > > selectMethod=cursor.  So you should try removing that parameter.
> > > >
> > > > Thanks,
> > > > Shawn
> > > >
> > > >
> >

Re: SOLR 7.0 DIH out of memory issue with sqlserver

[Tanya Bompi]

Hi Erick,
  Thank you for the follow-up. I have resolved the issue with the increase
in heapSize and I am able to set the SOLR VM to initialize with a  3G heap
size and the subset of 1 mil records was fetched successfully. Although it
fails with the entire 3 mil records. So something is off with the adaptive
buffering setting as I see  it is not helping. And I also set the
autosoftcommit param. I might have to increase the heap size further to see
if it helps. I will keep posted if my issue doesnt resolve.

Thanks,
Tanya

On Wed, Sep 19, 2018 at 8:22 AM Erick Erickson 
wrote:

> Has this ever worked? IOW, is this something that's changed or has
> just never worked?
>
> The obvious first step is to start Solr with more than 1G of memory.
> Solr _likes_ memory and a 1G heap is quite small. But you say:
> "Increasing the heap size further doesnt start SOLR instance itself.".
> How much RAM do you have on your machine? What other programs are
> running? You should be able to increase the heap and start Solr if you
> have the RAM on your machine so I'd figure out what's behind that
> issue first. I regularly start Solr with 16 or 32G of memory on my
> local machines, I know of installations running Solr with 60G heaps so
> this points to something really odd about your environment.
>
> When you "increase it further", exactly _how_ does Solr fail to start?
> What appears in the Solr logs? etc. Really, troubleshoot that issue
> first I'd recommend.
>
> If DIH still needs a ridiculous amount of memory, it's usually the
> JDBC driver trying to read all the rows into memory at once and you'll
> have to explore the jdbc driver settings in detail.
>
> Best,
> Erick
> On Tue, Sep 18, 2018 at 11:16 PM Tanya Bompi 
> wrote:
> >
> > Hi,
> >   I am using the Microsoft Jdbc driver 6.4 version in Solr 7.4.0 . I have
> > tried removing the selectMethod=Cursor and still it runs out of heap
> space.
> > Do we have anyone who has faced similar issue.
> >
> > Thanks
> > Tanya
> >
> >
> > On Tue, Sep 18, 2018 at 6:38 PM Shawn Heisey 
> wrote:
> >
> > > On 9/18/2018 4:48 PM, Tanya Bompi wrote:
> > > >I have the SOLR 7.0 setup with the DataImportHandler connecting
> to the
> > > > sql server db. I keep getting OutOfMemory: Java Heap Space when
> doing a
> > > > full import. The size of the records is around 3 million so not very
> > > huge.
> > > > I tried the following steps and nothing helped thus far.
> > >
> > > See this wiki page:
> > >
> > > https://wiki.apache.org/solr/DataImportHandlerFaq
> > >
> > > You already have the suggested fix -- setting responseBuffering to
> > > adaptive.  You might try upgrading the driver.  If that doesn't work,
> > > you're probably going to need to talk to Microsoft about what you need
> > > to do differently on the JDBC url.
> > >
> > > I did find this page:
> > >
> > >
> > >
> https://docs.microsoft.com/en-us/sql/connect/jdbc/using-adaptive-buffering?view=sql-server-2017
> > >
> > > This says that when using adaptive buffering, you should avoid using
> > > selectMethod=cursor.  So you should try removing that parameter.
> > >
> > > Thanks,
> > > Shawn
> > >
> > >
>

Re: SOLR 7.0 DIH out of memory issue with sqlserver

[Erick Erickson]

Has this ever worked? IOW, is this something that's changed or has
just never worked?

The obvious first step is to start Solr with more than 1G of memory.
Solr _likes_ memory and a 1G heap is quite small. But you say:
"Increasing the heap size further doesnt start SOLR instance itself.".
How much RAM do you have on your machine? What other programs are
running? You should be able to increase the heap and start Solr if you
have the RAM on your machine so I'd figure out what's behind that
issue first. I regularly start Solr with 16 or 32G of memory on my
local machines, I know of installations running Solr with 60G heaps so
this points to something really odd about your environment.

When you "increase it further", exactly _how_ does Solr fail to start?
What appears in the Solr logs? etc. Really, troubleshoot that issue
first I'd recommend.

If DIH still needs a ridiculous amount of memory, it's usually the
JDBC driver trying to read all the rows into memory at once and you'll
have to explore the jdbc driver settings in detail.

Best,
Erick
On Tue, Sep 18, 2018 at 11:16 PM Tanya Bompi  wrote:
>
> Hi,
>   I am using the Microsoft Jdbc driver 6.4 version in Solr 7.4.0 . I have
> tried removing the selectMethod=Cursor and still it runs out of heap space.
> Do we have anyone who has faced similar issue.
>
> Thanks
> Tanya
>
>
> On Tue, Sep 18, 2018 at 6:38 PM Shawn Heisey  wrote:
>
> > On 9/18/2018 4:48 PM, Tanya Bompi wrote:
> > >I have the SOLR 7.0 setup with the DataImportHandler connecting to the
> > > sql server db. I keep getting OutOfMemory: Java Heap Space when doing a
> > > full import. The size of the records is around 3 million so not very
> > huge.
> > > I tried the following steps and nothing helped thus far.
> >
> > See this wiki page:
> >
> > https://wiki.apache.org/solr/DataImportHandlerFaq
> >
> > You already have the suggested fix -- setting responseBuffering to
> > adaptive.  You might try upgrading the driver.  If that doesn't work,
> > you're probably going to need to talk to Microsoft about what you need
> > to do differently on the JDBC url.
> >
> > I did find this page:
> >
> >
> > https://docs.microsoft.com/en-us/sql/connect/jdbc/using-adaptive-buffering?view=sql-server-2017
> >
> > This says that when using adaptive buffering, you should avoid using
> > selectMethod=cursor.  So you should try removing that parameter.
> >
> > Thanks,
> > Shawn
> >
> >

Re: SOLR 7.0 DIH out of memory issue with sqlserver

[Tanya Bompi]

Hi,
  I am using the Microsoft Jdbc driver 6.4 version in Solr 7.4.0 . I have
tried removing the selectMethod=Cursor and still it runs out of heap space.
Do we have anyone who has faced similar issue.

Thanks
Tanya


On Tue, Sep 18, 2018 at 6:38 PM Shawn Heisey  wrote:

> On 9/18/2018 4:48 PM, Tanya Bompi wrote:
> >I have the SOLR 7.0 setup with the DataImportHandler connecting to the
> > sql server db. I keep getting OutOfMemory: Java Heap Space when doing a
> > full import. The size of the records is around 3 million so not very
> huge.
> > I tried the following steps and nothing helped thus far.
>
> See this wiki page:
>
> https://wiki.apache.org/solr/DataImportHandlerFaq
>
> You already have the suggested fix -- setting responseBuffering to
> adaptive.  You might try upgrading the driver.  If that doesn't work,
> you're probably going to need to talk to Microsoft about what you need
> to do differently on the JDBC url.
>
> I did find this page:
>
>
> https://docs.microsoft.com/en-us/sql/connect/jdbc/using-adaptive-buffering?view=sql-server-2017
>
> This says that when using adaptive buffering, you should avoid using
> selectMethod=cursor.  So you should try removing that parameter.
>
> Thanks,
> Shawn
>
>

Re: SOLR 7.0 DIH out of memory issue with sqlserver

[Shawn Heisey]

On 9/18/2018 4:48 PM, Tanya Bompi wrote:

   I have the SOLR 7.0 setup with the DataImportHandler connecting to the
sql server db. I keep getting OutOfMemory: Java Heap Space when doing a
full import. The size of the records is around 3 million so not very huge.
I tried the following steps and nothing helped thus far.


See this wiki page:

https://wiki.apache.org/solr/DataImportHandlerFaq

You already have the suggested fix -- setting responseBuffering to 
adaptive.  You might try upgrading the driver.  If that doesn't work, 
you're probably going to need to talk to Microsoft about what you need 
to do differently on the JDBC url.


I did find this page:

https://docs.microsoft.com/en-us/sql/connect/jdbc/using-adaptive-buffering?view=sql-server-2017

This says that when using adaptive buffering, you should avoid using 
selectMethod=cursor.  So you should try removing that parameter.


Thanks,
Shawn

SOLR 7.0 DIH out of memory issue with sqlserver

[Tanya Bompi]

Hi,
  I have the SOLR 7.0 setup with the DataImportHandler connecting to the
sql server db. I keep getting OutOfMemory: Java Heap Space when doing a
full import. The size of the records is around 3 million so not very huge.
I tried the following steps and nothing helped thus far.

1. Setting the "responseBuffering=adaptive;selectMethod=Cursor" in the jdbc
connection string.
2. Setting the batchSize="-1" which hasnt helped
3. Increasing the heap size at solr startup by issuing the command \solr
start -m 1024m -p 8983
Increasing the heap size further doesnt start SOLR instance itself.

I am wondering what could be causing the issue and how to resolve this.
Below is the data-config :


  
  

Thanks,
Tanya

Re: 7.3.1: Query of death - all nodes ran out of memory and had to be shut down

[Ash Ramesh]

Thank you all :) We have made the necessary changes to mitigate this issue

On Wed, Aug 22, 2018 at 6:01 AM Shawn Heisey  wrote:

> On 8/20/2018 9:55 PM, Ash Ramesh wrote:
> > We ran a bunch of deep paginated queries (offset of 1,000,000) with a
> > filter query. We set the timeout to 5 seconds and it did timeout. We
> aren't
> > sure if this is what caused the irrecoverable failure, but by reading
> this
> > -
> >
> https://lucene.apache.org/solr/guide/7_4/pagination-of-results.html#performance-problems-with-deep-paging
> > , we feel that this was the cause.
>
> Yes, this is most likely the cause.
>
> Since you have three shards, the problem is even worse than Erick
> described.  Those 110 results will be returned by EVERY shard, and
> consolidated on the machine that's actually making the query.  So it
> will have three million results in memory that it must sort.
>
> Unless you're running on Windows, the bin/solr script will configure
> Java to kill itself when OutOfMemoryError occurs.  It does this because
> program behavior after OOME occurs is completely unpredictable, so
> there's a good chance that if it keeps running, it will corrupt the index.
>
> If you're going to be doing queries like this, you need a larger heap.
> There's no way around that.
>
> Thanks,
> Shawn
>
>

-- 
*P.S. We've launched a new blog to share the latest ideas and case studies 
from our team. Check it out here: product.canva.com 
. ***
** Empowering the world 
to design
Also, we're hiring. Apply here! 

  
  

Re: 7.3.1: Query of death - all nodes ran out of memory and had to be shut down

[Shawn Heisey]

On 8/20/2018 9:55 PM, Ash Ramesh wrote:

We ran a bunch of deep paginated queries (offset of 1,000,000) with a
filter query. We set the timeout to 5 seconds and it did timeout. We aren't
sure if this is what caused the irrecoverable failure, but by reading this
-
https://lucene.apache.org/solr/guide/7_4/pagination-of-results.html#performance-problems-with-deep-paging
, we feel that this was the cause.


Yes, this is most likely the cause.

Since you have three shards, the problem is even worse than Erick 
described.  Those 110 results will be returned by EVERY shard, and 
consolidated on the machine that's actually making the query.  So it 
will have three million results in memory that it must sort.


Unless you're running on Windows, the bin/solr script will configure 
Java to kill itself when OutOfMemoryError occurs.  It does this because 
program behavior after OOME occurs is completely unpredictable, so 
there's a good chance that if it keeps running, it will corrupt the index.


If you're going to be doing queries like this, you need a larger heap.  
There's no way around that.


Thanks,
Shawn

Re: 7.3.1: Query of death - all nodes ran out of memory and had to be shut down

[Erick Erickson]

bq. I meant to ask whether there is a high probability that that could
be the correlated cause for the issue.

Yes, I do tend to be pedantic on occasion, a personal failing ;)


bq. Do you know why Solr itself isn't able to recover or is that to be
expected with allowing such deep pagination.

The general problem isn't deep paging itself, but the fact that in
your case it generates an OOM. After an OOM there is no way to
recover; the state of the program is unknown. This is pretty much true
of all Java programs, which is why there's an OOM-killer script you
can configure. That won't help your situation since you'll probably
dive right back into an OOM, but at least it doesn't continue to try
to work with a program that's in an unknown state.

Best,
Erick

On Tue, Aug 21, 2018 at 2:08 AM, Ere Maijala  wrote:
> Hi,
>
> Just my short comment here. It's difficult to say for someone else, but we
> identified deep paging as the definite reason for running out of memory or
> at least grinding to semi-halt because of long stop-the-world garbage
> collection pauses in an application running on a similar SolrCloud. You can
> often get away without issues as long as you only have a single shard, but
> for the reason Erick mentioned deep paging in a sharded index is a heavy
> operation.
>
> Regards,
> Ere
>
> Ash Ramesh kirjoitti 21.8.2018 klo 8.09:
>>
>> Hi Erick,
>>
>> Sorry I phrased that the wrong way. I meant to ask whether there is a high
>> probability that that could be the correlated cause for the issue. Do you
>> know why Solr itself isn't able to recover or is that to be expected with
>> allowing such deep pagination. We are going to be removing it going
>> forwards, but want to make sure that we find the root cause.
>>
>> Appreciate your help as always :)
>>
>> Ash
>>
>> On Tue, Aug 21, 2018 at 2:59 PM Erick Erickson 
>> wrote:
>>
>>> Did the large offsets _definitely_ cause the OOM? How do you expect
>>> that to be answerable? It's likely though. To return rows 1,000,000
>>> through 1,000,010 the system has to keep a list of 1,000,010 top
>>> documents. It has to be this way because you don't know (and can't
>>> guess) the score or a doc prior to, well, scoring it. And these very
>>> large structures are kept for every query being processed. Not only
>>> will that chew up memory, it'll chew up CPU cycles as well as this an
>>> ordered list.
>>>
>>> This is an anti-pattern, cursors were invented because this pattern is
>>> very costly (as you're finding out).
>>>
>>> Further, 4G isn't very much memory by modern standards.
>>>
>>> So it's very likely (but not guaranteed) that using cursors will fix
>>> this problem.
>>>
>>> Best,
>>> Erick
>>>
>>>
>>>
>>> On Mon, Aug 20, 2018 at 8:55 PM, Ash Ramesh  wrote:
>>>>
>>>> Hi everyone,
>>>>
>>>> We ran into an issue yesterday where all our ec2 machines, running solr,
>>>> ran out of memory and could not heal themselves. I'll try break down
>>>> what
>>>> happened here.
>>>>
>>>> *System Architecture:*
>>>>
>>>> - Solr Version: 7.3.1
>>>> - Replica Types: TLOG/PULL
>>>> - Num Shards: 8 (default hashing mechanism)
>>>> - Doc Count: > 20m
>>>> - Index Size: 17G
>>>> - EC2 Machine Spec: 16 Core | 32G ram | 100G SSD
>>>> - Num EC2 Machines: 7+ (scales up and down)
>>>> - Max Shards per node (one node per EC2 instance): 8 (some nodes had 4,
>>>> some had 8)
>>>> - Num TLOG shard replicas: 3 (3 copies of each shard as TLOG)
>>>> - Num PULL shard replicas: 3+
>>>> - Heap: 4G
>>>>
>>>> *What was run prior to the issue:*
>>>>
>>>> We ran these queries around 2.55pm
>>>>
>>>> We ran a bunch of deep paginated queries (offset of 1,000,000) with a
>>>> filter query. We set the timeout to 5 seconds and it did timeout. We
>>>
>>> aren't
>>>>
>>>> sure if this is what caused the irrecoverable failure, but by reading
>>>
>>> this
>>>>
>>>> -
>>>>
>>>
>>> https://lucene.apache.org/solr/guide/7_4/pagination-of-results.html#performance-problems-with-deep-paging
>>>>
>>>> , we feel that this was the cause.
>>>>
>>>> We did not use a cursor.
>&

Re: 7.3.1: Query of death - all nodes ran out of memory and had to be shut down

[Jan Høydahl]

The solution is to move to cursors, but you may as a safety net try to apply 
the RequestSanitizerComponent to disallow large offsets, see 
https://github.com/cominvent/request-sanitizer-component 
<https://github.com/cominvent/request-sanitizer-component>

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 21. aug. 2018 kl. 11:08 skrev Ere Maijala :
> 
> Hi,
> 
> Just my short comment here. It's difficult to say for someone else, but we 
> identified deep paging as the definite reason for running out of memory or at 
> least grinding to semi-halt because of long stop-the-world garbage collection 
> pauses in an application running on a similar SolrCloud. You can often get 
> away without issues as long as you only have a single shard, but for the 
> reason Erick mentioned deep paging in a sharded index is a heavy operation.
> 
> Regards,
> Ere
> 
> Ash Ramesh kirjoitti 21.8.2018 klo 8.09:
>> Hi Erick,
>> Sorry I phrased that the wrong way. I meant to ask whether there is a high
>> probability that that could be the correlated cause for the issue. Do you
>> know why Solr itself isn't able to recover or is that to be expected with
>> allowing such deep pagination. We are going to be removing it going
>> forwards, but want to make sure that we find the root cause.
>> Appreciate your help as always :)
>> Ash
>> On Tue, Aug 21, 2018 at 2:59 PM Erick Erickson 
>> wrote:
>>> Did the large offsets _definitely_ cause the OOM? How do you expect
>>> that to be answerable? It's likely though. To return rows 1,000,000
>>> through 1,000,010 the system has to keep a list of 1,000,010 top
>>> documents. It has to be this way because you don't know (and can't
>>> guess) the score or a doc prior to, well, scoring it. And these very
>>> large structures are kept for every query being processed. Not only
>>> will that chew up memory, it'll chew up CPU cycles as well as this an
>>> ordered list.
>>> 
>>> This is an anti-pattern, cursors were invented because this pattern is
>>> very costly (as you're finding out).
>>> 
>>> Further, 4G isn't very much memory by modern standards.
>>> 
>>> So it's very likely (but not guaranteed) that using cursors will fix
>>> this problem.
>>> 
>>> Best,
>>> Erick
>>> 
>>> 
>>> 
>>> On Mon, Aug 20, 2018 at 8:55 PM, Ash Ramesh  wrote:
>>>> Hi everyone,
>>>> 
>>>> We ran into an issue yesterday where all our ec2 machines, running solr,
>>>> ran out of memory and could not heal themselves. I'll try break down what
>>>> happened here.
>>>> 
>>>> *System Architecture:*
>>>> 
>>>> - Solr Version: 7.3.1
>>>> - Replica Types: TLOG/PULL
>>>> - Num Shards: 8 (default hashing mechanism)
>>>> - Doc Count: > 20m
>>>> - Index Size: 17G
>>>> - EC2 Machine Spec: 16 Core | 32G ram | 100G SSD
>>>> - Num EC2 Machines: 7+ (scales up and down)
>>>> - Max Shards per node (one node per EC2 instance): 8 (some nodes had 4,
>>>> some had 8)
>>>> - Num TLOG shard replicas: 3 (3 copies of each shard as TLOG)
>>>> - Num PULL shard replicas: 3+
>>>> - Heap: 4G
>>>> 
>>>> *What was run prior to the issue:*
>>>> 
>>>> We ran these queries around 2.55pm
>>>> 
>>>> We ran a bunch of deep paginated queries (offset of 1,000,000) with a
>>>> filter query. We set the timeout to 5 seconds and it did timeout. We
>>> aren't
>>>> sure if this is what caused the irrecoverable failure, but by reading
>>> this
>>>> -
>>>> 
>>> https://lucene.apache.org/solr/guide/7_4/pagination-of-results.html#performance-problems-with-deep-paging
>>>> , we feel that this was the cause.
>>>> 
>>>> We did not use a cursor.
>>>> 
>>>> This cluster was healthy for about 1 week, but we noticed the degradation
>>>> soon after (within 30min) of running the offset queries mentioned above.
>>> We
>>>> currently use a single sharded collection in production, however are
>>>> transitioning to an 8 shard cluster. We hit this issue in a controlled 8
>>>> sharded environment, but don't notice any issues on our production
>>> (single
>>>> sharded) cluster. On production the query still timed out (with same num
>>>> docs etc.) but didn

Re: 7.3.1: Query of death - all nodes ran out of memory and had to be shut down

[Ere Maijala]

Hi,

Just my short comment here. It's difficult to say for someone else, but 
we identified deep paging as the definite reason for running out of 
memory or at least grinding to semi-halt because of long stop-the-world 
garbage collection pauses in an application running on a similar 
SolrCloud. You can often get away without issues as long as you only 
have a single shard, but for the reason Erick mentioned deep paging in a 
sharded index is a heavy operation.


Regards,
Ere

Ash Ramesh kirjoitti 21.8.2018 klo 8.09:

Hi Erick,

Sorry I phrased that the wrong way. I meant to ask whether there is a high
probability that that could be the correlated cause for the issue. Do you
know why Solr itself isn't able to recover or is that to be expected with
allowing such deep pagination. We are going to be removing it going
forwards, but want to make sure that we find the root cause.

Appreciate your help as always :)

Ash

On Tue, Aug 21, 2018 at 2:59 PM Erick Erickson 
wrote:


Did the large offsets _definitely_ cause the OOM? How do you expect
that to be answerable? It's likely though. To return rows 1,000,000
through 1,000,010 the system has to keep a list of 1,000,010 top
documents. It has to be this way because you don't know (and can't
guess) the score or a doc prior to, well, scoring it. And these very
large structures are kept for every query being processed. Not only
will that chew up memory, it'll chew up CPU cycles as well as this an
ordered list.

This is an anti-pattern, cursors were invented because this pattern is
very costly (as you're finding out).

Further, 4G isn't very much memory by modern standards.

So it's very likely (but not guaranteed) that using cursors will fix
this problem.

Best,
Erick



On Mon, Aug 20, 2018 at 8:55 PM, Ash Ramesh  wrote:

Hi everyone,

We ran into an issue yesterday where all our ec2 machines, running solr,
ran out of memory and could not heal themselves. I'll try break down what
happened here.

*System Architecture:*

- Solr Version: 7.3.1
- Replica Types: TLOG/PULL
- Num Shards: 8 (default hashing mechanism)
- Doc Count: > 20m
- Index Size: 17G
- EC2 Machine Spec: 16 Core | 32G ram | 100G SSD
- Num EC2 Machines: 7+ (scales up and down)
- Max Shards per node (one node per EC2 instance): 8 (some nodes had 4,
some had 8)
- Num TLOG shard replicas: 3 (3 copies of each shard as TLOG)
- Num PULL shard replicas: 3+
- Heap: 4G

*What was run prior to the issue:*

We ran these queries around 2.55pm

We ran a bunch of deep paginated queries (offset of 1,000,000) with a
filter query. We set the timeout to 5 seconds and it did timeout. We

aren't

sure if this is what caused the irrecoverable failure, but by reading

this

-


https://lucene.apache.org/solr/guide/7_4/pagination-of-results.html#performance-problems-with-deep-paging

, we feel that this was the cause.

We did not use a cursor.

This cluster was healthy for about 1 week, but we noticed the degradation
soon after (within 30min) of running the offset queries mentioned above.

We

currently use a single sharded collection in production, however are
transitioning to an 8 shard cluster. We hit this issue in a controlled 8
sharded environment, but don't notice any issues on our production

(single

sharded) cluster. On production the query still timed out (with same num
docs etc.) but didn't go into a crazy state.

*What Happened:*

- All the EC2 instances started logging OOM error. None of the nodes were
responsive to new requests.
- We saw that the Heap usage jumped from an average of 2.7G to the max of
4G within a 5 minute window.
- CPU across all 16 cores was at 100%
- We saw that the distributed requests were timing out across all

machines.

- We shutdown all the machines that only had PULL replicas on them and it
still didn't 'fix' itself.
- Eventually we shut down SOLR on the main node which had all the master
TLOG replicas. Once restarted, the machine started working again.


*Questions:*
- Did this deep pagination query *DEFINITELY* cause this issue?
- Is each node single threaded? I don't think so, but I'd like to confirm
that.
- Is there any configuration that we could use to avoid this in the

future?

- Why could the nodes not recover by themselves? When we ran the same

query

on the single shard cluster it failed and didn't spin out of control.

Thanks for all your help, Logs are pasted below from different

timestamps.


Regards,
Ash

*Logs:*

Here are some logs we collected. Not sure if it tells a lot outside of

what

we know.

*Time: 2.55pm ~ Requests are failing to complete in time*


ERROR RequestHandlerBase org.apache.solr.common.SolrException:
org.apache.solr.client.solrj.SolrServerException: Time allowed to handle
this request exceeded:[
http://10.0.9.204:8983/solr/media_shard1_replica_p57,
http://10.0.9.204:8983/solr/media_shard4_replica_p80,
http://10.0.9.20

Re: 7.3.1: Query of death - all nodes ran out of memory and had to be shut down

[Ash Ramesh]

Hi Erick,

Sorry I phrased that the wrong way. I meant to ask whether there is a high
probability that that could be the correlated cause for the issue. Do you
know why Solr itself isn't able to recover or is that to be expected with
allowing such deep pagination. We are going to be removing it going
forwards, but want to make sure that we find the root cause.

Appreciate your help as always :)

Ash

On Tue, Aug 21, 2018 at 2:59 PM Erick Erickson 
wrote:

> Did the large offsets _definitely_ cause the OOM? How do you expect
> that to be answerable? It's likely though. To return rows 1,000,000
> through 1,000,010 the system has to keep a list of 1,000,010 top
> documents. It has to be this way because you don't know (and can't
> guess) the score or a doc prior to, well, scoring it. And these very
> large structures are kept for every query being processed. Not only
> will that chew up memory, it'll chew up CPU cycles as well as this an
> ordered list.
>
> This is an anti-pattern, cursors were invented because this pattern is
> very costly (as you're finding out).
>
> Further, 4G isn't very much memory by modern standards.
>
> So it's very likely (but not guaranteed) that using cursors will fix
> this problem.
>
> Best,
> Erick
>
>
>
> On Mon, Aug 20, 2018 at 8:55 PM, Ash Ramesh  wrote:
> > Hi everyone,
> >
> > We ran into an issue yesterday where all our ec2 machines, running solr,
> > ran out of memory and could not heal themselves. I'll try break down what
> > happened here.
> >
> > *System Architecture:*
> >
> > - Solr Version: 7.3.1
> > - Replica Types: TLOG/PULL
> > - Num Shards: 8 (default hashing mechanism)
> > - Doc Count: > 20m
> > - Index Size: 17G
> > - EC2 Machine Spec: 16 Core | 32G ram | 100G SSD
> > - Num EC2 Machines: 7+ (scales up and down)
> > - Max Shards per node (one node per EC2 instance): 8 (some nodes had 4,
> > some had 8)
> > - Num TLOG shard replicas: 3 (3 copies of each shard as TLOG)
> > - Num PULL shard replicas: 3+
> > - Heap: 4G
> >
> > *What was run prior to the issue:*
> >
> > We ran these queries around 2.55pm
> >
> > We ran a bunch of deep paginated queries (offset of 1,000,000) with a
> > filter query. We set the timeout to 5 seconds and it did timeout. We
> aren't
> > sure if this is what caused the irrecoverable failure, but by reading
> this
> > -
> >
> https://lucene.apache.org/solr/guide/7_4/pagination-of-results.html#performance-problems-with-deep-paging
> > , we feel that this was the cause.
> >
> > We did not use a cursor.
> >
> > This cluster was healthy for about 1 week, but we noticed the degradation
> > soon after (within 30min) of running the offset queries mentioned above.
> We
> > currently use a single sharded collection in production, however are
> > transitioning to an 8 shard cluster. We hit this issue in a controlled 8
> > sharded environment, but don't notice any issues on our production
> (single
> > sharded) cluster. On production the query still timed out (with same num
> > docs etc.) but didn't go into a crazy state.
> >
> > *What Happened:*
> >
> > - All the EC2 instances started logging OOM error. None of the nodes were
> > responsive to new requests.
> > - We saw that the Heap usage jumped from an average of 2.7G to the max of
> > 4G within a 5 minute window.
> > - CPU across all 16 cores was at 100%
> > - We saw that the distributed requests were timing out across all
> machines.
> > - We shutdown all the machines that only had PULL replicas on them and it
> > still didn't 'fix' itself.
> > - Eventually we shut down SOLR on the main node which had all the master
> > TLOG replicas. Once restarted, the machine started working again.
> >
> >
> > *Questions:*
> > - Did this deep pagination query *DEFINITELY* cause this issue?
> > - Is each node single threaded? I don't think so, but I'd like to confirm
> > that.
> > - Is there any configuration that we could use to avoid this in the
> future?
> > - Why could the nodes not recover by themselves? When we ran the same
> query
> > on the single shard cluster it failed and didn't spin out of control.
> >
> > Thanks for all your help, Logs are pasted below from different
> timestamps.
> >
> > Regards,
> > Ash
> >
> > *Logs:*
> >
> > Here are some logs we collected. Not sure if it tells a lot outside of
> what
> > we know.
> >
> > *Time: 2.55pm ~ Requests are failing to

Re: 7.3.1: Query of death - all nodes ran out of memory and had to be shut down

[Erick Erickson]

Did the large offsets _definitely_ cause the OOM? How do you expect
that to be answerable? It's likely though. To return rows 1,000,000
through 1,000,010 the system has to keep a list of 1,000,010 top
documents. It has to be this way because you don't know (and can't
guess) the score or a doc prior to, well, scoring it. And these very
large structures are kept for every query being processed. Not only
will that chew up memory, it'll chew up CPU cycles as well as this an
ordered list.

This is an anti-pattern, cursors were invented because this pattern is
very costly (as you're finding out).

Further, 4G isn't very much memory by modern standards.

So it's very likely (but not guaranteed) that using cursors will fix
this problem.

Best,
Erick



On Mon, Aug 20, 2018 at 8:55 PM, Ash Ramesh  wrote:
> Hi everyone,
>
> We ran into an issue yesterday where all our ec2 machines, running solr,
> ran out of memory and could not heal themselves. I'll try break down what
> happened here.
>
> *System Architecture:*
>
> - Solr Version: 7.3.1
> - Replica Types: TLOG/PULL
> - Num Shards: 8 (default hashing mechanism)
> - Doc Count: > 20m
> - Index Size: 17G
> - EC2 Machine Spec: 16 Core | 32G ram | 100G SSD
> - Num EC2 Machines: 7+ (scales up and down)
> - Max Shards per node (one node per EC2 instance): 8 (some nodes had 4,
> some had 8)
> - Num TLOG shard replicas: 3 (3 copies of each shard as TLOG)
> - Num PULL shard replicas: 3+
> - Heap: 4G
>
> *What was run prior to the issue:*
>
> We ran these queries around 2.55pm
>
> We ran a bunch of deep paginated queries (offset of 1,000,000) with a
> filter query. We set the timeout to 5 seconds and it did timeout. We aren't
> sure if this is what caused the irrecoverable failure, but by reading this
> -
> https://lucene.apache.org/solr/guide/7_4/pagination-of-results.html#performance-problems-with-deep-paging
> , we feel that this was the cause.
>
> We did not use a cursor.
>
> This cluster was healthy for about 1 week, but we noticed the degradation
> soon after (within 30min) of running the offset queries mentioned above. We
> currently use a single sharded collection in production, however are
> transitioning to an 8 shard cluster. We hit this issue in a controlled 8
> sharded environment, but don't notice any issues on our production (single
> sharded) cluster. On production the query still timed out (with same num
> docs etc.) but didn't go into a crazy state.
>
> *What Happened:*
>
> - All the EC2 instances started logging OOM error. None of the nodes were
> responsive to new requests.
> - We saw that the Heap usage jumped from an average of 2.7G to the max of
> 4G within a 5 minute window.
> - CPU across all 16 cores was at 100%
> - We saw that the distributed requests were timing out across all machines.
> - We shutdown all the machines that only had PULL replicas on them and it
> still didn't 'fix' itself.
> - Eventually we shut down SOLR on the main node which had all the master
> TLOG replicas. Once restarted, the machine started working again.
>
>
> *Questions:*
> - Did this deep pagination query *DEFINITELY* cause this issue?
> - Is each node single threaded? I don't think so, but I'd like to confirm
> that.
> - Is there any configuration that we could use to avoid this in the future?
> - Why could the nodes not recover by themselves? When we ran the same query
> on the single shard cluster it failed and didn't spin out of control.
>
> Thanks for all your help, Logs are pasted below from different timestamps.
>
> Regards,
> Ash
>
> *Logs:*
>
> Here are some logs we collected. Not sure if it tells a lot outside of what
> we know.
>
> *Time: 2.55pm ~ Requests are failing to complete in time*
>
>> ERROR RequestHandlerBase org.apache.solr.common.SolrException:
>> org.apache.solr.client.solrj.SolrServerException: Time allowed to handle
>> this request exceeded:[
>> http://10.0.9.204:8983/solr/media_shard1_replica_p57,
>> http://10.0.9.204:8983/solr/media_shard4_replica_p80,
>> http://10.0.9.204:8983/solr/media_shard3_replica_p73,
>> http://10.0.9.204:8983/solr/media_shard2_replica_p68]
>> #011at
>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:410)
>> #011at
>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
>> #011at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)
>> #011at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)
>> #011at org.apache.s...
>>  #011at
>> org.eclipse.jetty.server.handler.ScopedHandler.nextHan

7.3.1: Query of death - all nodes ran out of memory and had to be shut down

[Ash Ramesh]

Hi everyone,

We ran into an issue yesterday where all our ec2 machines, running solr,
ran out of memory and could not heal themselves. I'll try break down what
happened here.

*System Architecture:*

- Solr Version: 7.3.1
- Replica Types: TLOG/PULL
- Num Shards: 8 (default hashing mechanism)
- Doc Count: > 20m
- Index Size: 17G
- EC2 Machine Spec: 16 Core | 32G ram | 100G SSD
- Num EC2 Machines: 7+ (scales up and down)
- Max Shards per node (one node per EC2 instance): 8 (some nodes had 4,
some had 8)
- Num TLOG shard replicas: 3 (3 copies of each shard as TLOG)
- Num PULL shard replicas: 3+
- Heap: 4G

*What was run prior to the issue:*

We ran these queries around 2.55pm

We ran a bunch of deep paginated queries (offset of 1,000,000) with a
filter query. We set the timeout to 5 seconds and it did timeout. We aren't
sure if this is what caused the irrecoverable failure, but by reading this
-
https://lucene.apache.org/solr/guide/7_4/pagination-of-results.html#performance-problems-with-deep-paging
, we feel that this was the cause.

We did not use a cursor.

This cluster was healthy for about 1 week, but we noticed the degradation
soon after (within 30min) of running the offset queries mentioned above. We
currently use a single sharded collection in production, however are
transitioning to an 8 shard cluster. We hit this issue in a controlled 8
sharded environment, but don't notice any issues on our production (single
sharded) cluster. On production the query still timed out (with same num
docs etc.) but didn't go into a crazy state.

*What Happened:*

- All the EC2 instances started logging OOM error. None of the nodes were
responsive to new requests.
- We saw that the Heap usage jumped from an average of 2.7G to the max of
4G within a 5 minute window.
- CPU across all 16 cores was at 100%
- We saw that the distributed requests were timing out across all machines.
- We shutdown all the machines that only had PULL replicas on them and it
still didn't 'fix' itself.
- Eventually we shut down SOLR on the main node which had all the master
TLOG replicas. Once restarted, the machine started working again.


*Questions:*
- Did this deep pagination query *DEFINITELY* cause this issue?
- Is each node single threaded? I don't think so, but I'd like to confirm
that.
- Is there any configuration that we could use to avoid this in the future?
- Why could the nodes not recover by themselves? When we ran the same query
on the single shard cluster it failed and didn't spin out of control.

Thanks for all your help, Logs are pasted below from different timestamps.

Regards,
Ash

*Logs:*

Here are some logs we collected. Not sure if it tells a lot outside of what
we know.

*Time: 2.55pm ~ Requests are failing to complete in time*

> ERROR RequestHandlerBase org.apache.solr.common.SolrException:
> org.apache.solr.client.solrj.SolrServerException: Time allowed to handle
> this request exceeded:[
> http://10.0.9.204:8983/solr/media_shard1_replica_p57,
> http://10.0.9.204:8983/solr/media_shard4_replica_p80,
> http://10.0.9.204:8983/solr/media_shard3_replica_p73,
> http://10.0.9.204:8983/solr/media_shard2_replica_p68]
> #011at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:410)
> #011at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
> #011at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)
> #011at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:711)
> #011at org.apache.s...
>  #011at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
>  #011at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
>  #011at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
>  #011at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
>  #011at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
>  #011at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
>  #011at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>  #011at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
>  #011at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
>  #011at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>  #011at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
>  #011at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
>  #011at org.eclipse.jetty.server.Server.handle(Server.java:530)
>  #011at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
>  #011at
> org.ecli

Re: Solr Nodes Killed During a ReIndexing Process on New VMs Out of Memory Error

[THADC]

Thanks, made heap size considerably larger and its fine now. Thank you



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr Nodes Killed During a ReIndexing Process on New VMs Out of Memory Error

[Shawn Heisey]

On 7/18/2018 8:31 AM, THADC wrote:

Thanks for the reply. I read the link you provided. I am currently not
specifying a heap size with solr so my understanding is that by default it
will just grow automatically. If I add more physical memory to the VM
without doing anything with heap size, won't that possibly fix the problem?


No, that is not how it works.  If Java is not given a heap size, then it 
will choose the heap size for you based on how much memory the machine 
has and its own internal algorithms, and limit itself to that amount.


Solr 5.0 and later, when started using the included scripts, asks Java 
for a 512MB heap by default.  This is an extremely small heap.  Nearly 
all Solr users must increase the heap size beyond 512MB.


Thanks,
Shawn

Re: Solr Nodes Killed During a ReIndexing Process on New VMs Out of Memory Error

[THADC]

Thanks for the reply. I read the link you provided. I am currently not
specifying a heap size with solr so my understanding is that by default it
will just grow automatically. If I add more physical memory to the VM
without doing anything with heap size, won't that possibly fix the problem?

Thanks



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr Nodes Killed During a ReIndexing Process on New VMs Out of Memory Error

[Shawn Heisey]

On 7/18/2018 7:10 AM, THADC wrote:

We performed a full reindex for the first time against our largest database
and on two new VMs dedicated to solr indexing. We have two solr nodes
(solrCloud/solr7.3) with a zookeeper cluster. Several hours into the
reindexing process, both solr nodes shut down with:

java.long.OutOfMemoryError: Java heap space

Running OOM killer script for process blah on port blah

Does this definitely indicate we need more memory or could it just be a heap
space adjustment issue? Is there a way to better diagnose what to do?


https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap

There are exactly two ways to deal with OOME:  Increase the available 
amount of the resource that's running out (heap space in this case), or 
change something so the program requires less of that resource.  
Depending on the hardware and software configuration, either of these 
options might turn out to be impossible.


The rest of that wiki page discusses memory in general as well as heap 
memory.  If you have questions after reading the page, go ahead and ask 
them.


Thanks,
Shawn

Solr Nodes Killed During a ReIndexing Process on New VMs Out of Memory Error

[THADC]

Hi,

We performed a full reindex for the first time against our largest database
and on two new VMs dedicated to solr indexing. We have two solr nodes
(solrCloud/solr7.3) with a zookeeper cluster. Several hours into the
reindexing process, both solr nodes shut down with:

java.long.OutOfMemoryError: Java heap space

Running OOM killer script for process blah on port blah

Does this definitely indicate we need more memory or could it just be a heap
space adjustment issue? Is there a way to better diagnose what to do?

Thanks for any reply.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Multiple consecutive wildcards (**) causes Out-of-memory

[Bjarke Buur Mortensen]

Just to clarify:
I can only cause this to happen when using the complexphrase query parser.
Lucene/dismax/edismax parsers are not affected.

2018-02-07 13:09 GMT+01:00 Bjarke Buur Mortensen :

> Hello list,
>
> Whenever I make a query for ** (two consecutive wildcards) it causes my
> Solr to run out of memory.
>
> http://localhost:8983/solr/select?q=**
>
> Why is that?
>
> I realize that this is not a reasonable query to make, but the system
> supports input from users, and they might by accident input this query,
> causing Solr to crash.
>
> I should add that we use the complexphrase query parser as the default
> parser on a Solr 7.1.
>
> Can anyone repro this or explain what causes the problem?
>
> Thanks in advance,
> Bjarke Buur Mortensen
> Senior Software Engineer
> Eluence A/S
>
>
>
>
>
>
>

Multiple consecutive wildcards (**) causes Out-of-memory

[Bjarke Buur Mortensen]

Hello list,

Whenever I make a query for ** (two consecutive wildcards) it causes my
Solr to run out of memory.

http://localhost:8983/solr/select?q=**

Why is that?

I realize that this is not a reasonable query to make, but the system
supports input from users, and they might by accident input this query,
causing Solr to crash.

I should add that we use the complexphrase query parser as the default
parser on a Solr 7.1.

Can anyone repro this or explain what causes the problem?

Thanks in advance,
Bjarke Buur Mortensen
Senior Software Engineer
Eluence A/S

Re: With 100% CPU usage giving out of memory exception and solr is not responding

[prathap]

What is your Xmx?  - 20GB
* How many documents in your index? 12GB
* What is your filterCache size? 512 MB



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: With 100% CPU usage giving out of memory exception and solr is not responding

[prathap]

What is your Xmx? 20GB
* How many documents in your index? 12GB
* What is your filterCache size? 512 MB



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: With 100% CPU usage giving out of memory exception and solr is not responding

[Toke Eskildsen]

prathap  wrote:
> ERROR - 2017-12-21 08:39:13.326; org.apache.solr.common.SolrException;
> null:java.lang.RuntimeException: java.lang.OutOfMemoryError: GC overhead
> limit exceeded
...
> INFO  - 2017-12-21 08:43:43.351; org.apache.solr.core.SolrCore;
> [SA_PROD_SPK_QC] webapp=/solr path=/select
> params={q=430+Gl&fq=channel:(20052)&fq=languageId:(9)&rows=6&wt=xml
> &version=2.2} hits=0 status=0 QTime=6732700

Seems like your heap is too small for what your are doing. Common culprits are 
a combination of many documents in the index and a large filterCache or 
faceting on a high-cardinality field.

If the query above is representative of your general queries, I'll guess it's 
the many docs + large filterCache one. It's fairly easy to check:
* What is your Xmx?
* How many documents in your index?
* What is your filterCache size?

- Toke Eskildsen

With 100% CPU usage giving out of memory exception and solr is not responding

[prathap]

ERROR - 2017-12-21 08:39:13.326; org.apache.solr.common.SolrException;
null:java.lang.RuntimeException: java.lang.OutOfMemoryError: GC overhead
limit exceeded
at
org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:793)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:434)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Unknown
Source)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(Unknown 
Source)
at org.apache.catalina.core.StandardWrapperValve.invoke(Unknown Source)
at org.apache.catalina.core.StandardContextValve.invoke(Unknown Source)
at org.apache.catalina.core.StandardHostValve.invoke(Unknown Source)
at org.apache.catalina.valves.ErrorReportValve.invoke(Unknown Source)
at org.apache.catalina.valves.AccessLogValve.invoke(Unknown Source)
at org.apache.catalina.core.StandardEngineValve.invoke(Unknown Source)
at org.apache.catalina.connector.CoyoteAdapter.service(Unknown Source)
at org.apache.coyote.http11.AbstractHttp11Processor.process(Unknown 
Source)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Unknown
Source)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(Unknown
Source)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Unknown
Source)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

INFO  - 2017-12-21 08:43:43.351; org.apache.solr.core.SolrCore;
[SA_PROD_SPK_QC] webapp=/solr path=/select
params={q=430+Gl&fq=channel:(20052)&fq=languageId:(9)&rows=6&wt=xml&version=2.2}
hits=0 status=0 QTime=6732700 
ERROR - 2017-12-21 08:43:43.351; org.apache.solr.common.SolrException;
null:org.apache.catalina.connector.ClientAbortException: java.io.IOException
at org.apache.catalina.connector.OutputBuffer.doFlush(Unknown Source)
at org.apache.catalina.connector.OutputBuffer.flush(Unknown Source)
at org.apache.catalina.connector.CoyoteOutputStream.flush(Unknown 
Source)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:297)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
at org.apache.solr.util.FastWriter.flush(FastWriter.java:137)
at
org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:766)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:426)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Unknown
Source)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(Unknown 
Source)
at org.apache.catalina.core.StandardWrapperValve.invoke(Unknown Source)
at org.apache.catalina.core.StandardContextValve.invoke(Unknown Source)
at org.apache.catalina.core.StandardHostValve.invoke(Unknown Source)
at org.apache.catalina.valves.ErrorReportValve.invoke(Unknown Source)
at org.apache.catalina.valves.AccessLogValve.invoke(Unknown Source)
at org.apache.catalina.core.StandardEngineValve.invoke(Unknown Source)
at org.apache.catalina.connector.CoyoteAdapter.service(Unknown Source)
at org.apache.coyote.http11.AbstractHttp11Processor.process(Unknown 
Source)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Unknown
Source)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(Unknown
Source)
at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Unknown
Source)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException
at org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(Unknown
Source)
at org.apache.coyote.http11.InternalAprOutputBuffer.flush(Unknown 
Source)
at org.apache.coyote.http11.AbstractHttp11Processor.action(Unknown 
Source)
at org.apache.coyote.Response.action(Unknown Source)
... 27 more



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Solr5.1 delete out of memory

[Shawn Heisey]

On 12/17/2017 9:36 PM, soul wrote:

hi! I'm using solr5.1. There have about 0.3 billion doc in my solr.
I can insert and  select doc in my solr, while failed to delete doc. It
remind me that this writer hit an OutOfMemoryError : cannot commit.
I am curious that what cause this reason?


The nature of the problem is in the exception class name.  I can be 
fairly sure that the message following the error is NOT "cannot commit", 
though.  Can you make the full error message available, with the 
possibly dozens of lines of stacktrace?


The most common message that people get with OOME is this:

java.lang.OutOfMemoryError: Java heap space

But there are several other messages you can get with OutOfMemoryError. 
Another common message is "unable to create new native Thread".


If you are getting the heap space message, then there are exactly two 
ways to deal with it.  One is to increase the heap size.  The other is 
to change Solr's configuration or the nature of your queries to reduce 
the amount of memory that Solr needs.


https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap

My *guess* is that you're going to need to make Solr's heap larger, but 
depending on the exact error you're getting, this may not help.


Thanks,
Shawn

Solr5.1 delete out of memory

[soul]

hi! I'm using solr5.1. There have about 0.3 billion doc in my solr.
I can insert and  select doc in my solr, while failed to delete doc. It
remind me that this writer hit an OutOfMemoryError : cannot commit.
I am curious that what cause this reason?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

RE: DataImport Handler Out of Memory

[Allison, Timothy B.]

https://wiki.apache.org/solr/DataImportHandlerFaq#I.27m_using_DataImportHandler_with_a_MySQL_database._My_table_is_huge_and_DataImportHandler_is_going_out_of_memory._Why_does_DataImportHandler_bring_everything_to_memory.3F


-Original Message-
From: Deeksha Sharma [mailto:dsha...@flexera.com] 
Sent: Wednesday, September 27, 2017 1:40 PM
To: solr-user@lucene.apache.org
Subject: DataImport Handler Out of Memory

I am trying to create indexes using dataimport handler (Solr 5.2.1). Data is in 
mysql db and the number of records are more than 3.5 million. My solr server 
stops due to OOM (out of memory error). I tried starting solr by giving 12GB of 
RAM but still no luck.


Also, I see that Solr fetches all the documents in 1 request. Is there a way to 
configure Solr to stream the data from DB or any other solution somewhere may 
have tried?


Note: When my records are nearly 2 Million, I am able to create indexes by 
giving Solr 10GB of RAM.


Your help is appreciated.



Thanks

Deeksha

DataImport Handler Out of Memory

[Deeksha Sharma]

I am trying to create indexes using dataimport handler (Solr 5.2.1). Data is in 
mysql db and the number of records are more than 3.5 million. My solr server 
stops due to OOM (out of memory error). I tried starting solr by giving 12GB of 
RAM but still no luck.


Also, I see that Solr fetches all the documents in 1 request. Is there a way to 
configure Solr to stream the data from DB or any other solution somewhere may 
have tried?


Note: When my records are nearly 2 Million, I am able to create indexes by 
giving Solr 10GB of RAM.


Your help is appreciated.



Thanks

Deeksha

Re: Out of Memory Errors

[Susheel Kumar]

The attachment will not come thru.  Can you upload thru dropbox / other
sharing sites etc.

On Wed, Jun 14, 2017 at 12:41 PM, Satya Marivada 
wrote:

> Susheel, Please see attached. There  heap towards the end of graph has
> spiked
>
>
>
> On Wed, Jun 14, 2017 at 11:46 AM Susheel Kumar 
> wrote:
>
>> You may have gc logs saved when OOM happened. Can you draw it in GC Viewer
>> or so and share.
>>
>> Thnx
>>
>> On Wed, Jun 14, 2017 at 11:26 AM, Satya Marivada <
>> satya.chaita...@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > I am getting Out of Memory Errors after a while on solr-6.3.0.
>> > The -XX:OnOutOfMemoryError=/sanfs/mnt/vol01/solr/solr-6.3.0/bin/
>> oom_solr.sh
>> > just kills the jvm right after.
>> > Using Jconsole, I see the nice triangle pattern, where it uses the heap
>> > and being reclaimed back.
>> >
>> > The heap size is set at 3g. The index size hosted on that particular
>> node
>> > is 17G.
>> >
>> > java -server -Xms3g -Xmx3g -XX:NewRatio=3 -XX:SurvivorRatio=4
>> > -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
>> > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4
>> > -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark
>> > -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:
>> > CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000
>> >
>> > Looking at the solr_gc.log.0, the eden space is being used 100% all the
>> > while and being successfully reclaimed. So don't think that has go to do
>> > with it.
>> >
>> > Apart from that in the solr.log, I see exceptions that are aftermath of
>> > killing the jvm
>> >
>> > org.eclipse.jetty.io.EofException: Closed
>> > at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.
>> java:383)
>> > at org.apache.commons.io.output.ProxyOutputStream.write(
>> > ProxyOutputStream.java:90)
>> > at org.apache.solr.common.util.FastOutputStream.flush(
>> > FastOutputStream.java:213)
>> > at org.apache.solr.common.util.FastOutputStream.flushBuffer(
>> > FastOutputStream.java:206)
>> > at org.apache.solr.common.util.JavaBinCodec.marshal(
>> > JavaBinCodec.java:136)
>> >
>> > Any suggestions on how to go about it.
>> >
>> > Thanks,
>> > Satya
>> >
>>
>

Re: Out of Memory Errors

[Satya Marivada]

Susheel, Please see attached. There  heap towards the end of graph has
spiked



On Wed, Jun 14, 2017 at 11:46 AM Susheel Kumar 
wrote:

> You may have gc logs saved when OOM happened. Can you draw it in GC Viewer
> or so and share.
>
> Thnx
>
> On Wed, Jun 14, 2017 at 11:26 AM, Satya Marivada <
> satya.chaita...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I am getting Out of Memory Errors after a while on solr-6.3.0.
> > The
> -XX:OnOutOfMemoryError=/sanfs/mnt/vol01/solr/solr-6.3.0/bin/oom_solr.sh
> > just kills the jvm right after.
> > Using Jconsole, I see the nice triangle pattern, where it uses the heap
> > and being reclaimed back.
> >
> > The heap size is set at 3g. The index size hosted on that particular node
> > is 17G.
> >
> > java -server -Xms3g -Xmx3g -XX:NewRatio=3 -XX:SurvivorRatio=4
> > -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> > -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4
> > -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark
> > -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:
> > CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000
> >
> > Looking at the solr_gc.log.0, the eden space is being used 100% all the
> > while and being successfully reclaimed. So don't think that has go to do
> > with it.
> >
> > Apart from that in the solr.log, I see exceptions that are aftermath of
> > killing the jvm
> >
> > org.eclipse.jetty.io.EofException: Closed
> > at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:383)
> > at org.apache.commons.io.output.ProxyOutputStream.write(
> > ProxyOutputStream.java:90)
> > at org.apache.solr.common.util.FastOutputStream.flush(
> > FastOutputStream.java:213)
> > at org.apache.solr.common.util.FastOutputStream.flushBuffer(
> > FastOutputStream.java:206)
> > at org.apache.solr.common.util.JavaBinCodec.marshal(
> > JavaBinCodec.java:136)
> >
> > Any suggestions on how to go about it.
> >
> > Thanks,
> > Satya
> >
>

Re: Out of Memory Errors

[Susheel Kumar]

You may have gc logs saved when OOM happened. Can you draw it in GC Viewer
or so and share.

Thnx

On Wed, Jun 14, 2017 at 11:26 AM, Satya Marivada 
wrote:

> Hi,
>
> I am getting Out of Memory Errors after a while on solr-6.3.0.
> The -XX:OnOutOfMemoryError=/sanfs/mnt/vol01/solr/solr-6.3.0/bin/oom_solr.sh
> just kills the jvm right after.
> Using Jconsole, I see the nice triangle pattern, where it uses the heap
> and being reclaimed back.
>
> The heap size is set at 3g. The index size hosted on that particular node
> is 17G.
>
> java -server -Xms3g -Xmx3g -XX:NewRatio=3 -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4
> -XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark
> -XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly -XX:
> CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000
>
> Looking at the solr_gc.log.0, the eden space is being used 100% all the
> while and being successfully reclaimed. So don't think that has go to do
> with it.
>
> Apart from that in the solr.log, I see exceptions that are aftermath of
> killing the jvm
>
> org.eclipse.jetty.io.EofException: Closed
> at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:383)
> at org.apache.commons.io.output.ProxyOutputStream.write(
> ProxyOutputStream.java:90)
> at org.apache.solr.common.util.FastOutputStream.flush(
> FastOutputStream.java:213)
> at org.apache.solr.common.util.FastOutputStream.flushBuffer(
> FastOutputStream.java:206)
> at org.apache.solr.common.util.JavaBinCodec.marshal(
> JavaBinCodec.java:136)
>
> Any suggestions on how to go about it.
>
> Thanks,
> Satya
>

Out of Memory Errors

[Satya Marivada]

Hi,

I am getting Out of Memory Errors after a while on solr-6.3.0.
The -XX:OnOutOfMemoryError=/sanfs/mnt/vol01/solr/solr-6.3.0/bin/oom_solr.sh
just kills the jvm right after.
Using Jconsole, I see the nice triangle pattern, where it uses the heap and
being reclaimed back.

The heap size is set at 3g. The index size hosted on that particular node
is 17G.

java -server -Xms3g -Xmx3g -XX:NewRatio=3 -XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ConcGCThreads=4
-XX:ParallelGCThreads=4 -XX:+CMSScavengeBeforeRemark
-XX:PretenureSizeThreshold=64m -XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50 -XX:CMSMaxAbortablePrecleanTime=6000

Looking at the solr_gc.log.0, the eden space is being used 100% all the
while and being successfully reclaimed. So don't think that has go to do
with it.

Apart from that in the solr.log, I see exceptions that are aftermath of
killing the jvm

org.eclipse.jetty.io.EofException: Closed
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:383)
at
org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:90)
at
org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:213)
at
org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:206)
at
org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:136)

Any suggestions on how to go about it.

Thanks,
Satya

Re: Solr Delete By Id Out of memory issue

[Rohit Kanchan]

Thanks everyone for replying to this issue. Just a final comment on this
issue which I was closely working on. We have fixed this issue. It was a
bug in our custom component which we wrote to convert delete by query to
delete by id. We were using BytesRef differently, we were not making a deep
copy. It was causing OOM. We have changed that and now making a deep copy.
Now It seems it is restricting old deletes map to capacity 1K.

After deployment of this change, we took another heap dump and did not find
this as leak suspects.  Please let me know if anyone have questions.

Thanks
Rohit


On Mon, Mar 27, 2017 at 11:56 AM, Rohit Kanchan 
wrote:

> Thanks Erick for replying back. I have deployed changes to production, we
> will figure it out soon if it is still causing OOM or not. And for commits
> we are doing auto commits after 10K docs or 30 secs.
> If I get time I will try to run a local test to check if we will hit OOM
> because of 1K map entries or not. I will update this thread about my
> findings. I really appreciate yours and Chris response.
>
> Thanks
> Rohit
>
>
> On Mon, Mar 27, 2017 at 10:47 AM, Erick Erickson 
> wrote:
>
>> Rohit:
>>
>> Well, whenever I see something like "I have this custom component..."
>> I immediately want the problem to be demonstrated without that custom
>> component before trying to debug Solr.
>>
>> As Chris explained, we can't clear the 1K entries. It's hard to
>> imagine why keeping the last 1,000 entries around would cause OOMs.
>>
>> You haven't demonstrated yet that after your latest change you still
>> get OOMs, you've just assumed so. After running for a "long time" do
>> you still see the problem after your changes?
>>
>> So before assuming it's a Solr bug, and after you demonstrate that
>> your latest change didn't solve the problem, you should try two
>> things:
>>
>> 1> as I suggested and Chris endorsed, try committing upon occasion
>> from your custom component. Or set your autocommit settings
>> appropriately if you haven't already.
>>
>> 2> run your deletes from the client as a test. You've created a custom
>> URP component because you "didn't want to run the queries from the
>> client". That's perfectly reasonable, it's just that to know where you
>> should be looking deleting from the client would eliminate your custom
>> code and tell us where to focus.
>>
>> Best,
>> Erick
>>
>>
>>
>> On Sat, Mar 25, 2017 at 1:21 PM, Rohit Kanchan 
>> wrote:
>> > I think we figure out the issue, When we were conventing delete by
>> query in
>> > a Solr Handler we were not making a deep copy of BytesRef. We were
>> making
>> > reference of same object, which was causing old deletes(LinkedHasmap)
>> > adding more than 1K entries.
>> >
>> > But I think it is still not clearing those 1K entries. Eventually it
>> will
>> > throw OOM because UpdateLog is not singleton and when there will be many
>> > delete by id and server is not re started for very long time then
>> > eventually throw OOM. I think we should clear this map when we are
>> > committing. I am not a committer,  it would be great if I get reply
>> from a
>> > committer.  What do you guys think?
>> >
>> > Thanks
>> > Rohit
>> >
>> >
>> > On Wed, Mar 22, 2017 at 1:36 PM, Rohit Kanchan 
>> > wrote:
>> >
>> >> For commits we are relying on auto commits. We have define following in
>> >> configs:
>> >>
>> >>
>> >>
>> >> 1
>> >>
>> >> 3
>> >>
>> >> false
>> >>
>> >> 
>> >>
>> >> 
>> >>
>> >> 15000
>> >>
>> >> 
>> >>
>> >> One thing which I would like to mention is that we are not calling
>> >> directly deleteById from client. We have created an  update chain and
>> added
>> >> a processor there. In this processor we are querying first and
>> collecting
>> >> all byteRefHash and get each byteRef out of it and set it to indexedId.
>> >> After collecting indexedId we are using those ids to call delete byId.
>> We
>> >> are doing this because we do not want query solr before deleting at
>> client
>> >> side. It is possible that there is a bug in this code but I am not
>> sure,
>> >> because when I run tests in my local it is not showing any issues. I am
>> >> trying to remote debug now.
>> >>
>> >> Thanks
>> >> Rohit
>> >>
>> >>
>> >> On Wed, Mar 22, 2017 at 9:57 AM, Chris Hostetter <
>> hossman_luc...@fucit.org
>> >> > wrote:
>> >>
>> >>>
>> >>> : OK, The whole DBQ thing baffles the heck out of me so this may be
>> >>> : totally off base. But would committing help here? Or at least be
>> worth
>> >>> : a test?
>> >>>
>> >>> ths isn't DBQ -- the OP specifically said deleteById, and that the
>> >>> oldDeletes map (only used for DBI) was the problem acording to the
>> heap
>> >>> dumps they looked at.
>> >>>
>> >>> I suspect you are correct about the root cause of the OOMs ...
>> perhaps the
>> >>> OP isn't using hard/soft commits effectively enough and the
>> uncommitted
>> >>> data is what's causing the OOM ... hard to say w/o more deta

Re: Solr Delete By Id Out of memory issue

[Rohit Kanchan]

Thanks Erick for replying back. I have deployed changes to production, we
will figure it out soon if it is still causing OOM or not. And for commits
we are doing auto commits after 10K docs or 30 secs.
If I get time I will try to run a local test to check if we will hit OOM
because of 1K map entries or not. I will update this thread about my
findings. I really appreciate yours and Chris response.

Thanks
Rohit


On Mon, Mar 27, 2017 at 10:47 AM, Erick Erickson 
wrote:

> Rohit:
>
> Well, whenever I see something like "I have this custom component..."
> I immediately want the problem to be demonstrated without that custom
> component before trying to debug Solr.
>
> As Chris explained, we can't clear the 1K entries. It's hard to
> imagine why keeping the last 1,000 entries around would cause OOMs.
>
> You haven't demonstrated yet that after your latest change you still
> get OOMs, you've just assumed so. After running for a "long time" do
> you still see the problem after your changes?
>
> So before assuming it's a Solr bug, and after you demonstrate that
> your latest change didn't solve the problem, you should try two
> things:
>
> 1> as I suggested and Chris endorsed, try committing upon occasion
> from your custom component. Or set your autocommit settings
> appropriately if you haven't already.
>
> 2> run your deletes from the client as a test. You've created a custom
> URP component because you "didn't want to run the queries from the
> client". That's perfectly reasonable, it's just that to know where you
> should be looking deleting from the client would eliminate your custom
> code and tell us where to focus.
>
> Best,
> Erick
>
>
>
> On Sat, Mar 25, 2017 at 1:21 PM, Rohit Kanchan 
> wrote:
> > I think we figure out the issue, When we were conventing delete by query
> in
> > a Solr Handler we were not making a deep copy of BytesRef. We were making
> > reference of same object, which was causing old deletes(LinkedHasmap)
> > adding more than 1K entries.
> >
> > But I think it is still not clearing those 1K entries. Eventually it will
> > throw OOM because UpdateLog is not singleton and when there will be many
> > delete by id and server is not re started for very long time then
> > eventually throw OOM. I think we should clear this map when we are
> > committing. I am not a committer,  it would be great if I get reply from
> a
> > committer.  What do you guys think?
> >
> > Thanks
> > Rohit
> >
> >
> > On Wed, Mar 22, 2017 at 1:36 PM, Rohit Kanchan 
> > wrote:
> >
> >> For commits we are relying on auto commits. We have define following in
> >> configs:
> >>
> >>
> >>
> >> 1
> >>
> >> 3
> >>
> >> false
> >>
> >> 
> >>
> >> 
> >>
> >> 15000
> >>
> >> 
> >>
> >> One thing which I would like to mention is that we are not calling
> >> directly deleteById from client. We have created an  update chain and
> added
> >> a processor there. In this processor we are querying first and
> collecting
> >> all byteRefHash and get each byteRef out of it and set it to indexedId.
> >> After collecting indexedId we are using those ids to call delete byId.
> We
> >> are doing this because we do not want query solr before deleting at
> client
> >> side. It is possible that there is a bug in this code but I am not sure,
> >> because when I run tests in my local it is not showing any issues. I am
> >> trying to remote debug now.
> >>
> >> Thanks
> >> Rohit
> >>
> >>
> >> On Wed, Mar 22, 2017 at 9:57 AM, Chris Hostetter <
> hossman_luc...@fucit.org
> >> > wrote:
> >>
> >>>
> >>> : OK, The whole DBQ thing baffles the heck out of me so this may be
> >>> : totally off base. But would committing help here? Or at least be
> worth
> >>> : a test?
> >>>
> >>> ths isn't DBQ -- the OP specifically said deleteById, and that the
> >>> oldDeletes map (only used for DBI) was the problem acording to the heap
> >>> dumps they looked at.
> >>>
> >>> I suspect you are correct about the root cause of the OOMs ... perhaps
> the
> >>> OP isn't using hard/soft commits effectively enough and the uncommitted
> >>> data is what's causing the OOM ... hard to say w/o more details. or
> >>> confirmation of exactly what the OP was looking at in their claim below
> >>> about the heap dump
> >>>
> >>>
> >>> : > : Thanks for replying. We are using Solr 6.1 version. Even I saw
> that
> >>> it is
> >>> : > : bounded by 1K count, but after looking at heap dump I was amazed
> >>> how can it
> >>> : > : keep more than 1K entries. But Yes I see around 7M entries
> >>> according to
> >>> : > : heap dump and around 17G of memory occupied by BytesRef there.
> >>> : >
> >>> : > what exactly are you looking at when you say you see "7M entries" ?
> >>> : >
> >>> : > are you sure you aren't confusing the keys in oldDeletes with other
> >>> : > instances of BytesRef in the JVM?
> >>>
> >>>
> >>> -Hoss
> >>> http://www.lucidworks.com/
> >>>
> >>
> >>
>

Re: Solr Delete By Id Out of memory issue

[Erick Erickson]

Rohit:

Well, whenever I see something like "I have this custom component..."
I immediately want the problem to be demonstrated without that custom
component before trying to debug Solr.

As Chris explained, we can't clear the 1K entries. It's hard to
imagine why keeping the last 1,000 entries around would cause OOMs.

You haven't demonstrated yet that after your latest change you still
get OOMs, you've just assumed so. After running for a "long time" do
you still see the problem after your changes?

So before assuming it's a Solr bug, and after you demonstrate that
your latest change didn't solve the problem, you should try two
things:

1> as I suggested and Chris endorsed, try committing upon occasion
from your custom component. Or set your autocommit settings
appropriately if you haven't already.

2> run your deletes from the client as a test. You've created a custom
URP component because you "didn't want to run the queries from the
client". That's perfectly reasonable, it's just that to know where you
should be looking deleting from the client would eliminate your custom
code and tell us where to focus.

Best,
Erick



On Sat, Mar 25, 2017 at 1:21 PM, Rohit Kanchan  wrote:
> I think we figure out the issue, When we were conventing delete by query in
> a Solr Handler we were not making a deep copy of BytesRef. We were making
> reference of same object, which was causing old deletes(LinkedHasmap)
> adding more than 1K entries.
>
> But I think it is still not clearing those 1K entries. Eventually it will
> throw OOM because UpdateLog is not singleton and when there will be many
> delete by id and server is not re started for very long time then
> eventually throw OOM. I think we should clear this map when we are
> committing. I am not a committer,  it would be great if I get reply from a
> committer.  What do you guys think?
>
> Thanks
> Rohit
>
>
> On Wed, Mar 22, 2017 at 1:36 PM, Rohit Kanchan 
> wrote:
>
>> For commits we are relying on auto commits. We have define following in
>> configs:
>>
>>
>>
>> 1
>>
>> 3
>>
>> false
>>
>> 
>>
>> 
>>
>> 15000
>>
>> 
>>
>> One thing which I would like to mention is that we are not calling
>> directly deleteById from client. We have created an  update chain and added
>> a processor there. In this processor we are querying first and collecting
>> all byteRefHash and get each byteRef out of it and set it to indexedId.
>> After collecting indexedId we are using those ids to call delete byId. We
>> are doing this because we do not want query solr before deleting at client
>> side. It is possible that there is a bug in this code but I am not sure,
>> because when I run tests in my local it is not showing any issues. I am
>> trying to remote debug now.
>>
>> Thanks
>> Rohit
>>
>>
>> On Wed, Mar 22, 2017 at 9:57 AM, Chris Hostetter > > wrote:
>>
>>>
>>> : OK, The whole DBQ thing baffles the heck out of me so this may be
>>> : totally off base. But would committing help here? Or at least be worth
>>> : a test?
>>>
>>> ths isn't DBQ -- the OP specifically said deleteById, and that the
>>> oldDeletes map (only used for DBI) was the problem acording to the heap
>>> dumps they looked at.
>>>
>>> I suspect you are correct about the root cause of the OOMs ... perhaps the
>>> OP isn't using hard/soft commits effectively enough and the uncommitted
>>> data is what's causing the OOM ... hard to say w/o more details. or
>>> confirmation of exactly what the OP was looking at in their claim below
>>> about the heap dump
>>>
>>>
>>> : > : Thanks for replying. We are using Solr 6.1 version. Even I saw that
>>> it is
>>> : > : bounded by 1K count, but after looking at heap dump I was amazed
>>> how can it
>>> : > : keep more than 1K entries. But Yes I see around 7M entries
>>> according to
>>> : > : heap dump and around 17G of memory occupied by BytesRef there.
>>> : >
>>> : > what exactly are you looking at when you say you see "7M entries" ?
>>> : >
>>> : > are you sure you aren't confusing the keys in oldDeletes with other
>>> : > instances of BytesRef in the JVM?
>>>
>>>
>>> -Hoss
>>> http://www.lucidworks.com/
>>>
>>
>>

Re: Solr Delete By Id Out of memory issue

[Rohit Kanchan]

I think we figure out the issue, When we were conventing delete by query in
a Solr Handler we were not making a deep copy of BytesRef. We were making
reference of same object, which was causing old deletes(LinkedHasmap)
adding more than 1K entries.

But I think it is still not clearing those 1K entries. Eventually it will
throw OOM because UpdateLog is not singleton and when there will be many
delete by id and server is not re started for very long time then
eventually throw OOM. I think we should clear this map when we are
committing. I am not a committer,  it would be great if I get reply from a
committer.  What do you guys think?

Thanks
Rohit


On Wed, Mar 22, 2017 at 1:36 PM, Rohit Kanchan 
wrote:

> For commits we are relying on auto commits. We have define following in
> configs:
>
>
>
> 1
>
> 3
>
> false
>
> 
>
> 
>
> 15000
>
> 
>
> One thing which I would like to mention is that we are not calling
> directly deleteById from client. We have created an  update chain and added
> a processor there. In this processor we are querying first and collecting
> all byteRefHash and get each byteRef out of it and set it to indexedId.
> After collecting indexedId we are using those ids to call delete byId. We
> are doing this because we do not want query solr before deleting at client
> side. It is possible that there is a bug in this code but I am not sure,
> because when I run tests in my local it is not showing any issues. I am
> trying to remote debug now.
>
> Thanks
> Rohit
>
>
> On Wed, Mar 22, 2017 at 9:57 AM, Chris Hostetter  > wrote:
>
>>
>> : OK, The whole DBQ thing baffles the heck out of me so this may be
>> : totally off base. But would committing help here? Or at least be worth
>> : a test?
>>
>> ths isn't DBQ -- the OP specifically said deleteById, and that the
>> oldDeletes map (only used for DBI) was the problem acording to the heap
>> dumps they looked at.
>>
>> I suspect you are correct about the root cause of the OOMs ... perhaps the
>> OP isn't using hard/soft commits effectively enough and the uncommitted
>> data is what's causing the OOM ... hard to say w/o more details. or
>> confirmation of exactly what the OP was looking at in their claim below
>> about the heap dump
>>
>>
>> : > : Thanks for replying. We are using Solr 6.1 version. Even I saw that
>> it is
>> : > : bounded by 1K count, but after looking at heap dump I was amazed
>> how can it
>> : > : keep more than 1K entries. But Yes I see around 7M entries
>> according to
>> : > : heap dump and around 17G of memory occupied by BytesRef there.
>> : >
>> : > what exactly are you looking at when you say you see "7M entries" ?
>> : >
>> : > are you sure you aren't confusing the keys in oldDeletes with other
>> : > instances of BytesRef in the JVM?
>>
>>
>> -Hoss
>> http://www.lucidworks.com/
>>
>
>

Re: Solr Delete By Id Out of memory issue

[Rohit Kanchan]

For commits we are relying on auto commits. We have define following in
configs:

   

1

3

false





15000



One thing which I would like to mention is that we are not calling directly
deleteById from client. We have created an  update chain and added a
processor there. In this processor we are querying first and collecting all
byteRefHash and get each byteRef out of it and set it to indexedId. After
collecting indexedId we are using those ids to call delete byId. We are
doing this because we do not want query solr before deleting at client
side. It is possible that there is a bug in this code but I am not sure,
because when I run tests in my local it is not showing any issues. I am
trying to remote debug now.

Thanks
Rohit


On Wed, Mar 22, 2017 at 9:57 AM, Chris Hostetter 
wrote:

>
> : OK, The whole DBQ thing baffles the heck out of me so this may be
> : totally off base. But would committing help here? Or at least be worth
> : a test?
>
> ths isn't DBQ -- the OP specifically said deleteById, and that the
> oldDeletes map (only used for DBI) was the problem acording to the heap
> dumps they looked at.
>
> I suspect you are correct about the root cause of the OOMs ... perhaps the
> OP isn't using hard/soft commits effectively enough and the uncommitted
> data is what's causing the OOM ... hard to say w/o more details. or
> confirmation of exactly what the OP was looking at in their claim below
> about the heap dump
>
>
> : > : Thanks for replying. We are using Solr 6.1 version. Even I saw that
> it is
> : > : bounded by 1K count, but after looking at heap dump I was amazed how
> can it
> : > : keep more than 1K entries. But Yes I see around 7M entries according
> to
> : > : heap dump and around 17G of memory occupied by BytesRef there.
> : >
> : > what exactly are you looking at when you say you see "7M entries" ?
> : >
> : > are you sure you aren't confusing the keys in oldDeletes with other
> : > instances of BytesRef in the JVM?
>
>
> -Hoss
> http://www.lucidworks.com/
>

Re: Solr Delete By Id Out of memory issue

[Chris Hostetter]

: OK, The whole DBQ thing baffles the heck out of me so this may be
: totally off base. But would committing help here? Or at least be worth
: a test?

ths isn't DBQ -- the OP specifically said deleteById, and that the 
oldDeletes map (only used for DBI) was the problem acording to the heap 
dumps they looked at.

I suspect you are correct about the root cause of the OOMs ... perhaps the 
OP isn't using hard/soft commits effectively enough and the uncommitted 
data is what's causing the OOM ... hard to say w/o more details. or 
confirmation of exactly what the OP was looking at in their claim below 
about the heap dump


: > : Thanks for replying. We are using Solr 6.1 version. Even I saw that it is
: > : bounded by 1K count, but after looking at heap dump I was amazed how can 
it
: > : keep more than 1K entries. But Yes I see around 7M entries according to
: > : heap dump and around 17G of memory occupied by BytesRef there.
: >
: > what exactly are you looking at when you say you see "7M entries" ?
: >
: > are you sure you aren't confusing the keys in oldDeletes with other
: > instances of BytesRef in the JVM?


-Hoss
http://www.lucidworks.com/

Re: Solr Delete By Id Out of memory issue

[Erick Erickson]

Chris:

OK, The whole DBQ thing baffles the heck out of me so this may be
totally off base. But would committing help here? Or at least be worth
a test?

On Tue, Mar 21, 2017 at 4:28 PM, Chris Hostetter
 wrote:
>
> : Thanks for replying. We are using Solr 6.1 version. Even I saw that it is
> : bounded by 1K count, but after looking at heap dump I was amazed how can it
> : keep more than 1K entries. But Yes I see around 7M entries according to
> : heap dump and around 17G of memory occupied by BytesRef there.
>
> what exactly are you looking at when you say you see "7M entries" ?
>
> are you sure you aren't confusing the keys in oldDeletes with other
> instances of BytesRef in the JVM?
>
> : It would be better to know why old deletes Map is used there. I am still
> : digging, If I find something then I will share that.
>
> See UpdateLog.lookupVersion for where the mapping is used if you are
> curious. Replicas recieving updates need to be able to verify (and in some
> casees ask the leader via RTG) for the version used when a doc was deleted
> in the even of concurrent/out-of-order updates for the same uniqueKey.
>
>
>
> -Hoss
> http://www.lucidworks.com/

Re: Solr Delete By Id Out of memory issue

[Chris Hostetter]

: Thanks for replying. We are using Solr 6.1 version. Even I saw that it is
: bounded by 1K count, but after looking at heap dump I was amazed how can it
: keep more than 1K entries. But Yes I see around 7M entries according to
: heap dump and around 17G of memory occupied by BytesRef there.

what exactly are you looking at when you say you see "7M entries" ?

are you sure you aren't confusing the keys in oldDeletes with other 
instances of BytesRef in the JVM?

: It would be better to know why old deletes Map is used there. I am still
: digging, If I find something then I will share that.

See UpdateLog.lookupVersion for where the mapping is used if you are 
curious. Replicas recieving updates need to be able to verify (and in some 
casees ask the leader via RTG) for the version used when a doc was deleted 
in the even of concurrent/out-of-order updates for the same uniqueKey.



-Hoss
http://www.lucidworks.com/

Re: Solr Delete By Id Out of memory issue

[Rohit Kanchan]

Hi Chris,

Thanks for replying. We are using Solr 6.1 version. Even I saw that it is
bounded by 1K count, but after looking at heap dump I was amazed how can it
keep more than 1K entries. But Yes I see around 7M entries according to
heap dump and around 17G of memory occupied by BytesRef there.

It would be better to know why old deletes Map is used there. I am still
digging, If I find something then I will share that.

Thanks
Rohit


On Tue, Mar 21, 2017 at 4:00 PM, Chris Hostetter 
wrote:

>
> : facing. We are storing messages in solr as documents. We are running a
> : pruning job every night to delete old message documents. We are deleting
> : old documents by calling multiple delete by id query to solr. Document
> : count can be in millions which we are deleting using SolrJ client. We are
> : using delete by id because it is faster than delete by query. It works
> : great for few days but after a week these delete by id get accumulated in
> : Linked hash map of UpdateLog (variable name as olddeletes). Once this map
> : is full then we are seeing out of memory.
>
> first off: what version of Solr are you running?
>
> UpdateLog.oldDeletes is bounded at numDeletesToKeep=1000 entries -- any
> more then that and the oldest entry is automatically deleted when more
> items are added.  So it doesn't really make sense to me that you would be
> seeing OOMs from this map filling up endlessly.
>
> Are you seeing more then 1000 entries in this map when you look at your
> heap dumps?
>
> : I am not sure why it is keeping the reference of all old deletes.
>
> It's complicated -- the short answer is that it's protection against out
> of order updates ariving from other nodes in SolrCloud under highly
> concurrent updates.
>
>
>
> -Hoss
> http://www.lucidworks.com/
>

Re: Solr Delete By Id Out of memory issue

[Chris Hostetter]

: facing. We are storing messages in solr as documents. We are running a
: pruning job every night to delete old message documents. We are deleting
: old documents by calling multiple delete by id query to solr. Document
: count can be in millions which we are deleting using SolrJ client. We are
: using delete by id because it is faster than delete by query. It works
: great for few days but after a week these delete by id get accumulated in
: Linked hash map of UpdateLog (variable name as olddeletes). Once this map
: is full then we are seeing out of memory.

first off: what version of Solr are you running?

UpdateLog.oldDeletes is bounded at numDeletesToKeep=1000 entries -- any 
more then that and the oldest entry is automatically deleted when more 
items are added.  So it doesn't really make sense to me that you would be 
seeing OOMs from this map filling up endlessly.

Are you seeing more then 1000 entries in this map when you look at your 
heap dumps?

: I am not sure why it is keeping the reference of all old deletes.

It's complicated -- the short answer is that it's protection against out 
of order updates ariving from other nodes in SolrCloud under highly 
concurrent updates.



-Hoss
http://www.lucidworks.com/

Solr Delete By Id Out of memory issue

[Rohit Kanchan]

Hi All,

I am looking for some help to solve an out of memory issue which we are
facing. We are storing messages in solr as documents. We are running a
pruning job every night to delete old message documents. We are deleting
old documents by calling multiple delete by id query to solr. Document
count can be in millions which we are deleting using SolrJ client. We are
using delete by id because it is faster than delete by query. It works
great for few days but after a week these delete by id get accumulated in
Linked hash map of UpdateLog (variable name as olddeletes). Once this map
is full then we are seeing out of memory.

We did look into heap dump then we found that this map is storing BytesRef
as key and LogPtr as value. BytesRef is the one which is taking a lot of
memory. BytesRef is storing reference of all the ids which we are deleting.
I am not sure why it is keeping the reference of all old deletes.

I looked at solr code but I could not trace how it is cleaning this map.
There is a deleteAll method in UpdateLog and only test cases are calling
this method.

Did anyone face the same issue? I really appreciate a reply for this
problem.

*Note*: We are running Solr in cloud and because of this there is high gc
pause which is causing first replica go in recovery then leader and replica
crashes.

-
Rohit

RE: Solr Cloud with 5 servers cluster failed due to Leader out of memory

[Tim Chen]

Guys, (@Erick & @Shawn),

Thanks for the great suggestions!

I have increased Tomcat MaxThreads from 200 to 1 on our staging 
environment. So far so good.

I will perform some more indexing test and see how it goes.

Many thanks,
Tim

-Original Message-
From: Shawn Heisey [mailto:apa...@elyograg.org]
Sent: Monday, 8 August 2016 11:44 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Cloud with 5 servers cluster failed due to Leader out of 
memory

On 8/7/2016 6:53 PM, Tim Chen wrote:
> Exception in thread "http-bio-8983-exec-6571" java.lang.OutOfMemoryError: 
> unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at 
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
> at 
> java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at 
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.lang.Thread.run(Thread.java:745)

I find myself chasing Erick once again. :)  Supplementing what he told you:

There are two things that might be happening here.

1) The Tomcat setting "maxThreads" may limiting the number of threads.
This defaults to 200, and should be increased to 1.  The specific error 
doesn't sound like an application limit, though -- it acts more like Java 
itself can't create the thread.  If you have already adjusted maxThreads, then 
it's more likely to be the second option:

2) The operating system may be imposing a limit on the number of 
processes/threads a user is allowed to start.  On Linux systems, this is 
typically 1024.  For other operating systems, I am not sure what the default 
limit is.

Thanks,
Shawn



[Premiere League Starts Saturday 13 August 9.30pm on 
SBS]<http://theworldgame.sbs.com.au/>

Re: Solr Cloud with 5 servers cluster failed due to Leader out of memory

[Shawn Heisey]

On 8/8/2016 11:09 AM, Ritesh Kumar (Avanade) wrote:
> This is great but where can I do this change in SOLR 6 as I have
> implemented CDCR.

In Solr 6, the chance of using Tomcat will be near zero, and the
maxThreads setting in Solr's Jetty config should already be set to 1.

If you're seeing this same OOME (can't create a new thread) in Solr 6,
then the problem is most likely going to be at the operating system
level.  Exactly how to increase the number of processes/threads that
Solr can create will vary depending on the operating system you're
running.  For help, consult documentation or support resources for your
OS, or maybe Google.

If you're seeing a different problem, then please send a brand new
message to the list detailing your problem.

http://people.apache.org/~hossman/#threadhijack

Thanks,
Shawn

RE: Solr Cloud with 5 servers cluster failed due to Leader out of memory

[Ritesh Kumar (Avanade)]

This is great but where can I do this change in SOLR 6 as I have implemented 
CDCR.

Ritesh K
Infrastructure Sr. Engineer – Jericho Team
Sales & Marketing Digital Services
t +91-7799936921   v-kur...@microsoft.com

-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com] 
Sent: 08 August 2016 21:30
To: solr-user 
Subject: Re: Solr Cloud with 5 servers cluster failed due to Leader out of 
memory

Yeah, Shawn, but you, like, know something about Tomcat and actually provide 
useful advice ;)

On Mon, Aug 8, 2016 at 6:44 AM, Shawn Heisey  wrote:
> On 8/7/2016 6:53 PM, Tim Chen wrote:
>> Exception in thread "http-bio-8983-exec-6571" java.lang.OutOfMemoryError: 
>> unable to create new native thread
>> at java.lang.Thread.start0(Native Method)
>> at java.lang.Thread.start(Thread.java:714)
>> at 
>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>> at 
>> java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
>> at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
>> at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at 
>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>> at java.lang.Thread.run(Thread.java:745)
>
> I find myself chasing Erick once again. :)  Supplementing what he told you:
>
> There are two things that might be happening here.
>
> 1) The Tomcat setting "maxThreads" may limiting the number of threads.
> This defaults to 200, and should be increased to 1.  The specific 
> error doesn't sound like an application limit, though -- it acts more 
> like Java itself can't create the thread.  If you have already 
> adjusted maxThreads, then it's more likely to be the second option:
>
> 2) The operating system may be imposing a limit on the number of 
> processes/threads a user is allowed to start.  On Linux systems, this 
> is typically 1024.  For other operating systems, I am not sure what 
> the default limit is.
>
> Thanks,
> Shawn
>

Re: Solr Cloud with 5 servers cluster failed due to Leader out of memory

[Erick Erickson]

Yeah, Shawn, but you, like, know something about Tomcat and
actually provide useful advice ;)

On Mon, Aug 8, 2016 at 6:44 AM, Shawn Heisey  wrote:
> On 8/7/2016 6:53 PM, Tim Chen wrote:
>> Exception in thread "http-bio-8983-exec-6571" java.lang.OutOfMemoryError: 
>> unable to create new native thread
>> at java.lang.Thread.start0(Native Method)
>> at java.lang.Thread.start(Thread.java:714)
>> at 
>> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
>> at 
>> java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
>> at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
>> at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at 
>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>> at java.lang.Thread.run(Thread.java:745)
>
> I find myself chasing Erick once again. :)  Supplementing what he told you:
>
> There are two things that might be happening here.
>
> 1) The Tomcat setting "maxThreads" may limiting the number of threads.
> This defaults to 200, and should be increased to 1.  The specific
> error doesn't sound like an application limit, though -- it acts more
> like Java itself can't create the thread.  If you have already adjusted
> maxThreads, then it's more likely to be the second option:
>
> 2) The operating system may be imposing a limit on the number of
> processes/threads a user is allowed to start.  On Linux systems, this is
> typically 1024.  For other operating systems, I am not sure what the
> default limit is.
>
> Thanks,
> Shawn
>

Re: Solr Cloud with 5 servers cluster failed due to Leader out of memory

[Shawn Heisey]

On 8/7/2016 6:53 PM, Tim Chen wrote:
> Exception in thread "http-bio-8983-exec-6571" java.lang.OutOfMemoryError: 
> unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> at 
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
> at 
> java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at 
> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> at java.lang.Thread.run(Thread.java:745)

I find myself chasing Erick once again. :)  Supplementing what he told you:

There are two things that might be happening here.

1) The Tomcat setting "maxThreads" may limiting the number of threads. 
This defaults to 200, and should be increased to 1.  The specific
error doesn't sound like an application limit, though -- it acts more
like Java itself can't create the thread.  If you have already adjusted
maxThreads, then it's more likely to be the second option:

2) The operating system may be imposing a limit on the number of
processes/threads a user is allowed to start.  On Linux systems, this is
typically 1024.  For other operating systems, I am not sure what the
default limit is.

Thanks,
Shawn

Re: Solr Cloud with 5 servers cluster failed due to Leader out of memory

[Erick Erickson]

You're correct on maxWarmingSarchers, and autowarming isn't really a concern
since you're not exceeding maxWarmingSearchers.

Wait... the error is a _tomcat_ error according to the
stack trace. If this were the internals of Solr you'd be seeing
org.apache.solr in there somewhere. I've seen the
"unable o create native thread" problem when there were a
bazillion replicas on a single Solr JVM, but it doesn't sound like
this is your case.

You say you have a heavy indexing load... Are you by any chance
sending a single document per update? I usually start with sending
docs (assuming SolrJ) with the CloudSolrClient.add(docList) version
and my docList straw-man number is 1,000. If the docs are
very large, that can be too many, but "very large" here is 50K or so.

In the case I saw, bumping the JVM won't help, the -Xss _might_ help
but the pathological case I was testing didn't get solved that way.

You're right, there are (somehow) just too many threads being created,
but I've seen very heavy indexing rates without this problem, so I'd
guess there's something magic here.

Of the three choices, (a) is probably best assuming you're already
batching up docs for indexing.

Best,
Erick

On Sun, Aug 7, 2016 at 6:53 PM, Tim Chen  wrote:
> Sorry Erick, forgot to answer your question:
>
> No, I didn't increase the maxWarmingSearchers. It is set to 
> 2. I read it somewhere that 
> increasing this is a risk.
>
> Just to make sure, you didn't mean the "autowarmCount " in the 
> 
> Thanks,
> Tim
>
> Reference:
>
>   size="512"
>  initialSize="512"
>  autowarmCount="0"/>
>
> 
>   size="512"
>  initialSize="512"
>  autowarmCount="32"/>
>
> 
> size="512"
>initialSize="512"
>autowarmCount="0"/>
>
> -Original Message-
> From: Erick Erickson [mailto:erickerick...@gmail.com]
> Sent: Saturday, 6 August 2016 2:31 AM
> To: solr-user
> Subject: Re: Solr Cloud with 5 servers cluster failed due to Leader out of 
> memory
>
> You don't really have to worry that much about memory consumed during 
> indexing.
> The ramBufferSizeMB setting in solrconfig.xml pretty much limits the amount 
> of RAM consumed, when adding a doc if that limit is exceeded then the buffer 
> is flushed.
>
> So you can reduce that number, but it's default is 100M and if you're running 
> that close to your limits I suspect you'd get, at best, a bit more runway 
> before you hit the problem again.
>
> NOTE: that number isn't an absolute limit, IIUC the algorithm is
>> index a doc to the in-memory structures check if the limit is exceeded
>> and flush if so.
>
> So say you were at 99% of your ramBufferSizeMB setting and then indexed a 
> ginormous doc your in-memory stuff might be significantly bigger.
>
> Searching usually is the bigger RAM consumer, so when I say "a bit more 
> runway" what I'm thinking about is that when you start _searching_ the data 
> your memory requirements will continue to grow and you'll be back where you 
> started.
>
> And just as a sanity check: You didn't perchance increase the 
> maxWarmingSearchers parameter in solrconfig.xml, did you? If so, that's 
> really a red flag.
>
> Best,
> Erick
>
> On Fri, Aug 5, 2016 at 12:41 AM, Tim Chen  wrote:
>> Thanks Guys. Very very helpful.
>>
>> I will probably look at consolidate 4 Solr servers into 2 bigger/better 
>> server - it gives more memory, and it cut down the replica the Leader needs 
>> to manage.
>>
>> Also, I may look into write a script to monitor the tomcat log and if there 
>> is OOM, kill tomcat, then restart it. A bit dirty, but may work for a short 
>> term.
>>
>> I don't know too much about how documents indexed, and how to save memory 
>> from that. Will probably work with a developer on this as well.
>>
>> Many Thanks guys.
>>
>> Cheers,
>> Tim
>>
>> -Original Message-
>> From: Shawn Heisey [mailto:apa...@elyograg.org]
>> Sent: Friday, 5 August 2016 4:55 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Solr Cloud with 5 servers cluster failed due to Leader
>> out of memory
>>
>> On 8/4/2016 8:14 PM, Tim Chen wrote:
>>> Couple of thoughts: 1, If Leader goes down, it should just go down,
>>> like dead down, so other servers can do the electi

RE: Solr Cloud with 5 servers cluster failed due to Leader out of memory

[Tim Chen]

Sorry Erick, forgot to answer your question:

No, I didn't increase the maxWarmingSearchers. It is set to 
2. I read it somewhere that 
increasing this is a risk.

Just to make sure, you didn't mean the "autowarmCount " in the 








-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com]
Sent: Saturday, 6 August 2016 2:31 AM
To: solr-user
Subject: Re: Solr Cloud with 5 servers cluster failed due to Leader out of 
memory

You don't really have to worry that much about memory consumed during indexing.
The ramBufferSizeMB setting in solrconfig.xml pretty much limits the amount of 
RAM consumed, when adding a doc if that limit is exceeded then the buffer is 
flushed.

So you can reduce that number, but it's default is 100M and if you're running 
that close to your limits I suspect you'd get, at best, a bit more runway 
before you hit the problem again.

NOTE: that number isn't an absolute limit, IIUC the algorithm is
> index a doc to the in-memory structures check if the limit is exceeded
> and flush if so.

So say you were at 99% of your ramBufferSizeMB setting and then indexed a 
ginormous doc your in-memory stuff might be significantly bigger.

Searching usually is the bigger RAM consumer, so when I say "a bit more runway" 
what I'm thinking about is that when you start _searching_ the data your memory 
requirements will continue to grow and you'll be back where you started.

And just as a sanity check: You didn't perchance increase the 
maxWarmingSearchers parameter in solrconfig.xml, did you? If so, that's really 
a red flag.

Best,
Erick

On Fri, Aug 5, 2016 at 12:41 AM, Tim Chen  wrote:
> Thanks Guys. Very very helpful.
>
> I will probably look at consolidate 4 Solr servers into 2 bigger/better 
> server - it gives more memory, and it cut down the replica the Leader needs 
> to manage.
>
> Also, I may look into write a script to monitor the tomcat log and if there 
> is OOM, kill tomcat, then restart it. A bit dirty, but may work for a short 
> term.
>
> I don't know too much about how documents indexed, and how to save memory 
> from that. Will probably work with a developer on this as well.
>
> Many Thanks guys.
>
> Cheers,
> Tim
>
> -Original Message-
> From: Shawn Heisey [mailto:apa...@elyograg.org]
> Sent: Friday, 5 August 2016 4:55 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Cloud with 5 servers cluster failed due to Leader
> out of memory
>
> On 8/4/2016 8:14 PM, Tim Chen wrote:
>> Couple of thoughts: 1, If Leader goes down, it should just go down,
>> like dead down, so other servers can do the election and choose the
>> new leader. This at least avoids bringing down the whole cluster. Am
>> I right?
>
> Supplementing what Erick told you:
>
> When a typical Java program throws OutOfMemoryError, program behavior is 
> completely unpredictable.  There are programming techniques that can be used 
> so that behavior IS predictable, but writing that code can be challenging.
>
> Solr 5.x and 6.x, when they are started on a UNIX/Linux system, use a Java 
> option to execute a script when OutOfMemoryError happens.  This script kills 
> Solr completely.  We are working on adding this capability when running on 
> Windows.
>
>> 2, Apparently we should not pushing too many documents to Solr, how
>> do you guys handle this? Set a limit somewhere?
>
> There are exactly two ways to deal with OOME problems: Increase the heap or 
> reduce Solr's memory requirements.  The number of documents you push to Solr 
> is unlikely to have a large effect on the amount of memory that Solr 
> requires.  Here's some information on this topic:
>
> https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap
>
> Thanks,
> Shawn
>
>
>
> [Premiere League Starts Saturday 13 August 9.30pm on
> SBS]<http://theworldgame.sbs.com.au/>


[Premiere League Starts Saturday 13 August 9.30pm on 
SBS]<http://theworldgame.sbs.com.au/>

RE: Solr Cloud with 5 servers cluster failed due to Leader out of memory

[Tim Chen]

Hi Erick, Shawn,

Thanks for following this up.

1,
For some reason, ramBufferSizeMB in our solrconfig.xml is not set to 100MB, but 
32MB.

In that case, considering we have 10G for JVM, my understanding is we should 
not run out of memory due to large number of documents being added to Solr.

Just to make sure I understand it correctly, the documents adding to Solr will 
be stored in an internal queue in Solr, and Solr will only use that 32MB (or 
99% of 32M + one extra document memory) for indexing documents. The documents 
in the queue will be indexed one by one.

2,
Based on our tomcat (Solr) access_log and website peak hours, the time we had 
our cluster failure is not likely because of _searching_traffic. Eg, we can see 
much more Solr requests with 'update' keyword, but as usual number of requests 
with 'select' keyword.

3,
Now, this leads me to the only reason I can think of: (you mentioned this 
earlier as well):
Since each Shard has 4 replicas in our setup, when there are large number of 
documents being add, the Leader will create a lot of threads to send the 
document to other replica servers. All these threads are the one consumed all 
the memory on Leader server, and leads to OOM.

If my assumption was right, to try or fix this issue, is to:
a): still need to limit the documents being add to Solr
b): change to 2 replica for each shard (loss of data reliability, but..)
c): bump up server memory.

Am I going the right way? Any advice and suggestions are much appreciated!!

Also attached part of catalina.out OOM log for reference:

Exception in thread "http-bio-8983-exec-6571" java.lang.OutOfMemoryError: 
unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at 
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
at 
java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "http-bio-8983-exec-6861" java.lang.OutOfMemoryError: 
unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at 
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
at 
java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Exception in thread "http-bio-8983-exec-6671" java.lang.OutOfMemoryError: 
unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at 
java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:949)
at 
java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1017)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1163)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

Many thanks,
Tim


-Original Message-
From: Erick Erickson [mailto:erickerick...@gmail.com]
Sent: Saturday, 6 August 2016 2:31 AM
To: solr-user
Subject: Re: Solr Cloud with 5 servers cluster failed due to Leader out of 
memory

You don't really have to worry that much about memory consumed during indexing.
The ramBufferSizeMB setting in solrconfig.xml pretty much limits the amount of 
RAM consumed, when adding a doc if that limit is exceeded then the buffer is 
flushed.

So you can reduce that number, but it's default is 100M and if you're running 
that close to your limits I suspect you'd get, at best, a bit more runway 
before you hit the problem again.

NOTE: that number isn't an absolute limit, IIUC the algorithm is
> index a doc to the in-memory structures check if the limit is exceeded
> and flush if so.

So say you were at 99% of your ramBufferSizeMB setting and then indexed a 
ginormous doc your in-memory stuff might be significantly bigger.

Searching usually is the bigger RAM consumer, so when I say "a bit more runway" 
what I'm thinking about is that when you start _searching_ the data your memory 
requ

Re: Solr Cloud with 5 servers cluster failed due to Leader out of memory

[Erick Erickson]

You don't really have to worry that much about memory consumed during indexing.
The ramBufferSizeMB setting in solrconfig.xml pretty much limits the amount of
RAM consumed, when adding a doc if that limit is exceeded then the
buffer is flushed.

So you can reduce that number, but it's default is 100M and if you're
running that close
to your limits I suspect you'd get, at best, a bit more runway before
you hit the problem again.

NOTE: that number isn't an absolute limit, IIUC the algorithm is
> index a doc to the in-memory structures
> check if the limit is exceeded and flush if so.

So say you were at 99% of your ramBufferSizeMB setting and then indexed a
ginormous doc your in-memory stuff might be significantly bigger.

Searching usually is the bigger RAM consumer, so when I say "a bit
more runway" what
I'm thinking about is that when you start _searching_ the data your
memory requirements
will continue to grow and you'll be back where you started.

And just as a sanity check: You didn't perchance increase the
maxWarmingSearchers
parameter in solrconfig.xml, did you? If so, that's really a red flag.

Best,
Erick

On Fri, Aug 5, 2016 at 12:41 AM, Tim Chen  wrote:
> Thanks Guys. Very very helpful.
>
> I will probably look at consolidate 4 Solr servers into 2 bigger/better 
> server - it gives more memory, and it cut down the replica the Leader needs 
> to manage.
>
> Also, I may look into write a script to monitor the tomcat log and if there 
> is OOM, kill tomcat, then restart it. A bit dirty, but may work for a short 
> term.
>
> I don't know too much about how documents indexed, and how to save memory 
> from that. Will probably work with a developer on this as well.
>
> Many Thanks guys.
>
> Cheers,
> Tim
>
> -Original Message-
> From: Shawn Heisey [mailto:apa...@elyograg.org]
> Sent: Friday, 5 August 2016 4:55 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr Cloud with 5 servers cluster failed due to Leader out of 
> memory
>
> On 8/4/2016 8:14 PM, Tim Chen wrote:
>> Couple of thoughts: 1, If Leader goes down, it should just go down,
>> like dead down, so other servers can do the election and choose the
>> new leader. This at least avoids bringing down the whole cluster. Am I
>> right?
>
> Supplementing what Erick told you:
>
> When a typical Java program throws OutOfMemoryError, program behavior is 
> completely unpredictable.  There are programming techniques that can be used 
> so that behavior IS predictable, but writing that code can be challenging.
>
> Solr 5.x and 6.x, when they are started on a UNIX/Linux system, use a Java 
> option to execute a script when OutOfMemoryError happens.  This script kills 
> Solr completely.  We are working on adding this capability when running on 
> Windows.
>
>> 2, Apparently we should not pushing too many documents to Solr, how do
>> you guys handle this? Set a limit somewhere?
>
> There are exactly two ways to deal with OOME problems: Increase the heap or 
> reduce Solr's memory requirements.  The number of documents you push to Solr 
> is unlikely to have a large effect on the amount of memory that Solr 
> requires.  Here's some information on this topic:
>
> https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap
>
> Thanks,
> Shawn
>
>
>
> [Premiere League Starts Saturday 13 August 9.30pm on 
> SBS]<http://theworldgame.sbs.com.au/>

RE: Solr Cloud with 5 servers cluster failed due to Leader out of memory

[Tim Chen]

Thanks Guys. Very very helpful.

I will probably look at consolidate 4 Solr servers into 2 bigger/better server 
- it gives more memory, and it cut down the replica the Leader needs to manage.

Also, I may look into write a script to monitor the tomcat log and if there is 
OOM, kill tomcat, then restart it. A bit dirty, but may work for a short term.

I don't know too much about how documents indexed, and how to save memory from 
that. Will probably work with a developer on this as well.

Many Thanks guys.

Cheers,
Tim

-Original Message-
From: Shawn Heisey [mailto:apa...@elyograg.org]
Sent: Friday, 5 August 2016 4:55 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr Cloud with 5 servers cluster failed due to Leader out of 
memory

On 8/4/2016 8:14 PM, Tim Chen wrote:
> Couple of thoughts: 1, If Leader goes down, it should just go down,
> like dead down, so other servers can do the election and choose the
> new leader. This at least avoids bringing down the whole cluster. Am I
> right?

Supplementing what Erick told you:

When a typical Java program throws OutOfMemoryError, program behavior is 
completely unpredictable.  There are programming techniques that can be used so 
that behavior IS predictable, but writing that code can be challenging.

Solr 5.x and 6.x, when they are started on a UNIX/Linux system, use a Java 
option to execute a script when OutOfMemoryError happens.  This script kills 
Solr completely.  We are working on adding this capability when running on 
Windows.

> 2, Apparently we should not pushing too many documents to Solr, how do
> you guys handle this? Set a limit somewhere?

There are exactly two ways to deal with OOME problems: Increase the heap or 
reduce Solr's memory requirements.  The number of documents you push to Solr is 
unlikely to have a large effect on the amount of memory that Solr requires.  
Here's some information on this topic:

https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap

Thanks,
Shawn



[Premiere League Starts Saturday 13 August 9.30pm on 
SBS]<http://theworldgame.sbs.com.au/>

Re: Solr Cloud with 5 servers cluster failed due to Leader out of memory

[Shawn Heisey]

On 8/4/2016 8:14 PM, Tim Chen wrote:
> Couple of thoughts: 1, If Leader goes down, it should just go down,
> like dead down, so other servers can do the election and choose the
> new leader. This at least avoids bringing down the whole cluster. Am I
> right? 

Supplementing what Erick told you:

When a typical Java program throws OutOfMemoryError, program behavior is
completely unpredictable.  There are programming techniques that can be
used so that behavior IS predictable, but writing that code can be
challenging.

Solr 5.x and 6.x, when they are started on a UNIX/Linux system, use a
Java option to execute a script when OutOfMemoryError happens.  This
script kills Solr completely.  We are working on adding this capability
when running on Windows.

> 2, Apparently we should not pushing too many documents to Solr, how do
> you guys handle this? Set a limit somewhere? 

There are exactly two ways to deal with OOME problems: Increase the heap
or reduce Solr's memory requirements.  The number of documents you push
to Solr is unlikely to have a large effect on the amount of memory that
Solr requires.  Here's some information on this topic:

https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap

Thanks,
Shawn

Re: Solr Cloud with 5 servers cluster failed due to Leader out of memory

[Erick Erickson]

The fact that all the shards have the same leader is somewhat of a red
herring. Until you get hundreds of shards (perhaps across a _lot_ of
collections), the additional load on the leaders is hard to measure.
If you really see this as a problem, consider the BALANCESHARDUNIQUE
and REBALANCELEADERS Collection API commands, see:
https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-BalanceSliceUnique

That said, your OOM errors indicate you simply have too many Solr
collections doing too many things with too little memory.

bq: All other non-leader server are relying on Leader to finish the
new document index.

This is not the case. The indexing is done independently on all
replicas. What's _probably_ happening here is that the leaders are
spinning off threads to pass the data on to the replicas and you're
running so close to the heap limit that spinning up those threads is
pushing you to OOM errors.

And, if my hypothesis is true you'll soon run into problems on the
non-leaders as you index more and more documents to your collections.
Consider some serious effort in terms of determining your hardware/JVM
needs, see: 
https://lucidworks.com/blog/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/

Best,
Erick

Solr Cloud with 5 servers cluster failed due to Leader out of memory

[Tim Chen]

Hi Guys,

Me again. :)

We have 5 Solr servers:
01 -04 running Solr version 4.10 and ZooKeeper service
05 running ZooKeeper only.

JVM Max Memory set to 10G.

We have around 20 collections, and for each collection, there are 4 shards, for 
each shard, there are 4 replica sitting across on 4 Solr servers.

Unfortunately most of time, all the Shards have the same Leader (eg, Solr 
server 01).

Now, If we are adding a lot of documents to Solr, and eventually Solr 01 (All 
Shard's Leader) throws Out of memory in Tomcat log, and service goes down (but 
8983 port is still responding to telnet).
At this moment, I went to see logs on Solr02, Solr03, Solr04, and there are a 
lot of "Connection time out", in another 2 minutes, all these three Solr 
servers' service goes down too!

My feeling is that, when there are a lot of documents pushing in, Leader will 
be busy with indexing, and also requesting other (non-leader) servers to do the 
index as well. All other non-leader server are relying on Leader to finish the 
new document index. At a certain point, that Solr01 (Leader) server has no more 
memory, it gives up, but other (non-leader) servers are still waiting for 
Leader to respond. The whole Solr Cloud cluster breaks from here  No more 
requests being served.

Couple of thoughts:
1, If Leader goes down, it should just go down, like dead down, so other 
servers can do the election and choose the new leader. This at least avoids 
bringing down the whole cluster. Am I right?
2, Apparently we should not pushing too many documents to Solr, how do you guys 
handle this? Set a limit somewhere?

Thanks,
Tim




[Premiere League Starts Saturday 13 August 9.30pm on 
SBS]<http://theworldgame.sbs.com.au/>

Re: Solr JSON facet range out of memory exception

[Toke Eskildsen]

On Mon, 2016-04-11 at 13:31 +0430, Ali Nazemian wrote:
> http: //10.102.1.5: 8983/solr/edgeIndex/select?q=*%3A*&fq=stat_owner_id:
> 122952&rows=0&wt=json&indent=true&facet=true&json.facet=%7bresult: %7b
> type: range,
> field: stat_date,
> start: 146027158386,
> end: 1460271583864,
> gap: 1
> %7d%7d

(1460271583864-146027158386)/1 = 131424442 (132 million) buckets.

I do not know the internal JSON code well enough, but if it creates an
object for each of the 132 million buckets, I can understand why it
OOMs. Even if it doesn't, you could easily be looking at 50K buckets,
which does seems excessive.
 
- Toke Eskildsen, State and University Library, Denmark

Re: Solr JSON facet range out of memory exception

[Ali Nazemian]

Dear Yonik,
Hi,

The entire index has 50k documents not the faceted one.  It is just a test
case right now! I used the JSON facet API, here is my query after encoding:

http: //10.102.1.5: 8983/solr/edgeIndex/select?q=*%3A*&fq=stat_owner_id:
122952&rows=0&wt=json&indent=true&facet=true&json.facet=%7bresult: %7b
type: range,
field: stat_date,
start: 146027158386,
end: 1460271583864,
gap: 1
%7d%7d

Sincerely,


On Sun, Apr 10, 2016 at 4:56 PM, Yonik Seeley  wrote:

> On Sun, Apr 10, 2016 at 3:47 AM, Ali Nazemian 
> wrote:
> > Dear all Solr users/developeres,
> > Hi,
> > I am going to use Solr JSON facet range on a date filed which is stored
> as
> > long milis. Unfortunately I got java heap space exception no matter how
> > much memory assigned to Solr Java heap! I already test that with 2g heap
> > space for Solr core with 50k documents!!
>
> You mean the entire index is 50K documents? Or do you mean the subset
> of documents to be faceted?
> If you're getting an OOM with the former (with a 2G heap), it sounds
> like you've hit some sort of bug.
>
> What does your faceting command look like?
>
> -Yonik
>



-- 
A.Nazemian

Re: Solr JSON facet range out of memory exception

[Yonik Seeley]

On Sun, Apr 10, 2016 at 3:47 AM, Ali Nazemian  wrote:
> Dear all Solr users/developeres,
> Hi,
> I am going to use Solr JSON facet range on a date filed which is stored as
> long milis. Unfortunately I got java heap space exception no matter how
> much memory assigned to Solr Java heap! I already test that with 2g heap
> space for Solr core with 50k documents!!

You mean the entire index is 50K documents? Or do you mean the subset
of documents to be faceted?
If you're getting an OOM with the former (with a 2G heap), it sounds
like you've hit some sort of bug.

What does your faceting command look like?

-Yonik

Solr JSON facet range out of memory exception

[Ali Nazemian]

Dear all Solr users/developeres,
Hi,
I am going to use Solr JSON facet range on a date filed which is stored as
long milis. Unfortunately I got java heap space exception no matter how
much memory assigned to Solr Java heap! I already test that with 2g heap
space for Solr core with 50k documents!! Is there any performance concern
regarding the JSON facet range on long field?

Sincerely,

-- 
A.Nazemian

Re: Out of memory error during full import

[Shawn Heisey]

On 2/4/2016 12:18 AM, Srinivas Kashyap wrote:
> I have implemented 'SortedMapBackedCache' in my SqlEntityProcessor for the 
> child entities in data-config.xml. When i try to do full import, i'm getting 
> OutOfMemory error(Java Heap Space). I increased the HEAP allocation to the 
> maximum extent possible. Is there a workaround to do initial data load 
> without running into this error?
>
> I found that 'batchSize=-1' parameter needs to be specified in the datasource 
> for MySql, is there a way to specify for others Databases as well?

Setting batchSize to -1 in the DIH config translates to a 'setFetchSize'
on the JDBC object of Integer.MIN_VALUE.  This is how to turn on result
streaming in MySQL.

The method for doing this with other JDBC implementations is likely to
be different.  The Microsoft driver for SQL Server uses a URL parameter,
and newer versions of that particular driver have the streaming behavior
as default.  I have no idea how to do it for any other driver, you would
need to ask the author of the driver.

When you turn on caching (SortedMapBackedCache), you are asking Solr to
put all of the data received into memory -- very similar to what happens
if result streaming is not turned on.  When the SQL result is very
large, this can require a LOT of memory.  In situations like that,
you'll just have to remove the caching.  One alternative to child
entities is to do a query using JOIN in a single entity, so that all the
data you need is returned by a single SQL query, where the heavy lifting
is done by the database server instead of Solr.

The MySQL database that serves as the information source for *my* Solr
index is hundreds of gigabytes in size, so caching it is not possible
for me.  The batchSize=-1 option is the only way to get the import to work.

Thanks,
Shawn

Out of memory error during full import

[Srinivas Kashyap]

Hello,

I have implemented 'SortedMapBackedCache' in my SqlEntityProcessor for the 
child entities in data-config.xml. When i try to do full import, i'm getting 
OutOfMemory error(Java Heap Space). I increased the HEAP allocation to the 
maximum extent possible. Is there a workaround to do initial data load without 
running into this error?

I found that 'batchSize=-1' parameter needs to be specified in the datasource 
for MySql, is there a way to specify for others Databases as well?

Thanks and Regards,
Srinivas Kashyap
DISCLAIMER: E-mails and attachments from Bamboo Rose, Inc. are confidential. If 
you are not the intended recipient, please notify the sender immediately by 
replying to the e-mail, and then delete it without making copies or using it in 
any way. No representation is made that this email or any attachments are free 
of viruses. Virus scanning is recommended and is the responsibility of the 
recipient.

Out of memory error during full import

[Srinivas Kashyap]

Hello,

I have implemented 'SortedMapBackedCache' in my SqlEntityProcessor for the 
child entities in data-config.xml. When i try to do full import, i'm getting 
OutOfMemory error(Java Heap Space). I increased the HEAP allocation to the 
maximum extent possible. Is there a workaround to do initial data load without 
running into this error?

I found that 'batchSize=-1' parameter needs to be specified in the datasource 
for MySql, is there a way to specify for others Databases as well?

Thanks and Regards,
Srinivas Kashyap

DISCLAIMER: 
E-mails and attachments from TradeStone Software, Inc. are confidential.
If you are not the intended recipient, please notify the sender immediately by
replying to the e-mail, and then delete it without making copies or using it
in any way. No representation is made that this email or any attachments are
free of viruses. Virus scanning is recommended and is the responsibility of
the recipient.

Re: Help on Out of memory when using Cursor with sort on Unique Key

[Naresh Yadav]

docvalues with reindexing does not seem viable option for me as of
now...regarding second question on Xmx4G so i tried
various options Xmx8G, Xmx10G, Xmx12G all not worked except Xmx14G which
not seem practical for production with 16gb ram.

While searching i came across :
https://issues.apache.org/jira/browse/SOLR-6121 &
https://issues.apache.org/jira/browse/SOLR-6277
I think i am also in same boat where for Cursor use i have to specify
unique key field in SORT and that leads to heap space.

Please suggest some workaround for now as SOLR-6121, SOLR-627 are still in
discussion.


On Tue, Sep 8, 2015 at 7:42 PM, Raja Pothuganti <
rpothuga...@competitrack.com> wrote:

> Hi Naresh
>
> 1) For 'sort by' fields, have you considered using DocValue=true for in
> schema definition.
> If you  are changing schema definition, you would need redo full reindex
> after backing up & deleting current index from dataDir.
> Also note that, adding docValue=true would increase size of index.
>
> 2)>Each node memory parameter : -Xms2g, -Xmx4g
> What is the basis choosing above memory sizes? Have you observed through
> jconsole or visual vm?
>
> Raja
> On 9/8/15, 8:57 AM, "Naresh Yadav"  wrote:
>
> >Cluster details :
> >
> >Solr Version  : solr-4.10.4
> >No of nodes : 2 each 16 GB RAM
> >Node of shards : 2
> >Replication : 1
> >Each node memory parameter : -Xms2g, -Xmx4g
> >
> >Collection details :
> >
> >No of docs in my collection : 12.31 million
> >Indexed field per document : 2
> >Unique key field : tids
> >Stored filed per document : varies 30- 40
> >Total index size node1+node2 = 13gb+13gb=26gb
> >
> >Query throwing Heap Space : /select?q=*:*&sort=tids+desc&rows=100&fl=tids
> >
> >Query working* : */select?q=*:*&rows=100&fl=tids
> >
> >I am using sort on unique key field tids for Cursor based pagination of
> >100
> >size.
> >
> >Already tried :
> >
> >I also tried tweaking Xmx but problem not solved..
> >I also tried q with criteria of indexed filed with only 4200 hits that
> >also
> >not working
> >when sort parameter included.
> >
> >Please help me here as i am clueless why OOM error in getting 100
> >documents.
> >
> >Thanks
> >Naresh
>
>

Re: Help on Out of memory when using Cursor with sort on Unique Key

[Raja Pothuganti]

Hi Naresh

1) For 'sort by' fields, have you considered using DocValue=true for in
schema definition.
If you  are changing schema definition, you would need redo full reindex
after backing up & deleting current index from dataDir.
Also note that, adding docValue=true would increase size of index.

2)>Each node memory parameter : -Xms2g, -Xmx4g
What is the basis choosing above memory sizes? Have you observed through
jconsole or visual vm?

Raja
On 9/8/15, 8:57 AM, "Naresh Yadav"  wrote:

>Cluster details :
>
>Solr Version  : solr-4.10.4
>No of nodes : 2 each 16 GB RAM
>Node of shards : 2
>Replication : 1
>Each node memory parameter : -Xms2g, -Xmx4g
>
>Collection details :
>
>No of docs in my collection : 12.31 million
>Indexed field per document : 2
>Unique key field : tids
>Stored filed per document : varies 30- 40
>Total index size node1+node2 = 13gb+13gb=26gb
>
>Query throwing Heap Space : /select?q=*:*&sort=tids+desc&rows=100&fl=tids
>
>Query working* : */select?q=*:*&rows=100&fl=tids
>
>I am using sort on unique key field tids for Cursor based pagination of
>100
>size.
>
>Already tried :
>
>I also tried tweaking Xmx but problem not solved..
>I also tried q with criteria of indexed filed with only 4200 hits that
>also
>not working
>when sort parameter included.
>
>Please help me here as i am clueless why OOM error in getting 100
>documents.
>
>Thanks
>Naresh

Help on Out of memory when using Cursor with sort on Unique Key

[Naresh Yadav]

Cluster details :

Solr Version  : solr-4.10.4
No of nodes : 2 each 16 GB RAM
Node of shards : 2
Replication : 1
Each node memory parameter : -Xms2g, -Xmx4g

Collection details :

No of docs in my collection : 12.31 million
Indexed field per document : 2
Unique key field : tids
Stored filed per document : varies 30- 40
Total index size node1+node2 = 13gb+13gb=26gb

Query throwing Heap Space : /select?q=*:*&sort=tids+desc&rows=100&fl=tids

Query working* : */select?q=*:*&rows=100&fl=tids

I am using sort on unique key field tids for Cursor based pagination of 100
size.

Already tried :

I also tried tweaking Xmx but problem not solved..
I also tried q with criteria of indexed filed with only 4200 hits that also
not working
when sort parameter included.

Please help me here as i am clueless why OOM error in getting 100 documents.

Thanks
Naresh

Re: out of memory when trying to sort by id in a 1.5 billion index

[Chris Hostetter]

: For sorting DocValues are the best option I think.

yep, definitely a good idea.

: > I have a usecase for using cursorpage and when tried to check this, I got
: > outOfMemory just for sorting by id.

what does the field/fieldType for your uniqueKey field look like?

If you aren't using DocValues, then the amount of RAM needed is going to 
vary widely depending on the datatype used by the FieldCache.

-Hoss
http://www.lucidworks.com/

Re: out of memory when trying to sort by id in a 1.5 billion index

[Yago Riveiro]

For sorting DocValues are the best option I think.

—
/Yago Riveiro

On Fri, Nov 7, 2014 at 12:45 PM, adfel70  wrote:

> hi
> I have 11 machines in my cluster.
> each machine 128GB memory, 2 solr jvm's with 12gb heap each.
> cluster has 7 shard, 3 replicas.
> 1.5 billion docs total.
> most user queries are pretty simple for now, sorting by date fields and
> another field the has around 1000 unique values.
> I have a usecase for using cursorpage and when tried to check this, I got
> outOfMemory just for sorting by id.
> I read in old posts that I should add heap memory, and I can do that, but I
> would rather not .
> All other usecases I have are using stable 8gb heap .
> Any other way to handle this in solr 4.8?
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/out-of-memory-when-trying-to-sort-by-id-in-a-1-5-billion-index-tp4168156.html
> Sent from the Solr - User mailing list archive at Nabble.com.

out of memory when trying to sort by id in a 1.5 billion index

[adfel70]

hi
I have 11 machines in my cluster.
each machine 128GB memory, 2 solr jvm's with 12gb heap each.
cluster has 7 shard, 3 replicas.
1.5 billion docs total.
most user queries are pretty simple for now, sorting by date fields and
another field the has around 1000 unique values.

I have a usecase for using cursorpage and when tried to check this, I got
outOfMemory just for sorting by id.
I read in old posts that I should add heap memory, and I can do that, but I
would rather not .
All other usecases I have are using stable 8gb heap .
Any other way to handle this in solr 4.8?



--
View this message in context: 
http://lucene.472066.n3.nabble.com/out-of-memory-when-trying-to-sort-by-id-in-a-1-5-billion-index-tp4168156.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr/lucene 4.10 out of memory issues

[rulinma]

mark.



--
View this message in context: 
http://lucene.472066.n3.nabble.com/solr-lucene-4-10-out-of-memory-issues-tp4158262p4159829.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Large Transaction Logs & Out of memory

[Shawn Heisey]

On 9/18/2014 5:36 AM, or gerson wrote:
> what is the cause of the other? the outOfMemory causes the tlog to get
> bigger (cause the autoCommit does not happen) or the tlog size causes the
> outOfMemorty error?

Program behavior (any Java program, not just Solr) becomes completely
undefined when you get out of memory errors.  It's exceptionally
difficult to write programs that have defined behavior in an OOM
situation.  Not impossible, but *very* hard, so I'm reasonably sure that
no attempt has been made in the Lucene/Solr codebase.

To eliminate the OOM, you need to either make your Java heap larger, or
find ways to reduce Solr's heap requirements.

http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap

Thanks,
Shawn

Large Transaction Logs & Out of memory

[or gerson]

Hello,

i have encountered this problem where one of the transaction logs size
reached about 60GB
in the solr log i have seen  outOfMemeoryError during delta import.

autoCommit  is configured as :

 2
 3
 false
   
* i think a 15min time for hard commit might be better

buffers are configured as:

128
2

solr is used for indexing a small amount of documents,  few thousands at
most.

jvm is running with -XX:MaxPermSize=256m -Xms512m -Xmx2048m

the delta import is scheduled to run every minute.
-

what is the cause of the other? the outOfMemory causes the tlog to get
bigger (cause the autoCommit does not happen) or the tlog size causes the
outOfMemorty error?

another weird thing im seeing is that every minute in the solr log im seeing
the exact same lines
13 Sep 2014 00:04:50,845 [Thread-119233] INFO  DocBuilder - Completed
ModifiedRowKey for Entity: document rows obtained : 6203
13 Sep 2014 00:04:50,845 [Thread-119233] INFO  DocBuilder - Completed
DeletedRowKey for Entity: document rows obtained : 308
13 Sep 2014 00:39:43,618 [Thread-119303] INFO  DocBuilder - Completed
ModifiedRowKey for Entity: permission_set rows obtained : 12699

i think this shows that the "last_index_time" property is not being updated
(maybe due to outOfMem error)
since the same data is being retrieved over and over.


Thanks for the help










--
View this message in context: 
http://lucene.472066.n3.nabble.com/Large-Transaction-Logs-Out-of-memory-tp4159636.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: solr/lucene 4.10 out of memory issues

[Luis Carlos Guerrero]

I checked and these 'insanity' cached keys correspond to fields we use for
both grouping and faceting. The same behavior is documented here:
https://issues.apache.org/jira/browse/SOLR-4866, although I have single
shards for every replica which the jira says is a setup which should not
generate these issues.

What I don't get is why the cluster was running fine with solr 4.4,
although double checking I was using LUCENE_40 as the match version. If I
use this match version in my current running 4.10 cluster will it make a
difference, or will I experience more issues than if I just roll back to
4.4 with LUCENE_40 match version? The problem in the end is that the
fieldcache grows unlimitedly. I'm thinking its because of the insanity
entries but I'm not really sure. It seem like a really big problem to leave
unattended or is the use case for faceting and grouping on the same field
not that common?

On Tue, Sep 16, 2014 at 11:06 AM, Luis Carlos Guerrero <
lcguerreroc...@gmail.com> wrote:

> Thanks for the response, I've been working on solving some of the most
> evident issues and I also added your garbage collector parameters. First of
> all the Lucene field cache is being filled with some entries which are
> marked as 'insanity'. Some of these were related to a custom field that we
> use for our ranking. We fixed our custom plugin classes so that we wouldn't
> see any entries related to those fields there, but it seems there are other
> related problems with the field cache. Mainly the cache is being filled
> with these types of insanity entries:
>
> 'SUBREADER: Found caches for descendants of StandardDirectoryReader'
>
> They are all related to standard solr fields. Could it be that our current
> schemas and configs have some incorrect setting that is not compliant with
> this lucene version? I'll keep investigating the subject but if there is
> any additional information you can give me about these types of field cache
> insanity warnings it would be really helpful.
>
> On Thu, Sep 11, 2014 at 3:00 PM, Timothy Potter 
> wrote:
>
>> Probably need to look at it running with a profiler to see what's up.
>> Here's a few additional flags that might help the GC work better for
>> you (which is not to say there isn't a leak somewhere):
>>
>> -XX:MaxTenuringThreshold=8 -XX:CMSInitiatingOccupancyFraction=40
>>
>> This should lead to a nice up-and-down GC profile over time.
>>
>> On Thu, Sep 11, 2014 at 10:52 AM, Luis Carlos Guerrero
>>  wrote:
>> > hey guys,
>> >
>> > I'm running a solrcloud cluster consisting of five nodes. My largest
>> index
>> > contains 2.5 million documents and occupies about 6 gigabytes of disk
>> > space. We recently switched to the latest solr version (4.10) from
>> version
>> > 4.4.1 which we ran successfully for about a year without any major
>> issues.
>> > From the get go we started having memory problems caused by the CMS old
>> > heap usage being filled up incrementally. It starts out with a very low
>> > memory consumption and after 12 hours or so it ends up using up all
>> > available heap space. We thought it could be one of the caches we had
>> > configured, so we reduced our main core filter cache max size from 1024
>> to
>> > 512 elements. The only thing we accomplished was that the cluster ran
>> for a
>> > longer time than before.
>> >
>> > I generated several heapdumps and basically what is filling up the heap
>> is
>> > lucene's field cache. it gets bigger and bigger until it fills up all
>> > available memory.
>> >
>> > My jvm memory settings are the following:
>> >
>> > -Xms15g -Xmx15g -XX:PermSize=512m -XX:MaxPermSize=512m -XX:NewSize=5g
>> > -XX:MaxNewSize=5g
>> > -XX:+UseParNewGC -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDateStamps
>> > -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError
>> -XX:+UseConcMarkSweepGC
>> > What's weird to me is that we didn't have this problem before, I'm
>> thinking
>> > this is some kind of memory leak issue present in the new lucene. We ran
>> > our old cluster for several weeks at a time without having to redeploy
>> > because of config changes or other reasons. Was there some issue
>> reported
>> > related to elevated memory consumption by the field cache?
>> >
>> > any help would be greatly appreciated.
>> >
>> > regards,
>> >
>> > --
>> > Luis Carlos Guerrero
>> > about.me/luis.guerrero
>>
>
>
>
> --
> Luis Carlos Guerrero
> about.me/luis.guerrero
>



-- 
Luis Carlos Guerrero
about.me/luis.guerrero

Re: solr/lucene 4.10 out of memory issues

[Luis Carlos Guerrero]

Thanks for the response, I've been working on solving some of the most
evident issues and I also added your garbage collector parameters. First of
all the Lucene field cache is being filled with some entries which are
marked as 'insanity'. Some of these were related to a custom field that we
use for our ranking. We fixed our custom plugin classes so that we wouldn't
see any entries related to those fields there, but it seems there are other
related problems with the field cache. Mainly the cache is being filled
with these types of insanity entries:

'SUBREADER: Found caches for descendants of StandardDirectoryReader'

They are all related to standard solr fields. Could it be that our current
schemas and configs have some incorrect setting that is not compliant with
this lucene version? I'll keep investigating the subject but if there is
any additional information you can give me about these types of field cache
insanity warnings it would be really helpful.

On Thu, Sep 11, 2014 at 3:00 PM, Timothy Potter 
wrote:

> Probably need to look at it running with a profiler to see what's up.
> Here's a few additional flags that might help the GC work better for
> you (which is not to say there isn't a leak somewhere):
>
> -XX:MaxTenuringThreshold=8 -XX:CMSInitiatingOccupancyFraction=40
>
> This should lead to a nice up-and-down GC profile over time.
>
> On Thu, Sep 11, 2014 at 10:52 AM, Luis Carlos Guerrero
>  wrote:
> > hey guys,
> >
> > I'm running a solrcloud cluster consisting of five nodes. My largest
> index
> > contains 2.5 million documents and occupies about 6 gigabytes of disk
> > space. We recently switched to the latest solr version (4.10) from
> version
> > 4.4.1 which we ran successfully for about a year without any major
> issues.
> > From the get go we started having memory problems caused by the CMS old
> > heap usage being filled up incrementally. It starts out with a very low
> > memory consumption and after 12 hours or so it ends up using up all
> > available heap space. We thought it could be one of the caches we had
> > configured, so we reduced our main core filter cache max size from 1024
> to
> > 512 elements. The only thing we accomplished was that the cluster ran
> for a
> > longer time than before.
> >
> > I generated several heapdumps and basically what is filling up the heap
> is
> > lucene's field cache. it gets bigger and bigger until it fills up all
> > available memory.
> >
> > My jvm memory settings are the following:
> >
> > -Xms15g -Xmx15g -XX:PermSize=512m -XX:MaxPermSize=512m -XX:NewSize=5g
> > -XX:MaxNewSize=5g
> > -XX:+UseParNewGC -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDateStamps
> > -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError
> -XX:+UseConcMarkSweepGC
> > What's weird to me is that we didn't have this problem before, I'm
> thinking
> > this is some kind of memory leak issue present in the new lucene. We ran
> > our old cluster for several weeks at a time without having to redeploy
> > because of config changes or other reasons. Was there some issue reported
> > related to elevated memory consumption by the field cache?
> >
> > any help would be greatly appreciated.
> >
> > regards,
> >
> > --
> > Luis Carlos Guerrero
> > about.me/luis.guerrero
>



-- 
Luis Carlos Guerrero
about.me/luis.guerrero

Re: solr/lucene 4.10 out of memory issues

[Timothy Potter]

Probably need to look at it running with a profiler to see what's up.
Here's a few additional flags that might help the GC work better for
you (which is not to say there isn't a leak somewhere):

-XX:MaxTenuringThreshold=8 -XX:CMSInitiatingOccupancyFraction=40

This should lead to a nice up-and-down GC profile over time.

On Thu, Sep 11, 2014 at 10:52 AM, Luis Carlos Guerrero
 wrote:
> hey guys,
>
> I'm running a solrcloud cluster consisting of five nodes. My largest index
> contains 2.5 million documents and occupies about 6 gigabytes of disk
> space. We recently switched to the latest solr version (4.10) from version
> 4.4.1 which we ran successfully for about a year without any major issues.
> From the get go we started having memory problems caused by the CMS old
> heap usage being filled up incrementally. It starts out with a very low
> memory consumption and after 12 hours or so it ends up using up all
> available heap space. We thought it could be one of the caches we had
> configured, so we reduced our main core filter cache max size from 1024 to
> 512 elements. The only thing we accomplished was that the cluster ran for a
> longer time than before.
>
> I generated several heapdumps and basically what is filling up the heap is
> lucene's field cache. it gets bigger and bigger until it fills up all
> available memory.
>
> My jvm memory settings are the following:
>
> -Xms15g -Xmx15g -XX:PermSize=512m -XX:MaxPermSize=512m -XX:NewSize=5g
> -XX:MaxNewSize=5g
> -XX:+UseParNewGC -XX:+ExplicitGCInvokesConcurrent -XX:+PrintGCDateStamps
> -XX:+PrintGCDetails -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC
> What's weird to me is that we didn't have this problem before, I'm thinking
> this is some kind of memory leak issue present in the new lucene. We ran
> our old cluster for several weeks at a time without having to redeploy
> because of config changes or other reasons. Was there some issue reported
> related to elevated memory consumption by the field cache?
>
> any help would be greatly appreciated.
>
> regards,
>
> --
> Luis Carlos Guerrero
> about.me/luis.guerrero