[jira] [Updated] (SOLR-11196) Solr 6.5.0 consuming entire Heap suddenly while working smoothly on Solr 6.1.0
[
https://issues.apache.org/jira/browse/SOLR-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomás Fernández Löbbe updated SOLR-11196:
-
Priority: Major (was: Critical)
> Solr 6.5.0 consuming entire Heap suddenly while working smoothly on Solr 6.1.0
> --
>
> Key: SOLR-11196
> URL: https://issues.apache.org/jira/browse/SOLR-11196
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 6.5, 6.6
>Reporter: Amit
>
> Please note, this issue does not occurs on Solr-6.1.0 while the same occurs
> on Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0
> version.
> We have been hit by a Solr Behavior in production which we are unable to
> debug. To start with here are the configurations for solr:
> Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
> below.
> *JVM Config:*
>
> {code:java}
> -Xms2048m
> -Xmx4096m
> -XX:+ParallelRefProcEnabled
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:CMSInitiatingOccupancyFraction=50
> {code}
> Rest all are default values.
> *Solr Config* :
>
> {code:java}
>
>
> {solr.autoCommit.maxTime:30}
> false
>
>
>
> {solr.autoSoftCommit.maxTime:90}
>
>
>
> 1024
>autowarmCount="0" />
>autowarmCount="0" />
>autowarmCount="0" />
>initialSize="0" autowarmCount="10" regenerator="solr.NoOpRegenerator" />
> true
> 20
> ${solr.query.max.docs:40}
>
> false
> 2
>
> {code}
> *The Host (AWS) configurations are:*
> RAM: 7.65GB
> Cores: 4
> Now, our solr works perfectly fine for hours and sometimes for days but
> sometimes suddenly memory jumps up and the GC kicks in causing long big
> pauses with not much to recover. We are seeing this happening most often when
> one or multiple segments gets added or deleted post a hard commit. It doesn't
> matter how many documents got indexed. The images attached shows that just 1
> document was indexed, causing an addition of one segment and it all got
> messed up till we restarted the Solr.
> Here are the images from NewRelic and Sematext (Kindly click on the links to
> view):
> [JVM Heap Memory Image | https://i.stack.imgur.com/9dQAy.png]
> [1 Document and 1 Segment addition Image |
> https://i.stack.imgur.com/6N4FC.png]
> Update: Here is the JMap output when SOLR last died, we have now increased
> the JVM memory to xmx of 12GB:
>
> {code:java}
> num #instances #bytes class name
> --
> 1: 11210921 1076248416
> org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat$IntBlockTermState
> 2: 10623486 934866768 [Lorg.apache.lucene.index.TermState;
> 3: 15567646 475873992 [B
> 4: 10623485 424939400
> org.apache.lucene.search.spans.SpanTermQuery$SpanTermWeight
> 5: 15508972 372215328 org.apache.lucene.util.BytesRef
> 6: 15485834 371660016 org.apache.lucene.index.Term
> 7: 15477679 371464296
> org.apache.lucene.search.spans.SpanTermQuery
> 8: 10623486 339951552 org.apache.lucene.index.TermContext
> 9: 1516724 150564320 [Ljava.lang.Object;
> 10:724486 50948800 [C
> 11: 1528110 36674640 java.util.ArrayList
> 12:849884 27196288
> org.apache.lucene.search.spans.SpanNearQuery
> 13:582008 23280320
> org.apache.lucene.search.spans.SpanNearQuery$SpanNearWeight
> 14:481601 23116848 org.apache.lucene.document.FieldType
> 15:623073 19938336 org.apache.lucene.document.StoredField
> 16:721649 17319576 java.lang.String
> 17: 327297329640 [J
> 18: 146435788376 [F
> {code}
> The load on Solr is not much - max it goes to 2000 requests per minute. The
> indexing load can sometimes be in burst but most of the time its pretty low.
> But as mentioned above sometimes even a single document indexing can put solr
> into tizzy and sometimes it just works like a charm.
> Edit :
> The last configuration on which 6.1 works but not 6.5 is:
> *JVM Config:*
>
> {code:java}
> Xms: 2 GB
> Xmx: 12 GB
> {code}
> *Solr Config:*
> We also removed soft commit.
> {code:java}
>
>
>${solr.autoCommit.maxTime:30}
>true
>
> {code}
> *The Host (AWS) configurations:*
> RAM: 16GB
> Cores: 4
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.
[jira] [Updated] (SOLR-11196) Solr 6.5.0 consuming entire Heap suddenly while working smoothly on Solr 6.1.0
[
https://issues.apache.org/jira/browse/SOLR-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit updated SOLR-11196:
Description:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
{code:java}
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
{code}
Rest all are default values.
*Solr Config* :
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
{code}
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the Solr.
Here are the images from NewRelic and Sematext (Kindly click on the links to
view):
[JVM Heap Memory Image | https://i.stack.imgur.com/9dQAy.png]
[1 Document and 1 Segment addition Image | https://i.stack.imgur.com/6N4FC.png]
Update: Here is the JMap output when SOLR last died, we have now increased the
JVM memory to xmx of 12GB:
{code:java}
num #instances #bytes class name
--
1: 11210921 1076248416
org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat$IntBlockTermState
2: 10623486 934866768 [Lorg.apache.lucene.index.TermState;
3: 15567646 475873992 [B
4: 10623485 424939400
org.apache.lucene.search.spans.SpanTermQuery$SpanTermWeight
5: 15508972 372215328 org.apache.lucene.util.BytesRef
6: 15485834 371660016 org.apache.lucene.index.Term
7: 15477679 371464296 org.apache.lucene.search.spans.SpanTermQuery
8: 10623486 339951552 org.apache.lucene.index.TermContext
9: 1516724 150564320 [Ljava.lang.Object;
10:724486 50948800 [C
11: 1528110 36674640 java.util.ArrayList
12:849884 27196288 org.apache.lucene.search.spans.SpanNearQuery
13:582008 23280320
org.apache.lucene.search.spans.SpanNearQuery$SpanNearWeight
14:481601 23116848 org.apache.lucene.document.FieldType
15:623073 19938336 org.apache.lucene.document.StoredField
16:721649 17319576 java.lang.String
17: 327297329640 [J
18: 146435788376 [F
{code}
The load on Solr is not much - max it goes to 2000 requests per minute. The
indexing load can sometimes be in burst but most of the time its pretty low.
But as mentioned above sometimes even a single document indexing can put solr
into tizzy and sometimes it just works like a charm.
Edit :
The last configuration on which 6.1 works but not 6.5 is:
*JVM Config:*
{code:java}
Xms: 2 GB
Xmx: 12 GB
{code}
*Solr Config:*
We also removed soft commit.
{code:java}
${solr.autoCommit.maxTime:30}
true
{code}
*The Host (AWS) configurations:*
RAM: 16GB
Cores: 4
was:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
{code:java}
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
{code}
Rest all are default values.
*Solr Config* :
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
{code}
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing
[jira] [Updated] (SOLR-11196) Solr 6.5.0 consuming entire Heap suddenly while working smoothly on Solr 6.1.0
[
https://issues.apache.org/jira/browse/SOLR-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit updated SOLR-11196:
Description:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
{code:java}
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
{code}
Rest all are default values.
*Solr Config* :
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
{code}
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the Solr.
Here are the images from NewRelic and Sematext (Kindly click on the links to
view):
[JVM Heap Memory Image | https://i.stack.imgur.com/9dQAy.png]
[1 Document and 1 Segment addition Image | https://i.stack.imgur.com/6N4FC.png]
Update: Here is the JMap output when SOLR last died, we have now increased the
JVM memory to xmx of 12GB:
{code:java}
num #instances #bytes class name
--
1: 11210921 1076248416
org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat$IntBlockTermState
2: 10623486 934866768 [Lorg.apache.lucene.index.TermState;
3: 15567646 475873992 [B
4: 10623485 424939400
org.apache.lucene.search.spans.SpanTermQuery$SpanTermWeight
5: 15508972 372215328 org.apache.lucene.util.BytesRef
6: 15485834 371660016 org.apache.lucene.index.Term
7: 15477679 371464296 org.apache.lucene.search.spans.SpanTermQuery
8: 10623486 339951552 org.apache.lucene.index.TermContext
9: 1516724 150564320 [Ljava.lang.Object;
10:724486 50948800 [C
11: 1528110 36674640 java.util.ArrayList
12:849884 27196288 org.apache.lucene.search.spans.SpanNearQuery
13:582008 23280320
org.apache.lucene.search.spans.SpanNearQuery$SpanNearWeight
14:481601 23116848 org.apache.lucene.document.FieldType
15:623073 19938336 org.apache.lucene.document.StoredField
16:721649 17319576 java.lang.String
17: 327297329640 [J
18: 146435788376 [F
{code}
The load on Solr is not much - max it goes to 2000 requests per minute. The
indexing load can sometimes be in burst but most of the time its pretty low.
But as mentioned above sometimes even a single document indexing can put solr
into tizzy and sometimes it just works like a charm.
Edit :
The last configuration on which 6.1 works but not 6.5 is:
*JVM Config:*
{code:java}
Xms: 2 GB
Xmx : 12 GB
{code}
*Solr Config:*
We also removed soft commit.
{code:java}
${solr.autoCommit.maxTime:30}
true
{code}
*The Host (AWS) configurations:*
RAM: 16GB
Cores: 4
was:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
{code:java}
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
{code}
Rest all are default values.
*Solr Config* :
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
{code}
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causin
[jira] [Updated] (SOLR-11196) Solr 6.5.0 consuming entire Heap suddenly while working smoothly on Solr 6.1.0
[
https://issues.apache.org/jira/browse/SOLR-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit updated SOLR-11196:
Description:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
{code:java}
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
{code}
Rest all are default values.
*Solr Config* :
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
{code}
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the Solr.
Here are the images from NewRelic and Sematext (Kindly click on the links to
view):
[JVM Heap Memory Image | https://i.stack.imgur.com/9dQAy.png]
[1 Document and 1 Segment addition Image | https://i.stack.imgur.com/6N4FC.png]
Update: Here is the JMap output when SOLR last died, we have now increased the
JVM memory to xmx of 12GB:
{code:java}
num #instances #bytes class name
--
1: 11210921 1076248416
org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat$IntBlockTermState
2: 10623486 934866768 [Lorg.apache.lucene.index.TermState;
3: 15567646 475873992 [B
4: 10623485 424939400
org.apache.lucene.search.spans.SpanTermQuery$SpanTermWeight
5: 15508972 372215328 org.apache.lucene.util.BytesRef
6: 15485834 371660016 org.apache.lucene.index.Term
7: 15477679 371464296 org.apache.lucene.search.spans.SpanTermQuery
8: 10623486 339951552 org.apache.lucene.index.TermContext
9: 1516724 150564320 [Ljava.lang.Object;
10:724486 50948800 [C
11: 1528110 36674640 java.util.ArrayList
12:849884 27196288 org.apache.lucene.search.spans.SpanNearQuery
13:582008 23280320
org.apache.lucene.search.spans.SpanNearQuery$SpanNearWeight
14:481601 23116848 org.apache.lucene.document.FieldType
15:623073 19938336 org.apache.lucene.document.StoredField
16:721649 17319576 java.lang.String
17: 327297329640 [J
18: 146435788376 [F
{code}
The load on Solr is not much - max it goes to 2000 requests per minute. The
indexing load can sometimes be in burst but most of the time its pretty low.
But as mentioned above sometimes even a single document indexing can put solr
into tizzy and sometimes it just works like a charm.
Edit :
The last configuration on which 6.1 works but not 6.5 is:
*JVM Config:*
{code:java}
Xms: 2 GB
Xmx : 12 GB
{code}
*Solr Config: *
We also removed soft commit.
{code:java}
${solr.autoCommit.maxTime:30}
true
{code}
*The Host (AWS) configurations:*
RAM: 16GB
Cores: 4
was:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
{code:java}
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
{code}
Rest all are default values.
*Solr Config:*
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
{code}
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causin
[jira] [Updated] (SOLR-11196) Solr 6.5.0 consuming entire Heap suddenly while working smoothly on Solr 6.1.0
[
https://issues.apache.org/jira/browse/SOLR-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit updated SOLR-11196:
Description:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
{code:java}
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
{code}
Rest all are default values.
*Solr Config:*
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
{code}
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the Solr.
Here are the images from NewRelic and Sematext (Kindly click on the links to
view):
[JVM Heap Memory Image | https://i.stack.imgur.com/9dQAy.png]
[1 Document and 1 Segment addition Image | https://i.stack.imgur.com/6N4FC.png]
Update: Here is the JMap output when SOLR last died, we have now increased the
JVM memory to xmx of 12GB:
{code:java}
num #instances #bytes class name
--
1: 11210921 1076248416
org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat$IntBlockTermState
2: 10623486 934866768 [Lorg.apache.lucene.index.TermState;
3: 15567646 475873992 [B
4: 10623485 424939400
org.apache.lucene.search.spans.SpanTermQuery$SpanTermWeight
5: 15508972 372215328 org.apache.lucene.util.BytesRef
6: 15485834 371660016 org.apache.lucene.index.Term
7: 15477679 371464296 org.apache.lucene.search.spans.SpanTermQuery
8: 10623486 339951552 org.apache.lucene.index.TermContext
9: 1516724 150564320 [Ljava.lang.Object;
10:724486 50948800 [C
11: 1528110 36674640 java.util.ArrayList
12:849884 27196288 org.apache.lucene.search.spans.SpanNearQuery
13:582008 23280320
org.apache.lucene.search.spans.SpanNearQuery$SpanNearWeight
14:481601 23116848 org.apache.lucene.document.FieldType
15:623073 19938336 org.apache.lucene.document.StoredField
16:721649 17319576 java.lang.String
17: 327297329640 [J
18: 146435788376 [F
{code}
The load on Solr is not much - max it goes to 2000 requests per minute. The
indexing load can sometimes be in burst but most of the time its pretty low.
But as mentioned above sometimes even a single document indexing can put solr
into tizzy and sometimes it just works like a charm.
Edit :
The last configuration on which 6.1 works but not 6.5 is:
*JVM Config:*
{code:java}
Xms: 2 GB
Xmx : 12 GB
{code}
*Solr Config: *
We also removed soft commit.
{code:java}
${solr.autoCommit.maxTime:30}
true
{code}
*The Host (AWS) configurations:*
RAM: 16GB
Cores: 4
was:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
{code:java}
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
{code}
Rest all are default values.
*Solr Config:*
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
{code}
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing
Re: [jira] [Updated] (SOLR-11196) Solr 6.5.0 consuming entire Heap suddenly while working smoothly on Solr 6.1.0
1. This should be a question on the solr-u...@lucene.apache.org, not a bug
report.
2. A 12 GB heap on an instance with 7.65 GB of RAM is a fatal configuration. A
full GC will cause lots of swapping and an extreme slowdown.
3. A 4 GB heap on an instance with 7.65 GB of RAM is not a good configuration.
That does not leave enough room for the OS, other processes, and file buffers
to cache Solr’s index files.
4. That instance is pretty small for Solr. The smallest AWS instance we run has
15 GB of RAM. We run an 8 GB heap. Check the disk access on New Relic during
the slowdown.
5. Does this instance swap to magnetic disk? Are the Solr indexes on magnetic
ephemeral or magnetic EBS? Check the iops on New Relic. When you hit the max
iops for a disk volume, very bad performance things happen.
6. Set -Xms equal to -Xmx. Growing the heap to max at startup is a waste of
time and makes Solr slow at the beginning. The heap will always get to max.
7. Setting a longer time for auto soft commit than for auto hard commit is
nonsense. Just don’t do the soft commit.
wunder
Walter Underwood
wun...@wunderwood.org
http://observer.wunderwood.org/ (my blog)
> On Aug 4, 2017, at 7:13 AM, Amit (JIRA) wrote:
>
>
> [
> https://issues.apache.org/jira/browse/SOLR-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
> ]
>
> Amit updated SOLR-11196:
>
>Description:
> Please note, this issue does not occurs on Solr-6.1.0 while the same occurs
> on Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0
> version.
>
> We have been hit by a Solr Behavior in production which we are unable to
> debug. To start with here are the configurations for solr:
>
> Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
> below.
>
> *JVM Config:*
>
>
> {code:java}
> -Xms2048m
> -Xmx4096m
> -XX:+ParallelRefProcEnabled
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:CMSInitiatingOccupancyFraction=50
> {code}
>
> Rest all are default values.
>
> *Solr Config:*
>
>
> {code:java}
>
>
> {solr.autoCommit.maxTime:30}
> false
>
>
>
> {solr.autoSoftCommit.maxTime:90}
>
>
>
>
> 1024
> autowarmCount="0" />
> autowarmCount="0" />
> autowarmCount="0" />
> initialSize="0" autowarmCount="10" regenerator="solr.NoOpRegenerator" />
> true
> 20
> ${solr.query.max.docs:40}
>
> false
> 2
>
> {code}
>
> *The Host (AWS) configurations are:*
>
> RAM: 7.65GB
> Cores: 4
>
> Now, our solr works perfectly fine for hours and sometimes for days but
> sometimes suddenly memory jumps up and the GC kicks in causing long big
> pauses with not much to recover. We are seeing this happening most often when
> one or multiple segments gets added or deleted post a hard commit. It doesn't
> matter how many documents got indexed. The images attached shows that just 1
> document was indexed, causing an addition of one segment and it all got
> messed up till we restarted the Solr.
>
> Here are the images from NewRelic and Sematext (Kindly click on the links to
> view):
>
> JVM Heap Memory Image : [https://i.stack.imgur.com/9dQAy.png]
> 1 Document and 1 Segment addition Image: [https://i.stack.imgur.com/6N4FC.png]
>
> Update: Here is the JMap output when SOLR last died, we have now increased
> the JVM memory to xmx of 12GB:
>
>
> {code:java}
> num #instances #bytes class name
> --
> 1: 11210921 1076248416
> org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat$IntBlockTermState
> 2: 10623486 934866768 [Lorg.apache.lucene.index.TermState;
> 3: 15567646 475873992 [B
> 4: 10623485 424939400
> org.apache.lucene.search.spans.SpanTermQuery$SpanTermWeight
> 5: 15508972 372215328 org.apache.lucene.util.BytesRef
> 6: 15485834 371660016 org.apache.lucene.index.Term
> 7: 15477679 371464296 org.apache.lucene.search.spans.SpanTermQuery
> 8: 10623486 339951552 org.apache.lucene.index.TermContext
> 9: 1516724 150564320 [Ljava.lang.Object;
> 10:724486 50948800 [C
> 11: 1528110 36674640 java.util.ArrayList
> 12:849884 27196288 org.apache.lucene.search.spans.SpanNearQuery
> 13:582008 23280320
> org.apache.lucene.search.spans.SpanNearQuery$SpanNearWeight
> 14:481601 23116848 org.apache.lucene.document.FieldType
> 15:623073 19938336 org.apache.lucene.document.StoredField
> 16:721649 17319576 java.lang.String
> 17: 327297329640 [J
> 18: 146435788376 [F
> {code}
>
>
> The load on Solr is not much - max it goes to 2000 requests per minute. The
> indexing load can sometimes be in burst but most of the time its pretty low.
> But as mentioned
[jira] [Updated] (SOLR-11196) Solr 6.5.0 consuming entire Heap suddenly while working smoothly on Solr 6.1.0
[
https://issues.apache.org/jira/browse/SOLR-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit updated SOLR-11196:
Description:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
{code:java}
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
{code}
Rest all are default values.
*Solr Config:*
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
{code}
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the Solr.
Here are the images from NewRelic and Sematext (Kindly click on the links to
view):
[JVM Heap Memory Image | https://i.stack.imgur.com/9dQAy.png]
[1 Document and 1 Segment addition Image | https://i.stack.imgur.com/6N4FC.png]
Update: Here is the JMap output when SOLR last died, we have now increased the
JVM memory to xmx of 12GB:
{code:java}
num #instances #bytes class name
--
1: 11210921 1076248416
org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat$IntBlockTermState
2: 10623486 934866768 [Lorg.apache.lucene.index.TermState;
3: 15567646 475873992 [B
4: 10623485 424939400
org.apache.lucene.search.spans.SpanTermQuery$SpanTermWeight
5: 15508972 372215328 org.apache.lucene.util.BytesRef
6: 15485834 371660016 org.apache.lucene.index.Term
7: 15477679 371464296 org.apache.lucene.search.spans.SpanTermQuery
8: 10623486 339951552 org.apache.lucene.index.TermContext
9: 1516724 150564320 [Ljava.lang.Object;
10:724486 50948800 [C
11: 1528110 36674640 java.util.ArrayList
12:849884 27196288 org.apache.lucene.search.spans.SpanNearQuery
13:582008 23280320
org.apache.lucene.search.spans.SpanNearQuery$SpanNearWeight
14:481601 23116848 org.apache.lucene.document.FieldType
15:623073 19938336 org.apache.lucene.document.StoredField
16:721649 17319576 java.lang.String
17: 327297329640 [J
18: 146435788376 [F
{code}
The load on Solr is not much - max it goes to 2000 requests per minute. The
indexing load can sometimes be in burst but most of the time its pretty low.
But as mentioned above sometimes even a single document indexing can put solr
into tizzy and sometimes it just works like a charm.
was:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
{code:java}
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
{code}
Rest all are default values.
*Solr Config:*
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
{code}
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed
[jira] [Updated] (SOLR-11196) Solr 6.5.0 consuming entire Heap suddenly while working smoothly on Solr 6.1.0
[
https://issues.apache.org/jira/browse/SOLR-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit updated SOLR-11196:
Description:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
{code:java}
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
{code}
Rest all are default values.
*Solr Config:*
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
{code}
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the Solr.
Here are the images from NewRelic and Sematext (Kindly click on the links to
view):
[JVM Heap Memory Image | https://i.stack.imgur.com/9dQAy.png]
1 Document and 1 Segment addition Image: [https://i.stack.imgur.com/6N4FC.png]
Update: Here is the JMap output when SOLR last died, we have now increased the
JVM memory to xmx of 12GB:
{code:java}
num #instances #bytes class name
--
1: 11210921 1076248416
org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat$IntBlockTermState
2: 10623486 934866768 [Lorg.apache.lucene.index.TermState;
3: 15567646 475873992 [B
4: 10623485 424939400
org.apache.lucene.search.spans.SpanTermQuery$SpanTermWeight
5: 15508972 372215328 org.apache.lucene.util.BytesRef
6: 15485834 371660016 org.apache.lucene.index.Term
7: 15477679 371464296 org.apache.lucene.search.spans.SpanTermQuery
8: 10623486 339951552 org.apache.lucene.index.TermContext
9: 1516724 150564320 [Ljava.lang.Object;
10:724486 50948800 [C
11: 1528110 36674640 java.util.ArrayList
12:849884 27196288 org.apache.lucene.search.spans.SpanNearQuery
13:582008 23280320
org.apache.lucene.search.spans.SpanNearQuery$SpanNearWeight
14:481601 23116848 org.apache.lucene.document.FieldType
15:623073 19938336 org.apache.lucene.document.StoredField
16:721649 17319576 java.lang.String
17: 327297329640 [J
18: 146435788376 [F
{code}
The load on Solr is not much - max it goes to 2000 requests per minute. The
indexing load can sometimes be in burst but most of the time its pretty low.
But as mentioned above sometimes even a single document indexing can put solr
into tizzy and sometimes it just works like a charm.
was:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
{code:java}
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
{code}
Rest all are default values.
*Solr Config:*
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
{code}
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed u
[jira] [Updated] (SOLR-11196) Solr 6.5.0 consuming entire Heap suddenly while working smoothly on Solr 6.1.0
[
https://issues.apache.org/jira/browse/SOLR-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit updated SOLR-11196:
Description:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
{code:java}
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
{code}
Rest all are default values.
*Solr Config:*
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
{code}
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the Solr.
Here are the images from NewRelic and Sematext (Kindly click on the links to
view):
JVM Heap Memory Image : [https://i.stack.imgur.com/9dQAy.png]
1 Document and 1 Segment addition Image: [https://i.stack.imgur.com/6N4FC.png]
Update: Here is the JMap output when SOLR last died, we have now increased the
JVM memory to xmx of 12GB:
{code:java}
num #instances #bytes class name
--
1: 11210921 1076248416
org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat$IntBlockTermState
2: 10623486 934866768 [Lorg.apache.lucene.index.TermState;
3: 15567646 475873992 [B
4: 10623485 424939400
org.apache.lucene.search.spans.SpanTermQuery$SpanTermWeight
5: 15508972 372215328 org.apache.lucene.util.BytesRef
6: 15485834 371660016 org.apache.lucene.index.Term
7: 15477679 371464296 org.apache.lucene.search.spans.SpanTermQuery
8: 10623486 339951552 org.apache.lucene.index.TermContext
9: 1516724 150564320 [Ljava.lang.Object;
10:724486 50948800 [C
11: 1528110 36674640 java.util.ArrayList
12:849884 27196288 org.apache.lucene.search.spans.SpanNearQuery
13:582008 23280320
org.apache.lucene.search.spans.SpanNearQuery$SpanNearWeight
14:481601 23116848 org.apache.lucene.document.FieldType
15:623073 19938336 org.apache.lucene.document.StoredField
16:721649 17319576 java.lang.String
17: 327297329640 [J
18: 146435788376 [F
{code}
The load on Solr is not much - max it goes to 2000 requests per minute. The
indexing load can sometimes be in burst but most of the time its pretty low.
But as mentioned above sometimes even a single document indexing can put solr
into tizzy and sometimes it just works like a charm.
was:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
{code:java}
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
{code}
Rest all are default values.
*Solr Config:*
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
{code}
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed u
[jira] [Updated] (SOLR-11196) Solr 6.5.0 consuming entire Heap suddenly while working smoothly on Solr 6.1.0
[
https://issues.apache.org/jira/browse/SOLR-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit updated SOLR-11196:
Description:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
{code:java}
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
{code}
Rest all are default values.
*Solr Config:*
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
{code}
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the Solr.
Here are the images from NewRelic and Sematext (Kindly click on the links to
view):
[JVM Heap Memory Image : https://i.stack.imgur.com/9dQAy.png]
1 Document and 1 Segment addition Image: [https://i.stack.imgur.com/6N4FC.png]
Update: Here is the JMap output when SOLR last died, we have now increased the
JVM memory to xmx of 12GB:
{code:java}
num #instances #bytes class name
--
1: 11210921 1076248416
org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat$IntBlockTermState
2: 10623486 934866768 [Lorg.apache.lucene.index.TermState;
3: 15567646 475873992 [B
4: 10623485 424939400
org.apache.lucene.search.spans.SpanTermQuery$SpanTermWeight
5: 15508972 372215328 org.apache.lucene.util.BytesRef
6: 15485834 371660016 org.apache.lucene.index.Term
7: 15477679 371464296 org.apache.lucene.search.spans.SpanTermQuery
8: 10623486 339951552 org.apache.lucene.index.TermContext
9: 1516724 150564320 [Ljava.lang.Object;
10:724486 50948800 [C
11: 1528110 36674640 java.util.ArrayList
12:849884 27196288 org.apache.lucene.search.spans.SpanNearQuery
13:582008 23280320
org.apache.lucene.search.spans.SpanNearQuery$SpanNearWeight
14:481601 23116848 org.apache.lucene.document.FieldType
15:623073 19938336 org.apache.lucene.document.StoredField
16:721649 17319576 java.lang.String
17: 327297329640 [J
18: 146435788376 [F
{code}
The load on Solr is not much - max it goes to 2000 requests per minute. The
indexing load can sometimes be in burst but most of the time its pretty low.
But as mentioned above sometimes even a single document indexing can put solr
into tizzy and sometimes it just works like a charm.
was:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
{code:java}
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
{code}
Rest all are default values.
*Solr Config:*
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
{code}
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed u
[jira] [Updated] (SOLR-11196) Solr 6.5.0 consuming entire Heap suddenly while working smoothly on Solr 6.1.0
[
https://issues.apache.org/jira/browse/SOLR-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit updated SOLR-11196:
Description:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
{code:java}
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
{code}
Rest all are default values.
*Solr Config:*
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
{code}
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the Solr.
Here are the images from NewRelic and Sematext (Kindly click on the links to
view):
JVM Heap Memory Image : [https://i.stack.imgur.com/9dQAy.png]
1 Document and 1 Segment addition Image: [https://i.stack.imgur.com/6N4FC.png]
Update: Here is the JMap output when SOLR last died, we have now increased the
JVM memory to xmx of 12GB:
{code:java}
num #instances #bytes class name
--
1: 11210921 1076248416
org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat$IntBlockTermState
2: 10623486 934866768 [Lorg.apache.lucene.index.TermState;
3: 15567646 475873992 [B
4: 10623485 424939400
org.apache.lucene.search.spans.SpanTermQuery$SpanTermWeight
5: 15508972 372215328 org.apache.lucene.util.BytesRef
6: 15485834 371660016 org.apache.lucene.index.Term
7: 15477679 371464296 org.apache.lucene.search.spans.SpanTermQuery
8: 10623486 339951552 org.apache.lucene.index.TermContext
9: 1516724 150564320 [Ljava.lang.Object;
10:724486 50948800 [C
11: 1528110 36674640 java.util.ArrayList
12:849884 27196288 org.apache.lucene.search.spans.SpanNearQuery
13:582008 23280320
org.apache.lucene.search.spans.SpanNearQuery$SpanNearWeight
14:481601 23116848 org.apache.lucene.document.FieldType
15:623073 19938336 org.apache.lucene.document.StoredField
16:721649 17319576 java.lang.String
17: 327297329640 [J
18: 146435788376 [F
{code}
The load on Solr is not much - max it goes to 2000 requests per minute. The
indexing load can sometimes be in burst but most of the time its pretty low.
But as mentioned above sometimes even a single document indexing can put solr
into tizzy and sometimes it just works like a charm.
was:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
{code:java}
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
{code}
Rest all are default values.
*Solr Config:*
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
{code}
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up
[jira] [Updated] (SOLR-11196) Solr 6.5.0 consuming entire Heap suddenly while working smoothly on Solr 6.1.0
[
https://issues.apache.org/jira/browse/SOLR-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit updated SOLR-11196:
Description:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
{code:java}
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
{code}
Rest all are default values.
*Solr Config:*
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
{code}
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the Solr.
Here are the images from NewRelic and Sematext (Kindly click on the links to
view):
JVM Heap Memory Image : [https://i.stack.imgur.com/9dQAy.png]
1 Document and 1 Segment addition Image: [https://i.stack.imgur.com/6N4FC.png]
Update: Here is the JMap output when SOLR last died, we have now increased the
JVM memory to xmx of 12GB:
{code:java}
num #instances #bytes class name
--
1: 11210921 1076248416
org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat$IntBlockTermState
2: 10623486 934866768 [Lorg.apache.lucene.index.TermState;
3: 15567646 475873992 [B
4: 10623485 424939400
org.apache.lucene.search.spans.SpanTermQuery$SpanTermWeight
5: 15508972 372215328 org.apache.lucene.util.BytesRef
6: 15485834 371660016 org.apache.lucene.index.Term
7: 15477679 371464296 org.apache.lucene.search.spans.SpanTermQuery
8: 10623486 339951552 org.apache.lucene.index.TermContext
9: 1516724 150564320 [Ljava.lang.Object;
10:724486 50948800 [C
11: 1528110 36674640 java.util.ArrayList
12:849884 27196288 org.apache.lucene.search.spans.SpanNearQuery
13:582008 23280320
org.apache.lucene.search.spans.SpanNearQuery$SpanNearWeight
14:481601 23116848 org.apache.lucene.document.FieldType
15:623073 19938336 org.apache.lucene.document.StoredField
16:721649 17319576 java.lang.String
17: 327297329640 [J
18: 146435788376 [F
{code}
The load on Solr is not much - max it goes to 2000 requests per minute. The
indexing load can sometimes be in burst but most of the time its pretty low.
But as mentioned above sometimes even a single document indexing can put solr
into tizzy and sometimes it just works like a charm.
was:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
Rest all are default values.
*Solr Config:*
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
{code}
1024
true
20
${solr.query.max.docs:40}
false
2
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the
[jira] [Updated] (SOLR-11196) Solr 6.5.0 consuming entire Heap suddenly while working smoothly on Solr 6.1.0
[
https://issues.apache.org/jira/browse/SOLR-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit updated SOLR-11196:
Description:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
Rest all are default values.
*Solr Config:*
{code:java}
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
{code}
1024
true
20
${solr.query.max.docs:40}
false
2
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the Solr.
Here are the images from NewRelic and Sematext (Kindly click on the links to
view):
JVM Heap Memory Image : [https://i.stack.imgur.com/9dQAy.png]
1 Document and 1 Segment addition Image: [https://i.stack.imgur.com/6N4FC.png]
Update: Here is the JMap output when SOLR last died, we have now increased the
JVM memory to xmx of 12GB:
num #instances #bytes class name
--
1: 11210921 1076248416
org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat$IntBlockTermState
2: 10623486 934866768 [Lorg.apache.lucene.index.TermState;
3: 15567646 475873992 [B
4: 10623485 424939400
org.apache.lucene.search.spans.SpanTermQuery$SpanTermWeight
5: 15508972 372215328 org.apache.lucene.util.BytesRef
6: 15485834 371660016 org.apache.lucene.index.Term
7: 15477679 371464296 org.apache.lucene.search.spans.SpanTermQuery
8: 10623486 339951552 org.apache.lucene.index.TermContext
9: 1516724 150564320 [Ljava.lang.Object;
10:724486 50948800 [C
11: 1528110 36674640 java.util.ArrayList
12:849884 27196288 org.apache.lucene.search.spans.SpanNearQuery
13:582008 23280320
org.apache.lucene.search.spans.SpanNearQuery$SpanNearWeight
14:481601 23116848 org.apache.lucene.document.FieldType
15:623073 19938336 org.apache.lucene.document.StoredField
16:721649 17319576 java.lang.String
17: 327297329640 [J
18: 146435788376 [F
The load on Solr is not much - max it goes to 2000 requests per minute. The
indexing load can sometimes be in burst but most of the time its pretty low.
But as mentioned above sometimes even a single document indexing can put solr
into tizzy and sometimes it just works like a charm.
was:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
Rest all are default values.
*Solr Config:*
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the Solr.
Here are the images from NewRelic and Sematext (Kindly
[jira] [Updated] (SOLR-11196) Solr 6.5.0 consuming entire Heap suddenly while working smoothly on Solr 6.1.0
[
https://issues.apache.org/jira/browse/SOLR-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit updated SOLR-11196:
Description:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
Rest all are default values.
*Solr Config:*
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the Solr.
Here are the images from NewRelic and Sematext (Kindly click on the links to
view):
JVM Heap Memory Image : [https://i.stack.imgur.com/9dQAy.png]
1 Document and 1 Segment addition Image: [https://i.stack.imgur.com/6N4FC.png]
Update: Here is the JMap output when SOLR last died, we have now increased the
JVM memory to xmx of 12GB:
num #instances #bytes class name
--
1: 11210921 1076248416
org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat$IntBlockTermState
2: 10623486 934866768 [Lorg.apache.lucene.index.TermState;
3: 15567646 475873992 [B
4: 10623485 424939400
org.apache.lucene.search.spans.SpanTermQuery$SpanTermWeight
5: 15508972 372215328 org.apache.lucene.util.BytesRef
6: 15485834 371660016 org.apache.lucene.index.Term
7: 15477679 371464296 org.apache.lucene.search.spans.SpanTermQuery
8: 10623486 339951552 org.apache.lucene.index.TermContext
9: 1516724 150564320 [Ljava.lang.Object;
10:724486 50948800 [C
11: 1528110 36674640 java.util.ArrayList
12:849884 27196288 org.apache.lucene.search.spans.SpanNearQuery
13:582008 23280320
org.apache.lucene.search.spans.SpanNearQuery$SpanNearWeight
14:481601 23116848 org.apache.lucene.document.FieldType
15:623073 19938336 org.apache.lucene.document.StoredField
16:721649 17319576 java.lang.String
17: 327297329640 [J
18: 146435788376 [F
The load on Solr is not much - max it goes to 2000 requests per minute. The
indexing load can sometimes be in burst but most of the time its pretty low.
But as mentioned above sometimes even a single document indexing can put solr
into tizzy and sometimes it just works like a charm.
was:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
Rest all are default values.
*Solr Config:*
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the Solr.
Here are the images from NewRelic and Sematext (Kindly click on the links t
[jira] [Updated] (SOLR-11196) Solr 6.5.0 consuming entire Heap suddenly while working smoothly on Solr 6.1.0
[
https://issues.apache.org/jira/browse/SOLR-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit updated SOLR-11196:
Description:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
Rest all are default values.
*Solr Config:*
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the Solr.
Here are the images from NewRelic and Sematext (Kindly click on the links to
view):
[JVM Heap Memory Image][https://i.stack.imgur.com/9dQAy.png]
[1 Document and 1 Segment addition Image][2]
[1]: https://i.stack.imgur.com/9dQAy.png
[2]: https://i.stack.imgur.com/6N4FC.png
Update: Here is the JMap output when SOLR last died, we have now increased the
JVM memory to xmx of 12GB:
num #instances #bytes class name
--
1: 11210921 1076248416
org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat$IntBlockTermState
2: 10623486 934866768 [Lorg.apache.lucene.index.TermState;
3: 15567646 475873992 [B
4: 10623485 424939400
org.apache.lucene.search.spans.SpanTermQuery$SpanTermWeight
5: 15508972 372215328 org.apache.lucene.util.BytesRef
6: 15485834 371660016 org.apache.lucene.index.Term
7: 15477679 371464296 org.apache.lucene.search.spans.SpanTermQuery
8: 10623486 339951552 org.apache.lucene.index.TermContext
9: 1516724 150564320 [Ljava.lang.Object;
10:724486 50948800 [C
11: 1528110 36674640 java.util.ArrayList
12:849884 27196288 org.apache.lucene.search.spans.SpanNearQuery
13:582008 23280320
org.apache.lucene.search.spans.SpanNearQuery$SpanNearWeight
14:481601 23116848 org.apache.lucene.document.FieldType
15:623073 19938336 org.apache.lucene.document.StoredField
16:721649 17319576 java.lang.String
17: 327297329640 [J
18: 146435788376 [F
The load on Solr is not much - max it goes to 2000 requests per minute. The
indexing load can sometimes be in burst but most of the time its pretty low.
But as mentioned above sometimes even a single document indexing can put solr
into tizzy and sometimes it just works like a charm.
was:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
Rest all are default values.
*Solr Config:*
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the Solr.
Here are the images fro
[jira] [Updated] (SOLR-11196) Solr 6.5.0 consuming entire Heap suddenly while working smoothly on Solr 6.1.0
[
https://issues.apache.org/jira/browse/SOLR-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit updated SOLR-11196:
Description:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
Rest all are default values.
*Solr Config:*
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the Solr.
Here are the images from NewRelic and Sematext (Kindly click on the links to
view):
[#JVM Heap Memory Image]
[1 Document and 1 Segment addition Image][2]
[1]: https://i.stack.imgur.com/9dQAy.png
[2]: https://i.stack.imgur.com/6N4FC.png
Update: Here is the JMap output when SOLR last died, we have now increased the
JVM memory to xmx of 12GB:
num #instances #bytes class name
--
1: 11210921 1076248416
org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat$IntBlockTermState
2: 10623486 934866768 [Lorg.apache.lucene.index.TermState;
3: 15567646 475873992 [B
4: 10623485 424939400
org.apache.lucene.search.spans.SpanTermQuery$SpanTermWeight
5: 15508972 372215328 org.apache.lucene.util.BytesRef
6: 15485834 371660016 org.apache.lucene.index.Term
7: 15477679 371464296 org.apache.lucene.search.spans.SpanTermQuery
8: 10623486 339951552 org.apache.lucene.index.TermContext
9: 1516724 150564320 [Ljava.lang.Object;
10:724486 50948800 [C
11: 1528110 36674640 java.util.ArrayList
12:849884 27196288 org.apache.lucene.search.spans.SpanNearQuery
13:582008 23280320
org.apache.lucene.search.spans.SpanNearQuery$SpanNearWeight
14:481601 23116848 org.apache.lucene.document.FieldType
15:623073 19938336 org.apache.lucene.document.StoredField
16:721649 17319576 java.lang.String
17: 327297329640 [J
18: 146435788376 [F
The load on Solr is not much - max it goes to 2000 requests per minute. The
indexing load can sometimes be in burst but most of the time its pretty low.
But as mentioned above sometimes even a single document indexing can put solr
into tizzy and sometimes it just works like a charm.
was:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
Rest all are default values.
*Solr Config:*
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the Solr.
Here are the images from NewRelic and Sematext (Kindly clic
[jira] [Updated] (SOLR-11196) Solr 6.5.0 consuming entire Heap suddenly while working smoothly on Solr 6.1.0
[
https://issues.apache.org/jira/browse/SOLR-11196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amit updated SOLR-11196:
Description:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
Rest all are default values.
*Solr Config:*
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the Solr.
Here are the images from NewRelic and Sematext (Kindly click on the links to
view):
[JVM Heap Memory Image]: https://i.stack.imgur.com/9dQAy.png
[1 Document and 1 Segment addition Image][2]
[1]: https://i.stack.imgur.com/9dQAy.png
[2]: https://i.stack.imgur.com/6N4FC.png
Update: Here is the JMap output when SOLR last died, we have now increased the
JVM memory to xmx of 12GB:
num #instances #bytes class name
--
1: 11210921 1076248416
org.apache.lucene.codecs.lucene50.Lucene50PostingsFormat$IntBlockTermState
2: 10623486 934866768 [Lorg.apache.lucene.index.TermState;
3: 15567646 475873992 [B
4: 10623485 424939400
org.apache.lucene.search.spans.SpanTermQuery$SpanTermWeight
5: 15508972 372215328 org.apache.lucene.util.BytesRef
6: 15485834 371660016 org.apache.lucene.index.Term
7: 15477679 371464296 org.apache.lucene.search.spans.SpanTermQuery
8: 10623486 339951552 org.apache.lucene.index.TermContext
9: 1516724 150564320 [Ljava.lang.Object;
10:724486 50948800 [C
11: 1528110 36674640 java.util.ArrayList
12:849884 27196288 org.apache.lucene.search.spans.SpanNearQuery
13:582008 23280320
org.apache.lucene.search.spans.SpanNearQuery$SpanNearWeight
14:481601 23116848 org.apache.lucene.document.FieldType
15:623073 19938336 org.apache.lucene.document.StoredField
16:721649 17319576 java.lang.String
17: 327297329640 [J
18: 146435788376 [F
The load on Solr is not much - max it goes to 2000 requests per minute. The
indexing load can sometimes be in burst but most of the time its pretty low.
But as mentioned above sometimes even a single document indexing can put solr
into tizzy and sometimes it just works like a charm.
was:
Please note, this issue does not occurs on Solr-6.1.0 while the same occurs on
Solr-6.5.0 and above. To fix this we had to move back to Solr-6.1.0 version.
We have been hit by a Solr Behavior in production which we are unable to debug.
To start with here are the configurations for solr:
Solr Version: 6.5, Master with 1 Slave of the same configuration as mentioned
below.
*JVM Config:*
-Xms2048m
-Xmx4096m
-XX:+ParallelRefProcEnabled
-XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=50
Rest all are default values.
*Solr Config:*
{solr.autoCommit.maxTime:30}
false
{solr.autoSoftCommit.maxTime:90}
1024
true
20
${solr.query.max.docs:40}
false
2
*The Host (AWS) configurations are:*
RAM: 7.65GB
Cores: 4
Now, our solr works perfectly fine for hours and sometimes for days but
sometimes suddenly memory jumps up and the GC kicks in causing long big pauses
with not much to recover. We are seeing this happening most often when one or
multiple segments gets added or deleted post a hard commit. It doesn't matter
how many documents got indexed. The images attached shows that just 1 document
was indexed, causing an addition of one segment and it all got messed up till
we restarted the Solr.
Here are the images fro
