Re: Solr memory leak
I didn't meant to say that the fix is not in 7.0. I just stated that I do not see it listed in the release notes (https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310230=12335718). Thanks for explaining the release process. regards, Hendrik On 10.09.2017 17:32, Erick Erickson wrote: There will be no 6.7. Once the X+1 version is released, all past fixes are applied to as minor releases to the last released version of the previous major release. So now that 7.0 has been cut, there might be a 6.6.2 (6.6.1 was just released) but no 6.7. Current un-released JIRAs are parked on the 6.x (as opposed to branch_6_6) for convenience. If anyone steps up to release 6.6.2, they can include ths. Why do you say this isn't in 7.0? The "Fix Versions" clearly states so, as does CHANGES.txt for 7.0. The new file is is in the 7.0 branch. If you need it in 6x you have a couple of options: 1> agitate fo ra 6.6.2 with this included 2> apply the patch yourself and compile it locally Best, Erick On Sun, Sep 10, 2017 at 6:04 AM, Hendrik Haddorp <hendrik.hadd...@gmx.net> wrote: Hi, looks like SOLR-10506 didn't make it into 6.6.1. I do however also not see it listen in the current release notes for 6.7 nor 7.0: https://issues.apache.org/jira/projects/SOLR/versions/12340568 https://issues.apache.org/jira/projects/SOLR/versions/12335718 Is there any any rough idea already when 6.7 or 7.0 will be released? thanks, Hendrik On 28.08.2017 18:31, Erick Erickson wrote: Varun Thacker is the RM for Solr 6.6.1, I've pinged him about including it. On Mon, Aug 28, 2017 at 8:52 AM, Walter Underwood <wun...@wunderwood.org> wrote: That would be a really good reason for a 6.7. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) On Aug 28, 2017, at 8:48 AM, Markus Jelsma <markus.jel...@openindex.io> wrote: It is, unfortunately, not committed for 6.7. -Original message- From:Markus Jelsma <markus.jel...@openindex.io> Sent: Monday 28th August 2017 17:46 To: solr-user@lucene.apache.org Subject: RE: Solr memory leak See https://issues.apache.org/jira/browse/SOLR-10506 Fixed for 7.0 Markus -Original message- From:Hendrik Haddorp <hendrik.hadd...@gmx.net> Sent: Monday 28th August 2017 17:42 To: solr-user@lucene.apache.org Subject: Solr memory leak Hi, we noticed that triggering collection reloads on many collections has a good chance to result in an OOM-Error. To investigate that further I did a simple test: - Start solr with a 2GB heap and 1GB Metaspace - create a trivial collection with a few documents (I used only 2 fields and 100 documents) - trigger a collection reload in a loop (I used SolrJ for this) Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6 worked better but also failed after 1100 loops. When looking at the memory usage on the Solr dashboard it looks like the space left after GC cycles gets less and less. Then Solr gets very slow, as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In my last run this was actually for the Metaspace. So it looks like more and more heap and metaspace is being used by just constantly reloading a trivial collection. regards, Hendrik
Re: Solr memory leak
There will be no 6.7. Once the X+1 version is released, all past fixes are applied to as minor releases to the last released version of the previous major release. So now that 7.0 has been cut, there might be a 6.6.2 (6.6.1 was just released) but no 6.7. Current un-released JIRAs are parked on the 6.x (as opposed to branch_6_6) for convenience. If anyone steps up to release 6.6.2, they can include ths. Why do you say this isn't in 7.0? The "Fix Versions" clearly states so, as does CHANGES.txt for 7.0. The new file is is in the 7.0 branch. If you need it in 6x you have a couple of options: 1> agitate fo ra 6.6.2 with this included 2> apply the patch yourself and compile it locally Best, Erick On Sun, Sep 10, 2017 at 6:04 AM, Hendrik Haddorp <hendrik.hadd...@gmx.net> wrote: > Hi, > > looks like SOLR-10506 didn't make it into 6.6.1. I do however also not see > it listen in the current release notes for 6.7 nor 7.0: > https://issues.apache.org/jira/projects/SOLR/versions/12340568 > https://issues.apache.org/jira/projects/SOLR/versions/12335718 > > Is there any any rough idea already when 6.7 or 7.0 will be released? > > thanks, > Hendrik > > > On 28.08.2017 18:31, Erick Erickson wrote: >> >> Varun Thacker is the RM for Solr 6.6.1, I've pinged him about including >> it. >> >> On Mon, Aug 28, 2017 at 8:52 AM, Walter Underwood <wun...@wunderwood.org> >> wrote: >>> >>> That would be a really good reason for a 6.7. >>> >>> wunder >>> Walter Underwood >>> wun...@wunderwood.org >>> http://observer.wunderwood.org/ (my blog) >>> >>> >>>> On Aug 28, 2017, at 8:48 AM, Markus Jelsma <markus.jel...@openindex.io> >>>> wrote: >>>> >>>> It is, unfortunately, not committed for 6.7. >>>> >>>> >>>> >>>> >>>> >>>> -Original message- >>>>> >>>>> From:Markus Jelsma <markus.jel...@openindex.io> >>>>> Sent: Monday 28th August 2017 17:46 >>>>> To: solr-user@lucene.apache.org >>>>> Subject: RE: Solr memory leak >>>>> >>>>> See https://issues.apache.org/jira/browse/SOLR-10506 >>>>> Fixed for 7.0 >>>>> >>>>> Markus >>>>> >>>>> >>>>> >>>>> -Original message- >>>>>> >>>>>> From:Hendrik Haddorp <hendrik.hadd...@gmx.net> >>>>>> Sent: Monday 28th August 2017 17:42 >>>>>> To: solr-user@lucene.apache.org >>>>>> Subject: Solr memory leak >>>>>> >>>>>> Hi, >>>>>> >>>>>> we noticed that triggering collection reloads on many collections has >>>>>> a >>>>>> good chance to result in an OOM-Error. To investigate that further I >>>>>> did >>>>>> a simple test: >>>>>> - Start solr with a 2GB heap and 1GB Metaspace >>>>>> - create a trivial collection with a few documents (I used only 2 >>>>>> fields and 100 documents) >>>>>> - trigger a collection reload in a loop (I used SolrJ for this) >>>>>> >>>>>> Using Solr 6.3 the test started to fail after about 250 loops. Solr >>>>>> 6.6 >>>>>> worked better but also failed after 1100 loops. >>>>>> >>>>>> When looking at the memory usage on the Solr dashboard it looks like >>>>>> the >>>>>> space left after GC cycles gets less and less. Then Solr gets very >>>>>> slow, >>>>>> as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In >>>>>> my last run this was actually for the Metaspace. So it looks like more >>>>>> and more heap and metaspace is being used by just constantly reloading >>>>>> a >>>>>> trivial collection. >>>>>> >>>>>> regards, >>>>>> Hendrik >>>>>> >
Re: Solr memory leak
Hi, looks like SOLR-10506 didn't make it into 6.6.1. I do however also not see it listen in the current release notes for 6.7 nor 7.0: https://issues.apache.org/jira/projects/SOLR/versions/12340568 https://issues.apache.org/jira/projects/SOLR/versions/12335718 Is there any any rough idea already when 6.7 or 7.0 will be released? thanks, Hendrik On 28.08.2017 18:31, Erick Erickson wrote: Varun Thacker is the RM for Solr 6.6.1, I've pinged him about including it. On Mon, Aug 28, 2017 at 8:52 AM, Walter Underwood <wun...@wunderwood.org> wrote: That would be a really good reason for a 6.7. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) On Aug 28, 2017, at 8:48 AM, Markus Jelsma <markus.jel...@openindex.io> wrote: It is, unfortunately, not committed for 6.7. -Original message- From:Markus Jelsma <markus.jel...@openindex.io> Sent: Monday 28th August 2017 17:46 To: solr-user@lucene.apache.org Subject: RE: Solr memory leak See https://issues.apache.org/jira/browse/SOLR-10506 Fixed for 7.0 Markus -Original message- From:Hendrik Haddorp <hendrik.hadd...@gmx.net> Sent: Monday 28th August 2017 17:42 To: solr-user@lucene.apache.org Subject: Solr memory leak Hi, we noticed that triggering collection reloads on many collections has a good chance to result in an OOM-Error. To investigate that further I did a simple test: - Start solr with a 2GB heap and 1GB Metaspace - create a trivial collection with a few documents (I used only 2 fields and 100 documents) - trigger a collection reload in a loop (I used SolrJ for this) Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6 worked better but also failed after 1100 loops. When looking at the memory usage on the Solr dashboard it looks like the space left after GC cycles gets less and less. Then Solr gets very slow, as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In my last run this was actually for the Metaspace. So it looks like more and more heap and metaspace is being used by just constantly reloading a trivial collection. regards, Hendrik
Re: Solr memory leak
Did you get an answer? Would really be nice to have that in the next release. On 28.08.2017 18:31, Erick Erickson wrote: Varun Thacker is the RM for Solr 6.6.1, I've pinged him about including it. On Mon, Aug 28, 2017 at 8:52 AM, Walter Underwood <wun...@wunderwood.org> wrote: That would be a really good reason for a 6.7. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) On Aug 28, 2017, at 8:48 AM, Markus Jelsma <markus.jel...@openindex.io> wrote: It is, unfortunately, not committed for 6.7. -Original message- From:Markus Jelsma <markus.jel...@openindex.io> Sent: Monday 28th August 2017 17:46 To: solr-user@lucene.apache.org Subject: RE: Solr memory leak See https://issues.apache.org/jira/browse/SOLR-10506 Fixed for 7.0 Markus -Original message- From:Hendrik Haddorp <hendrik.hadd...@gmx.net> Sent: Monday 28th August 2017 17:42 To: solr-user@lucene.apache.org Subject: Solr memory leak Hi, we noticed that triggering collection reloads on many collections has a good chance to result in an OOM-Error. To investigate that further I did a simple test: - Start solr with a 2GB heap and 1GB Metaspace - create a trivial collection with a few documents (I used only 2 fields and 100 documents) - trigger a collection reload in a loop (I used SolrJ for this) Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6 worked better but also failed after 1100 loops. When looking at the memory usage on the Solr dashboard it looks like the space left after GC cycles gets less and less. Then Solr gets very slow, as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In my last run this was actually for the Metaspace. So it looks like more and more heap and metaspace is being used by just constantly reloading a trivial collection. regards, Hendrik
Re: Solr memory leak
Varun Thacker is the RM for Solr 6.6.1, I've pinged him about including it. On Mon, Aug 28, 2017 at 8:52 AM, Walter Underwood <wun...@wunderwood.org> wrote: > That would be a really good reason for a 6.7. > > wunder > Walter Underwood > wun...@wunderwood.org > http://observer.wunderwood.org/ (my blog) > > >> On Aug 28, 2017, at 8:48 AM, Markus Jelsma <markus.jel...@openindex.io> >> wrote: >> >> It is, unfortunately, not committed for 6.7. >> >> >> >> >> >> -Original message- >>> From:Markus Jelsma <markus.jel...@openindex.io> >>> Sent: Monday 28th August 2017 17:46 >>> To: solr-user@lucene.apache.org >>> Subject: RE: Solr memory leak >>> >>> See https://issues.apache.org/jira/browse/SOLR-10506 >>> Fixed for 7.0 >>> >>> Markus >>> >>> >>> >>> -Original message- >>>> From:Hendrik Haddorp <hendrik.hadd...@gmx.net> >>>> Sent: Monday 28th August 2017 17:42 >>>> To: solr-user@lucene.apache.org >>>> Subject: Solr memory leak >>>> >>>> Hi, >>>> >>>> we noticed that triggering collection reloads on many collections has a >>>> good chance to result in an OOM-Error. To investigate that further I did >>>> a simple test: >>>> - Start solr with a 2GB heap and 1GB Metaspace >>>> - create a trivial collection with a few documents (I used only 2 >>>> fields and 100 documents) >>>> - trigger a collection reload in a loop (I used SolrJ for this) >>>> >>>> Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6 >>>> worked better but also failed after 1100 loops. >>>> >>>> When looking at the memory usage on the Solr dashboard it looks like the >>>> space left after GC cycles gets less and less. Then Solr gets very slow, >>>> as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In >>>> my last run this was actually for the Metaspace. So it looks like more >>>> and more heap and metaspace is being used by just constantly reloading a >>>> trivial collection. >>>> >>>> regards, >>>> Hendrik >>>> >>> >
Re: Solr memory leak
That would be a really good reason for a 6.7. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Aug 28, 2017, at 8:48 AM, Markus Jelsma <markus.jel...@openindex.io> wrote: > > It is, unfortunately, not committed for 6.7. > > > > > > -Original message- >> From:Markus Jelsma <markus.jel...@openindex.io> >> Sent: Monday 28th August 2017 17:46 >> To: solr-user@lucene.apache.org >> Subject: RE: Solr memory leak >> >> See https://issues.apache.org/jira/browse/SOLR-10506 >> Fixed for 7.0 >> >> Markus >> >> >> >> -Original message- >>> From:Hendrik Haddorp <hendrik.hadd...@gmx.net> >>> Sent: Monday 28th August 2017 17:42 >>> To: solr-user@lucene.apache.org >>> Subject: Solr memory leak >>> >>> Hi, >>> >>> we noticed that triggering collection reloads on many collections has a >>> good chance to result in an OOM-Error. To investigate that further I did >>> a simple test: >>> - Start solr with a 2GB heap and 1GB Metaspace >>> - create a trivial collection with a few documents (I used only 2 >>> fields and 100 documents) >>> - trigger a collection reload in a loop (I used SolrJ for this) >>> >>> Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6 >>> worked better but also failed after 1100 loops. >>> >>> When looking at the memory usage on the Solr dashboard it looks like the >>> space left after GC cycles gets less and less. Then Solr gets very slow, >>> as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In >>> my last run this was actually for the Metaspace. So it looks like more >>> and more heap and metaspace is being used by just constantly reloading a >>> trivial collection. >>> >>> regards, >>> Hendrik >>> >>
RE: Solr memory leak
It is, unfortunately, not committed for 6.7. -Original message- > From:Markus Jelsma <markus.jel...@openindex.io> > Sent: Monday 28th August 2017 17:46 > To: solr-user@lucene.apache.org > Subject: RE: Solr memory leak > > See https://issues.apache.org/jira/browse/SOLR-10506 > Fixed for 7.0 > > Markus > > > > -Original message- > > From:Hendrik Haddorp <hendrik.hadd...@gmx.net> > > Sent: Monday 28th August 2017 17:42 > > To: solr-user@lucene.apache.org > > Subject: Solr memory leak > > > > Hi, > > > > we noticed that triggering collection reloads on many collections has a > > good chance to result in an OOM-Error. To investigate that further I did > > a simple test: > > - Start solr with a 2GB heap and 1GB Metaspace > > - create a trivial collection with a few documents (I used only 2 > > fields and 100 documents) > > - trigger a collection reload in a loop (I used SolrJ for this) > > > > Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6 > > worked better but also failed after 1100 loops. > > > > When looking at the memory usage on the Solr dashboard it looks like the > > space left after GC cycles gets less and less. Then Solr gets very slow, > > as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In > > my last run this was actually for the Metaspace. So it looks like more > > and more heap and metaspace is being used by just constantly reloading a > > trivial collection. > > > > regards, > > Hendrik > > >
RE: Solr memory leak
See https://issues.apache.org/jira/browse/SOLR-10506 Fixed for 7.0 Markus -Original message- > From:Hendrik Haddorp <hendrik.hadd...@gmx.net> > Sent: Monday 28th August 2017 17:42 > To: solr-user@lucene.apache.org > Subject: Solr memory leak > > Hi, > > we noticed that triggering collection reloads on many collections has a > good chance to result in an OOM-Error. To investigate that further I did > a simple test: > - Start solr with a 2GB heap and 1GB Metaspace > - create a trivial collection with a few documents (I used only 2 > fields and 100 documents) > - trigger a collection reload in a loop (I used SolrJ for this) > > Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6 > worked better but also failed after 1100 loops. > > When looking at the memory usage on the Solr dashboard it looks like the > space left after GC cycles gets less and less. Then Solr gets very slow, > as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In > my last run this was actually for the Metaspace. So it looks like more > and more heap and metaspace is being used by just constantly reloading a > trivial collection. > > regards, > Hendrik >
Solr memory leak
Hi, we noticed that triggering collection reloads on many collections has a good chance to result in an OOM-Error. To investigate that further I did a simple test: - Start solr with a 2GB heap and 1GB Metaspace - create a trivial collection with a few documents (I used only 2 fields and 100 documents) - trigger a collection reload in a loop (I used SolrJ for this) Using Solr 6.3 the test started to fail after about 250 loops. Solr 6.6 worked better but also failed after 1100 loops. When looking at the memory usage on the Solr dashboard it looks like the space left after GC cycles gets less and less. Then Solr gets very slow, as the JVM is busy with the GC. A bit later Solr gets an OOM-Error. In my last run this was actually for the Metaspace. So it looks like more and more heap and metaspace is being used by just constantly reloading a trivial collection. regards, Hendrik
Re: [/solr] memory leak prevent tomcat shutdown
any input on this? thanks Jie -- View this message in context: http://lucene.472066.n3.nabble.com/solr-memory-leak-prevent-tomcat-shutdown-tp4014788p4015265.html Sent from the Solr - User mailing list archive at Nabble.com.
[/solr] memory leak prevent tomcat shutdown
very often when we try to shutdown tomcat, we got following error in catalina.out indicating a solr thread can not be stopped, the tomcat results hanging, we have to kill -9, which we think lead to some core corruptions in our production environment. please help ... catalina.out: ... ... Oct 19, 2012 10:17:22 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/solr] appears to have started a thread named [pool-69-thread-1] but has failed to stop it. This is very likely to create a memory leak. Then I used kill -3 to signal the thread dump, here is what I get (note the thread [pool-69-thread-1] is hanging) : 2012-10-19 10:18:39 Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.2-b06 mixed mode): DestroyJavaVM prio=10 tid=0x55b39800 nid=0x7e82 waiting on condition [0x] java.lang.Thread.State: RUNNABLE pool-69-thread-1 prio=10 tid=0x2aaabcb41800 nid=0x19fa waiting on condition [0x4205e000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x0006de699d80 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(Unknown Source) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) at java.util.concurrent.LinkedBlockingQueue.take(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.getTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) JDWP Transport Listener: dt_socket daemon prio=10 tid=0x578aa000 nid=0x19f9 runnable [0x] java.lang.Thread.State: RUNNABLE ... ... -- View this message in context: http://lucene.472066.n3.nabble.com/solr-memory-leak-prevent-tomcat-shutdown-tp4014788.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: [/solr] memory leak prevent tomcat shutdown
by the way, I am running tomcat 6, solr 3.5 on redhat 2.6.18-274.el5 #1 SMP Fri Jul 8 17:36:59 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux -- View this message in context: http://lucene.472066.n3.nabble.com/solr-memory-leak-prevent-tomcat-shutdown-tp4014788p4014792.html Sent from the Solr - User mailing list archive at Nabble.com.
