Re: 3.11.2 memory leak
Likely in the next few weeks. On Mon., 23 Jul. 2018, 01:17 Abdul Patel, wrote: > Any idea when 3.11.3 is coming in? > > On Tuesday, June 19, 2018, kurt greaves wrote: > >> At this point I'd wait for 3.11.3. If you can't, you can get away with >> backporting a few repair fixes or just doing sub range repairs on 3.11.2 >> >> On Wed., 20 Jun. 2018, 01:10 Abdul Patel, wrote: >> >>> Hi All, >>> >>> Do we kmow whats the stable version for now if u wish to upgrade ? >>> >>> On Tuesday, June 5, 2018, Steinmaurer, Thomas < >>> thomas.steinmau...@dynatrace.com> wrote: >>> >>>> Jeff, >>>> >>>> >>>> >>>> FWIW, when talking about >>>> https://issues.apache.org/jira/browse/CASSANDRA-13929, there is a >>>> patch available since March without getting further attention. >>>> >>>> >>>> >>>> Regards, >>>> >>>> Thomas >>>> >>>> >>>> >>>> *From:* Jeff Jirsa [mailto:jji...@gmail.com] >>>> *Sent:* Dienstag, 05. Juni 2018 00:51 >>>> *To:* cassandra >>>> *Subject:* Re: 3.11.2 memory leak >>>> >>>> >>>> >>>> There have been a few people who have reported it, but nobody (yet) has >>>> offered a patch to fix it. It would be good to have a reliable way to >>>> repro, and/or an analysis of a heap dump demonstrating the problem (what's >>>> actually retained at the time you're OOM'ing). >>>> >>>> >>>> >>>> On Mon, Jun 4, 2018 at 6:52 AM, Abdul Patel >>>> wrote: >>>> >>>> Hi All, >>>> >>>> >>>> >>>> I recently upgraded my non prod cluster from 3.10 to 3.11.2. >>>> >>>> It was working fine for a 1.5 weeks then suddenly nodetool info startee >>>> reporting 80% and more memory consumption. >>>> >>>> Intially it was 16gb configured, then i bumped to 20gb and rebooted all >>>> 4 nodes of cluster-single DC. >>>> >>>> Now after 8 days i again see 80% + usage and its 16gb and above ..which >>>> we never saw before . >>>> >>>> Seems like memory leak bug? >>>> >>>> Does anyone has any idea ? Our 3.11.2 release rollout has been halted >>>> because of this. >>>> >>>> If not 3.11.2 whats the next best stable release we have now? >>>> >>>> >>>> The contents of this e-mail are intended for the named addressee only. >>>> It contains information that may be confidential. Unless you are the named >>>> addressee or an authorized designee, you may not copy or use it, or >>>> disclose it to anyone else. If you received it in error please notify us >>>> immediately and then destroy it. Dynatrace Austria GmbH (registration >>>> number FN 91482h) is a company registered in Linz whose registered office >>>> is at 4040 Linz, Austria, Freistädterstraße 313 >>>> >>>
Re: 3.11.2 memory leak
Any idea when 3.11.3 is coming in? On Tuesday, June 19, 2018, kurt greaves wrote: > At this point I'd wait for 3.11.3. If you can't, you can get away with > backporting a few repair fixes or just doing sub range repairs on 3.11.2 > > On Wed., 20 Jun. 2018, 01:10 Abdul Patel, wrote: > >> Hi All, >> >> Do we kmow whats the stable version for now if u wish to upgrade ? >> >> On Tuesday, June 5, 2018, Steinmaurer, Thomas < >> thomas.steinmau...@dynatrace.com> wrote: >> >>> Jeff, >>> >>> >>> >>> FWIW, when talking about https://issues.apache.org/ >>> jira/browse/CASSANDRA-13929, there is a patch available since March >>> without getting further attention. >>> >>> >>> >>> Regards, >>> >>> Thomas >>> >>> >>> >>> *From:* Jeff Jirsa [mailto:jji...@gmail.com] >>> *Sent:* Dienstag, 05. Juni 2018 00:51 >>> *To:* cassandra >>> *Subject:* Re: 3.11.2 memory leak >>> >>> >>> >>> There have been a few people who have reported it, but nobody (yet) has >>> offered a patch to fix it. It would be good to have a reliable way to >>> repro, and/or an analysis of a heap dump demonstrating the problem (what's >>> actually retained at the time you're OOM'ing). >>> >>> >>> >>> On Mon, Jun 4, 2018 at 6:52 AM, Abdul Patel wrote: >>> >>> Hi All, >>> >>> >>> >>> I recently upgraded my non prod cluster from 3.10 to 3.11.2. >>> >>> It was working fine for a 1.5 weeks then suddenly nodetool info startee >>> reporting 80% and more memory consumption. >>> >>> Intially it was 16gb configured, then i bumped to 20gb and rebooted all >>> 4 nodes of cluster-single DC. >>> >>> Now after 8 days i again see 80% + usage and its 16gb and above ..which >>> we never saw before . >>> >>> Seems like memory leak bug? >>> >>> Does anyone has any idea ? Our 3.11.2 release rollout has been halted >>> because of this. >>> >>> If not 3.11.2 whats the next best stable release we have now? >>> >>> >>> The contents of this e-mail are intended for the named addressee only. >>> It contains information that may be confidential. Unless you are the named >>> addressee or an authorized designee, you may not copy or use it, or >>> disclose it to anyone else. If you received it in error please notify us >>> immediately and then destroy it. Dynatrace Austria GmbH (registration >>> number FN 91482h) is a company registered in Linz whose registered office >>> is at 4040 Linz, Austria, Freistädterstraße 313 >>> >>
Re: 3.11.2 memory leak
At this point I'd wait for 3.11.3. If you can't, you can get away with backporting a few repair fixes or just doing sub range repairs on 3.11.2 On Wed., 20 Jun. 2018, 01:10 Abdul Patel, wrote: > Hi All, > > Do we kmow whats the stable version for now if u wish to upgrade ? > > On Tuesday, June 5, 2018, Steinmaurer, Thomas < > thomas.steinmau...@dynatrace.com> wrote: > >> Jeff, >> >> >> >> FWIW, when talking about >> https://issues.apache.org/jira/browse/CASSANDRA-13929, there is a patch >> available since March without getting further attention. >> >> >> >> Regards, >> >> Thomas >> >> >> >> *From:* Jeff Jirsa [mailto:jji...@gmail.com] >> *Sent:* Dienstag, 05. Juni 2018 00:51 >> *To:* cassandra >> *Subject:* Re: 3.11.2 memory leak >> >> >> >> There have been a few people who have reported it, but nobody (yet) has >> offered a patch to fix it. It would be good to have a reliable way to >> repro, and/or an analysis of a heap dump demonstrating the problem (what's >> actually retained at the time you're OOM'ing). >> >> >> >> On Mon, Jun 4, 2018 at 6:52 AM, Abdul Patel wrote: >> >> Hi All, >> >> >> >> I recently upgraded my non prod cluster from 3.10 to 3.11.2. >> >> It was working fine for a 1.5 weeks then suddenly nodetool info startee >> reporting 80% and more memory consumption. >> >> Intially it was 16gb configured, then i bumped to 20gb and rebooted all 4 >> nodes of cluster-single DC. >> >> Now after 8 days i again see 80% + usage and its 16gb and above ..which >> we never saw before . >> >> Seems like memory leak bug? >> >> Does anyone has any idea ? Our 3.11.2 release rollout has been halted >> because of this. >> >> If not 3.11.2 whats the next best stable release we have now? >> >> >> The contents of this e-mail are intended for the named addressee only. It >> contains information that may be confidential. Unless you are the named >> addressee or an authorized designee, you may not copy or use it, or >> disclose it to anyone else. If you received it in error please notify us >> immediately and then destroy it. Dynatrace Austria GmbH (registration >> number FN 91482h) is a company registered in Linz whose registered office >> is at 4040 Linz, Austria, Freistädterstraße 313 >> >
Re: 3.11.2 memory leak
Hi All, Do we kmow whats the stable version for now if u wish to upgrade ? On Tuesday, June 5, 2018, Steinmaurer, Thomas < thomas.steinmau...@dynatrace.com> wrote: > Jeff, > > > > FWIW, when talking about https://issues.apache.org/ > jira/browse/CASSANDRA-13929, there is a patch available since March > without getting further attention. > > > > Regards, > > Thomas > > > > *From:* Jeff Jirsa [mailto:jji...@gmail.com] > *Sent:* Dienstag, 05. Juni 2018 00:51 > *To:* cassandra > *Subject:* Re: 3.11.2 memory leak > > > > There have been a few people who have reported it, but nobody (yet) has > offered a patch to fix it. It would be good to have a reliable way to > repro, and/or an analysis of a heap dump demonstrating the problem (what's > actually retained at the time you're OOM'ing). > > > > On Mon, Jun 4, 2018 at 6:52 AM, Abdul Patel wrote: > > Hi All, > > > > I recently upgraded my non prod cluster from 3.10 to 3.11.2. > > It was working fine for a 1.5 weeks then suddenly nodetool info startee > reporting 80% and more memory consumption. > > Intially it was 16gb configured, then i bumped to 20gb and rebooted all 4 > nodes of cluster-single DC. > > Now after 8 days i again see 80% + usage and its 16gb and above ..which we > never saw before . > > Seems like memory leak bug? > > Does anyone has any idea ? Our 3.11.2 release rollout has been halted > because of this. > > If not 3.11.2 whats the next best stable release we have now? > > > The contents of this e-mail are intended for the named addressee only. It > contains information that may be confidential. Unless you are the named > addressee or an authorized designee, you may not copy or use it, or > disclose it to anyone else. If you received it in error please notify us > immediately and then destroy it. Dynatrace Austria GmbH (registration > number FN 91482h) is a company registered in Linz whose registered office > is at 4040 Linz, Austria, Freistädterstraße 313 >
RE: 3.11.2 memory leak
Jeff, FWIW, when talking about https://issues.apache.org/jira/browse/CASSANDRA-13929, there is a patch available since March without getting further attention. Regards, Thomas From: Jeff Jirsa [mailto:jji...@gmail.com] Sent: Dienstag, 05. Juni 2018 00:51 To: cassandra Subject: Re: 3.11.2 memory leak There have been a few people who have reported it, but nobody (yet) has offered a patch to fix it. It would be good to have a reliable way to repro, and/or an analysis of a heap dump demonstrating the problem (what's actually retained at the time you're OOM'ing). On Mon, Jun 4, 2018 at 6:52 AM, Abdul Patel mailto:abd786...@gmail.com>> wrote: Hi All, I recently upgraded my non prod cluster from 3.10 to 3.11.2. It was working fine for a 1.5 weeks then suddenly nodetool info startee reporting 80% and more memory consumption. Intially it was 16gb configured, then i bumped to 20gb and rebooted all 4 nodes of cluster-single DC. Now after 8 days i again see 80% + usage and its 16gb and above ..which we never saw before . Seems like memory leak bug? Does anyone has any idea ? Our 3.11.2 release rollout has been halted because of this. If not 3.11.2 whats the next best stable release we have now? The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313
Re: 3.11.2 memory leak
There have been a few people who have reported it, but nobody (yet) has offered a patch to fix it. It would be good to have a reliable way to repro, and/or an analysis of a heap dump demonstrating the problem (what's actually retained at the time you're OOM'ing). On Mon, Jun 4, 2018 at 6:52 AM, Abdul Patel wrote: > Hi All, > > I recently upgraded my non prod cluster from 3.10 to 3.11.2. > It was working fine for a 1.5 weeks then suddenly nodetool info startee > reporting 80% and more memory consumption. > Intially it was 16gb configured, then i bumped to 20gb and rebooted all 4 > nodes of cluster-single DC. > Now after 8 days i again see 80% + usage and its 16gb and above ..which we > never saw before . > Seems like memory leak bug? > Does anyone has any idea ? Our 3.11.2 release rollout has been halted > because of this. > If not 3.11.2 whats the next best stable release we have now? >
Re: 3.11.2 memory leak
Are you seeing significant issues in terms of performance? Increased garbage collection, long pauses, or even OutOfMemory? Which garbage collector are you using and with what settings/thresholds? Since the JVM's garbage-collected, a bigger heap can mean a problem or it can just mean "hasn't gotten big enough for the collector to bother doing any work" If it's genuinely having memory/heap pressure problems, it's probably worth getting a heap dump and poking through it to see what's using the space. For a heap that big, you'll probably need to run the Eclipse MAT CLI tools against it then open the result in the GUI. On Mon, Jun 4, 2018 at 6:52 AM, Abdul Patel wrote: > Hi All, > > I recently upgraded my non prod cluster from 3.10 to 3.11.2. > It was working fine for a 1.5 weeks then suddenly nodetool info startee > reporting 80% and more memory consumption. > Intially it was 16gb configured, then i bumped to 20gb and rebooted all 4 > nodes of cluster-single DC. > Now after 8 days i again see 80% + usage and its 16gb and above ..which we > never saw before . > Seems like memory leak bug? > Does anyone has any idea ? Our 3.11.2 release rollout has been halted > because of this. > If not 3.11.2 whats the next best stable release we have now? >