[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17623634#comment-17623634
]
Benedict Elliott Smith commented on CASSANDRA-14355:
I would recommend upgrading to the 4.x series. I can say that repair has
received a number of improvements since this ticket was filed, and I would say
it is somewhat likely not to be a problem. However, there was insufficient
information provided to reproduce this bug, so I cannot say for sure whether
any issue you may be experiencing is resolved in a later release.
CASSANDRA-15902 has been resolved since 3.11.9.
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Normal
> Fix For: 3.11.x
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png, LongGC_Dominator-Tree.png, LongGC_Histogram.png,
> LongGC_Problem-Suspect-1_FastThreadLocalThread.png, LongGC_nodetool_info.txt
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17623385#comment-17623385
]
Doug Whitfield commented on CASSANDRA-14355:
It has been over two years since there has been any comment on this. Is it safe
to assume this was fixed in 3.11.5?
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Normal
> Fix For: 3.11.x
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png, LongGC_Dominator-Tree.png, LongGC_Histogram.png,
> LongGC_Problem-Suspect-1_FastThreadLocalThread.png, LongGC_nodetool_info.txt
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148333#comment-17148333
]
Swen Fuhrmann commented on CASSANDRA-14355:
---
We are facing the same issue as described by [~padakwaak]. I created a new
ticked specific for that issue: CASSANDRA-15902
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Normal
> Fix For: 3.11.x
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png, LongGC_Dominator-Tree.png, LongGC_Histogram.png,
> LongGC_Problem-Suspect-1_FastThreadLocalThread.png, LongGC_nodetool_info.txt
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992980#comment-16992980
]
Benedict Elliott Smith commented on CASSANDRA-14355:
If there's an issue around Cassandra failing to clean up after a terminated
repair, that's certainly a bug you should file.
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Normal
> Fix For: 3.11.x
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png, LongGC_Dominator-Tree.png, LongGC_Histogram.png,
> LongGC_Problem-Suspect-1_FastThreadLocalThread.png, LongGC_nodetool_info.txt
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992973#comment-16992973
]
Chris Kistner commented on CASSANDRA-14355:
---
[~benedict], thank you for the feedback.
After we've done some extensive investigation on our end, we pinned our issue
to Cassandra Reaper terminating the Cassandra repair sessions if they don't
finish within 30 minutes ("hangingRepairTimeoutMins" default), and then those
repair threads aren't closed by Cassandra 3.11.4 and thus retain ~200MB of data
on heap in our case. The other annoying part when this happens is that both
Cassandra & Cassandra Reaper logs it as if the repair session was successful,
unless you go and look at the logs in detail.
I think I should rather create a new bug for Cassandra not closing the Repair
session thread correctly, since the original issue's
"io.netty.util.concurrent.FastThreadLocalThread" class was referenced from
"Native-Transport-Requests" and "ReadStage" threads, where as ours were all
from "Repair" threads.
We might be able to see if we can reproduce the Cassandra issue where it is not
closing Repair threads properly on 3.11.4 and then on 3.11.5.
So far we've only experienced these OOME issues in client production
environments and right now all our clients are in code/change freeze, so we
won't be able to test 3.11.5 for about a month in a large production
environment.
So for now we'll just schedule per host & per table repairs using cron scripts
and doing a different host each day of the week instead of using Cassandra
Reaper that might terminate our repair sessions.
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Normal
> Fix For: 3.11.x
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png, LongGC_Dominator-Tree.png, LongGC_Histogram.png,
> LongGC_Problem-Suspect-1_FastThreadLocalThread.png, LongGC_nodetool_info.txt
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992599#comment-16992599
]
Benedict Elliott Smith commented on CASSANDRA-14355:
This looks like the known issues with repair, which has been partially
addressed in both CASSANDRA-14096 and CASSANDRA-15202.
The best solution to this problem until then is to repair smaller ranges at a
time, or to upgrade to a version with CASSANDRA-14096 (CASSANDRA-15202 won't be
released for a while, though it could easily be back ported, and might
justifiably be considered for that).
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Normal
> Fix For: 3.11.x
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png, LongGC_Dominator-Tree.png, LongGC_Histogram.png,
> LongGC_Problem-Suspect-1_FastThreadLocalThread.png, LongGC_nodetool_info.txt
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16992394#comment-16992394
]
Chris Kistner commented on CASSANDRA-14355:
---
We have now experienced an issue that might be related to this, however our
Cassandra did not crash yet - it just had frequent (every ~ 2 minutes)
"ConcurrentMarkSweep GC" events of 16+ seconds!
eg.:
{noformat}
Line 78776: WARN [Service Thread] 2019-12-10 08:03:19,969 GCInspector.java:282
- ConcurrentMarkSweep GC in 19129ms. CMS Old Gen: 7547650016 -> 7547650048;
Par Eden Space: 671088640 -> 251798544; Par Survivor Space: 83886048 -> 0
Line 79080: WARN [Service Thread] 2019-12-10 08:03:37,565
GCInspector.java:282 - ConcurrentMarkSweep GC in 16379ms. Par Eden Space:
671088640 -> 254509608; Par Survivor Space: 83886032 -> 0{noformat}
Sometimes it went back down to 200ms again, and after we did a "nodetool drain"
and then removed the node from the cluster the GC time remained sub 250ms.
Our setup is:
* 5 nodes in dc1, 5 nodes in dc2.
* RF: dc1=5, dc2=5
* CL = Local Quorum
* Host with 32GB of RAM -> Cassandra allocates 8GB to heap
* Java version: java-1.8.0-openjdk-1.8.0.151-5.b12
* Using Cassandra Reaper 4.6.1 where we scheduled a repair with 32
segments/node (364 segments in total)
I have attached some screenshots from our~11GB heap dump, where
io.netty.util.concurrent.FastThreadLocalThread contributed towards 6.4GB of the
heap size:
* Problem Suspect 1: LongGC_Problem-Suspect-1_FastThreadLocalThread.png
* Dominator Tree: LongGC_Dominator-Tree.png
* Histogram: LongGC_Histogram.png
I have also attached the output of "nodetool status": LongGC_nodetool_info.txt
We have not tried out Cassandra 3.11.5, which apparently solved the Repair OOME
issue: CASSANDRA-14096
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Normal
> Fix For: 3.11.x
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png, LongGC_Dominator-Tree.png, LongGC_Histogram.png,
> LongGC_Problem-Suspect-1_FastThreadLocalThread.png, LongGC_nodetool_info.txt
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503228#comment-16503228
]
Luke Arrigoni commented on CASSANDRA-14355:
---
Has anyone determined a method of recreating this leak under load test
conditions to bring the reproducing time down? This ticket would be easier to
work on if we could recreate within hours instead of weeks.
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Major
> Fix For: 3.11.3
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500249#comment-16500249
]
Abdul Patel commented on CASSANDRA-14355:
-
If 3.11.2 release has memory leak issues, whats the next best patch version ?
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Major
> Fix For: 3.11.3
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499444#comment-16499444
]
Abdul Patel commented on CASSANDRA-14355:
-
I have seen nodetool info reporting high memory usage after a weeks of
upgrading t0 3.11.2, i rebooted all nodes 1 week no issues, later agian same ,
no errors in errorlog .
is this related to this issue?
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Major
> Fix For: 3.11.3
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493457#comment-16493457
]
Jan Dörrenhaus commented on CASSANDRA-14355:
This issue has not been touched in almost two months now. It is the last open
issue for 3.11.3, and 3.11.3 turns out to be critical for us, since
CASSANDRA-14087 is a showstopper for us. Can this issue be fixed soon?
Alternatively, is it possible to push it to a 3.11.4 release, and have 3.11.3
now?
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Major
> Fix For: 3.11.3
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428135#comment-16428135
]
Thomas Steinmaurer commented on CASSANDRA-14355:
[~zznate]:
{quote}
Also, Thomas Steinmaurer this issue looks different from what you experienced
in CASSANDRA-13929, but have you all encountered anything like this before?
{quote}
No, sorry. Looks different.
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Major
> Fix For: 3.11.3
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426397#comment-16426397
]
Eric Evans commented on CASSANDRA-14355:
Our production cluster is on 3.11.0 and is running fine, yes. This staging
environment was first setup on 3.11.0 where these OOMs started, and has since
been upgraded to 3.11.2. Both versions OOM (regularly, on about 1.5 hour
intervals, in fact).
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Major
> Fix For: 3.11.3
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426387#comment-16426387
]
Nate McCall commented on CASSANDRA-14355:
-
{quote}I am (so far) at a loss to understand what is different about this
environment.
{quote}
So other cluster(s) with the same version are running fine?
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Major
> Fix For: 3.11.3
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426179#comment-16426179
]
Eric Evans commented on CASSANDRA-14355:
Also, I can make a heap available, but be warned, they are 12GB in size.
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Major
> Fix For: 3.11.3
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426178#comment-16426178
]
Eric Evans commented on CASSANDRA-14355:
{quote}[~urandom] Did this just start happening again or is this a continuation
of the issues you saw with CASSANDRA-13754?
{quote}
This is happening in a newly minted staging cluster, and is (readily)
reproducible on both the same 3.11.0 build we use in production, and 3.11.2. I
am (so far) at a loss to understand what is different about this environment.
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Major
> Fix For: 3.11.3
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14355) Memory leak
[
https://issues.apache.org/jira/browse/CASSANDRA-14355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426118#comment-16426118
]
Nate McCall commented on CASSANDRA-14355:
-
[~urandom] Did this just start happening again or is this a continuation of the
issues you saw with CASSANDRA-13754?
Also, [~tsteinmaurer] this issue looks different from what you experienced in
CASSANDRA-13929, but have you all encountered anything like this before?
> Memory leak
> ---
>
> Key: CASSANDRA-14355
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14355
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian Jessie, OpenJDK 1.8.0_151
>Reporter: Eric Evans
>Priority: Major
> Fix For: 3.11.3
>
> Attachments: 01_Screenshot from 2018-04-04 14-24-00.png,
> 02_Screenshot from 2018-04-04 14-28-33.png, 03_Screenshot from 2018-04-04
> 14-24-50.png
>
>
> We're seeing regular, frequent {{OutOfMemoryError}} exceptions. Similar to
> CASSANDRA-13754, an analysis of the heap dumps shows the heap consumed by the
> {{threadLocals}} member of the instances of
> {{io.netty.util.concurrent.FastThreadLocalThread}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
