[jira] [Commented] (CASSANDRA-13754) FastThreadLocal leaks memory
[
https://issues.apache.org/jira/browse/CASSANDRA-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150611#comment-16150611
]
Jeremiah Jordan commented on CASSANDRA-13754:
-
and +1 for the patch.
> FastThreadLocal leaks memory
>
>
> Key: CASSANDRA-13754
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13754
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 3.11.0, Netty 4.0.44.Final, OpenJDK 8u141-b15
>Reporter: Eric Evans
>Assignee: Robert Stupp
> Fix For: 3.11.1
>
>
> After a chronic bout of {{OutOfMemoryError}} in our development environment,
> a heap analysis is showing that more than 10G of our 12G heaps are consumed
> by the {{threadLocals}} members (instances of {{java.lang.ThreadLocalMap}})
> of various {{io.netty.util.concurrent.FastThreadLocalThread}} instances.
> Reverting
> [cecbe17|https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=cecbe17e3eafc052acc13950494f7dddf026aa54]
> fixes the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13754) FastThreadLocal leaks memory
[
https://issues.apache.org/jira/browse/CASSANDRA-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150609#comment-16150609
]
Jeremiah Jordan commented on CASSANDRA-13754:
-
+1 for just
https://github.com/apache/cassandra/commit/2cafd0b6b4bbc5a6ec5726d47d0093bdac3af19c
to fix this and splitting out the other changes to a new ticket.
> FastThreadLocal leaks memory
>
>
> Key: CASSANDRA-13754
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13754
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 3.11.0, Netty 4.0.44.Final, OpenJDK 8u141-b15
>Reporter: Eric Evans
>Assignee: Robert Stupp
> Fix For: 3.11.1
>
>
> After a chronic bout of {{OutOfMemoryError}} in our development environment,
> a heap analysis is showing that more than 10G of our 12G heaps are consumed
> by the {{threadLocals}} members (instances of {{java.lang.ThreadLocalMap}})
> of various {{io.netty.util.concurrent.FastThreadLocalThread}} instances.
> Reverting
> [cecbe17|https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=cecbe17e3eafc052acc13950494f7dddf026aa54]
> fixes the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13754) FastThreadLocal leaks memory
[
https://issues.apache.org/jira/browse/CASSANDRA-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150416#comment-16150416
]
Markus Dlugi commented on CASSANDRA-13754:
--
Your latest patch which resets the entire {{BTree$Builder.values}} array seemed
to do the trick, entire load test is now running smoothly. No more crazy GCing
and most importantly no {{OutOfMemoryError}} s. Thanks a lot for the fast
support and help!
> FastThreadLocal leaks memory
>
>
> Key: CASSANDRA-13754
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13754
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 3.11.0, Netty 4.0.44.Final, OpenJDK 8u141-b15
>Reporter: Eric Evans
>Assignee: Robert Stupp
> Fix For: 3.11.1
>
>
> After a chronic bout of {{OutOfMemoryError}} in our development environment,
> a heap analysis is showing that more than 10G of our 12G heaps are consumed
> by the {{threadLocals}} members (instances of {{java.lang.ThreadLocalMap}})
> of various {{io.netty.util.concurrent.FastThreadLocalThread}} instances.
> Reverting
> [cecbe17|https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=cecbe17e3eafc052acc13950494f7dddf026aa54]
> fixes the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13754) FastThreadLocal leaks memory
[
https://issues.apache.org/jira/browse/CASSANDRA-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150338#comment-16150338
]
Robert Stupp commented on CASSANDRA-13754:
--
Your observation regarding {{BTree.Builder.values[]}} seems correct.
However, {{SEPWorker}} must *not* remove the thread locals - it's the intention
of these thread-locals to be kept for reuse.
> FastThreadLocal leaks memory
>
>
> Key: CASSANDRA-13754
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13754
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 3.11.0, Netty 4.0.44.Final, OpenJDK 8u141-b15
>Reporter: Eric Evans
>Assignee: Robert Stupp
> Fix For: 3.11.1
>
>
> After a chronic bout of {{OutOfMemoryError}} in our development environment,
> a heap analysis is showing that more than 10G of our 12G heaps are consumed
> by the {{threadLocals}} members (instances of {{java.lang.ThreadLocalMap}})
> of various {{io.netty.util.concurrent.FastThreadLocalThread}} instances.
> Reverting
> [cecbe17|https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=cecbe17e3eafc052acc13950494f7dddf026aa54]
> fixes the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13754) FastThreadLocal leaks memory
[
https://issues.apache.org/jira/browse/CASSANDRA-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150172#comment-16150172
]
Markus Dlugi commented on CASSANDRA-13754:
--
[~snazy], I don't think the node is overloaded. I originally thought so as
well, so I made a little experiment where I included a cap in our load test
limiting the {{INSERT}} s per minute from ~25,000 to ~10,000. As a consequence,
the node survived a little longer, but in the end it still died with an
{{OutOfMemoryError}} after more data had been inserted. So it's not that there
are too many active writes, it's just that the node fails after a certain
amount of total writes, which indicates to me that a memory leak is indeed
happening.
I also had another look into the heap dump I sent you, and you are correct that
the heap is mostly filled with {{BTree$Builder}} instances that still have
stuff in their {{values}} array. However, if you look closer, you will notice
that for each of these instances, the {{values}} array always contains {{null}}
for the first couple of entries, and only after those there is still actual
content. For some reason, the actual content always starts at index 28, whereas
indices 0 - 27 are {{null}} - not sure if this is a coincidence? But you can
also see that for all the {{BTree$Builder}} objects, the {{count}} attribute is
0, which also indicates to me that {{BTree$Builder.cleanup()}} has already run
and those are not active writes. This theory is supported by the fact that my
little workaround of manually calling {{FastThreadLocal.removeAll()}} actually
works, because this means that no other objects except the {{FastThreadLocal}}
s still have references to the builders.
Therefore, I think we have two issues here:
# {{SEPWorker}} is never cleaning the {{FastThreadLocal}} s, therefore
accumulating references to otherwise dead objects - maybe we can include
something to at least remove non-static entries regularly?
# {{BTree$Builder}} seems to have an issue properly cleaning up after building,
so the objects referenced by the {{FastThreadLocal}} s of the {{SEPWorker}}
threads are very large and thus ultimately lead to the {{OutOfMemoryError}} s
> FastThreadLocal leaks memory
>
>
> Key: CASSANDRA-13754
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13754
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 3.11.0, Netty 4.0.44.Final, OpenJDK 8u141-b15
>Reporter: Eric Evans
>Assignee: Robert Stupp
> Fix For: 3.11.1
>
>
> After a chronic bout of {{OutOfMemoryError}} in our development environment,
> a heap analysis is showing that more than 10G of our 12G heaps are consumed
> by the {{threadLocals}} members (instances of {{java.lang.ThreadLocalMap}})
> of various {{io.netty.util.concurrent.FastThreadLocalThread}} instances.
> Reverting
> [cecbe17|https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=cecbe17e3eafc052acc13950494f7dddf026aa54]
> fixes the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13754) FastThreadLocal leaks memory
[
https://issues.apache.org/jira/browse/CASSANDRA-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149207#comment-16149207
]
Robert Stupp commented on CASSANDRA-13754:
--
Well, yea. Looking at the heap dump, that [~markusdlugi] provided, is looks
like the node is "just" overloaded with too many and maybe too big writes in
combination with a small heap. There are lots of {{BTree$Builder}} instances
with live references in their {{Object[] values}} array to {{HeapByteBuffer}}
instances, each holding a 1MB {{byte[]}}.
{{BTree$Builder}} instances reset the {{Object[] values}} when finished - i.e.
those builders are actively doing something (writes are happening at that time).
TL;DR I don't think this is actually related to the issue that [~urandom]
describes.
[~urandom], can you explain what actually what these {{ThreadLocal}} instances
referenced?
> FastThreadLocal leaks memory
>
>
> Key: CASSANDRA-13754
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13754
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 3.11.0, Netty 4.0.44.Final, OpenJDK 8u141-b15
>Reporter: Eric Evans
>Assignee: Robert Stupp
> Fix For: 3.11.1
>
>
> After a chronic bout of {{OutOfMemoryError}} in our development environment,
> a heap analysis is showing that more than 10G of our 12G heaps are consumed
> by the {{threadLocals}} members (instances of {{java.lang.ThreadLocalMap}})
> of various {{io.netty.util.concurrent.FastThreadLocalThread}} instances.
> Reverting
> [cecbe17|https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=cecbe17e3eafc052acc13950494f7dddf026aa54]
> fixes the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13754) FastThreadLocal leaks memory
[
https://issues.apache.org/jira/browse/CASSANDRA-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149023#comment-16149023
]
Robert Stupp commented on CASSANDRA-13754:
--
[~markusdlugi] do you have a heap dump for me or references to what's actually
in these {{FastThreadLocal}} s or which FTLs actually cause this? I suspect
these are FTLs created in non-static fields, which probably need a different
handling.
> FastThreadLocal leaks memory
>
>
> Key: CASSANDRA-13754
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13754
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 3.11.0, Netty 4.0.44.Final, OpenJDK 8u141-b15
>Reporter: Eric Evans
>Assignee: Robert Stupp
> Fix For: 3.11.1
>
>
> After a chronic bout of {{OutOfMemoryError}} in our development environment,
> a heap analysis is showing that more than 10G of our 12G heaps are consumed
> by the {{threadLocals}} members (instances of {{java.lang.ThreadLocalMap}})
> of various {{io.netty.util.concurrent.FastThreadLocalThread}} instances.
> Reverting
> [cecbe17|https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=cecbe17e3eafc052acc13950494f7dddf026aa54]
> fixes the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13754) FastThreadLocal leaks memory
[
https://issues.apache.org/jira/browse/CASSANDRA-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149002#comment-16149002
]
Markus Dlugi commented on CASSANDRA-13754:
--
[~snazy], I checked out the 13754-ftl-leak-3.11 branch and ran our load test
with your patches. Unfortunately, the memory leak is still there, Cassandra
first starts GCing like crazy and then throws an {{OutOfMemoryError}} after a
while. So it seems like the call to {{FastThreadLocal.removeAll()}} is not
properly executed.
Actually, while writing this it just struck me why that is the case. The thread
created in {{SEPWorker}} does not stop after it finishes, but is instead
returned to the pool and picks up new tasks. So it will never leave the
{{run()}} method, and therefore also never execute
{{FastThreadLocal.removeAll()}}. So I think we will have to include that call
in the {{SEPWorker}} itself.
> FastThreadLocal leaks memory
>
>
> Key: CASSANDRA-13754
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13754
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 3.11.0, Netty 4.0.44.Final, OpenJDK 8u141-b15
>Reporter: Eric Evans
>Assignee: Robert Stupp
> Fix For: 3.11.1
>
>
> After a chronic bout of {{OutOfMemoryError}} in our development environment,
> a heap analysis is showing that more than 10G of our 12G heaps are consumed
> by the {{threadLocals}} members (instances of {{java.lang.ThreadLocalMap}})
> of various {{io.netty.util.concurrent.FastThreadLocalThread}} instances.
> Reverting
> [cecbe17|https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=cecbe17e3eafc052acc13950494f7dddf026aa54]
> fixes the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13754) FastThreadLocal leaks memory
[
https://issues.apache.org/jira/browse/CASSANDRA-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148858#comment-16148858
]
Robert Stupp commented on CASSANDRA-13754:
--
Yea, that {{SEPWorker}} plus a couple more places (see
[here|https://github.com/apache/cassandra/compare/cassandra-3.11...snazy:13754-ftl-leak-3.11?expand=1]
and
[here|https://github.com/apache/cassandra/compare/trunk...snazy:13754-ftl-leak-trunk?expand=1]).
It's actually difficult to tell whether there was really something broken in
all these places in the patch, but those changes don't (should not) harm. But
if someone introduces a {{FastThreadLocal}} there in the future, it should have
no negative impact.
Our CI's currently checking the patches.
[~markusdlugi], do you have any chance to verify the patches?
> FastThreadLocal leaks memory
>
>
> Key: CASSANDRA-13754
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13754
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 3.11.0, Netty 4.0.44.Final, OpenJDK 8u141-b15
>Reporter: Eric Evans
>Assignee: Robert Stupp
> Fix For: 3.11.1
>
>
> After a chronic bout of {{OutOfMemoryError}} in our development environment,
> a heap analysis is showing that more than 10G of our 12G heaps are consumed
> by the {{threadLocals}} members (instances of {{java.lang.ThreadLocalMap}})
> of various {{io.netty.util.concurrent.FastThreadLocalThread}} instances.
> Reverting
> [cecbe17|https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=cecbe17e3eafc052acc13950494f7dddf026aa54]
> fixes the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13754) FastThreadLocal leaks memory
[
https://issues.apache.org/jira/browse/CASSANDRA-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148823#comment-16148823
]
Markus Dlugi commented on CASSANDRA-13754:
--
I see, you are talking about {{NamedTheadFactory.threadLocalDeallocator()}},
right? That should help for any threads created there, but as stated above,
{{FastThreadLocalThread}} is also used in some other classes. Heap-wise, the
most problematic one for us was the usage in {{SEPWorker}} I think, and this
seems to have only been introduced with commit
[1e92ce43a5a730f81d3f6cfd72e7f4b126db788a|https://github.com/apache/cassandra/commit/1e92ce43a5a730f81d3f6cfd72e7f4b126db788a]
by [~tjake]. Maybe something similar can be done there and in all other
classes making use of {{FastThreadLocalThread}}.
> FastThreadLocal leaks memory
>
>
> Key: CASSANDRA-13754
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13754
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 3.11.0, Netty 4.0.44.Final, OpenJDK 8u141-b15
>Reporter: Eric Evans
>Assignee: Robert Stupp
> Fix For: 3.11.1
>
>
> After a chronic bout of {{OutOfMemoryError}} in our development environment,
> a heap analysis is showing that more than 10G of our 12G heaps are consumed
> by the {{threadLocals}} members (instances of {{java.lang.ThreadLocalMap}})
> of various {{io.netty.util.concurrent.FastThreadLocalThread}} instances.
> Reverting
> [cecbe17|https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=cecbe17e3eafc052acc13950494f7dddf026aa54]
> fixes the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13754) FastThreadLocal leaks memory
[
https://issues.apache.org/jira/browse/CASSANDRA-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148769#comment-16148769
]
Robert Stupp commented on CASSANDRA-13754:
--
Thanks for the investigation, [~markusdlugi], I'm going to check what's
happening. All {{Runnables}} should have been decorated with a {{try-finally}}
to call {{FastThreadLocal.removeAll()}} to prevent exactly that.
> FastThreadLocal leaks memory
>
>
> Key: CASSANDRA-13754
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13754
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 3.11.0, Netty 4.0.44.Final, OpenJDK 8u141-b15
>Reporter: Eric Evans
> Fix For: 3.11.1
>
>
> After a chronic bout of {{OutOfMemoryError}} in our development environment,
> a heap analysis is showing that more than 10G of our 12G heaps are consumed
> by the {{threadLocals}} members (instances of {{java.lang.ThreadLocalMap}})
> of various {{io.netty.util.concurrent.FastThreadLocalThread}} instances.
> Reverting
> [cecbe17|https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=cecbe17e3eafc052acc13950494f7dddf026aa54]
> fixes the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13754) FastThreadLocal leaks memory
[
https://issues.apache.org/jira/browse/CASSANDRA-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148580#comment-16148580
]
Markus Dlugi commented on CASSANDRA-13754:
--
[~jjirsa], we are experiencing the same issue on Cassandra 3.11.0 and Netty
4.0.44. We have some custom triggers which create additional rows when
{{INSERT}}s on specific CFs are executed. When inserting a lot of data, the
Cassandra would constantly crash with {{OutOfMemoryError}}s. We analyzed a heap
dump and came to the same conclusion, which is that the instances of
{{FastThreadLocalThread}} were responsible for this behaviour. As a quick
workaround, we added a call to {{FastThreadLocal.removeAll()}} to our triggers,
which alleviates the problem for us.
It seems that there already was an issue once with {{FastThreadLocal}}s, as
described in CASSANDRA-13033, but apparently the change by [~snazy] in
CASSANDRA-13034 resurfaced the issue. For a proper fix, I think all
{{FastThreadLocalThread}} instances should call {{FastThreadLocal.removeAll()}}
when they are done with their work. A quick glance at the code indicates that
this affects the classes {{CompressedInputStream}},
{{StreamingInboundHandler}}, {{NamedThreadFactory}} and {{SEPWorker}}.
> FastThreadLocal leaks memory
>
>
> Key: CASSANDRA-13754
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13754
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 3.11.0, Netty 4.0.44.Final, OpenJDK 8u141-b15
>Reporter: Eric Evans
> Fix For: 3.11.1
>
>
> After a chronic bout of {{OutOfMemoryError}} in our development environment,
> a heap analysis is showing that more than 10G of our 12G heaps are consumed
> by the {{threadLocals}} members (instances of {{java.lang.ThreadLocalMap}})
> of various {{io.netty.util.concurrent.FastThreadLocalThread}} instances.
> Reverting
> [cecbe17|https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=cecbe17e3eafc052acc13950494f7dddf026aa54]
> fixes the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13754) FastThreadLocal leaks memory
[
https://issues.apache.org/jira/browse/CASSANDRA-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122390#comment-16122390
]
Eric Evans commented on CASSANDRA-13754:
Cassandra 3.11.0, Netty 4.0.44.Final
> FastThreadLocal leaks memory
>
>
> Key: CASSANDRA-13754
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13754
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: OpenJDK 8u141-b15
>Reporter: Eric Evans
> Fix For: 3.11.1
>
>
> After a chronic bout of {{OutOfMemoryError}} in our development environment,
> a heap analysis is showing that more than 10G of our 12G heaps are consumed
> by the {{threadLocals}} members (instances of {{java.lang.ThreadLocalMap}})
> of various {{io.netty.util.concurrent.FastThreadLocalThread}} instances.
> Reverting
> [cecbe17|https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=cecbe17e3eafc052acc13950494f7dddf026aa54]
> fixes the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13754) FastThreadLocal leaks memory
[
https://issues.apache.org/jira/browse/CASSANDRA-13754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16122012#comment-16122012
]
Jeff Jirsa commented on CASSANDRA-13754:
What version are you on [~urandom] (or really, which version of netty is in the
classpath) ?
> FastThreadLocal leaks memory
>
>
> Key: CASSANDRA-13754
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13754
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: OpenJDK 8u141-b15
>Reporter: Eric Evans
> Fix For: 3.11.1
>
>
> After a chronic bout of {{OutOfMemoryError}} in our development environment,
> a heap analysis is showing that more than 10G of our 12G heaps are consumed
> by the {{threadLocals}} members (instances of {{java.lang.ThreadLocalMap}})
> of various {{io.netty.util.concurrent.FastThreadLocalThread}} instances.
> Reverting
> [cecbe17|https://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=cecbe17e3eafc052acc13950494f7dddf026aa54]
> fixes the issue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
