[jira] [Commented] (CASSANDRA-11870) Consider allocating direct buffers bypassing ByteBuffer.allocateDirect
[
https://issues.apache.org/jira/browse/CASSANDRA-11870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15344121#comment-15344121
]
Robert Stupp commented on CASSANDRA-11870:
--
It's a bit difficult to tell whether it's experienced in the wild, since before
3.x Netty buffers were on-heap and C* only had somewhat infrequent direct
memory allocations. However, with 3.0 and especially 3.6, we have more frequent
off-heap memory allocations. At least from my point of view, the majority is
still on 2.1.
What I intend to tackle is to prevent the (ugly) call to
{{java.nio.Bits#reserveMemory}} and also the global linked queue via
{{sun.misc.Cleaner}}, which has static synchronized methods {{add}} +
{{remove}}. My objection about {{ByteBuffer.allocateDirect()}} in general is
that if one thread runs into an _out-of-direct-memory_ situations, other thread
will likely run into that situation, too - and that will cause one CMS-GC
({{System.gc()}}) per thread.
However, I agree that maintaining a separate {{max_direct_memory_in_mb}} config
variable is probably not the best way. A better option could be to use the
already existing options that define off-heap memory bounds like
{{file_cache_size_in_mb}}, {{memtable_offheap_space_in_mb}},
{{index_summary_capacity_in_mb}}, sum them and use that value. WDYT?
I'm not sure, whether the following is a good idea: while testing OHC a while
ago with >30G of small allocations, it worked really nice (OHC does not use
BB.allocateDirect - that time I just realized that it's slow). What I'm
thinking of, is to not pool off-heap memory at all (and just keep track of the
total allocated amount, as currently in the patch) and leave alloc()/free() up
to the OS or jemalloc.
Your idea to centralize BB allocations in BufferPool sounds good!
> Consider allocating direct buffers bypassing ByteBuffer.allocateDirect
> --
>
> Key: CASSANDRA-11870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11870
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Minor
> Fix For: 3.x
>
>
> As outlined in CASSANDRA-11818, {{ByteBuffer.allocateDirect}} uses
> {{Bits.reserveMemory}}, which is there to respect the JVM setting
> {{-XX:MaxDirectMemorySize=...}}.
> {{Bits.reserveMemory}} first tries an "optimistic" {{tryReserveMemory}} and
> exits immediately on success. However, if that somehow doesn't succeed, it
> triggers a {{System.gc()}}, which is bad IMO (however, kind of how direct
> buffers work in Java). After that GC it sleeps and tries to reserve the
> memory up to 9 times - up to 511 ms - and then throws
> {{OutOfMemoryError("Direct buffer memory")}}.
> This is unnecessary for us since we always immediately "free" direct buffers
> as soon as we no longer need them.
> Proposal: Manage direct-memory reservations in our own code and skip
> {{Bits.reserveMemory}} that way.
> (However, Netty direct buffers are not under our control.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11870) Consider allocating direct buffers bypassing ByteBuffer.allocateDirect
[
https://issues.apache.org/jira/browse/CASSANDRA-11870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15340326#comment-15340326
]
T Jake Luciani commented on CASSANDRA-11870:
I'm trying to weigh the value of this approach, is this something people
experience in the wild? It seems if you hit this you would simply bump the JVM
flag. You also have your own flag you need to size appropriately so how is
this really any operationally better?
This might not be appropriate for this issue but looking at the patch it seems
we should try and consolidate calls to ByteBuffer.allocateDirect() with
BufferPool.tryGet()/put() which gracefully handles running out of DirectMemory
and recycling(). But is currently sized by the
DatabaseDescriptor.getFileCacheSizeInMB() which seems wrong to me, since it's a
generic buffer pool.
> Consider allocating direct buffers bypassing ByteBuffer.allocateDirect
> --
>
> Key: CASSANDRA-11870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11870
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Minor
> Fix For: 3.x
>
>
> As outlined in CASSANDRA-11818, {{ByteBuffer.allocateDirect}} uses
> {{Bits.reserveMemory}}, which is there to respect the JVM setting
> {{-XX:MaxDirectMemorySize=...}}.
> {{Bits.reserveMemory}} first tries an "optimistic" {{tryReserveMemory}} and
> exits immediately on success. However, if that somehow doesn't succeed, it
> triggers a {{System.gc()}}, which is bad IMO (however, kind of how direct
> buffers work in Java). After that GC it sleeps and tries to reserve the
> memory up to 9 times - up to 511 ms - and then throws
> {{OutOfMemoryError("Direct buffer memory")}}.
> This is unnecessary for us since we always immediately "free" direct buffers
> as soon as we no longer need them.
> Proposal: Manage direct-memory reservations in our own code and skip
> {{Bits.reserveMemory}} that way.
> (However, Netty direct buffers are not under our control.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11870) Consider allocating direct buffers bypassing ByteBuffer.allocateDirect
[
https://issues.apache.org/jira/browse/CASSANDRA-11870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15339098#comment-15339098
]
Robert Stupp commented on CASSANDRA-11870:
--
It's default is calculated in {{DatabaseDescriptor.applyConfig()}} to at least
1GB. 1GB is the default max memory setting for direct memory in Java AFAIK.
> Consider allocating direct buffers bypassing ByteBuffer.allocateDirect
> --
>
> Key: CASSANDRA-11870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11870
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Minor
> Fix For: 3.x
>
>
> As outlined in CASSANDRA-11818, {{ByteBuffer.allocateDirect}} uses
> {{Bits.reserveMemory}}, which is there to respect the JVM setting
> {{-XX:MaxDirectMemorySize=...}}.
> {{Bits.reserveMemory}} first tries an "optimistic" {{tryReserveMemory}} and
> exits immediately on success. However, if that somehow doesn't succeed, it
> triggers a {{System.gc()}}, which is bad IMO (however, kind of how direct
> buffers work in Java). After that GC it sleeps and tries to reserve the
> memory up to 9 times - up to 511 ms - and then throws
> {{OutOfMemoryError("Direct buffer memory")}}.
> This is unnecessary for us since we always immediately "free" direct buffers
> as soon as we no longer need them.
> Proposal: Manage direct-memory reservations in our own code and skip
> {{Bits.reserveMemory}} that way.
> (However, Netty direct buffers are not under our control.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11870) Consider allocating direct buffers bypassing ByteBuffer.allocateDirect
[
https://issues.apache.org/jira/browse/CASSANDRA-11870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15338935#comment-15338935
]
Mahdi Mohammadi commented on CASSANDRA-11870:
-
Shouldn't we set a default value for `max_direct_memory_in_mb` in Config.java?
> Consider allocating direct buffers bypassing ByteBuffer.allocateDirect
> --
>
> Key: CASSANDRA-11870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11870
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Minor
> Fix For: 3.x
>
>
> As outlined in CASSANDRA-11818, {{ByteBuffer.allocateDirect}} uses
> {{Bits.reserveMemory}}, which is there to respect the JVM setting
> {{-XX:MaxDirectMemorySize=...}}.
> {{Bits.reserveMemory}} first tries an "optimistic" {{tryReserveMemory}} and
> exits immediately on success. However, if that somehow doesn't succeed, it
> triggers a {{System.gc()}}, which is bad IMO (however, kind of how direct
> buffers work in Java). After that GC it sleeps and tries to reserve the
> memory up to 9 times - up to 511 ms - and then throws
> {{OutOfMemoryError("Direct buffer memory")}}.
> This is unnecessary for us since we always immediately "free" direct buffers
> as soon as we no longer need them.
> Proposal: Manage direct-memory reservations in our own code and skip
> {{Bits.reserveMemory}} that way.
> (However, Netty direct buffers are not under our control.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11870) Consider allocating direct buffers bypassing ByteBuffer.allocateDirect
[
https://issues.apache.org/jira/browse/CASSANDRA-11870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15338546#comment-15338546
]
Robert Stupp commented on CASSANDRA-11870:
--
(removed link to CASSANDRA-11921)
This code change by itself is independent, but related to CASSANDRA-12032 +
CASSANDRA-12033.
> Consider allocating direct buffers bypassing ByteBuffer.allocateDirect
> --
>
> Key: CASSANDRA-11870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11870
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Minor
> Fix For: 3.x
>
>
> As outlined in CASSANDRA-11818, {{ByteBuffer.allocateDirect}} uses
> {{Bits.reserveMemory}}, which is there to respect the JVM setting
> {{-XX:MaxDirectMemorySize=...}}.
> {{Bits.reserveMemory}} first tries an "optimistic" {{tryReserveMemory}} and
> exits immediately on success. However, if that somehow doesn't succeed, it
> triggers a {{System.gc()}}, which is bad IMO (however, kind of how direct
> buffers work in Java). After that GC it sleeps and tries to reserve the
> memory up to 9 times - up to 511 ms - and then throws
> {{OutOfMemoryError("Direct buffer memory")}}.
> This is unnecessary for us since we always immediately "free" direct buffers
> as soon as we no longer need them.
> Proposal: Manage direct-memory reservations in our own code and skip
> {{Bits.reserveMemory}} that way.
> (However, Netty direct buffers are not under our control.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11870) Consider allocating direct buffers bypassing ByteBuffer.allocateDirect
[
https://issues.apache.org/jira/browse/CASSANDRA-11870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306486#comment-15306486
]
Robert Stupp commented on CASSANDRA-11870:
--
As outlined in CASSANDRA-11818, it is beneficial for stability and performance
to introduce this.
To achieve that, Netty 4.1 with that patch plus code changes to work around
{{Bits.reserveMemory}}+{{Cleaner}} are required.
> Consider allocating direct buffers bypassing ByteBuffer.allocateDirect
> --
>
> Key: CASSANDRA-11870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11870
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Robert Stupp
>Priority: Minor
> Fix For: 3.x
>
>
> As outlined in CASSANDRA-11818, {{ByteBuffer.allocateDirect}} uses
> {{Bits.reserveMemory}}, which is there to respect the JVM setting
> {{-XX:MaxDirectMemorySize=...}}.
> {{Bits.reserveMemory}} first tries an "optimistic" {{tryReserveMemory}} and
> exits immediately on success. However, if that somehow doesn't succeed, it
> triggers a {{System.gc()}}, which is bad IMO (however, kind of how direct
> buffers work in Java). After that GC it sleeps and tries to reserve the
> memory up to 9 times - up to 511 ms - and then throws
> {{OutOfMemoryError("Direct buffer memory")}}.
> This is unnecessary for us since we always immediately "free" direct buffers
> as soon as we no longer need them.
> Proposal: Manage direct-memory reservations in our own code and skip
> {{Bits.reserveMemory}} that way.
> (However, Netty direct buffers are not under our control.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
