Hi Erik, The slabs will always have a high in-degree because we allocate space for data inside them. We generally don't use unslabbed because of the high GC overhead; it would be interesting to see what kind of performance one gets with unslabbed compared to heap buffers with this GC improvement and whether it makes unslabbed a realistic option to use.
The other way to avoid the high in-degree of slabs is to use the offheap_objects option. This choice is used very often in production systems. It takes the trie buffers and cell data off-heap, but still has quite a bit of ordering structure on the heap. The above applies to all memtable implementations in Cassandra; they share the same cell data management logic. I will take removing the cycle in ApplyState into account for the next stages of trie memtable development. Regards, Branimir On Mon, Jul 27, 2026 at 3:32 PM Erik Osterlund via dev < [email protected]> wrote: > Hi Dmitry, > > Thanks for the reply and the pointers, it’s very helpful. As a GC > developer, I naturally get interested when people are moving data off-heap > because of GC issues. > > I tried out the Trie memtable data structure. Unfortunately, it looks like > there is a cycle caused by InMemoryTrie.ApplyState which has an implicit > this$0 field due to being an internal class. I tried patching it up a bit > locally to pass that around as an argument instead. Still some slab > allocator (not sure what it’s fore) seemingly causing in-degree > 6 which > the algorithm is a bit sensitive to right now. But with the patched Trie > and unslabbed_heap_buffers it works beautifully well and I see several GB > reclaimed old gen memory in young collections, without having to trace the > old generation. It’s CPU spending is now invariant of the old gen live set > size, as long as we can stick to these rules I mentioned. > > This feels rather encouraging IMO. :-) > > Thanks, > /Erik > > > On 27 Jul 2026, at 12:09, Dmitry Konstantinov <[email protected]> wrote: > > This Message Is From a New Sender > You have not previously corresponded with the sender of this message. > Report Suspicious > > <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/ACWV5N9M2RV99hQ!Op20OCZG1EegmzBtO1N5l_0NlI-ToMvOCf6nTg_MDwHHjAYeHpOwVpF74U5Z9U9hvMImUXCmFpRqWmpISVsORhymuGGN-sm7Tr4G63oCvH3p-fuixd7qWTAOw5ah2Ug$> > > Hi Erik, > Thanks for the email! We're always happy to welcome JVM developers here. > We also have Trie memtable implementation for memtable <partition key > ---> partition data> map - [1],[2],[5] > It can keep some portion of data off-heap also but the underlying objects > are still in heap: > > <image.png> > > There is an idea about extending Trie implementation and moving more parts > of it to off-heap - [3]. > There is also another idea for the map but it has not been implemented so > far: [4] > > [1] > https://cassandra.apache.org/_/blog/Apache-Cassandra-5.0-Features-Trie-Memtables-and-Trie-Indexed-SSTables.html > <https://urldefense.com/v3/__https://cassandra.apache.org/_/blog/Apache-Cassandra-5.0-Features-Trie-Memtables-and-Trie-Indexed-SSTables.html__;!!ACWV5N9M2RV99hQ!PczTox7K8CxUiR-8i1oQAWeuiwNxISCGjq8moRxQRGYGjt7DVNe2Uk51DQscE8g13blXNeWAG6E6w58ED4ei$> > > [2] https://www.vldb.org/pvldb/vol15/p3359-lambov.pdf > <https://urldefense.com/v3/__https://www.vldb.org/pvldb/vol15/p3359-lambov.pdf__;!!ACWV5N9M2RV99hQ!PczTox7K8CxUiR-8i1oQAWeuiwNxISCGjq8moRxQRGYGjt7DVNe2Uk51DQscE8g13blXNeWAG6E6wwydJXc_$> > > [3] > https://cwiki.apache.org/confluence/spaces/CASSANDRA/pages/392038876/CEP-57+Flat+keys+and+trie+interfaces > <https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/CASSANDRA/pages/392038876/CEP-57*Flat*keys*and*trie*interfaces__;KysrKys!!ACWV5N9M2RV99hQ!PczTox7K8CxUiR-8i1oQAWeuiwNxISCGjq8moRxQRGYGjt7DVNe2Uk51DQscE8g13blXNeWAG6E6w3z0dj6w$> > > [4] https://issues.apache.org/jira/browse/CASSANDRA-7282 > <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CASSANDRA-7282__;!!ACWV5N9M2RV99hQ!PczTox7K8CxUiR-8i1oQAWeuiwNxISCGjq8moRxQRGYGjt7DVNe2Uk51DQscE8g13blXNeWAG6E6w33qzYRA$> > [5] > https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/memtable/TrieMemtable.java > <https://urldefense.com/v3/__https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/memtable/TrieMemtable.java__;!!ACWV5N9M2RV99hQ!PczTox7K8CxUiR-8i1oQAWeuiwNxISCGjq8moRxQRGYGjt7DVNe2Uk51DQscE8g13blXNeWAG6E6w3Jt98gW$> > > Regards, > Dmitry > > On Mon, 27 Jul 2026 at 10:45, Erik Osterlund via dev < > [email protected]> wrote: > >> Hi, >> >> I’m one of the ZGC devs. Have been playing around a bit with an >> interesting algorithmic extension of generational ZGC using a form of >> reference counting in the old generation. This allows us to essentially >> have young generation collections reclaim zero reference count old objects, >> and free up acyclic garbage in the old generation as well, just before it >> promotes objects from the young generation to the old generation. The >> consequence is that as long as long-lived garbage is acyclic, we pretty >> much don’t need to do major collections other than to defragment memory >> every now and then. But the actual pressure to reclaim old garbage vanishes >> and is handled from the young collections. >> >> One quirk is that in this scheme, we can reclaim objects that never have >> an in-degree higher than 6, because then accounting isn’t free any longer >> (we are using available object header bits). >> >> I was delighted to find that cassandra memtables using skip lists are >> acyclic. But unfortunately I think the base nodes of the skip lists get too >> high in-degree and ultimately end up spoiling eager reclamation. >> >> So I’m wondering, how annoying would it be to change this map >> implementation to something that is both acyclic and has ref counts lower >> than 7 for any internal nodes? If that’s not too tricky to do, I suspect we >> could collapse the GC overheads related to having memtables in-heap. Maybe >> a tree structure or something? What do you guys think? If it’s too annoying >> that’s good feedback too. >> >> Please let me know if this is the wrong place for this kind of discussion. >> >> Thanks, >> /Erik > > > > -- > Dmitry Konstantinov > > >
