Could I ask how the high in-degree to slabs affects GC? The references to the slab are acyclic, and so I would expect the incoming references to be collected directly, eventually reducing the in-degree? I suppose the in-degree counter saturating once disables the feature for the object entirely?
One other significant problem with unslabbed buffers is that the _object_ overhead is higher; offheap_objects was introduced in large part to address this problem (there remains significant object overhead, but a small fraction). However, every mode besides unslabbed_buffers has a problem with overwrite workloads, since you cannot reclaim any released space until the memtable is flushed. For these workloads the object overhead might be a price worth paying if you can flush less often. On 2026/07/28 07:12:42 Branimir Lambov wrote: > Hi Erik, > > The slabs will always have a high in-degree because we allocate space for > data inside them. We generally don't use unslabbed because of the high GC > overhead; it would be interesting to see what kind of performance one gets > with unslabbed compared to heap buffers with this GC improvement and > whether it makes unslabbed a realistic option to use. > > The other way to avoid the high in-degree of slabs is to use the > offheap_objects option. This choice is used very often in production > systems. It takes the trie buffers and cell data off-heap, but still has > quite a bit of ordering structure on the heap. > > The above applies to all memtable implementations in Cassandra; they share > the same cell data management logic. > > I will take removing the cycle in ApplyState into account for the next > stages of trie memtable development. > > Regards, > Branimir > > On Mon, Jul 27, 2026 at 3:32 PM Erik Osterlund via dev < > [email protected]> wrote: > > > Hi Dmitry, > > > > Thanks for the reply and the pointers, it’s very helpful. As a GC > > developer, I naturally get interested when people are moving data off-heap > > because of GC issues. > > > > I tried out the Trie memtable data structure. Unfortunately, it looks like > > there is a cycle caused by InMemoryTrie.ApplyState which has an implicit > > this$0 field due to being an internal class. I tried patching it up a bit > > locally to pass that around as an argument instead. Still some slab > > allocator (not sure what it’s fore) seemingly causing in-degree > 6 which > > the algorithm is a bit sensitive to right now. But with the patched Trie > > and unslabbed_heap_buffers it works beautifully well and I see several GB > > reclaimed old gen memory in young collections, without having to trace the > > old generation. It’s CPU spending is now invariant of the old gen live set > > size, as long as we can stick to these rules I mentioned. > > > > This feels rather encouraging IMO. :-) > > > > Thanks, > > /Erik > > > > > > On 27 Jul 2026, at 12:09, Dmitry Konstantinov <[email protected]> wrote: > > > > This Message Is From a New Sender > > You have not previously corresponded with the sender of this message. > > Report Suspicious > > > > <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/ACWV5N9M2RV99hQ!Op20OCZG1EegmzBtO1N5l_0NlI-ToMvOCf6nTg_MDwHHjAYeHpOwVpF74U5Z9U9hvMImUXCmFpRqWmpISVsORhymuGGN-sm7Tr4G63oCvH3p-fuixd7qWTAOw5ah2Ug$> > > > > Hi Erik, > > Thanks for the email! We're always happy to welcome JVM developers here. > > We also have Trie memtable implementation for memtable <partition key > > ---> partition data> map - [1],[2],[5] > > It can keep some portion of data off-heap also but the underlying objects > > are still in heap: > > > > <image.png> > > > > There is an idea about extending Trie implementation and moving more parts > > of it to off-heap - [3]. > > There is also another idea for the map but it has not been implemented so > > far: [4] > > > > [1] > > https://cassandra.apache.org/_/blog/Apache-Cassandra-5.0-Features-Trie-Memtables-and-Trie-Indexed-SSTables.html > > <https://urldefense.com/v3/__https://cassandra.apache.org/_/blog/Apache-Cassandra-5.0-Features-Trie-Memtables-and-Trie-Indexed-SSTables.html__;!!ACWV5N9M2RV99hQ!PczTox7K8CxUiR-8i1oQAWeuiwNxISCGjq8moRxQRGYGjt7DVNe2Uk51DQscE8g13blXNeWAG6E6w58ED4ei$> > > > > [2] https://www.vldb.org/pvldb/vol15/p3359-lambov.pdf > > <https://urldefense.com/v3/__https://www.vldb.org/pvldb/vol15/p3359-lambov.pdf__;!!ACWV5N9M2RV99hQ!PczTox7K8CxUiR-8i1oQAWeuiwNxISCGjq8moRxQRGYGjt7DVNe2Uk51DQscE8g13blXNeWAG6E6wwydJXc_$> > > > > [3] > > https://cwiki.apache.org/confluence/spaces/CASSANDRA/pages/392038876/CEP-57+Flat+keys+and+trie+interfaces > > <https://urldefense.com/v3/__https://cwiki.apache.org/confluence/spaces/CASSANDRA/pages/392038876/CEP-57*Flat*keys*and*trie*interfaces__;KysrKys!!ACWV5N9M2RV99hQ!PczTox7K8CxUiR-8i1oQAWeuiwNxISCGjq8moRxQRGYGjt7DVNe2Uk51DQscE8g13blXNeWAG6E6w3z0dj6w$> > > > > [4] https://issues.apache.org/jira/browse/CASSANDRA-7282 > > <https://urldefense.com/v3/__https://issues.apache.org/jira/browse/CASSANDRA-7282__;!!ACWV5N9M2RV99hQ!PczTox7K8CxUiR-8i1oQAWeuiwNxISCGjq8moRxQRGYGjt7DVNe2Uk51DQscE8g13blXNeWAG6E6w33qzYRA$> > > [5] > > https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/memtable/TrieMemtable.java > > <https://urldefense.com/v3/__https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/memtable/TrieMemtable.java__;!!ACWV5N9M2RV99hQ!PczTox7K8CxUiR-8i1oQAWeuiwNxISCGjq8moRxQRGYGjt7DVNe2Uk51DQscE8g13blXNeWAG6E6w3Jt98gW$> > > > > Regards, > > Dmitry > > > > On Mon, 27 Jul 2026 at 10:45, Erik Osterlund via dev < > > [email protected]> wrote: > > > >> Hi, > >> > >> I’m one of the ZGC devs. Have been playing around a bit with an > >> interesting algorithmic extension of generational ZGC using a form of > >> reference counting in the old generation. This allows us to essentially > >> have young generation collections reclaim zero reference count old objects, > >> and free up acyclic garbage in the old generation as well, just before it > >> promotes objects from the young generation to the old generation. The > >> consequence is that as long as long-lived garbage is acyclic, we pretty > >> much don’t need to do major collections other than to defragment memory > >> every now and then. But the actual pressure to reclaim old garbage vanishes > >> and is handled from the young collections. > >> > >> One quirk is that in this scheme, we can reclaim objects that never have > >> an in-degree higher than 6, because then accounting isn’t free any longer > >> (we are using available object header bits). > >> > >> I was delighted to find that cassandra memtables using skip lists are > >> acyclic. But unfortunately I think the base nodes of the skip lists get too > >> high in-degree and ultimately end up spoiling eager reclamation. > >> > >> So I’m wondering, how annoying would it be to change this map > >> implementation to something that is both acyclic and has ref counts lower > >> than 7 for any internal nodes? If that’s not too tricky to do, I suspect we > >> could collapse the GC overheads related to having memtables in-heap. Maybe > >> a tree structure or something? What do you guys think? If it’s too annoying > >> that’s good feedback too. > >> > >> Please let me know if this is the wrong place for this kind of discussion. > >> > >> Thanks, > >> /Erik > > > > > > > > -- > > Dmitry Konstantinov > > > > > > >
