There are a handful of offheap uses - bloom filters (scale approximately linearly with data size based on your bloom filter false positive ratio) - compression metadata, scales linearly with data size based on your compression chunk size - Partition indexes - Netty For client / server requests. In 4.0 and higher, netty for internode - In 3.8? And higher the chunk pool, which allows things like sstablereader to pool direct memory - In 4.0 and higher, merkle trees during repair coordination
In my experience, bloom filters and compression chunks usually dominate the off heap usage - they can easily use a few gigs and they don’t drop in usage (though they can spike during compaction). The sstablereaders and netty tend to be much smaller contributors and clean up when theres lower traffic. > On Apr 17, 2020, at 11:06 PM, HImanshu Sharma <himanshusharma0...@gmail.com> > wrote: > > > From the codebase as much I understood, if once a buffer is being allocated, > then it is not freed and added to a recyclable pool. When a new request comes > effort is made to fetch memory from recyclable pool and if is not available > new allocation request is made. And while allocating a new request if memory > limit is breached then we get this oom error. > > I would like to know is my understanding correct > If what I am thinking is correct, is there way we can get this buffer pool > reduced when there is low traffic because what I have observed in my system > this memory remains static even if there is no traffic. > > Regards > Manish > >> On Sat, Apr 18, 2020 at 11:13 AM Erick Ramirez <erick.rami...@datastax.com> >> wrote: >> Like most things, it depends on (a) what you're allowing and (b) how much >> your nodes require. MaxDirectMemorySize is the upper-bound for off-heap >> memory used for the direct byte buffer. C* uses it for Netty so if your >> nodes are busy servicing requests, they'd have more IO threads consuming >> memory. >> >> During low traffic periods, there's less memory allocated to service >> requests and they eventually get freed up by GC tasks. But if traffic >> volumes are high, memory doesn't get freed up quick enough so the max is >> reached. When this happens, you'll see OOMs like "OutOfMemoryError: Direct >> buffer memory" show up in the logs. >> >> You can play around with different values but make sure you test it >> exhaustively before trying it out in production. Cheers! >> >> GOT QUESTIONS? Apache Cassandra experts from the community and DataStax have >> answers! Share your expertise on https://community.datastax.com/.
