I forget a very important part. There is a memory allocator used by the process. And this memory allocator either makes use of the program break or the mmap for an anonymous memory mapping.
The ByteBuffer.allocateDirect forwards allocation requests to this memory allocator; it will not directly allocate memory by changing the program break or add an anonymous memory mapping region (so what I said before was incorrect). This memory allocator could also have a big impact on how the system deals with fragmentation. On Fri, Oct 2, 2020 at 8:11 AM alarm...@gmail.com <alarmnum...@gmail.com> wrote: > I'm not an expert, so take my answer with a few grains of salt. Also the > following applies to Linux; I don't know anything about other OSs. > > On the physical memory level, memory for the array doesn't need to be > contiguous. That is the whole point of having virtual memory in the first > place. > > On the virtual memory level, the memory for array needs to be contiguous. > So what is likely to happen is that an anonymous memory mapping for a > particular region is created. This memory mapping is initialized with the > zero page (has 0's as content), so all page table entries for this region > are pointing to the same chunk of physical memory. Only when there is a > write on this zero page, the copy on write mechanism kicks in and a page > frame (physical memory) is allocated and assigned to that page table entry. > > Afaik ByteBuffer.allocate direct just forwards the request to the OS to do > an anonymous memory mapping. And since the virtual address space is huge, > it should be easy to find a contiguous region of virtual memory for that > array. > > When you use new Object[...] you deal with the memory management from the > JVM and then it depends on the GC algorithm. For example with CMS, there > needs to be an entry in the free list large enough to hold that array (it > needs to be contiguous). With G1 AFAIK it is less of an issue; if it is a > large array, one or more non-contiguous humongous regions can be used. With > parallel/serial, it should also not be an issue since these compact the > memory and therefore there is no fragmentation. > > Which GC algorithm are you using? > > On Thursday, October 1, 2020 at 9:43:59 PM UTC+3 Shevek wrote: > >> When I do new byte[N], I get OutOfMemoryError, despite that the VM >> claims to have more than enough free space (according to MemoryMXBean, >> Runtime.freeMemory, visualvm, etc). >> >> My working assumption is that while I have enough free memory, I don't >> have enough contiguous free memory. Is there a solution to this? Will I >> get better results from any of: >> >> * ByteBuffer.allocateDirect() - presumably yes, but has other issues >> relating to overall memory usage on the system >> * G1GC (or other GC which allocates (relocatable?) regions) - this is a >> deep hole I haven't yet explored. >> * Calling System.gc() before allocating a contiguous region [apparently >> doesn't help]. >> * Other? >> >> If we do follow a strategy using allocateDirect, will we end up with the >> same fragmentation issue in the native heap, along with committed >> off-heap memory which we can no longer effectively use, or is the >> off-heap memory managed in some manner which avoids this problem? >> >> Thank you. >> >> S. >> > -- > You received this message because you are subscribed to the Google Groups > "mechanical-sympathy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to mechanical-sympathy+unsubscr...@googlegroups.com. > To view this discussion on the web, visit > https://groups.google.com/d/msgid/mechanical-sympathy/99daf015-ae67-4fb7-948f-cfb4a2cc00d6n%40googlegroups.com > <https://groups.google.com/d/msgid/mechanical-sympathy/99daf015-ae67-4fb7-948f-cfb4a2cc00d6n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group. To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscr...@googlegroups.com. To view this discussion on the web, visit https://groups.google.com/d/msgid/mechanical-sympathy/CAGuAWdDX8FGoLNUA2i0QezFqtZKSrYs_cLVEN4zQ9Cp%2BTo23wA%40mail.gmail.com.
