Has this problem been fixed in OpenJDK? https://bugs.openjdk.java.net/browse/JDK-8149596 description says that "JDK-8141491 moves the copy functionality from libjava.so/Bits.c into the VM and Unsafe. The wrapper methods are no longer needed and the nio code should be updated to call them directly instead." However I cannot find any code that looks like safepoint poll insertion or copy chunking neither in Unsafe.java (http://hg.openjdk.java.net/jdk-updates/jdk9u/jdk/file/d54486c189e5/src/java.base/share/classes/jdk/internal/misc/Unsafe.java), unsafe.cpp (http://hg.openjdk.java.net/jdk-updates/jdk9u/hotspot/file/22d7a88dbe78/src/share/vm/prims/unsafe.cpp), nor copy.cpp (http://hg.openjdk.java.net/jdk-updates/jdk9u/hotspot/file/22d7a88dbe78/src/share/vm/utilities/copy.cpp).
On Thursday, 25 July 2013 09:47:34 UTC+2, Peter Lawrey wrote: > > > This can have all sorts of BAD consequences including hitting the OOM > killer. > > You would have to be pretty close to the OOM killer's limit. I suspect > the assumption is that you don't creates lots of these and you are no where > near your process limits. > > BTW If you can see better ways of doing this, you really should submit > RFEs. ;) > > > On 25 July 2013 04:57, Kevin Burton <burto...@gmail.com <javascript:>> > wrote: > >> Actually... there's a bug in the JVM here. >> >> They allocate capacity + pageSize but only 'reserve' capacity. >> >> This means that one MAX_DIRECT_MEMORY isn't yielded correctly and you >> could actually allocated MORE direct memory than allowed. >> >> This can have all sorts of BAD consequences including hitting the OOM >> killer. >> >> I'm really surprised by the sever LACK of quality in a lot of this code. >> I think a lot of it doesn't see the light of day and when you dive into it >> you really start to see lots of breakage. >> >> I've found 2-3 pretty severe problems in this code. >> >> Makes me think I should look at the source of Unsafe too. >> >> >> On Wednesday, July 24, 2013 8:51:24 PM UTC-7, Kevin Burton wrote: >>> >>> Ah... Actually yes. Diving into it DirectByteBuffer does align some >>> cache alignment. >>> >>> Here's the code in question from JDK 1.7: >>> >>> https://gist.github.com/burtonator/6076790 >>> >>> Very interesting! >>> >>> On Tuesday, July 23, 2013 9:49:58 PM UTC-7, mikeb01 wrote: >>>> >>>> It's word aligned but not cache aligned, if some of your operations >>>> are straddling cache-lines you could be paying some extra cost there. >>>> >>>> Mike. >>>> >>>> On 24 July 2013 16:45, Kevin Burton <burto...@gmail.com> wrote: >>>> > I think I'm going to verify this just to be sure. >>>> > >>>> > But Unsafe's documentation says that it's aligned. >>>> > >>>> > Here's a good deep dive into this topic btw: >>>> > >>>> > http://psy-lob-saw.blogspot.com/2013/01/direct-memory- >>>> alignment-in-java.html >>>> > >>>> > On Tuesday, July 23, 2013 9:10:22 PM UTC-7, mikeb01 wrote: >>>> >> >>>> >> You may want to compare the alignment of the DirectByteBuffer and >>>> the >>>> >> memory allocated by Unsafe. >>>> >> >>>> >> Mike. >>>> >> >>>> >> On 24 July 2013 15:11, Kevin Burton <burto...@gmail.com> wrote: >>>> >> > I got sucked into writing my own implementation which its really >>>> JUST >>>> >> > Unsafe… but I'm trying to avoid the lock in Bits since I do lots >>>> of >>>> >> > small >>>> >> > allocations. >>>> >> > >>>> >> > What's interesting is that my benchmarks show that Unsafe is >>>> *slightly* >>>> >> > slower than a LITTLE_ENDIAN DirectByteBuffer. I'm not sure why… >>>> I'm >>>> >> > going >>>> >> > to work on some code to get the benchmark smaller to see what's >>>> up… it's >>>> >> > not >>>> >> > AMAZINGLY slower… maybe like 5%. I just expected Unsafe to be 20% >>>> or so >>>> >> > faster. >>>> >> > >>>> >> > Another benefit is that you can work with >2GB mmap files or >>>> memory >>>> >> > slabs. >>>> >> > And in my use case that's another perk! >>>> >> > >>>> >> > On Tuesday, July 23, 2013 7:53:12 PM UTC-7, Vladimir Rodionov >>>> wrote: >>>> >> >> >>>> >> >> I have never seen any real issue with Unsafe.copyMemory (except >>>> the >>>> >> >> fact >>>> >> >> that it is not supported in some OpenJDK versions). It seems that >>>> all >>>> >> >> Unsafe >>>> >> >> is rock solid. But we are still in development-testing phase of >>>> the >>>> >> >> product. >>>> >> >> >>>> >> >> Custom All Java Off Heap Memory Management is built entirely on >>>> top of >>>> >> >> Unsafe. Tested up to 240GB with 20M alloc/free per sec during 24 >>>> hours. >>>> >> >> >>>> >> >> --best >>>> >> >> >>>> >> >> On Tuesday, July 16, 2013 12:45:34 PM UTC-7, Kevin Burton wrote: >>>> >> >>> >>>> >> >>> So I'm reading the JVM documentation dealing with copying memory >>>> and >>>> >> >>> Unsafe when I came across this: >>>> >> >>> >>>> >> >>> // This number limits the number of bytes to copy per call >>>> to >>>> >> >>> Unsafe's >>>> >> >>> // copyMemory method. A limit is imposed to allow for >>>> safepoint >>>> >> >>> polling >>>> >> >>> // during a large copy >>>> >> >>> >>>> >> >>> ... in Bits.java >>>> >> >>> >>>> >> >>> Basically, it looks like the copyArray methods use >>>> Unsafe.copyMemory >>>> >> >>> which I assume internally is using memcpy. >>>> >> >>> >>>> >> >>> And I imagine the reason they are paging over 1MB chunks is so >>>> that >>>> >> >>> when >>>> >> >>> the GC kicks in they can lower the latency for that method to >>>> return. >>>> >> >>> >>>> >> >>> If you're copying 100MB you're going to have a GC hiccup if you >>>> run >>>> >> >>> out >>>> >> >>> of memory. Better to just pause during a large copy since >>>> you're >>>> >> >>> doing 1MB >>>> >> >>> at a time. >>>> >> >>> >>>> >> >>> Seems reasonable of course but more for me to implement and >>>> certainly >>>> >> >>> ANOTHER argument for not using Unsafe if you can avoid it. >>>> >> >>> >>>> >> >>> I'm just sick of having to dive into the GC internals to figure >>>> out >>>> >> >>> what >>>> >> >>> brain damage I have to route around and this way I can just do >>>> my own >>>> >> >>> memory >>>> >> >>> management. >>>> >> > >>>> >> > -- >>>> >> > You received this message because you are subscribed to the Google >>>> >> > Groups >>>> >> > "mechanical-sympathy" group. >>>> >> > To unsubscribe from this group and stop receiving emails from it, >>>> send >>>> >> > an >>>> >> > email to mechanical-sympathy+unsubscr...@googlegroups.com. >>>> >> > For more options, visit https://groups.google.com/groups/opt_out. >>>> >> > >>>> >> > >>>> > >>>> > -- >>>> > You received this message because you are subscribed to the Google >>>> Groups >>>> > "mechanical-sympathy" group. >>>> > To unsubscribe from this group and stop receiving emails from it, >>>> send an >>>> > email to mechanical-sympathy+unsubscr...@googlegroups.com. >>>> > For more options, visit https://groups.google.com/groups/opt_out. >>>> > >>>> > >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "mechanical-sympathy" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to mechanical-sympathy+unsubscr...@googlegroups.com <javascript:>. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > -- You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group. To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
