Re: JVM crashed with memmap and unsafe operation for IPC

[Li Yunpeng]

Totally agree hopefully we have better solution on Java 9 regarding to
Usafe.


Thanks a lot
Yunpeng Li

On Feb 17, 2017 4:05 AM, "Gil Tene"  wrote:

>
>
> On Thursday, February 16, 2017 at 7:14:23 AM UTC-8, Yunpeng Li wrote:
>>
>> Thanks a lot. It works now. NPE always find a way to give some surprise
>> :P even with GC help.
>>
>
> It's hard to call this sort of crash "even with GC help". It's more like
> "when you insist on bypassing GC safety".
>
> You'll never (or should never) see a JVM crash with regular Java code.
> Seeing that would be an actual JVM bug.
>
> Unsafe is NOT regular Java code. And it's hard to label the implications
> of using it more clearly than it already is. When you insist on running
> with scissors and on bypassing the safety that the GC based system
> provides, a JVM crash can't be blamed on anyone other than the user of
> Unsafe. If you want to use Unsafe, it becomes *your* responsibility to take
> all impacts into account, including interactions with your own code AND any
> of the runtime components you may or may not be aware of. These include the
> GC, the JIT compilers (including interesting interactions with on-the-fly
> optimization and de-optimization), locking mechanisms, hashcode tracking
> mechanisms, class unloading mechanisms, finalization mechanisms, the
> potential for escape analysis not materializing objects in memory where and
> when you think, and many many other things that already exist if may pop up
> in the future. "Unsafe" means "you are responsible for understanding what
> this would do in a system full of (literally) moving targets".
>
>
>>
>> Thanks a lot
>> Yunpeng Li
>>
>> On Feb 16, 2017 1:04 PM, "Gil Tene"  wrote:
>>
>>> To prevent the unmapping, you can simply prevent
>>> the sun.nio.ch.DirectBuffer from going out of scope. Keep a static
>>> final sun.nio.ch.DirectBuffer around and assign it from the result
>>> of raf.getChannel().map(MapMode.READ_WRITE, 0, raf.length()), only only
>>> then get the address from it.
>>>
>>> That would work, technically. However, I feel obligated to point out the
>>> danger of assuming that the result of 
>>> raf.getChannel().map(MapMode.READ_WRITE,
>>> 0, raf.length()) actually implements sun.nio.ch.DirectBuffer. It is only
>>> known to be an instance of java.nio.MappedByteBuffer, and the fact that an
>>> implementation-specific version of it returns an internally implemented
>>> subclass of MappedByteBuffer that also implements sun.nio.ch.DirectBuffer
>>> is a happy (actually sad, if you think about it) accident. One that can
>>> change at any moment, and with any version of the JDK. In fact, I'd be
>>> surprised if this code would work on JDK9...
>>>
>>> On Wednesday, February 15, 2017 at 8:09:51 PM UTC-8, Yunpeng Li wrote:

 Thanks for point out the root cause, could you share what I'd the
 correct way to do it to make a pair of what you should and should not do
 samples :P

 Thanks a lot
 Yunpeng Li

 On Feb 15, 2017 11:35 PM, "Gil Tene"  wrote:

> This should crash. Every time. Reliably. It would be a JVM bug if it
> didn't.
>
> What your programs are doing is the logical equivalent to:
>
> T1:
>   long *address = mmap(/* the file */);
>   while (true) {
> long value = *address;
> count++;
>   }
>
> T2:
>   while (count < trigger) {
> sleep(1);
>   }
>   munmap(address...);
>
>
> Your pub and sub programs basically establish the unsafe addresses of
> temporarily mapped file regions that are dead immediately after being
> created. The programs will run long enough for the allocations caused by
> the printouts in your loops to trigger a couple of GCs, at which point the
> mapped file region gets cleaned up and properly unmapped. And the next
> access to those unmapped addresses crashes.
>
> On Wednesday, February 15, 2017 at 4:43:58 AM UTC-8, Yunpeng Li wrote:
>>
>> Hi there,
>>  I'm trying to copycat using memmap and unsafe to build an IPC
>> ping pong test. Attached the source codes and crash reports, 
>> unfortunately,
>> my code crashes JVM each time when ping pong reaches 73100(+- 10), Run
>> PublisherTest first then SubscriberTest in separated process. Sometimes 
>> one
>> process crashes sometime both crash at the sametime.
>>  Can someone help to check what's the reason it crashes at the
>> very place. I ran the test case inside eclipse and on Mint Linux 16(yeah
>> old version) and jvm 8.
>>  And one more question on how JVM work with OS on memmap swap,
>> e.g. in my case, the locks are more than 1 page away from data, I follow
>> the reserve-modify-publish pattern from disruptor, First updating data 
>> then
>> publish lock by unsafe putOrdered, How did jvm make sure the lock is
>> visible behind data from another 

Re: JVM crashed with memmap and unsafe operation for IPC

[Gil Tene]

On Thursday, February 16, 2017 at 7:14:23 AM UTC-8, Yunpeng Li wrote:
>
> Thanks a lot. It works now. NPE always find a way to give some surprise :P 
> even with GC help.
>

It's hard to call this sort of crash "even with GC help". It's more like 
"when you insist on bypassing GC safety".

You'll never (or should never) see a JVM crash with regular Java code. 
Seeing that would be an actual JVM bug.

Unsafe is NOT regular Java code. And it's hard to label the implications of 
using it more clearly than it already is. When you insist on running with 
scissors and on bypassing the safety that the GC based system provides, a 
JVM crash can't be blamed on anyone other than the user of Unsafe. If you 
want to use Unsafe, it becomes *your* responsibility to take all impacts 
into account, including interactions with your own code AND any of the 
runtime components you may or may not be aware of. These include the GC, 
the JIT compilers (including interesting interactions with on-the-fly 
optimization and de-optimization), locking mechanisms, hashcode tracking 
mechanisms, class unloading mechanisms, finalization mechanisms, the 
potential for escape analysis not materializing objects in memory where and 
when you think, and many many other things that already exist if may pop up 
in the future. "Unsafe" means "you are responsible for understanding what 
this would do in a system full of (literally) moving targets".
 

>
> Thanks a lot
> Yunpeng Li
>
> On Feb 16, 2017 1:04 PM, "Gil Tene"  wrote:
>
>> To prevent the unmapping, you can simply prevent 
>> the sun.nio.ch.DirectBuffer from going out of scope. Keep a static 
>> final sun.nio.ch.DirectBuffer around and assign it from the result 
>> of raf.getChannel().map(MapMode.READ_WRITE, 0, raf.length()), only only 
>> then get the address from it.
>>
>> That would work, technically. However, I feel obligated to point out the 
>> danger of assuming that the result 
>> of raf.getChannel().map(MapMode.READ_WRITE, 0, raf.length()) actually 
>> implements sun.nio.ch.DirectBuffer. It is only known to be an instance 
>> of java.nio.MappedByteBuffer, and the fact that an implementation-specific 
>> version of it returns an internally implemented subclass of 
>> MappedByteBuffer that also implements sun.nio.ch.DirectBuffer is a happy 
>> (actually sad, if you think about it) accident. One that can change at any 
>> moment, and with any version of the JDK. In fact, I'd be surprised if this 
>> code would work on JDK9...
>>
>> On Wednesday, February 15, 2017 at 8:09:51 PM UTC-8, Yunpeng Li wrote:
>>>
>>> Thanks for point out the root cause, could you share what I'd the 
>>> correct way to do it to make a pair of what you should and should not do 
>>> samples :P
>>>
>>> Thanks a lot
>>> Yunpeng Li
>>>
>>> On Feb 15, 2017 11:35 PM, "Gil Tene"  wrote:
>>>
 This should crash. Every time. Reliably. It would be a JVM bug if it 
 didn't.

 What your programs are doing is the logical equivalent to:

 T1:
   long *address = mmap(/* the file */);
   while (true) {
 long value = *address;
 count++;
   }

 T2:
   while (count < trigger) {
 sleep(1);
   }
   munmap(address...);


 Your pub and sub programs basically establish the unsafe addresses of 
 temporarily mapped file regions that are dead immediately after being 
 created. The programs will run long enough for the allocations caused by 
 the printouts in your loops to trigger a couple of GCs, at which point the 
 mapped file region gets cleaned up and properly unmapped. And the next 
 access to those unmapped addresses crashes.

 On Wednesday, February 15, 2017 at 4:43:58 AM UTC-8, Yunpeng Li wrote:
>
> Hi there,
>  I'm trying to copycat using memmap and unsafe to build an IPC 
> ping pong test. Attached the source codes and crash reports, 
> unfortunately, 
> my code crashes JVM each time when ping pong reaches 73100(+- 10), Run 
> PublisherTest first then SubscriberTest in separated process. Sometimes 
> one 
> process crashes sometime both crash at the sametime.
>  Can someone help to check what's the reason it crashes at the 
> very place. I ran the test case inside eclipse and on Mint Linux 16(yeah 
> old version) and jvm 8.
>  And one more question on how JVM work with OS on memmap swap, 
> e.g. in my case, the locks are more than 1 page away from data, I follow 
> the reserve-modify-publish pattern from disruptor, First updating data 
> then 
> publish lock by unsafe putOrdered, How did jvm make sure the lock is 
> visible behind data from another process, Given update dirty page is 
> "random" by OS.
>
> Thanks in advance.
> Yunpeng Li
>
 -- 
 You received this message because you are subscribed to a topic in the 
 Google Groups "mechanical-sympathy" group.

Re: JVM crashed with memmap and unsafe operation for IPC

[Li Yunpeng]

Thanks a lot. It works now. NPE always find a way to give some surprise :P
even with GC help.

Thanks a lot
Yunpeng Li

On Feb 16, 2017 1:04 PM, "Gil Tene"  wrote:

> To prevent the unmapping, you can simply prevent
> the sun.nio.ch.DirectBuffer from going out of scope. Keep a static
> final sun.nio.ch.DirectBuffer around and assign it from the result
> of raf.getChannel().map(MapMode.READ_WRITE, 0, raf.length()), only only
> then get the address from it.
>
> That would work, technically. However, I feel obligated to point out the
> danger of assuming that the result of raf.getChannel().map(MapMode.READ_WRITE,
> 0, raf.length()) actually implements sun.nio.ch.DirectBuffer. It is only
> known to be an instance of java.nio.MappedByteBuffer, and the fact that an
> implementation-specific version of it returns an internally implemented
> subclass of MappedByteBuffer that also implements sun.nio.ch.DirectBuffer
> is a happy (actually sad, if you think about it) accident. One that can
> change at any moment, and with any version of the JDK. In fact, I'd be
> surprised if this code would work on JDK9...
>
> On Wednesday, February 15, 2017 at 8:09:51 PM UTC-8, Yunpeng Li wrote:
>>
>> Thanks for point out the root cause, could you share what I'd the correct
>> way to do it to make a pair of what you should and should not do samples :P
>>
>> Thanks a lot
>> Yunpeng Li
>>
>> On Feb 15, 2017 11:35 PM, "Gil Tene"  wrote:
>>
>>> This should crash. Every time. Reliably. It would be a JVM bug if it
>>> didn't.
>>>
>>> What your programs are doing is the logical equivalent to:
>>>
>>> T1:
>>>   long *address = mmap(/* the file */);
>>>   while (true) {
>>> long value = *address;
>>> count++;
>>>   }
>>>
>>> T2:
>>>   while (count < trigger) {
>>> sleep(1);
>>>   }
>>>   munmap(address...);
>>>
>>>
>>> Your pub and sub programs basically establish the unsafe addresses of
>>> temporarily mapped file regions that are dead immediately after being
>>> created. The programs will run long enough for the allocations caused by
>>> the printouts in your loops to trigger a couple of GCs, at which point the
>>> mapped file region gets cleaned up and properly unmapped. And the next
>>> access to those unmapped addresses crashes.
>>>
>>> On Wednesday, February 15, 2017 at 4:43:58 AM UTC-8, Yunpeng Li wrote:

 Hi there,
  I'm trying to copycat using memmap and unsafe to build an IPC ping
 pong test. Attached the source codes and crash reports, unfortunately, my
 code crashes JVM each time when ping pong reaches 73100(+- 10), Run
 PublisherTest first then SubscriberTest in separated process. Sometimes one
 process crashes sometime both crash at the sametime.
  Can someone help to check what's the reason it crashes at the very
 place. I ran the test case inside eclipse and on Mint Linux 16(yeah old
 version) and jvm 8.
  And one more question on how JVM work with OS on memmap swap, e.g.
 in my case, the locks are more than 1 page away from data, I follow the
 reserve-modify-publish pattern from disruptor, First updating data then
 publish lock by unsafe putOrdered, How did jvm make sure the lock is
 visible behind data from another process, Given update dirty page is
 "random" by OS.

 Thanks in advance.
 Yunpeng Li

>>> --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "mechanical-sympathy" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/to
>>> pic/mechanical-sympathy/UCVLCHt_MIw/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> mechanical-sympathy+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
> You received this message because you are subscribed to a topic in the
> Google Groups "mechanical-sympathy" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/mechanical-sympathy/UCVLCHt_MIw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> mechanical-sympathy+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mechanical-sympathy+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: JVM crashed with memmap and unsafe operation for IPC

[Gil Tene]

This should crash. Every time. Reliably. It would be a JVM bug if it didn't.

What your programs are doing is the logical equivalent to:

T1:
  long *address = mmap(/* the file */);
  while (true) {
long value = *address;
count++;
  }

T2:
  while (count < trigger) {
sleep(1);
  }
  munmap(address...);


Your pub and sub programs basically establish the unsafe addresses of 
temporarily mapped file regions that are dead immediately after being 
created. The programs will run long enough for the allocations caused by 
the printouts in your loops to trigger a couple of GCs, at which point the 
mapped file region gets cleaned up and properly unmapped. And the next 
access to those unmapped addresses crashes.

On Wednesday, February 15, 2017 at 4:43:58 AM UTC-8, Yunpeng Li wrote:
>
> Hi there,
>  I'm trying to copycat using memmap and unsafe to build an IPC ping 
> pong test. Attached the source codes and crash reports, unfortunately, my 
> code crashes JVM each time when ping pong reaches 73100(+- 10), Run 
> PublisherTest first then SubscriberTest in separated process. Sometimes one 
> process crashes sometime both crash at the sametime.
>  Can someone help to check what's the reason it crashes at the very 
> place. I ran the test case inside eclipse and on Mint Linux 16(yeah old 
> version) and jvm 8.
>  And one more question on how JVM work with OS on memmap swap, e.g. in 
> my case, the locks are more than 1 page away from data, I follow the 
> reserve-modify-publish pattern from disruptor, First updating data then 
> publish lock by unsafe putOrdered, How did jvm make sure the lock is 
> visible behind data from another process, Given update dirty page is 
> "random" by OS.
>
> Thanks in advance.
> Yunpeng Li
>

-- 
You received this message because you are subscribed to the Google Groups 
"mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mechanical-sympathy+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: JVM crashed with memmap and unsafe operation for IPC

[Martin Thompson]

I might be related to the following bug that was discussed on another 
thread.

https://bugs.openjdk.java.net/browse/JDK-8087134

On Wednesday, 15 February 2017 12:43:58 UTC, Yunpeng Li wrote:
>
> Hi there,
>  I'm trying to copycat using memmap and unsafe to build an IPC ping 
> pong test. Attached the source codes and crash reports, unfortunately, my 
> code crashes JVM each time when ping pong reaches 73100(+- 10), Run 
> PublisherTest first then SubscriberTest in separated process. Sometimes one 
> process crashes sometime both crash at the sametime.
>  Can someone help to check what's the reason it crashes at the very 
> place. I ran the test case inside eclipse and on Mint Linux 16(yeah old 
> version) and jvm 8.
>  And one more question on how JVM work with OS on memmap swap, e.g. in 
> my case, the locks are more than 1 page away from data, I follow the 
> reserve-modify-publish pattern from disruptor, First updating data then 
> publish lock by unsafe putOrdered, How did jvm make sure the lock is 
> visible behind data from another process, Given update dirty page is 
> "random" by OS.
>
> Thanks in advance.
> Yunpeng Li
>

-- 
You received this message because you are subscribed to the Google Groups 
"mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mechanical-sympathy+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.