Re: JVM crashed with memmap and unsafe operation for IPC

[Gil Tene]

To prevent the unmapping, you can simply prevent 
the sun.nio.ch.DirectBuffer from going out of scope. Keep a static 
final sun.nio.ch.DirectBuffer around and assign it from the result 
of raf.getChannel().map(MapMode.READ_WRITE, 0, raf.length()), only only 
then get the address from it.

That would work, technically. However, I feel obligated to point out the 
danger of assuming that the result 
of raf.getChannel().map(MapMode.READ_WRITE, 0, raf.length()) actually 
implements sun.nio.ch.DirectBuffer. It is only known to be an instance 
of java.nio.MappedByteBuffer, and the fact that an implementation-specific 
version of it returns an internally implemented subclass of 
MappedByteBuffer that also implements sun.nio.ch.DirectBuffer is a happy 
(actually sad, if you think about it) accident. One that can change at any 
moment, and with any version of the JDK. In fact, I'd be surprised if this 
code would work on JDK9...

On Wednesday, February 15, 2017 at 8:09:51 PM UTC-8, Yunpeng Li wrote:
>
> Thanks for point out the root cause, could you share what I'd the correct 
> way to do it to make a pair of what you should and should not do samples :P
>
> Thanks a lot
> Yunpeng Li
>
> On Feb 15, 2017 11:35 PM, "Gil Tene" > wrote:
>
>> This should crash. Every time. Reliably. It would be a JVM bug if it 
>> didn't.
>>
>> What your programs are doing is the logical equivalent to:
>>
>> T1:
>>   long *address = mmap(/* the file */);
>>   while (true) {
>> long value = *address;
>> count++;
>>   }
>>
>> T2:
>>   while (count < trigger) {
>> sleep(1);
>>   }
>>   munmap(address...);
>>
>>
>> Your pub and sub programs basically establish the unsafe addresses of 
>> temporarily mapped file regions that are dead immediately after being 
>> created. The programs will run long enough for the allocations caused by 
>> the printouts in your loops to trigger a couple of GCs, at which point the 
>> mapped file region gets cleaned up and properly unmapped. And the next 
>> access to those unmapped addresses crashes.
>>
>> On Wednesday, February 15, 2017 at 4:43:58 AM UTC-8, Yunpeng Li wrote:
>>>
>>> Hi there,
>>>  I'm trying to copycat using memmap and unsafe to build an IPC ping 
>>> pong test. Attached the source codes and crash reports, unfortunately, my 
>>> code crashes JVM each time when ping pong reaches 73100(+- 10), Run 
>>> PublisherTest first then SubscriberTest in separated process. Sometimes one 
>>> process crashes sometime both crash at the sametime.
>>>  Can someone help to check what's the reason it crashes at the very 
>>> place. I ran the test case inside eclipse and on Mint Linux 16(yeah old 
>>> version) and jvm 8.
>>>  And one more question on how JVM work with OS on memmap swap, e.g. 
>>> in my case, the locks are more than 1 page away from data, I follow the 
>>> reserve-modify-publish pattern from disruptor, First updating data then 
>>> publish lock by unsafe putOrdered, How did jvm make sure the lock is 
>>> visible behind data from another process, Given update dirty page is 
>>> "random" by OS.
>>>
>>> Thanks in advance.
>>> Yunpeng Li
>>>
>> -- 
>> You received this message because you are subscribed to a topic in the 
>> Google Groups "mechanical-sympathy" group.
>> To unsubscribe from this topic, visit 
>> https://groups.google.com/d/topic/mechanical-sympathy/UCVLCHt_MIw/unsubscribe
>> .
>> To unsubscribe from this group and all its topics, send an email to 
>> mechanical-sympathy+unsubscr...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mechanical-sympathy+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: JVM crashed with memmap and unsafe operation for IPC

[Li Yunpeng]

Thanks for point out the root cause, could you share what I'd the correct
way to do it to make a pair of what you should and should not do samples :P

Thanks a lot
Yunpeng Li

On Feb 15, 2017 11:35 PM, "Gil Tene"  wrote:

> This should crash. Every time. Reliably. It would be a JVM bug if it
> didn't.
>
> What your programs are doing is the logical equivalent to:
>
> T1:
>   long *address = mmap(/* the file */);
>   while (true) {
> long value = *address;
> count++;
>   }
>
> T2:
>   while (count < trigger) {
> sleep(1);
>   }
>   munmap(address...);
>
>
> Your pub and sub programs basically establish the unsafe addresses of
> temporarily mapped file regions that are dead immediately after being
> created. The programs will run long enough for the allocations caused by
> the printouts in your loops to trigger a couple of GCs, at which point the
> mapped file region gets cleaned up and properly unmapped. And the next
> access to those unmapped addresses crashes.
>
> On Wednesday, February 15, 2017 at 4:43:58 AM UTC-8, Yunpeng Li wrote:
>>
>> Hi there,
>>  I'm trying to copycat using memmap and unsafe to build an IPC ping
>> pong test. Attached the source codes and crash reports, unfortunately, my
>> code crashes JVM each time when ping pong reaches 73100(+- 10), Run
>> PublisherTest first then SubscriberTest in separated process. Sometimes one
>> process crashes sometime both crash at the sametime.
>>  Can someone help to check what's the reason it crashes at the very
>> place. I ran the test case inside eclipse and on Mint Linux 16(yeah old
>> version) and jvm 8.
>>  And one more question on how JVM work with OS on memmap swap, e.g.
>> in my case, the locks are more than 1 page away from data, I follow the
>> reserve-modify-publish pattern from disruptor, First updating data then
>> publish lock by unsafe putOrdered, How did jvm make sure the lock is
>> visible behind data from another process, Given update dirty page is
>> "random" by OS.
>>
>> Thanks in advance.
>> Yunpeng Li
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "mechanical-sympathy" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/mechanical-sympathy/UCVLCHt_MIw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> mechanical-sympathy+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mechanical-sympathy+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Linux futex_wait() bug... [Yes. You read that right. UPDATE to LATEST PATCHES NOW].

[Longchao Dong]

In fact, I met a very strange problem.
My c ++ program now calls HDFS‘s interfaces via jni, but they are all
blocked by  the same java object lock. I obtained the state of the process
by jstack. All threads are waiting to lock the object(0x0006b30b3be8),
but no thread is holding it. Does anybody have clues?
The attachment is the output of jstack at that time.

On Wed, Feb 15, 2017 at 11:45 PM, Gil Tene  wrote:

> Don't know if this is the same bug. RHEL 7 kernel included fixes for this
> since some time in 2015.
>
> While one of my first courses of action when I see a suspicious FUTEX_WAIT
> hang situation is still to check kernel versions to rules this out (since
> this bug has wasted us a bunch of time in the past), keep in mind that not
> all things stuck in FUTEX_WAIT are futex_wait kernel bugs. The most likely
> explanations are usually actual application logic bugs involving actual
> deadlock or starvation.
>
> Does attaching and detaching from the process with gdb move it forward?
> [the original bug was missing the wakeup, and an attach/detach would "kick"
> the futex out of its slumber once]
>
> On Wednesday, February 15, 2017 at 6:33:45 AM UTC-8, Will Foster wrote:
>>
>>
>>
>> On Tuesday, February 14, 2017 at 4:01:52 PM UTC, Allen Reese wrote:
>>>
>>> This bug report seems to have a way to reproduce it:
>>> https://bugs.centos.org/view.php?id=8371
>>>
>>> Hope that helps.
>>>
>>> --Allen Reese
>>>
>>>
>>
>> I also see this on latest CentOS7.3 with Logstash, I've disabled huge
>> pages via
>> transparent_hugepage=never
>>
>> in grub.
>>
>> Here's what I get from strace against logstash (never fully comes up to
>> listen on TCP/5044)
>>
>> [root@host-01 ~]# strace -p 1292
>> Process 1292 attached
>> futex(0x7f80eff8a9d0, FUTEX_WAIT, 1312, NULL
>>
>>
>> I am hitting this issue on Logstash 5.2.1-1 while trying to upgrade my 
>> Ansible
>> playbooks  to the
>> latest ES versions.
>>
>>
>>
>>>
>>> --
>>> *From:* Longchao Dong 
>>> *To:* mechanical-sympathy 
>>> *Sent:* Monday, February 13, 2017 1:55 AM
>>> *Subject:* Re: Linux futex_wait() bug... [Yes. You read that right.
>>> UPDATE to LATEST PATCHES NOW].
>>>
>>> How to reproduce this issue ? Is it possible to show us the method ? I
>>> am also working on one strange pthread_cond_wait issue, but not sure
>>> if that one is related with this issue.
>>>
>>> On Wednesday, May 20, 2015 at 8:16:12 AM UTC+8, manis...@gmail.com
>>> wrote:
>>>
>>> I bumped on this error couple of months back when using CentOS 6.6 with
>>> 32 cores Dell server. After many days of debugging, I realized it to be a
>>> CentOS 6.6 bug and moved back to 6.5 and since then no such issues have
>>> been seen.
>>> I am able to reproduce this issue in 15 minutes of heavy load on my
>>> multi threaded c  code.
>>>
>>> On Wednesday, May 13, 2015 at 3:37:32 PM UTC-7, Gil Tene wrote:
>>>
>>> We had this one bite us hard and scare the %$^! out of us, so I figured
>>> I'd share the fear...
>>>
>>> The linux futex_wait call has been broken for about a year (in upstream
>>> since 3.14, around Jan 2014), and has just recently been fixed (in upstream
>>> 3.18, around October 2014). More importantly this breakage seems to have
>>> been back ported into major distros (e.g. into RHEL 6.6 and its cousins,
>>> released in October 2014), and the fix for it has only recently been back
>>> ported (e.g. RHEL 6.6.z and cousins have the fix).
>>>
>>> The impact of this kernel bug is very simple: user processes can
>>> deadlock and hang in seemingly impossible situations. A futex wait call
>>> (and anything using a futex wait) can stay blocked forever, even though it
>>> had been properly woken up by someone. Thread.park() in Java may stay
>>> parked. Etc. If you are lucky you may also find soft lockup messages in
>>> your dmesg logs. If you are not that lucky (like us, for example), you'll
>>> spend a couple of months of someone's time trying to find the fault in your
>>> code, when there is nothing there to find.
>>>
>>> This behavior seems to regularly appear in the wild on Haswell servers
>>> (all the machines where we have had customers hit it in the field and in
>>> labs been Haswells), and since Haswell servers are basically what you get
>>> if you buy a new machine now, or run on the cool new amazon EC2/GCE/Azure
>>> stuff, you are bound to experience some interesting behavior. I don't know
>>> of anyone that will see this as a good thing for production systems. Except
>>> for maybe Netflix (maybe we should call this the linux fumonkey).
>>>
>>> The commit for the *fix* is here:  https://github.com/torvalds/
>>> linux/commit/ 76835b0ebf8a7fe85beb03c7512141 9a7dec52f0
>>> 
>>>
>>> The commit explanation says that it fixes https://github.com/torvalds/
>>> linux/commit/ b0c29f79ecea0b6fbcefc999e70f28 43ae8306db
>>> 

Re: Linux futex_wait() bug... [Yes. You read that right. UPDATE to LATEST PATCHES NOW].

[Wojciech Kudla]

Just trying to eliminate the obvious. You should be stracing JVM threads by
referring their tids rather than parent process pid. That guy will pretty
much always show being blocked on a futex.

On Wed, 15 Feb 2017, 15:45 Gil Tene,  wrote:

> Don't know if this is the same bug. RHEL 7 kernel included fixes for this
> since some time in 2015.
>
> While one of my first courses of action when I see a suspicious FUTEX_WAIT
> hang situation is still to check kernel versions to rules this out (since
> this bug has wasted us a bunch of time in the past), keep in mind that not
> all things stuck in FUTEX_WAIT are futex_wait kernel bugs. The most likely
> explanations are usually actual application logic bugs involving actual
> deadlock or starvation.
>
> Does attaching and detaching from the process with gdb move it forward?
> [the original bug was missing the wakeup, and an attach/detach would "kick"
> the futex out of its slumber once]
>
> On Wednesday, February 15, 2017 at 6:33:45 AM UTC-8, Will Foster wrote:
>
>
>
> On Tuesday, February 14, 2017 at 4:01:52 PM UTC, Allen Reese wrote:
>
> This bug report seems to have a way to reproduce it:
> https://bugs.centos.org/view.php?id=8371
>
> Hope that helps.
>
> --Allen Reese
>
>
>
> I also see this on latest CentOS7.3 with Logstash, I've disabled huge
> pages via
> transparent_hugepage=never
>
> in grub.
>
> Here's what I get from strace against logstash (never fully comes up to
> listen on TCP/5044)
>
> [root@host-01 ~]# strace -p 1292
> Process 1292 attached
> futex(0x7f80eff8a9d0, FUTEX_WAIT, 1312, NULL
>
>
> I am hitting this issue on Logstash 5.2.1-1 while trying to upgrade my Ansible
> playbooks  to the
> latest ES versions.
>
>
>
>
> --
> *From:* Longchao Dong 
> *To:* mechanical-sympathy 
> *Sent:* Monday, February 13, 2017 1:55 AM
> *Subject:* Re: Linux futex_wait() bug... [Yes. You read that right.
> UPDATE to LATEST PATCHES NOW].
>
> How to reproduce this issue ? Is it possible to show us the method ? I am
> also working on one strange pthread_cond_wait issue, but not sure if that
> one is related with this issue.
>
> On Wednesday, May 20, 2015 at 8:16:12 AM UTC+8, manis...@gmail.com wrote:
>
> I bumped on this error couple of months back when using CentOS 6.6 with 32
> cores Dell server. After many days of debugging, I realized it to be a
> CentOS 6.6 bug and moved back to 6.5 and since then no such issues have
> been seen.
> I am able to reproduce this issue in 15 minutes of heavy load on my multi
> threaded c  code.
>
> On Wednesday, May 13, 2015 at 3:37:32 PM UTC-7, Gil Tene wrote:
>
> We had this one bite us hard and scare the %$^! out of us, so I figured
> I'd share the fear...
>
> The linux futex_wait call has been broken for about a year (in upstream
> since 3.14, around Jan 2014), and has just recently been fixed (in upstream
> 3.18, around October 2014). More importantly this breakage seems to have
> been back ported into major distros (e.g. into RHEL 6.6 and its cousins,
> released in October 2014), and the fix for it has only recently been back
> ported (e.g. RHEL 6.6.z and cousins have the fix).
>
> The impact of this kernel bug is very simple: user processes can deadlock
> and hang in seemingly impossible situations. A futex wait call (and
> anything using a futex wait) can stay blocked forever, even though it had
> been properly woken up by someone. Thread.park() in Java may stay parked.
> Etc. If you are lucky you may also find soft lockup messages in your dmesg
> logs. If you are not that lucky (like us, for example), you'll spend a
> couple of months of someone's time trying to find the fault in your code,
> when there is nothing there to find.
>
> This behavior seems to regularly appear in the wild on Haswell servers
> (all the machines where we have had customers hit it in the field and in
> labs been Haswells), and since Haswell servers are basically what you get
> if you buy a new machine now, or run on the cool new amazon EC2/GCE/Azure
> stuff, you are bound to experience some interesting behavior. I don't know
> of anyone that will see this as a good thing for production systems. Except
> for maybe Netflix (maybe we should call this the linux fumonkey).
>
> The commit for the *fix* is here:  https://github.com/torvalds/
> linux/commit/ 76835b0ebf8a7fe85beb03c7512141 9a7dec52f0
> 
>
> The commit explanation says that it fixes https://github.com/torvalds/
> linux/commit/ b0c29f79ecea0b6fbcefc999e70f28 43ae8306db
> 
> (presumably the bug introduced with that change), which was made in Jan of
> 2014into 3.14. That 3.14 code added logic to avoid taking a lock if the
> code knows that there are no waiters. It documents (pretty elaborately) how
> "…thus preventing tasks sleeping forever if wakers don

Re: Linux futex_wait() bug... [Yes. You read that right. UPDATE to LATEST PATCHES NOW].

[Gil Tene]

Don't know if this is the same bug. RHEL 7 kernel included fixes for this 
since some time in 2015.

While one of my first courses of action when I see a suspicious FUTEX_WAIT 
hang situation is still to check kernel versions to rules this out (since 
this bug has wasted us a bunch of time in the past), keep in mind that not 
all things stuck in FUTEX_WAIT are futex_wait kernel bugs. The most likely 
explanations are usually actual application logic bugs involving actual 
deadlock or starvation.

Does attaching and detaching from the process with gdb move it forward? 
[the original bug was missing the wakeup, and an attach/detach would "kick" 
the futex out of its slumber once]

On Wednesday, February 15, 2017 at 6:33:45 AM UTC-8, Will Foster wrote:
>
>
>
> On Tuesday, February 14, 2017 at 4:01:52 PM UTC, Allen Reese wrote:
>>
>> This bug report seems to have a way to reproduce it:
>> https://bugs.centos.org/view.php?id=8371
>>
>> Hope that helps.
>>
>> --Allen Reese
>>
>>
>
> I also see this on latest CentOS7.3 with Logstash, I've disabled huge 
> pages via 
> transparent_hugepage=never
>
> in grub.
>
> Here's what I get from strace against logstash (never fully comes up to 
> listen on TCP/5044)
>
> [root@host-01 ~]# strace -p 1292
> Process 1292 attached
> futex(0x7f80eff8a9d0, FUTEX_WAIT, 1312, NULL
>
>
> I am hitting this issue on Logstash 5.2.1-1 while trying to upgrade my 
> Ansible 
> playbooks  to the 
> latest ES versions.
>
>  
>
>>
>> --
>> *From:* Longchao Dong 
>> *To:* mechanical-sympathy  
>> *Sent:* Monday, February 13, 2017 1:55 AM
>> *Subject:* Re: Linux futex_wait() bug... [Yes. You read that right. 
>> UPDATE to LATEST PATCHES NOW].
>>
>> How to reproduce this issue ? Is it possible to show us the method ? I am 
>> also working on one strange pthread_cond_wait issue, but not sure if that 
>> one is related with this issue.
>>
>> On Wednesday, May 20, 2015 at 8:16:12 AM UTC+8, manis...@gmail.com wrote:
>>
>> I bumped on this error couple of months back when using CentOS 6.6 with 
>> 32 cores Dell server. After many days of debugging, I realized it to be a 
>> CentOS 6.6 bug and moved back to 6.5 and since then no such issues have 
>> been seen.
>> I am able to reproduce this issue in 15 minutes of heavy load on my multi 
>> threaded c  code.
>>
>> On Wednesday, May 13, 2015 at 3:37:32 PM UTC-7, Gil Tene wrote:
>>
>> We had this one bite us hard and scare the %$^! out of us, so I figured 
>> I'd share the fear...
>>
>> The linux futex_wait call has been broken for about a year (in upstream 
>> since 3.14, around Jan 2014), and has just recently been fixed (in upstream 
>> 3.18, around October 2014). More importantly this breakage seems to have 
>> been back ported into major distros (e.g. into RHEL 6.6 and its cousins, 
>> released in October 2014), and the fix for it has only recently been back 
>> ported (e.g. RHEL 6.6.z and cousins have the fix).
>>
>> The impact of this kernel bug is very simple: user processes can deadlock 
>> and hang in seemingly impossible situations. A futex wait call (and 
>> anything using a futex wait) can stay blocked forever, even though it had 
>> been properly woken up by someone. Thread.park() in Java may stay parked. 
>> Etc. If you are lucky you may also find soft lockup messages in your dmesg 
>> logs. If you are not that lucky (like us, for example), you'll spend a 
>> couple of months of someone's time trying to find the fault in your code, 
>> when there is nothing there to find. 
>>
>> This behavior seems to regularly appear in the wild on Haswell servers 
>> (all the machines where we have had customers hit it in the field and in 
>> labs been Haswells), and since Haswell servers are basically what you get 
>> if you buy a new machine now, or run on the cool new amazon EC2/GCE/Azure 
>> stuff, you are bound to experience some interesting behavior. I don't know 
>> of anyone that will see this as a good thing for production systems. Except 
>> for maybe Netflix (maybe we should call this the linux fumonkey).
>>
>> The commit for the *fix* is here:  https://github.com/torvalds/ 
>> linux/commit/ 76835b0ebf8a7fe85beb03c7512141 9a7dec52f0 
>> 
>>
>> The commit explanation says that it fixes https://github.com/torvalds/ 
>> linux/commit/ b0c29f79ecea0b6fbcefc999e70f28 43ae8306db 
>> 
>>  
>> (presumably the bug introduced with that change), which was made in Jan of 
>> 2014into 3.14. That 3.14 code added logic to avoid taking a lock if the 
>> code knows that there are no waiters. It documents (pretty elaborately) how 
>> "…thus preventing tasks sleeping forever if wakers don't acknowledge all 
>> possible waiters" with logic that explains how memory barriers guarantee 
>> the correct order (see paragraph at lin

Re: JVM crashed with memmap and unsafe operation for IPC

[Gil Tene]

This should crash. Every time. Reliably. It would be a JVM bug if it didn't.

What your programs are doing is the logical equivalent to:

T1:
  long *address = mmap(/* the file */);
  while (true) {
long value = *address;
count++;
  }

T2:
  while (count < trigger) {
sleep(1);
  }
  munmap(address...);


Your pub and sub programs basically establish the unsafe addresses of 
temporarily mapped file regions that are dead immediately after being 
created. The programs will run long enough for the allocations caused by 
the printouts in your loops to trigger a couple of GCs, at which point the 
mapped file region gets cleaned up and properly unmapped. And the next 
access to those unmapped addresses crashes.

On Wednesday, February 15, 2017 at 4:43:58 AM UTC-8, Yunpeng Li wrote:
>
> Hi there,
>  I'm trying to copycat using memmap and unsafe to build an IPC ping 
> pong test. Attached the source codes and crash reports, unfortunately, my 
> code crashes JVM each time when ping pong reaches 73100(+- 10), Run 
> PublisherTest first then SubscriberTest in separated process. Sometimes one 
> process crashes sometime both crash at the sametime.
>  Can someone help to check what's the reason it crashes at the very 
> place. I ran the test case inside eclipse and on Mint Linux 16(yeah old 
> version) and jvm 8.
>  And one more question on how JVM work with OS on memmap swap, e.g. in 
> my case, the locks are more than 1 page away from data, I follow the 
> reserve-modify-publish pattern from disruptor, First updating data then 
> publish lock by unsafe putOrdered, How did jvm make sure the lock is 
> visible behind data from another process, Given update dirty page is 
> "random" by OS.
>
> Thanks in advance.
> Yunpeng Li
>

-- 
You received this message because you are subscribed to the Google Groups 
"mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mechanical-sympathy+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: JVM crashed with memmap and unsafe operation for IPC

[Martin Thompson]

I might be related to the following bug that was discussed on another 
thread.

https://bugs.openjdk.java.net/browse/JDK-8087134

On Wednesday, 15 February 2017 12:43:58 UTC, Yunpeng Li wrote:
>
> Hi there,
>  I'm trying to copycat using memmap and unsafe to build an IPC ping 
> pong test. Attached the source codes and crash reports, unfortunately, my 
> code crashes JVM each time when ping pong reaches 73100(+- 10), Run 
> PublisherTest first then SubscriberTest in separated process. Sometimes one 
> process crashes sometime both crash at the sametime.
>  Can someone help to check what's the reason it crashes at the very 
> place. I ran the test case inside eclipse and on Mint Linux 16(yeah old 
> version) and jvm 8.
>  And one more question on how JVM work with OS on memmap swap, e.g. in 
> my case, the locks are more than 1 page away from data, I follow the 
> reserve-modify-publish pattern from disruptor, First updating data then 
> publish lock by unsafe putOrdered, How did jvm make sure the lock is 
> visible behind data from another process, Given update dirty page is 
> "random" by OS.
>
> Thanks in advance.
> Yunpeng Li
>

-- 
You received this message because you are subscribed to the Google Groups 
"mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mechanical-sympathy+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Linux futex_wait() bug... [Yes. You read that right. UPDATE to LATEST PATCHES NOW].

[Will Foster]

On Tuesday, February 14, 2017 at 4:01:52 PM UTC, Allen Reese wrote:
>
> This bug report seems to have a way to reproduce it:
> https://bugs.centos.org/view.php?id=8371
>
> Hope that helps.
>
> --Allen Reese
>
>

I also see this on latest CentOS7.3 with Logstash, I've disabled huge pages 
via 
transparent_hugepage=never

in grub.

Here's what I get from strace against logstash (never fully comes up to 
listen on TCP/5044)

[root@host-01 ~]# strace -p 1292
Process 1292 attached
futex(0x7f80eff8a9d0, FUTEX_WAIT, 1312, NULL


I am hitting this issue on Logstash 5.2.1-1 while trying to upgrade my Ansible 
playbooks  to the latest 
ES versions.

 

>
> --
> *From:* Longchao Dong >
> *To:* mechanical-sympathy > 
> *Sent:* Monday, February 13, 2017 1:55 AM
> *Subject:* Re: Linux futex_wait() bug... [Yes. You read that right. 
> UPDATE to LATEST PATCHES NOW].
>
> How to reproduce this issue ? Is it possible to show us the method ? I am 
> also working on one strange pthread_cond_wait issue, but not sure if that 
> one is related with this issue.
>
> On Wednesday, May 20, 2015 at 8:16:12 AM UTC+8, manis...@gmail.com wrote:
>
> I bumped on this error couple of months back when using CentOS 6.6 with 32 
> cores Dell server. After many days of debugging, I realized it to be a 
> CentOS 6.6 bug and moved back to 6.5 and since then no such issues have 
> been seen.
> I am able to reproduce this issue in 15 minutes of heavy load on my multi 
> threaded c  code.
>
> On Wednesday, May 13, 2015 at 3:37:32 PM UTC-7, Gil Tene wrote:
>
> We had this one bite us hard and scare the %$^! out of us, so I figured 
> I'd share the fear...
>
> The linux futex_wait call has been broken for about a year (in upstream 
> since 3.14, around Jan 2014), and has just recently been fixed (in upstream 
> 3.18, around October 2014). More importantly this breakage seems to have 
> been back ported into major distros (e.g. into RHEL 6.6 and its cousins, 
> released in October 2014), and the fix for it has only recently been back 
> ported (e.g. RHEL 6.6.z and cousins have the fix).
>
> The impact of this kernel bug is very simple: user processes can deadlock 
> and hang in seemingly impossible situations. A futex wait call (and 
> anything using a futex wait) can stay blocked forever, even though it had 
> been properly woken up by someone. Thread.park() in Java may stay parked. 
> Etc. If you are lucky you may also find soft lockup messages in your dmesg 
> logs. If you are not that lucky (like us, for example), you'll spend a 
> couple of months of someone's time trying to find the fault in your code, 
> when there is nothing there to find. 
>
> This behavior seems to regularly appear in the wild on Haswell servers 
> (all the machines where we have had customers hit it in the field and in 
> labs been Haswells), and since Haswell servers are basically what you get 
> if you buy a new machine now, or run on the cool new amazon EC2/GCE/Azure 
> stuff, you are bound to experience some interesting behavior. I don't know 
> of anyone that will see this as a good thing for production systems. Except 
> for maybe Netflix (maybe we should call this the linux fumonkey).
>
> The commit for the *fix* is here:  https://github.com/torvalds/ 
> linux/commit/ 76835b0ebf8a7fe85beb03c7512141 9a7dec52f0 
> 
>
> The commit explanation says that it fixes https://github.com/torvalds/ 
> linux/commit/ b0c29f79ecea0b6fbcefc999e70f28 43ae8306db 
> 
>  
> (presumably the bug introduced with that change), which was made in Jan of 
> 2014into 3.14. That 3.14 code added logic to avoid taking a lock if the 
> code knows that there are no waiters. It documents (pretty elaborately) how 
> "…thus preventing tasks sleeping forever if wakers don't acknowledge all 
> possible waiters" with logic that explains how memory barriers guarantee 
> the correct order (see paragraph at line 141), which includes the statement 
> "this is done by the barriers in get_futex_key_refs(), through either ihold 
> or atomic_inc, depending on the futex type." (this assumption is the actual 
> bug). The assumption is further reinforced in the fact that the change 
> added a comment to every calls to get_futex_key_refs() in the code that 
> says "/* implies MB (B) */".
>
> The problem was that get_futex_key_refs() did NOT imply a memory barrier. 
> It only included a memory barrier for two explicit cases in a switch 
> statement that checks the futex type, but did not have a default case 
> handler, and therefor did not apply a memory barrier for other fuxtex 
> types. Like private futexes. Which are a very commonly used type of futex.
>
> The fix is simple, an added default case for the switch that just has an 
> explicit smp_mb() in it. There was a missing

JVM crashed with memmap and unsafe operation for IPC

[Yunpeng Li]

Hi there,
 I'm trying to copycat using memmap and unsafe to build an IPC ping 
pong test. Attached the source codes and crash reports, unfortunately, my 
code crashes JVM each time when ping pong reaches 73100(+- 10), Run 
PublisherTest first then SubscriberTest in separated process. Sometimes one 
process crashes sometime both crash at the sametime.
 Can someone help to check what's the reason it crashes at the very 
place. I ran the test case inside eclipse and on Mint Linux 16(yeah old 
version) and jvm 8.
 And one more question on how JVM work with OS on memmap swap, e.g. in 
my case, the locks are more than 1 page away from data, I follow the 
reserve-modify-publish pattern from disruptor, First updating data then 
publish lock by unsafe putOrdered, How did jvm make sure the lock is 
visible behind data from another process, Given update dirty page is 
"random" by OS.

Thanks in advance.
Yunpeng Li

-- 
You received this message because you are subscribed to the Google Groups 
"mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to mechanical-sympathy+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
package org.loveyoupeng.play.ground.unsafe;

import java.io.File;
import java.io.RandomAccessFile;
import java.lang.reflect.Field;
import java.nio.channels.FileChannel.MapMode;

import org.junit.BeforeClass;
import org.junit.Test;

import sun.misc.Unsafe;

public class PublisherTest {
	private static long PUB_LOC;
	private static long SUB_LOC;
	private static long PUB_VALUE;
	private static long PUB_KEY;
	private static final Unsafe U;

	static {
		Unsafe tmp = null;
		try {
			Field field = Unsafe.class.getDeclaredField("theUnsafe");
			field.setAccessible(true);
			tmp = (Unsafe) field.get(null);
			field.setAccessible(false);
		} catch (Exception e) {
			e.printStackTrace();
		} finally {
			U = tmp;
		}
	}

	@BeforeClass
	public static void setupClass() throws Exception {
		File file = new File("data-back");
		file.delete();
		file.createNewFile();

		RandomAccessFile raf = new RandomAccessFile(file, "rw");
		raf.setLength(1024 * 20 + 50);
		PUB_LOC = ((sun.nio.ch.DirectBuffer) (raf.getChannel().map(MapMode.READ_WRITE, 0, raf.length(.address();
		PUB_LOC += 64 - (PUB_LOC & 63L);
		SUB_LOC = PUB_LOC + 256;
		PUB_KEY = SUB_LOC + 4 * 1024;
		PUB_VALUE = PUB_KEY + 8;
	}

	@Test
	public void test() throws Exception {
		System.out.println("waiting to publish");
		long value = 1L;
		while (true) {
			while (U.getLong(null, PUB_LOC) != 1)
Thread.yield();
			U.putLong(PUB_LOC, 0);
			U.putLong(PUB_KEY, value);
			U.putLong(PUB_VALUE, value);
			U.putOrderedLong(null, SUB_LOC, 1L);
			System.out.println("publish value=" + value++);
		}
	}
}
package org.loveyoupeng.play.ground.unsafe;

import java.io.File;
import java.io.RandomAccessFile;
import java.lang.reflect.Field;
import java.nio.channels.FileChannel.MapMode;

import org.junit.BeforeClass;
import org.junit.Test;

import sun.misc.Unsafe;

public class SubscriberTest {
	private static long PUB_LOC;
	private static long SUB_LOC;
	private static long PUB_VALUE;
	private static long PUB_KEY;
	private static final Unsafe U;

	static {
		Unsafe tmp = null;
		try {
			Field field = Unsafe.class.getDeclaredField("theUnsafe");
			field.setAccessible(true);
			tmp = (Unsafe) field.get(null);
			field.setAccessible(false);
		} catch (Exception e) {
			e.printStackTrace();
		} finally {
			U = tmp;
		}
	}

	@BeforeClass
	public static void setupClass() throws Exception {
		File file = new File("data-back");

		RandomAccessFile raf = new RandomAccessFile(file, "rw");
		PUB_LOC = ((sun.nio.ch.DirectBuffer) (raf.getChannel().map(MapMode.READ_WRITE, 0, raf.length(.address();
		PUB_LOC += 64 - (PUB_LOC & 63L);
		SUB_LOC = PUB_LOC + 256;
		PUB_KEY = SUB_LOC + 4 * 1024;
		PUB_VALUE = PUB_KEY + 8;
	}

	@Test
	public void test() throws Exception {
		System.out.println("waiting to subscribe");
		while (true) {
			U.putOrderedLong(null, PUB_LOC, 1L);
			while (U.getLong(null, SUB_LOC) != 1)
Thread.yield();
			U.putLong(SUB_LOC, 0);
			long key = U.getLong(PUB_KEY);
			long value = U.getLong(PUB_VALUE);
			if (key == value)
System.out.println("sub same value=" + value);
			else
System.out.println("different key=" + key + ", value=" + value);

		}
	}
}
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x7fc29d18be86, pid=6313, tid=140474240915200
#
# JRE version: Java(TM) SE Runtime Environment (8.0_74-b02) (build 1.8.0_74-b02)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.74-b02 mixed mode linux-amd64 
compressed oops)
# Problematic frame:
# J 205% C2 org.loveyoupeng.play.ground.unsafe.PublisherTest.test()V (101 
bytes) @ 0x7fc29d18be86 [0x7fc29d18bde0+0xa6]
#
# Failed to write core dump. Core dumps have been disabled. To enable core 
dumping, try "ulimit -c unlimited" before starting Java ag

Re: System.arraycopy vs Arrays.copyOf

[Nikolay Tsankov]

I strongly suspect that you are not giving it enough time to warm up.

The zero-ing happens when you allocate the destination arrays and hotspot
can often optimize it away if you copy into it right after allocation. As
Mr. Shipilev pointed there are cases when it doesn't yet, e.g. when you use
a field for the length parameter.
To test this assumption, I've added another benchmark to your test and run
it with a longer warmup:

@Benchmark
public long[] arraycopy_field() {
long[] dst = new long[size];
System.arraycopy(src, 0, dst, 0, size);
return dst;
}

Results:

Benchmark(size)  Mode  CntScore   Error  Units
ArrayBench.arraycopy_field  1048576  avgt   30  1907102.105 ±
30966.659  ns/op   <- zeroing slows it down
ArrayBench.arrcopy  1048576  avgt   30  1402851.015 ±
4220.862  ns/op   <- no zeroing
ArrayBench.syscopy  1048576  avgt   30  1422812.870 ±
26030.386  ns/op   <- no zeroing


I have not looked at the assembly though, so I might be wrong


On Fri, Feb 10, 2017 at 12:33 PM, Jean-Philippe BEMPEL 
wrote:

> Hi Carl,
>
> Did you read this: https://shipilev.net/blog/2016/arrays-wisdom-ancients/
> ? :)
>
> Cheers
>
>
> On Thursday, February 9, 2017 at 5:33:12 PM UTC+1, Carl Mastrangelo wrote:
>>
>> Hi,
>>
>> I'm trying to copy an array of bytes and return the copy.  I noticed that
>> there are two ways of doing this: System.arraycopy and Arrays.copyOf, both
>> of which have intrinsic versions in OpenJDK.  I was hoping that the
>> Arrays.copyOf version would skip the initial zeroing of the destination
>> array, and thus be slightly faster.  However, benchmarking them seems to
>> show that neither one is consistently faster than the other.  I'm not that
>> familiar with assembly, so I may be reading it wrong.  That said, reading
>> the "Hottest Region 2" below, it looks like copyOf is still zeroing?  Can
>> that be right?
>>
>> Benchmark:
>>
>> @State(Scope.Benchmark)
>> public class Benchmark {
>>
>>   @Param({"1048576"})
>>   public int size;
>>
>>   public long[] src;
>>
>>   @Setup
>>   public void setUp() throws Exception {
>> src = new long[size];
>> for (int i = 0; i < src.length; i++) {
>>   src[i] = i;
>> }
>>   }
>>
>>   /**
>>* Javadoc comment.
>>*/
>>   @Benchmark
>>   @BenchmarkMode(Mode.SampleTime)
>>   @OutputTimeUnit(TimeUnit.NANOSECONDS)
>>   public long[] syscopy() {
>> long[] dest = new long[src.length];
>> System.arraycopy(src, 0, dest, 0, src.length);
>> return dest;
>>   }
>>
>>   /**
>>* Javadoc comment.
>>*/
>>   @Benchmark
>>   @BenchmarkMode(Mode.SampleTime)
>>   @OutputTimeUnit(TimeUnit.NANOSECONDS)
>>   public long[] arrcopy() {
>> return Arrays.copyOf(src, src.length);
>>   }
>> }
>>
>> And the Output for arrcopy:
>>
>> # JMH 1.17.3 (released 56 days ago)
>> # VM version: JDK 1.8.0_92, VM 25.92-b14
>> # VM invoker: ~/Downloads/jdk1.8.0_92/jre/bin/java
>> # VM options: -server -Xms2g -Xmx2g -XX:+UnlockDiagnosticVMOptions
>> -XX:LogFile=/tmp/thelogs -XX:+LogCompilation -XX:+PrintNMethods
>> -XX:+PrintNativeNMethods -XX:+PrintAssembly -XX:+PrintInlining
>> -XX:PrintAssemblyOptions=syntax -XX:+PrintCompilation
>> # Warmup: 10 iterations, 1 s each
>> # Measurement: 1 iterations, 1 s each
>> # Timeout: 10 min per iteration
>> # Threads: 1 thread, will synchronize iterations
>> # Benchmark mode: Sampling time
>> # Benchmark: Benchmark.arrcopy
>> # Parameters: (size = 1048576)
>>
>> # Run progress: 0.00% complete, ETA 00:00:22
>> # Fork: 1 of 1
>> # Preparing profilers: LinuxPerfAsmProfiler
>> # Profilers consume stdout and stderr from target VM, use -v EXTRA to
>> copy to console
>> # Warmup Iteration   1: 3064247.755 ±(99.9%) 1101881.397 ns/op
>> # Warmup Iteration   2: 2104675.705 ±(99.9%) 116863.885 ns/op
>> # Warmup Iteration   3: 1933284.271 ±(99.9%) 73166.251 ns/op
>> # Warmup Iteration   4: 2022234.170 ±(99.9%) 31234.827 ns/op
>> # Warmup Iteration   5: 2054921.988 ±(99.9%) 171828.509 ns/op
>> # Warmup Iteration   6: 1984621.714 ±(99.9%) 70876.189 ns/op
>> # Warmup Iteration   7: 2029098.580 ±(99.9%) 32632.162 ns/op
>> # Warmup Iteration   8: 1888186.445 ±(99.9%) 27772.059 ns/op
>> # Warmup Iteration   9: 1956954.176 ±(99.9%) 110294.977 ns/op
>> # Warmup Iteration  10: 1961879.592 ±(99.9%) 75665.475 ns/op
>> Iteration   1: 2082120.533 ±(99.9%) 31688.951 ns/op
>>  arrcopy·p0.00:   1851392.000 ns/op
>>  arrcopy·p0.50:   2059264.000 ns/op
>>  arrcopy·p0.90:   2231910.400 ns/op
>>  arrcopy·p0.95:   2273280.000 ns/op
>>  arrcopy·p0.99:   3109150.720 ns/op
>>  arrcopy·p0.999:  4194304.000 ns/op
>>  arrcopy·p0.: 4194304.000 ns/op
>>  arrcopy·p1.00:   4194304.000 ns/op
>>
>> # Processing profiler results: LinuxPerfAsmProfiler
>>
>>
>> Result "arrcopy":
>>   N = 480
>>   mean = 2082120.533 ±(99.9%) 31688.951 ns/op
>>
>>   Histogram, ns/op: