Re: [OMPI devel] PMIX deadlock

Thanks, Nysal!! Good catch! Josh On Mon, Nov 9, 2015 at 2:27 PM, Mark Santcroos wrote: > It seems the change suggested by Nysal also allows me to run into the next > problem ;-) > > Mark > > > On 09 Nov 2015, at 20:19 , George Bosilca wrote: > > > > All 10k tests completed successfully. Nysal

Re: [OMPI devel] PMIX deadlock

It seems the change suggested by Nysal also allows me to run into the next problem ;-) Mark > On 09 Nov 2015, at 20:19 , George Bosilca wrote: > > All 10k tests completed successfully. Nysal pinpointed the real problem > behind the deadlocks. :+1: > > George. > > > On Mon, Nov 9, 2015 at

Re: [OMPI devel] PMIX deadlock

All 10k tests completed successfully. Nysal pinpointed the real problem behind the deadlocks. :+1: George. On Mon, Nov 9, 2015 at 1:13 PM, Ralph Castain wrote: > Looking at it, I think I see what was happening. The thread would start, > but then immediately see that the active flag was false

Re: [OMPI devel] PMIX deadlock

Looking at it, I think I see what was happening. The thread would start, but then immediately see that the active flag was false and would exit. This left the server without any listening thread - but it wouldn’t detect this had happened. It was therefore a race between whether the thread checke

Re: [OMPI devel] PMIX deadlock

Clearly Nyal got a valid point there. I launched a stress test with Nysal suggestion in the code, and so far it's up to few hundreds iterations without deadlock. I would not claim victory yet, I launched a 10k cycle to see where we stand (btw this never passed before). I'll let you know the outcome

Re: [OMPI devel] PMIX deadlock

2015-11-09 22:42 GMT+06:00 Artem Polyakov : > This is the very good point, Nysal! > > This is definitely a problem and I can say even more: avg. 3 from every 10 > tasks was affected by this bug. Once the PR ( > https://github.com/pmix/master/pull/8) was applied I was able to run 100 > testing task

Re: [OMPI devel] PMIX deadlock

This is the very good point, Nysal! This is definitely a problem and I can say even more: avg. 3 from every 10 tasks was affected by this bug. Once the PR ( https://github.com/pmix/master/pull/8) was applied I was able to run 100 testing tasks without any hangs. Here some more information on my s

Re: [OMPI devel] PMIX deadlock

Those are certainly valid suggestions, and I’ll incorporate them. However, I doubt that’s the root cause here as the thread gets started well before the first client is fork/exec’d by the daemon. > On Nov 8, 2015, at 8:53 PM, Nysal Jan K A wrote: > > In listen_thread(): > 194 while (pmix_

Re: [OMPI devel] PMIX deadlock

In listen_thread(): 194 while (pmix_server_globals.listen_thread_active) { 195 FD_ZERO(&readfds); 196 FD_SET(pmix_server_globals.listen_socket, &readfds); 197 max = pmix_server_globals.listen_socket; Is it possible that pmix_server_globals.listen_thread_active can be fa

Re: [OMPI devel] PMIX deadlock

We had a power outage last week and the local disks on our cluster were wiped out. My tester was in there. But, I can rewrite it after SC. George. On Sat, Nov 7, 2015 at 12:04 PM, Ralph Castain wrote: > Could you send me your stress test? I’m wondering if it is just something > about how we s

Re: [OMPI devel] PMIX deadlock

Could you send me your stress test? I’m wondering if it is just something about how we set socket options > On Nov 7, 2015, at 8:58 AM, George Bosilca wrote: > > I has to postpone this until after SC. However, I ran for 3 days a stress > test of UDS reproducing the opening and sending of data

Re: [OMPI devel] PMIX deadlock

I has to postpone this until after SC. However, I ran for 3 days a stress test of UDS reproducing the opening and sending of data (what Ralph said in his email) and I never could get a deadlock. George. On Sat, Nov 7, 2015 at 11:26 AM, Ralph Castain wrote: > George was looking into it, but I

Re: [OMPI devel] PMIX deadlock

George was looking into it, but I don’t know if he has had time recently to continue the investigation. We understand “what” is happening (accept sometimes ignores the connection), but we don’t yet know “why”. I’ve done some digging around the web, and found that sometimes you can try to talk to

Re: [OMPI devel] PMIX deadlock

Hello, is there any progress on this topic? This affects our PMIx measurements. 2015-10-30 21:21 GMT+06:00 Ralph Castain : > I’ve verified that the orte/util/listener thread is not being started, so > I don’t think it should be involved in this problem. > > HTH > Ralph > > On Oct 30, 2015, at 8:0

Re: [OMPI devel] PMIX deadlock

I’ve verified that the orte/util/listener thread is not being started, so I don’t think it should be involved in this problem. HTH Ralph > On Oct 30, 2015, at 8:07 AM, Ralph Castain wrote: > > Hmmm…there is a hook that would allow the PMIx server to utilize that > listener thread, but we aren

Re: [OMPI devel] PMIX deadlock

Hmmm…there is a hook that would allow the PMIx server to utilize that listener thread, but we aren’t currently using it. Each daemon plus mpirun will call orte_start_listener, but nothing is currently registering and so the listener in that code is supposed to just return without starting the th

Re: [OMPI devel] PMIX deadlock

Some progress, that puzzles me but might help you understand. Once the deadlock appears, if I manually kill the MPI process on the node where the deadlock was created, the local orte daemon doesn't notice and will just keep waiting. Quick question: I am under the impression that the issue is not i

Re: [OMPI devel] PMIX deadlock

Should have also clarified: the prior fixes are indeed in the current master. > On Oct 28, 2015, at 12:42 AM, Ralph Castain wrote: > > Nope - I was wrong. The correction on the client side consisted of attempting > to timeout if the blocking recv failed. We then modified the blocking > send/re

Re: [OMPI devel] PMIX deadlock

Nope - I was wrong. The correction on the client side consisted of attempting to timeout if the blocking recv failed. We then modified the blocking send/recv so they would handle errors. So that problem occurred -after- the server had correctly called accept. The listener code is in opal/mca/pm

Re: [OMPI devel] PMIX deadlock

Looking at the code, it appears that a fix was committed for this problem, and that we correctly resolved the issue found by Paul. The problem is that the fix didn’t get upstreamed, and so it was lost the next time we refreshed PMIx. Sigh. Let me try to recreate the fix and have you take a gande

Re: [OMPI devel] PMIX deadlock

Here is the discussion - afraid it is fairly lengthy. Ignore the hwloc references in it as that was a separate issue: http://www.open-mpi.org/community/lists/devel/2015/09/18074.php It definitely sounds like the same issue creepi

Re: [OMPI devel] PMIX deadlock

Interesting. Do you have a pointer to the commit (or/and to the discussion)? I looked at the PMIX code, and I have identified few issues, but unfortunately none of them seem to fix the problem for good. However, now I need more than 1000 runs to get a deadlock (instead of few tens). Looking with

Re: [OMPI devel] PMIX deadlock

Hmmm…this looks like it might be that problem we previously saw where the blocking recv hangs in a proc when the blocking send tries to send before the domain socket is actually ready, and so the send fails on the other end. As I recall, it was something to do with the socketoptions - and then P

Re: [OMPI devel] PMIX deadlock

It appear the branch solve the problem at least partially. I asked one of my students to hammer it pretty badly, and he reported that the deadlocks still occur. He also graciously provided some stacktraces: #0 0x7f4bd5274aed in nanosleep () from /lib64/libc.so.6 #1 0x7f4bd52a9c94 in usle

Re: [OMPI devel] PMIX deadlock

I haven’t been able to replicate this when using the branch in this PR: https://github.com/open-mpi/ompi/pull/1073 Would you mind giving it a try? It fixes some other race conditions and might pick this one up too. > On Oct 27, 2015, at 10:04 AM, R

Re: [OMPI devel] PMIX deadlock

Okay, I’ll take a look - I’ve been chasing a race condition that might be related > On Oct 27, 2015, at 9:54 AM, George Bosilca wrote: > > No, it's using 2 nodes. > George. > > > On Tue, Oct 27, 2015 at 12:35 PM, Ralph Castain > wrote: > Is this on a single node?

Re: [OMPI devel] PMIX deadlock

No, it's using 2 nodes. George. On Tue, Oct 27, 2015 at 12:35 PM, Ralph Castain wrote: > Is this on a single node? > > On Oct 27, 2015, at 9:25 AM, George Bosilca wrote: > > I get intermittent deadlocks wit the latest trunk. The smallest reproducer > is a shell for loop around a small (2 pro

Re: [OMPI devel] PMIX deadlock

Is this on a single node? > On Oct 27, 2015, at 9:25 AM, George Bosilca wrote: > > I get intermittent deadlocks wit the latest trunk. The smallest reproducer is > a shell for loop around a small (2 processes) short (20 seconds) MPI > application. After few tens of iterations the MPI_Init will

[OMPI devel] PMIX deadlock

I get intermittent deadlocks wit the latest trunk. The smallest reproducer is a shell for loop around a small (2 processes) short (20 seconds) MPI application. After few tens of iterations the MPI_Init will deadlock with the following backtrace: #0 0x7fa94b5d9aed in nanosleep () from /lib64/l