Re: [OMPI devel] 1.7.4rc: mpirun hangs on ia64
Some progress:
I fixed IA64.asm but still saw failures.
I realized I'd not checked the ia64/atomic.h file.
Lo and behold the origin of the bogus "sxt4" is a pair of improper casts,
removed by the following:
--- opal/include/opal/sys/ia64/atomic.h~2014-01-23
13:04:03.0 -0800
+++ opal/include/opal/sys/ia64/atomic.h 2014-01-23 13:04:42.0 -0800
@@ -119,7 +119,7 @@
__asm__ __volatile__ ("cmpxchg8.acq %0=[%1],%2,ar.ccv":
"=r"(ret) : "r"(addr), "r"(newval) : "memory");
-return ((int32_t)ret == oldval);
+return (ret == oldval);
}
@@ -132,7 +132,7 @@
__asm__ __volatile__ ("cmpxchg8.rel %0=[%1],%2,ar.ccv":
"=r"(ret) : "r"(addr), "r"(newval) : "memory");
-return ((int32_t)ret == oldval);
+return (ret == oldval);
}
#endif /* OMPI_GCC_INLINE_ASSEMBLY */
I will retest ASAP and report with, I hope, an attachment to fix both
IA64.asm and ia64/atomic.h
-Paul
On Wed, Jan 22, 2014 at 4:24 PM, Paul Hargrove wrote:
>
> On Wed, Jan 22, 2014 at 2:22 PM, Paul Hargrove wrote:
>
>> My ia64 asm is a bit rusty, but I'll give a quick look if/when I can.
>
>
> I had a look (in v1.7) and this is what I see:
>
> $cat -n IA64.asm | grep -A14 opal_atomic_cmpset_acq_64:
> 70 opal_atomic_cmpset_acq_64:
> 71 .prologue
> 72 .body
> 73 mov ar.ccv=r33;;
> 74 cmpxchg8.acq r32=[r32],r34,ar.ccv
> 75 ;;
> 76 sxt4 r32 = r32
> 77 ;;
> 78 cmp.eq p6, p7 = r33, r32
> 79 ;;
> 80 (p6) addl r8 = 1, r0
> 81 (p7) mov r8 = r0
> 82 br.ret.sptk.many b0
> 83 ;;
> 84 .endp opal_atomic_cmpset_acq_64#
>
> The (approximate and non-atomic) C equivalent is:
>
> // r32 = address
> // r33 = oldvalue
> // r34 = newvalue
> int opal_atomic_cmpset_acq_64(int64_t r32, int64_t r33, int64 r34) {
>int64_t ccv = r33; // L73
>if (*(int64_t *)r32 == ccv) *(int64_t *)r32 = r34; // L74
>
>r32 = (int64_t)(int32_t)r32; // L76 = sign-extend 32->64
>
>bool p6, p7;
>p7 = !(p6 = (r33 == r32)); // L78
>
>const int r0 = 0;
>int r8;
>if (p6) r8 = 1 + r0; // L80
>if (p7) r8 = r0; // L81
>return r8; // L82
> }
>
> Which is fine except that line 76 is totally wrong!!
> The "sxt4" instruction is "sign-extend from 4 bytes to 8 bytes".
> Thus the upper 32-bits of the value read from memory are lost!
> Unless the upper 33 bits off r33 (oldvalue) are all 0s or all 1s, the
> comparison on line 78 MUST fail.
> This explains the hang, as the lifo push will loop indefinitely waiting
> for the success of this cmpset.
>
> Note the same erroneous instruction is also present in the _rel variant
> (at line 94).
> The trunk has the same issue.
> This code has not changed at all since IA64.asm was added way back in
> r4471.
>
> I won't have access to the IA64 platform again until tomorrow AM.
> So, testing my hypothesis will need to wait.
>
> BTW:
> IFF I am right about the source of this problem, then it would be
> beneficial to have (and I may contribute) a stronger test (for "make
> check") that would detect this sort of bug in the atomics (specifically
> look for both false-positive and false-negative return value from 64-bit
> cmpset operations with values satisfying a range of "corner cases"). I
> think I have single-bit and double-bit "marching tests" for cmpset in my
> own arsenal of tests for GASNet's atomics. If I don't have time to
> contribute a complete test, I can at least contribute that logic for
> somebody else to port to the OPAL atomics.
>
> -Paul
>
> P.S.:
> The cmpxchgN for N in 1,2,4 are documented as ZERO-extending their loads
> to 64-bits.
> So, there is a slim chance that the sxt4 actually was intended for the
> 32-bit cmpset code.
> However, since the comparison used there is a "cmp4.eq" the "sxt4" would
> still not be needed.
>
>
> --
> Paul H. Hargrove phhargr...@lbl.gov
> Future Technologies Group
> Computer and Data Sciences Department Tel: +1-510-495-2352
> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
>
--
Paul H. Hargrove phhargr...@lbl.gov
Future Technologies Group
Computer and Data Sciences Department Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
Re: [OMPI devel] 1.7.4rc: mpirun hangs on ia64
On Wed, Jan 22, 2014 at 2:22 PM, Paul Hargrove wrote:
> My ia64 asm is a bit rusty, but I'll give a quick look if/when I can.
I had a look (in v1.7) and this is what I see:
$cat -n IA64.asm | grep -A14 opal_atomic_cmpset_acq_64:
70 opal_atomic_cmpset_acq_64:
71 .prologue
72 .body
73 mov ar.ccv=r33;;
74 cmpxchg8.acq r32=[r32],r34,ar.ccv
75 ;;
76 sxt4 r32 = r32
77 ;;
78 cmp.eq p6, p7 = r33, r32
79 ;;
80 (p6) addl r8 = 1, r0
81 (p7) mov r8 = r0
82 br.ret.sptk.many b0
83 ;;
84 .endp opal_atomic_cmpset_acq_64#
The (approximate and non-atomic) C equivalent is:
// r32 = address
// r33 = oldvalue
// r34 = newvalue
int opal_atomic_cmpset_acq_64(int64_t r32, int64_t r33, int64 r34) {
int64_t ccv = r33; // L73
if (*(int64_t *)r32 == ccv) *(int64_t *)r32 = r34; // L74
r32 = (int64_t)(int32_t)r32; // L76 = sign-extend 32->64
bool p6, p7;
p7 = !(p6 = (r33 == r32)); // L78
const int r0 = 0;
int r8;
if (p6) r8 = 1 + r0; // L80
if (p7) r8 = r0; // L81
return r8; // L82
}
Which is fine except that line 76 is totally wrong!!
The "sxt4" instruction is "sign-extend from 4 bytes to 8 bytes".
Thus the upper 32-bits of the value read from memory are lost!
Unless the upper 33 bits off r33 (oldvalue) are all 0s or all 1s, the
comparison on line 78 MUST fail.
This explains the hang, as the lifo push will loop indefinitely waiting for
the success of this cmpset.
Note the same erroneous instruction is also present in the _rel variant (at
line 94).
The trunk has the same issue.
This code has not changed at all since IA64.asm was added way back in r4471.
I won't have access to the IA64 platform again until tomorrow AM.
So, testing my hypothesis will need to wait.
BTW:
IFF I am right about the source of this problem, then it would be
beneficial to have (and I may contribute) a stronger test (for "make
check") that would detect this sort of bug in the atomics (specifically
look for both false-positive and false-negative return value from 64-bit
cmpset operations with values satisfying a range of "corner cases"). I
think I have single-bit and double-bit "marching tests" for cmpset in my
own arsenal of tests for GASNet's atomics. If I don't have time to
contribute a complete test, I can at least contribute that logic for
somebody else to port to the OPAL atomics.
-Paul
P.S.:
The cmpxchgN for N in 1,2,4 are documented as ZERO-extending their loads to
64-bits.
So, there is a slim chance that the sxt4 actually was intended for the
32-bit cmpset code.
However, since the comparison used there is a "cmp4.eq" the "sxt4" would
still not be needed.
--
Paul H. Hargrove phhargr...@lbl.gov
Future Technologies Group
Computer and Data Sciences Department Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
Re: [OMPI devel] 1.7.4rc: mpirun hangs on ia64
On Wed, Jan 22, 2014 at 1:59 PM, Ralph Castain wrote: > Huh - afraid I can't see anything wrong so far. All looks normal and then > it just hangs. Any chance you can "gdb" to the proc and see where it is > stuck? > Ralph, The gstack output below looks like one thread is spinning on an atomic of some sort. Running gstack repeatedly 100 times yields the following "histogram" of the top frame of Thread 1: 47 opal_atomic_lifo_push > opal_atomic_cmpset_ptr > opal_atomic_cmpset_acq_64 19 opal_atomic_lifo_push > opal_atomic_cmpset_ptr 6 opal_atomic_lifo_push > opal_atomic_wmb 28 opal_atomic_lifo_push A spin in a lifo push is not consistent (in my experience) with the possibility that the other thread and failed to post some event. So, the problem is probably in the atomics or lifo code, though "make check" passes just fine. My ia64 asm is a bit rusty, but I'll give a quick look if/when I can. I've implemented a lock-free LIFO for ia64 in the past and so have some idea what I am looking at/for. However, with my access window closing under 10 minutes from now, anything more than source inspection will need to wait until tomorrow. -Paul $ gstack 21094 Thread 2 (Thread 0x216bf200 (LWP 21095)): #0 0xa0010721 in __kernel_syscall_via_break () #1 0x205a00d0 in poll () from /lib/libc.so.6.1 #2 0x20a0c3e0 in poll_dispatch () from /eng/home/PHHargrove/OMPI/openmpi-1.7-latest-linux-ia64/INST/lib/libopen-pal.so.6 #3 0x209e5e90 in opal_libevent2021_event_base_loop () from /eng/home/PHHargrove/OMPI/openmpi-1.7-latest-linux-ia64/INST/lib/libopen-pal.so.6 #4 0x206bd8a0 in orte_progress_thread_engine () from /eng/home/PHHargrove/OMPI/openmpi-1.7-latest-linux-ia64/INST/lib/libopen-rte.so.7 #5 0x203dc310 in start_thread () from /lib/libpthread.so.0 #6 0x205b49a0 in __clone2 () from /lib/libc.so.6.1 #7 0x in ?? () Thread 1 (Thread 0x200566a0 (LWP 21094)): #0 0x200973f2 in opal_atomic_cmpset_acq_64 () from /eng/home/PHHargrove/OMPI/openmpi-1.7-latest-linux-ia64/INST/lib/libmpi.so.1 #1 0x20097350 in opal_atomic_cmpset_ptr () from /eng/home/PHHargrove/OMPI/openmpi-1.7-latest-linux-ia64/INST/lib/libmpi.so.1 #2 0x200995d0 in opal_atomic_lifo_push () from /eng/home/PHHargrove/OMPI/openmpi-1.7-latest-linux-ia64/INST/lib/libmpi.so.1 #3 0x20099030 in ompi_free_list_grow () from /eng/home/PHHargrove/OMPI/openmpi-1.7-latest-linux-ia64/INST/lib/libmpi.so.1 #4 0x2009a2a0 in ompi_rb_tree_init () from /eng/home/PHHargrove/OMPI/openmpi-1.7-latest-linux-ia64/INST/lib/libmpi.so.1 #5 0x2029ec10 in mca_mpool_base_tree_init () from /eng/home/PHHargrove/OMPI/openmpi-1.7-latest-linux-ia64/INST/lib/libmpi.so.1 #6 0x20299380 in mca_mpool_base_open () from /eng/home/PHHargrove/OMPI/openmpi-1.7-latest-linux-ia64/INST/lib/libmpi.so.1 #7 0x2098fd80 in mca_base_framework_open () from /eng/home/PHHargrove/OMPI/openmpi-1.7-latest-linux-ia64/INST/lib/libopen-pal.so.6 #8 0x2010d6b0 in ompi_mpi_init () from /eng/home/PHHargrove/OMPI/openmpi-1.7-latest-linux-ia64/INST/lib/libmpi.so.1 #9 0x201b3460 in PMPI_Init () from /eng/home/PHHargrove/OMPI/openmpi-1.7-latest-linux-ia64/INST/lib/libmpi.so.1 #10 0x4c00 in main () > > On Jan 22, 2014, at 11:39 AM, Paul Hargrove wrote: > > Ralph, > > Attached is the requested output with the addition of "-mca > grpcomm_base_verbose 5". > I have also attached a 2nd output with the further addition of "-mca > oob_tcp_if_include lo" to ensure that this is not related to the firewall > issues I've seen on other hosts. > > I have use of this host until 14:30 PST today, and then lose it for 12 > hours. > So, tests of the next tarball won't start until after 2:30am - which > probably means 10am. > > -Paul > > > On Wed, Jan 22, 2014 at 7:34 AM, Ralph Castain wrote: > >> Weird - everything looks completely normal. Can you add -mca >> grpcomm_base_verbose 5 to your cmd line? >> >> >> On Jan 22, 2014, at 1:38 AM, Paul Hargrove wrote: >> >> Following-up as promised: >> >> Output from an --enable-debug build is attached. >> >> -Paul >> >> >> On Tue, Jan 21, 2014 at 11:25 PM, Paul Hargrove wrote: >> >>> Yes, this is familiar. See: >>> http://www.open-mpi.org/community/lists/devel/2013/11/13182.php >>> >>> If I understand correctly, the thread ended with: >>> >>> On 03 December 2013, Sylvestre Ledru wrote: >>> FYI, Debian has stopped supporting ia64 for its next release So, I stopped working on that issue. >>> >>> >>> Well, I have access to a Linux/IA64 system and my trials with >>> openmpi-1.7.4rc2r30361 appear to hang, much as Sylvestre had reported w/ >>> 1.6.5. >>> >>> I am atatching output from a build W/O --enable debug for the command: >>> $ mpirun -mca plm_base_verbose 5 -mca ras_base_verbose 5 -mca >>> rmaps_base_verbose 5 -mca ess_base_verbose 5 -np 1 ./ring_c >>> >>> I will follow-up with an -
Re: [OMPI devel] 1.7.4rc: mpirun hangs on ia64
Huh - afraid I can't see anything wrong so far. All looks normal and then it just hangs. Any chance you can "gdb" to the proc and see where it is stuck? On Jan 22, 2014, at 11:39 AM, Paul Hargrove wrote: > Ralph, > > Attached is the requested output with the addition of "-mca > grpcomm_base_verbose 5". > I have also attached a 2nd output with the further addition of "-mca > oob_tcp_if_include lo" to ensure that this is not related to the firewall > issues I've seen on other hosts. > > I have use of this host until 14:30 PST today, and then lose it for 12 hours. > So, tests of the next tarball won't start until after 2:30am - which probably > means 10am. > > -Paul > > > On Wed, Jan 22, 2014 at 7:34 AM, Ralph Castain wrote: > Weird - everything looks completely normal. Can you add -mca > grpcomm_base_verbose 5 to your cmd line? > > > On Jan 22, 2014, at 1:38 AM, Paul Hargrove wrote: > >> Following-up as promised: >> >> Output from an --enable-debug build is attached. >> >> -Paul >> >> >> On Tue, Jan 21, 2014 at 11:25 PM, Paul Hargrove wrote: >> Yes, this is familiar. See: >> http://www.open-mpi.org/community/lists/devel/2013/11/13182.php >> >> If I understand correctly, the thread ended with: >> >> On 03 December 2013, Sylvestre Ledru wrote: >> FYI, Debian has stopped supporting ia64 for its next release >> So, I stopped working on that issue. >> >> >> Well, I have access to a Linux/IA64 system and my trials with >> openmpi-1.7.4rc2r30361 appear to hang, much as Sylvestre had reported w/ >> 1.6.5. >> >> I am atatching output from a build W/O --enable debug for the command: >> $ mpirun -mca plm_base_verbose 5 -mca ras_base_verbose 5 -mca >> rmaps_base_verbose 5 -mca ess_base_verbose 5 -np 1 ./ring_c >> >> I will follow-up with an --enable-debug build when possible. >> >> -Paul >> >> -- >> Paul H. Hargrove phhargr...@lbl.gov >> Future Technologies Group >> Computer and Data Sciences Department Tel: +1-510-495-2352 >> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900 >> >> >> >> -- >> Paul H. Hargrove phhargr...@lbl.gov >> Future Technologies Group >> Computer and Data Sciences Department Tel: +1-510-495-2352 >> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900 >> ___ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > > -- > Paul H. Hargrove phhargr...@lbl.gov > Future Technologies Group > Computer and Data Sciences Department Tel: +1-510-495-2352 > Lawrence Berkeley National Laboratory Fax: +1-510-486-6900 > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel
Re: [OMPI devel] 1.7.4rc: mpirun hangs on ia64
Ralph, Attached is the requested output with the addition of "-mca grpcomm_base_verbose 5". I have also attached a 2nd output with the further addition of "-mca oob_tcp_if_include lo" to ensure that this is not related to the firewall issues I've seen on other hosts. I have use of this host until 14:30 PST today, and then lose it for 12 hours. So, tests of the next tarball won't start until after 2:30am - which probably means 10am. -Paul On Wed, Jan 22, 2014 at 7:34 AM, Ralph Castain wrote: > Weird - everything looks completely normal. Can you add -mca > grpcomm_base_verbose 5 to your cmd line? > > > On Jan 22, 2014, at 1:38 AM, Paul Hargrove wrote: > > Following-up as promised: > > Output from an --enable-debug build is attached. > > -Paul > > > On Tue, Jan 21, 2014 at 11:25 PM, Paul Hargrove wrote: > >> Yes, this is familiar. See: >> http://www.open-mpi.org/community/lists/devel/2013/11/13182.php >> >> If I understand correctly, the thread ended with: >> >> On 03 December 2013, Sylvestre Ledru wrote: >> >>> FYI, Debian has stopped supporting ia64 for its next release >>> So, I stopped working on that issue. >> >> >> Well, I have access to a Linux/IA64 system and my trials with >> openmpi-1.7.4rc2r30361 appear to hang, much as Sylvestre had reported w/ >> 1.6.5. >> >> I am atatching output from a build W/O --enable debug for the command: >> $ mpirun -mca plm_base_verbose 5 -mca ras_base_verbose 5 -mca >> rmaps_base_verbose 5 -mca ess_base_verbose 5 -np 1 ./ring_c >> >> I will follow-up with an --enable-debug build when possible. >> >> -Paul >> >> -- >> Paul H. Hargrove phhargr...@lbl.gov >> Future Technologies Group >> Computer and Data Sciences Department Tel: +1-510-495-2352 >> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900 >> > > > > -- > Paul H. Hargrove phhargr...@lbl.gov > Future Technologies Group > Computer and Data Sciences Department Tel: +1-510-495-2352 > Lawrence Berkeley National Laboratory Fax: +1-510-486-6900 > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel > -- Paul H. Hargrove phhargr...@lbl.gov Future Technologies Group Computer and Data Sciences Department Tel: +1-510-495-2352 Lawrence Berkeley National Laboratory Fax: +1-510-486-6900 log.txt.bz2 Description: BZip2 compressed data log-incl-lo.txt.bz2 Description: BZip2 compressed data
Re: [OMPI devel] 1.7.4rc: mpirun hangs on ia64
Weird - everything looks completely normal. Can you add -mca grpcomm_base_verbose 5 to your cmd line? On Jan 22, 2014, at 1:38 AM, Paul Hargrove wrote: > Following-up as promised: > > Output from an --enable-debug build is attached. > > -Paul > > > On Tue, Jan 21, 2014 at 11:25 PM, Paul Hargrove wrote: > Yes, this is familiar. See: > http://www.open-mpi.org/community/lists/devel/2013/11/13182.php > > If I understand correctly, the thread ended with: > > On 03 December 2013, Sylvestre Ledru wrote: > FYI, Debian has stopped supporting ia64 for its next release > So, I stopped working on that issue. > > > Well, I have access to a Linux/IA64 system and my trials with > openmpi-1.7.4rc2r30361 appear to hang, much as Sylvestre had reported w/ > 1.6.5. > > I am atatching output from a build W/O --enable debug for the command: > $ mpirun -mca plm_base_verbose 5 -mca ras_base_verbose 5 -mca > rmaps_base_verbose 5 -mca ess_base_verbose 5 -np 1 ./ring_c > > I will follow-up with an --enable-debug build when possible. > > -Paul > > -- > Paul H. Hargrove phhargr...@lbl.gov > Future Technologies Group > Computer and Data Sciences Department Tel: +1-510-495-2352 > Lawrence Berkeley National Laboratory Fax: +1-510-486-6900 > > > > -- > Paul H. Hargrove phhargr...@lbl.gov > Future Technologies Group > Computer and Data Sciences Department Tel: +1-510-495-2352 > Lawrence Berkeley National Laboratory Fax: +1-510-486-6900 > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel
Re: [OMPI devel] 1.7.4rc: mpirun hangs on ia64
Following-up as promised: Output from an --enable-debug build is attached. -Paul On Tue, Jan 21, 2014 at 11:25 PM, Paul Hargrove wrote: > Yes, this is familiar. See: > http://www.open-mpi.org/community/lists/devel/2013/11/13182.php > > If I understand correctly, the thread ended with: > > On 03 December 2013, Sylvestre Ledru wrote: > >> FYI, Debian has stopped supporting ia64 for its next release >> So, I stopped working on that issue. > > > Well, I have access to a Linux/IA64 system and my trials with > openmpi-1.7.4rc2r30361 appear to hang, much as Sylvestre had reported w/ > 1.6.5. > > I am atatching output from a build W/O --enable debug for the command: > $ mpirun -mca plm_base_verbose 5 -mca ras_base_verbose 5 -mca > rmaps_base_verbose 5 -mca ess_base_verbose 5 -np 1 ./ring_c > > I will follow-up with an --enable-debug build when possible. > > -Paul > > -- > Paul H. Hargrove phhargr...@lbl.gov > Future Technologies Group > Computer and Data Sciences Department Tel: +1-510-495-2352 > Lawrence Berkeley National Laboratory Fax: +1-510-486-6900 > -- Paul H. Hargrove phhargr...@lbl.gov Future Technologies Group Computer and Data Sciences Department Tel: +1-510-495-2352 Lawrence Berkeley National Laboratory Fax: +1-510-486-6900 log.txt.bz2 Description: BZip2 compressed data
[OMPI devel] 1.7.4rc: mpirun hangs on ia64
Yes, this is familiar. See: http://www.open-mpi.org/community/lists/devel/2013/11/13182.php If I understand correctly, the thread ended with: On 03 December 2013, Sylvestre Ledru wrote: > FYI, Debian has stopped supporting ia64 for its next release > So, I stopped working on that issue. Well, I have access to a Linux/IA64 system and my trials with openmpi-1.7.4rc2r30361 appear to hang, much as Sylvestre had reported w/ 1.6.5. I am atatching output from a build W/O --enable debug for the command: $ mpirun -mca plm_base_verbose 5 -mca ras_base_verbose 5 -mca rmaps_base_verbose 5 -mca ess_base_verbose 5 -np 1 ./ring_c I will follow-up with an --enable-debug build when possible. -Paul -- Paul H. Hargrove phhargr...@lbl.gov Future Technologies Group Computer and Data Sciences Department Tel: +1-510-495-2352 Lawrence Berkeley National Laboratory Fax: +1-510-486-6900 log.txt.bz2 Description: BZip2 compressed data
