Re: [OMPI devel] large virtual memory consumption on smp nodes and gridengine problems
In addition to what Ralph said, I have the following random comments: 1. You'll have to ask on the GE mailing lists about the GE issues (2gb vs. 2000mb, etc.); I doubt we'll be of much help here on this list. 2. Do you have a very large SMP machine (i.e., 16 cores or more)? More specifically, how many MPI processes do you plan to run at once on a host? 3. Unrelated to the SMP issue, I see that you are using the InfiniBand Mellanox VAPI interface (mvapi BTL). Is there any chance that you can upgrade to the newer OpenFabrics kernel/user stack? All the IB vendors support it for their HPC customers. FWIW: all Open MPI InfiniBand work is being done in support of OpenFabrics; the "mvapi" BTL is only maintained for backward compatibility and has had no new work done on it in at least a year. See http://www.open- mpi.org/faq/?category=openfabrics#vapi-support. 4. Note that depending on your application (e.g., if it primarily sends large messages), it *may* be faster to use the DMA engine in your IB interface and not use Open MPI's shared memory interface. But there are a lot of factors involved here, such as the size of your typical messages, how many processes you run per host (i.e., I'm assuming you have one HCA that would need to service all the processes), etc. On Jun 10, 2007, at 6:04 PM, Ralph Castain wrote: Hi Markus There are two MCA params that can help you, I believe: 1. You to set the maximum size of the shared memory file with -mca mpool_sm_max_size xxx where xxx is the maximum memory file you want, expressed in bytes. The default value I see is 512MBytes. 2. You can set the size/peer of the file, again in bytes: -mca mpool_sm_per_peer_size xxx This will allocate a file that is xxx * num_procs_on_the_node on each node, up to the maximum file size (either the 512MB default or whatever you specified using the previous param). This defaults to 32MBytes/proc. I see that there is also a minimum (total, not per-proc) file size that defaults to 128MBytes. If that is still too large, you can adjust it using -mca mpool_sm_min_size yyy Hope that helps Ralph On 6/10/07 2:55 PM, "Markus Daene" halle.de> wrote: Dear all, I hope I am in the correct mailing list with my problem. I try to run openmpi with the gridengine(6.0u10, 6.1). Therefore I compiled openmpi (1.2.2), which has the gridengine support included, I have checked it with ompi_info. In principle, openmpi runs well. The gridengine is configured such that the user has to specify the memory consumption via the h_vmem option. Then I noticed that with a larger number of processes the job is killed by the gridengine because of taking too much memory. To take a closer look on that, I wrote a small and simple (Fortran) MPI program which has just a MPI_Init and a (static) array, in my case of 50MB, then the programm goes into a (infinite) loop, because it takes some time until the gridengine reports the maxvmem. I found, that if the processes run all on different nodes, there is only a offset per process, at least a linear scaling. But it becomes worse when the jobs run on one node. There it seems to be a quadratic scaling with the offset, in my case about 30M. I made a list of the virtual memory reported by the gridengine, I was running on a 16 processor node: #N procvirt. Mem[MB] 1 182 2 468 3 825 4 1065 5 1001 6 1378 7 1817 8 2303 12 4927 16 8559 the pure program should need N*50MB, for 16 it is only 800M, but it takes 10 times more, >7GB!!! Of course, the gridengine will kills the job is this overtaking is not taken into accout, because of too much virtual memory consumption. The momory consumtion is not related to the grid engine, it is the same if I run from the command line. I guess it might be related to the 'sm' component of the btl. Is it possible to avoid the quadratic scaling? Of course I could use the vapi/tcp component only like mpirun --mca btl mvapi -np 16 ./my_test_program in this case the virtual memory is fine, but it will not be what one wants on a smp node. then it becomes ever worse: openmpi nicely report the (max./act.) used virtual memory to the grid engine as sum of all processes. This value is the compared with the one the user has specified with the h_vmem option, but the gridengine takes this value per process for the allocation of the job (works) and does not multiply this with the number of processes. Maybe one should report this to the gridenging mailing list, but it could be related as well for the openmpi interface. The last thing I noticed: It seems that if the v_mem option for gridengine jobs is specified like '2.0G' my test job was immedialtely killed; but when I specify '2000M' (which is obviously less) it work. The gridengine puts the job allways on the correct node as requested, but I think there is might be a problem in th
Re: [OMPI devel] Patch to fix cross-compile failure
Paul -- Excellent; many thanks! Brian: this patch looks good to me, but I defer to the unOfficial OMPI ROMIO Maintainer (uOORM)... On Jun 8, 2007, at 3:33 PM, Paul Henning wrote: I've attached a patch relative to the revision 14962 version of ompi/mca/io/romio/romio/configure.in that fixes configuration errors when doing a cross-compile. It simply changes some tests for the number of parameters to aio_suspend and aio_write from AC_TRY_RUN to AC_TRY_COMPILE. Paul ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres Cisco Systems
Re: [OMPI devel] large virtual memory consumption on smp nodes and gridengine problems
Markus Daene wrote: then it becomes ever worse: openmpi nicely report the (max./act.) used virtual memory to the grid engine as sum of all processes. This value is the compared with the one the user has specified with the h_vmem option, but the gridengine takes this value per process for the allocation of the job (works) and does not multiply this with the number of processes. Maybe one should report this to the gridenging mailing list, but it could be related as well for the openmpi interface. Hi Markus, From the SGE 6.1 man page, it shows the hard virtual memory limit (h_vmem) is for the virtual memory consumed by all processes in a job. I don't think SGE does the fine-grained resource limit for each process anyway. Maybe you can verify this with the grid engine mailing list to just to confirm. N1GE 6 Last change: 2007/02/14 13:34:15 12 N1 Grid Engine File Formats QUEUE_CONF(5) The resource limit parameters s_vmem and h_vmem are imple- mented by N1 Grid Engine as a job limit. They impose a limit on the amount of combined virtual memory consumed by all the processes in the job. If h_vmem is exceeded by a job running in the queue, it is aborted via a SIGKILL signal (see kill(1)). If s_vmem is exceeded, the job is sent a SIGXCPU signal which can be caught by the job. If you wish to allow a job to be "warned" so it can exit gracefully before it is killed then you should set the s_vmem limit to a lower value than h_vmem. For parallel processes, the limit is applied per slot which means that the limit is multiplied by the number of slots being used by the job before being applied. ... h_cpu The per-job CPU time limit in seconds. h_dataThe per-job maximum memory limit in bytes. h_vmemThe same as h_data (if both are set the minimum is used). The last thing I noticed: It seems that if the v_mem option for gridengine jobs is specified like '2.0G' my test job was immedialtely killed; but when I specify '2000M' (which is obviously less) it work. The gridengine puts the job allways on the correct node as requested, but I think there is might be a problem in the openmpi interface. You should email the grid engine alias. This sounds like a SGE bug to me. It would be nice if someone could give some hints how to avoid the quadratic scaling or maybe to think if this is really neccessary in openmpi. Thanks. Markus Daene my compiling options: ./configure --prefix=/not_important --enable-static --with-f90-size=medium --with-f90-max-array-dim=7 --with-mpi-para m-check=always --enable-cxx-exceptions --with-mvapi --enable-mca-no-build=btl-tcp ompi_info output: Open MPI: 1.2.2 Open MPI SVN revision: r14613 Open RTE: 1.2.2 Open RTE SVN revision: r14613 OPAL: 1.2.2 OPAL SVN revision: r14613 Prefix: /usrurz/openmpi/1.2.2/pathscale_3.0 Configured architecture: x86_64-unknown-linux-gnu Configured by: root Configured on: Mon Jun 4 16:04:38 CEST 2007 Configure host: GE1N01 Built by: root Built on: Mon Jun 4 16:09:37 CEST 2007 Built host: GE1N01 C bindings: yes C++ bindings: yes Fortran77 bindings: yes (all) Fortran90 bindings: yes Fortran90 bindings size: small C compiler: pathcc C compiler absolute: /usrurz/pathscale/bin/pathcc C++ compiler: pathCC C++ compiler absolute: /usrurz/pathscale/bin/pathCC Fortran77 compiler: pathf90 Fortran77 compiler abs: /usrurz/pathscale/bin/pathf90 Fortran90 compiler: pathf90 Fortran90 compiler abs: /usrurz/pathscale/bin/pathf90 C profiling: yes C++ profiling: yes Fortran77 profiling: yes Fortran90 profiling: yes C++ exceptions: yes Thread support: posix (mpi: no, progress: no) Internal debug support: no MPI parameter check: always Memory profiling support: no Memory debugging support: no libltdl support: yes Heterogeneous support: yes mpirun default --prefix: no MCA backtrace: execinfo (MCA v1.0, API v1.0, Component v1.2.2) MCA memory: ptmalloc2 (MCA v1.0, API v1.0, Component v1.2.2) MCA paffinity: linux (MCA v1.0, API v1.0, Component v1.2.2) MCA maffinity: first_use (MCA v1.0, API v1.0, Component v1.2.2) MCA maffinity: libnuma (MCA v1.0, API v1.0, Component v1.2.2) MCA timer: linux (MCA v1.0, API v1.0, Component v1.2.2) MCA installdirs: env (MCA v1.0, API v1.0, Component v1.2.2) MCA installdirs: config (MCA v1.0, API v1.0, Component v1.2.2) MCA allocator: basic (MCA v1.0, API v1.0, Component v1.0) MCA allocator: bucket (MCA v1.0, API v1.0, Component v1.0)
[OMPI devel] threaded builds
Per the teleconf last week, I have started to revamp the Cisco MTT
infrastructure to do simplistic thread testing. Specifically, I'm
building the OMPI trunk and v1.2 branches with "--with-threads --
enable-mpi-threads".
I haven't switched this into my production MTT setup yet, but in the
first trial runs, I'm noticing a segv in the test/threads/
opal_condition program.
It seems that in the thr1 test on the v1.2 branch, when it calls
opal_progress() underneath the condition variable wait, at some point
in there current_base is getting to be NULL. Hence, the following
segv's because the passed in value of "base" is NULL (event.c):
int
opal_event_base_loop(struct event_base *base, int flags)
{
const struct opal_eventop *evsel = base->evsel;
...
Here's the full call stack:
#0 0x002a955a020e in opal_event_base_loop (base=0x0, flags=5)
at event.c:520
#1 0x002a955a01f9 in opal_event_loop (flags=5) at event.c:514
#2 0x002a95599111 in opal_progress () at runtime/opal_progress.c:
259
#3 0x004012c8 in opal_condition_wait (c=0x5025a0, m=0x502600)
at ../../opal/threads/condition.h:81
#4 0x00401146 in thr1_run (obj=0x503110) at opal_condition.c:46
#5 0x0036e290610a in start_thread () from /lib64/tls/
libpthread.so.0
#6 0x0036e1ec68c3 in clone () from /lib64/tls/libc.so.6
#7 0x in ?? ()
This test seems to work fine on the trunk (at least, it didn't segv
in my small number of trail runs).
Is this a known problem in the 1.2 branch? Should I skip the thread
testing on the 1.2 branch and concentrate on the trunk?
--
Jeff Squyres
Cisco Systems
Re: [OMPI devel] threaded builds
I had similar problems on the trunk, which was fixed by Brian with r14877.
Perhaps 1.2 needs something similar?
Tim
On Monday 11 June 2007 10:08:15 am Jeff Squyres wrote:
> Per the teleconf last week, I have started to revamp the Cisco MTT
> infrastructure to do simplistic thread testing. Specifically, I'm
> building the OMPI trunk and v1.2 branches with "--with-threads --
> enable-mpi-threads".
>
> I haven't switched this into my production MTT setup yet, but in the
> first trial runs, I'm noticing a segv in the test/threads/
> opal_condition program.
>
> It seems that in the thr1 test on the v1.2 branch, when it calls
> opal_progress() underneath the condition variable wait, at some point
> in there current_base is getting to be NULL. Hence, the following
> segv's because the passed in value of "base" is NULL (event.c):
>
> int
> opal_event_base_loop(struct event_base *base, int flags)
> {
> const struct opal_eventop *evsel = base->evsel;
> ...
>
> Here's the full call stack:
>
> #0 0x002a955a020e in opal_event_base_loop (base=0x0, flags=5)
> at event.c:520
> #1 0x002a955a01f9 in opal_event_loop (flags=5) at event.c:514
> #2 0x002a95599111 in opal_progress () at runtime/opal_progress.c:
> 259
> #3 0x004012c8 in opal_condition_wait (c=0x5025a0, m=0x502600)
> at ../../opal/threads/condition.h:81
> #4 0x00401146 in thr1_run (obj=0x503110) at opal_condition.c:46
> #5 0x0036e290610a in start_thread () from /lib64/tls/
> libpthread.so.0
> #6 0x0036e1ec68c3 in clone () from /lib64/tls/libc.so.6
> #7 0x in ?? ()
>
> This test seems to work fine on the trunk (at least, it didn't segv
> in my small number of trail runs).
>
> Is this a known problem in the 1.2 branch? Should I skip the thread
> testing on the 1.2 branch and concentrate on the trunk?
Re: [OMPI devel] threaded builds
Yes, this is a known issue. I don't know -- are we trying to make
threads work on the 1.2 branch, or just the trunk? I had thought
just the trunk?
Brian
On Jun 11, 2007, at 8:13 AM, Tim Prins wrote:
I had similar problems on the trunk, which was fixed by Brian with
r14877.
Perhaps 1.2 needs something similar?
Tim
On Monday 11 June 2007 10:08:15 am Jeff Squyres wrote:
Per the teleconf last week, I have started to revamp the Cisco MTT
infrastructure to do simplistic thread testing. Specifically, I'm
building the OMPI trunk and v1.2 branches with "--with-threads --
enable-mpi-threads".
I haven't switched this into my production MTT setup yet, but in the
first trial runs, I'm noticing a segv in the test/threads/
opal_condition program.
It seems that in the thr1 test on the v1.2 branch, when it calls
opal_progress() underneath the condition variable wait, at some point
in there current_base is getting to be NULL. Hence, the following
segv's because the passed in value of "base" is NULL (event.c):
int
opal_event_base_loop(struct event_base *base, int flags)
{
const struct opal_eventop *evsel = base->evsel;
...
Here's the full call stack:
#0 0x002a955a020e in opal_event_base_loop (base=0x0, flags=5)
at event.c:520
#1 0x002a955a01f9 in opal_event_loop (flags=5) at event.c:514
#2 0x002a95599111 in opal_progress () at runtime/
opal_progress.c:
259
#3 0x004012c8 in opal_condition_wait (c=0x5025a0,
m=0x502600)
at ../../opal/threads/condition.h:81
#4 0x00401146 in thr1_run (obj=0x503110) at
opal_condition.c:46
#5 0x0036e290610a in start_thread () from /lib64/tls/
libpthread.so.0
#6 0x0036e1ec68c3 in clone () from /lib64/tls/libc.so.6
#7 0x in ?? ()
This test seems to work fine on the trunk (at least, it didn't segv
in my small number of trail runs).
Is this a known problem in the 1.2 branch? Should I skip the thread
testing on the 1.2 branch and concentrate on the trunk?
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
Re: [OMPI devel] Patch to fix cross-compile failure
Argonne used AC_TRY_RUN instead of AC_TRY_COMPILE (I believe) because there are some places where aio functions behaved badly (returned ENOTIMPL or something). I was going to make it call AC_TRY_RUN if we weren't cross-compiling and AC_TRY_COMPILE if we were. I'll commit something this evening. Brian On Jun 11, 2007, at 6:07 AM, Jeff Squyres wrote: Paul -- Excellent; many thanks! Brian: this patch looks good to me, but I defer to the unOfficial OMPI ROMIO Maintainer (uOORM)... On Jun 8, 2007, at 3:33 PM, Paul Henning wrote: I've attached a patch relative to the revision 14962 version of ompi/mca/io/romio/romio/configure.in that fixes configuration errors when doing a cross-compile. It simply changes some tests for the number of parameters to aio_suspend and aio_write from AC_TRY_RUN to AC_TRY_COMPILE. Paul ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres Cisco Systems ___ devel mailing list de...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/devel
Re: [OMPI devel] threaded builds
I think that 1.2 is a lost cause in this regard - I thought we were just
looking forward on the trunk.
On 6/11/07 8:17 AM, "Brian Barrett" wrote:
> Yes, this is a known issue. I don't know -- are we trying to make
> threads work on the 1.2 branch, or just the trunk? I had thought
> just the trunk?
>
> Brian
>
>
> On Jun 11, 2007, at 8:13 AM, Tim Prins wrote:
>
>> I had similar problems on the trunk, which was fixed by Brian with
>> r14877.
>>
>> Perhaps 1.2 needs something similar?
>>
>> Tim
>>
>> On Monday 11 June 2007 10:08:15 am Jeff Squyres wrote:
>>> Per the teleconf last week, I have started to revamp the Cisco MTT
>>> infrastructure to do simplistic thread testing. Specifically, I'm
>>> building the OMPI trunk and v1.2 branches with "--with-threads --
>>> enable-mpi-threads".
>>>
>>> I haven't switched this into my production MTT setup yet, but in the
>>> first trial runs, I'm noticing a segv in the test/threads/
>>> opal_condition program.
>>>
>>> It seems that in the thr1 test on the v1.2 branch, when it calls
>>> opal_progress() underneath the condition variable wait, at some point
>>> in there current_base is getting to be NULL. Hence, the following
>>> segv's because the passed in value of "base" is NULL (event.c):
>>>
>>> int
>>> opal_event_base_loop(struct event_base *base, int flags)
>>> {
>>> const struct opal_eventop *evsel = base->evsel;
>>> ...
>>>
>>> Here's the full call stack:
>>>
>>> #0 0x002a955a020e in opal_event_base_loop (base=0x0, flags=5)
>>> at event.c:520
>>> #1 0x002a955a01f9 in opal_event_loop (flags=5) at event.c:514
>>> #2 0x002a95599111 in opal_progress () at runtime/
>>> opal_progress.c:
>>> 259
>>> #3 0x004012c8 in opal_condition_wait (c=0x5025a0,
>>> m=0x502600)
>>> at ../../opal/threads/condition.h:81
>>> #4 0x00401146 in thr1_run (obj=0x503110) at
>>> opal_condition.c:46
>>> #5 0x0036e290610a in start_thread () from /lib64/tls/
>>> libpthread.so.0
>>> #6 0x0036e1ec68c3 in clone () from /lib64/tls/libc.so.6
>>> #7 0x in ?? ()
>>>
>>> This test seems to work fine on the trunk (at least, it didn't segv
>>> in my small number of trail runs).
>>>
>>> Is this a known problem in the 1.2 branch? Should I skip the thread
>>> testing on the 1.2 branch and concentrate on the trunk?
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
Re: [OMPI devel] threaded builds
I leave it to the thread subgroup to decide... Should we discuss on
the call tomorrow?
I don't have a strong opinion; I was just testing both because it was
easy to do so. If we want to concentrate on the trunk, I can adjust
my MTT setup.
On Jun 11, 2007, at 10:17 AM, Brian Barrett wrote:
Yes, this is a known issue. I don't know -- are we trying to make
threads work on the 1.2 branch, or just the trunk? I had thought
just the trunk?
Brian
On Jun 11, 2007, at 8:13 AM, Tim Prins wrote:
I had similar problems on the trunk, which was fixed by Brian with
r14877.
Perhaps 1.2 needs something similar?
Tim
On Monday 11 June 2007 10:08:15 am Jeff Squyres wrote:
Per the teleconf last week, I have started to revamp the Cisco MTT
infrastructure to do simplistic thread testing. Specifically, I'm
building the OMPI trunk and v1.2 branches with "--with-threads --
enable-mpi-threads".
I haven't switched this into my production MTT setup yet, but in the
first trial runs, I'm noticing a segv in the test/threads/
opal_condition program.
It seems that in the thr1 test on the v1.2 branch, when it calls
opal_progress() underneath the condition variable wait, at some
point
in there current_base is getting to be NULL. Hence, the following
segv's because the passed in value of "base" is NULL (event.c):
int
opal_event_base_loop(struct event_base *base, int flags)
{
const struct opal_eventop *evsel = base->evsel;
...
Here's the full call stack:
#0 0x002a955a020e in opal_event_base_loop (base=0x0, flags=5)
at event.c:520
#1 0x002a955a01f9 in opal_event_loop (flags=5) at event.c:514
#2 0x002a95599111 in opal_progress () at runtime/
opal_progress.c:
259
#3 0x004012c8 in opal_condition_wait (c=0x5025a0,
m=0x502600)
at ../../opal/threads/condition.h:81
#4 0x00401146 in thr1_run (obj=0x503110) at
opal_condition.c:46
#5 0x0036e290610a in start_thread () from /lib64/tls/
libpthread.so.0
#6 0x0036e1ec68c3 in clone () from /lib64/tls/libc.so.6
#7 0x in ?? ()
This test seems to work fine on the trunk (at least, it didn't segv
in my small number of trail runs).
Is this a known problem in the 1.2 branch? Should I skip the thread
testing on the 1.2 branch and concentrate on the trunk?
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
--
Jeff Squyres
Cisco Systems
Re: [OMPI devel] threaded builds
On Jun 11, 2007, at 8:25 AM, Jeff Squyres wrote:
I leave it to the thread subgroup to decide... Should we discuss on
the call tomorrow?
I don't have a strong opinion; I was just testing both because it was
easy to do so. If we want to concentrate on the trunk, I can adjust
my MTT setup.
I think trying to worry about 1.2 would just be a time sink. We know
that there are architectural issues with threads in some parts of the
code. I don't see us re-architecting 1.2 in this regard.
Seems we should only focus on the trunk.
- Galen
On Jun 11, 2007, at 10:17 AM, Brian Barrett wrote:
Yes, this is a known issue. I don't know -- are we trying to make
threads work on the 1.2 branch, or just the trunk? I had thought
just the trunk?
Brian
On Jun 11, 2007, at 8:13 AM, Tim Prins wrote:
I had similar problems on the trunk, which was fixed by Brian with
r14877.
Perhaps 1.2 needs something similar?
Tim
On Monday 11 June 2007 10:08:15 am Jeff Squyres wrote:
Per the teleconf last week, I have started to revamp the Cisco MTT
infrastructure to do simplistic thread testing. Specifically, I'm
building the OMPI trunk and v1.2 branches with "--with-threads --
enable-mpi-threads".
I haven't switched this into my production MTT setup yet, but in
the
first trial runs, I'm noticing a segv in the test/threads/
opal_condition program.
It seems that in the thr1 test on the v1.2 branch, when it calls
opal_progress() underneath the condition variable wait, at some
point
in there current_base is getting to be NULL. Hence, the following
segv's because the passed in value of "base" is NULL (event.c):
int
opal_event_base_loop(struct event_base *base, int flags)
{
const struct opal_eventop *evsel = base->evsel;
...
Here's the full call stack:
#0 0x002a955a020e in opal_event_base_loop (base=0x0, flags=5)
at event.c:520
#1 0x002a955a01f9 in opal_event_loop (flags=5) at event.c:514
#2 0x002a95599111 in opal_progress () at runtime/
opal_progress.c:
259
#3 0x004012c8 in opal_condition_wait (c=0x5025a0,
m=0x502600)
at ../../opal/threads/condition.h:81
#4 0x00401146 in thr1_run (obj=0x503110) at
opal_condition.c:46
#5 0x0036e290610a in start_thread () from /lib64/tls/
libpthread.so.0
#6 0x0036e1ec68c3 in clone () from /lib64/tls/libc.so.6
#7 0x in ?? ()
This test seems to work fine on the trunk (at least, it didn't segv
in my small number of trail runs).
Is this a known problem in the 1.2 branch? Should I skip the
thread
testing on the 1.2 branch and concentrate on the trunk?
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
--
Jeff Squyres
Cisco Systems
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
Re: [OMPI devel] threaded builds
I would second this - thread safety should be a 1.3 item, unless someone has a
lot of spare time.
Rich
-Original Message-
From: devel-boun...@open-mpi.org
To: Open MPI Developers
Sent: Mon Jun 11 10:44:33 2007
Subject: Re: [OMPI devel] threaded builds
On Jun 11, 2007, at 8:25 AM, Jeff Squyres wrote:
> I leave it to the thread subgroup to decide... Should we discuss on
> the call tomorrow?
>
> I don't have a strong opinion; I was just testing both because it was
> easy to do so. If we want to concentrate on the trunk, I can adjust
> my MTT setup.
>
I think trying to worry about 1.2 would just be a time sink. We know
that there are architectural issues with threads in some parts of the
code. I don't see us re-architecting 1.2 in this regard.
Seems we should only focus on the trunk.
- Galen
>
> On Jun 11, 2007, at 10:17 AM, Brian Barrett wrote:
>
>> Yes, this is a known issue. I don't know -- are we trying to make
>> threads work on the 1.2 branch, or just the trunk? I had thought
>> just the trunk?
>>
>> Brian
>>
>>
>> On Jun 11, 2007, at 8:13 AM, Tim Prins wrote:
>>
>>> I had similar problems on the trunk, which was fixed by Brian with
>>> r14877.
>>>
>>> Perhaps 1.2 needs something similar?
>>>
>>> Tim
>>>
>>> On Monday 11 June 2007 10:08:15 am Jeff Squyres wrote:
Per the teleconf last week, I have started to revamp the Cisco MTT
infrastructure to do simplistic thread testing. Specifically, I'm
building the OMPI trunk and v1.2 branches with "--with-threads --
enable-mpi-threads".
I haven't switched this into my production MTT setup yet, but in
the
first trial runs, I'm noticing a segv in the test/threads/
opal_condition program.
It seems that in the thr1 test on the v1.2 branch, when it calls
opal_progress() underneath the condition variable wait, at some
point
in there current_base is getting to be NULL. Hence, the following
segv's because the passed in value of "base" is NULL (event.c):
int
opal_event_base_loop(struct event_base *base, int flags)
{
const struct opal_eventop *evsel = base->evsel;
...
Here's the full call stack:
#0 0x002a955a020e in opal_event_base_loop (base=0x0, flags=5)
at event.c:520
#1 0x002a955a01f9 in opal_event_loop (flags=5) at event.c:514
#2 0x002a95599111 in opal_progress () at runtime/
opal_progress.c:
259
#3 0x004012c8 in opal_condition_wait (c=0x5025a0,
m=0x502600)
at ../../opal/threads/condition.h:81
#4 0x00401146 in thr1_run (obj=0x503110) at
opal_condition.c:46
#5 0x0036e290610a in start_thread () from /lib64/tls/
libpthread.so.0
#6 0x0036e1ec68c3 in clone () from /lib64/tls/libc.so.6
#7 0x in ?? ()
This test seems to work fine on the trunk (at least, it didn't segv
in my small number of trail runs).
Is this a known problem in the 1.2 branch? Should I skip the
thread
testing on the 1.2 branch and concentrate on the trunk?
>>> ___
>>> devel mailing list
>>> de...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>
> --
> Jeff Squyres
> Cisco Systems
>
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
Re: [OMPI devel] threaded builds
If Jeff has the resources to run threaded tests against 1.2, *and* to
examine the results, then it might be valuable to have a summary the
known threading issues in 1.2 written down somewhere for the benefit of
those who don't chase the trunk.
-Paul
Graham, Richard L. wrote:
I would second this - thread safety should be a 1.3 item, unless someone has a
lot of spare time.
Rich
-Original Message-
From: devel-boun...@open-mpi.org
To: Open MPI Developers
Sent: Mon Jun 11 10:44:33 2007
Subject: Re: [OMPI devel] threaded builds
On Jun 11, 2007, at 8:25 AM, Jeff Squyres wrote:
I leave it to the thread subgroup to decide... Should we discuss on
the call tomorrow?
I don't have a strong opinion; I was just testing both because it was
easy to do so. If we want to concentrate on the trunk, I can adjust
my MTT setup.
I think trying to worry about 1.2 would just be a time sink. We know
that there are architectural issues with threads in some parts of the
code. I don't see us re-architecting 1.2 in this regard.
Seems we should only focus on the trunk.
- Galen
On Jun 11, 2007, at 10:17 AM, Brian Barrett wrote:
Yes, this is a known issue. I don't know -- are we trying to make
threads work on the 1.2 branch, or just the trunk? I had thought
just the trunk?
Brian
On Jun 11, 2007, at 8:13 AM, Tim Prins wrote:
I had similar problems on the trunk, which was fixed by Brian with
r14877.
Perhaps 1.2 needs something similar?
Tim
On Monday 11 June 2007 10:08:15 am Jeff Squyres wrote:
Per the teleconf last week, I have started to revamp the Cisco MTT
infrastructure to do simplistic thread testing. Specifically, I'm
building the OMPI trunk and v1.2 branches with "--with-threads --
enable-mpi-threads".
I haven't switched this into my production MTT setup yet, but in
the
first trial runs, I'm noticing a segv in the test/threads/
opal_condition program.
It seems that in the thr1 test on the v1.2 branch, when it calls
opal_progress() underneath the condition variable wait, at some
point
in there current_base is getting to be NULL. Hence, the following
segv's because the passed in value of "base" is NULL (event.c):
int
opal_event_base_loop(struct event_base *base, int flags)
{
const struct opal_eventop *evsel = base->evsel;
...
Here's the full call stack:
#0 0x002a955a020e in opal_event_base_loop (base=0x0, flags=5)
at event.c:520
#1 0x002a955a01f9 in opal_event_loop (flags=5) at event.c:514
#2 0x002a95599111 in opal_progress () at runtime/
opal_progress.c:
259
#3 0x004012c8 in opal_condition_wait (c=0x5025a0,
m=0x502600)
at ../../opal/threads/condition.h:81
#4 0x00401146 in thr1_run (obj=0x503110) at
opal_condition.c:46
#5 0x0036e290610a in start_thread () from /lib64/tls/
libpthread.so.0
#6 0x0036e1ec68c3 in clone () from /lib64/tls/libc.so.6
#7 0x in ?? ()
This test seems to work fine on the trunk (at least, it didn't segv
in my small number of trail runs).
Is this a known problem in the 1.2 branch? Should I skip the
thread
testing on the 1.2 branch and concentrate on the trunk?
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
--
Jeff Squyres
Cisco Systems
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
--
Paul H. Hargrove phhargr...@lbl.gov
Future Technologies Group
HPC Research Department Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
