Re: [OMPI devel] subcommunicator OpenMPI issues on K

[Edgar Gabriel]

My guess would be that both aspects (sorting + CID allocation) could be 
a problem.


There was a loong time back an effort to convert the sequence of 
allgather + qsort into a distributed sort (based on a paper by Moody et. 
al. where he demonstrated the benefits of this approach).  We didn't get 
unfortunately to implement this part due to timing constraints.


Also, we used to have a block allocation algorithm for CIDs that did 
save quite a bit in communication costs, but we had to abandon it 
because the implementation was wasting CIDs by not properly reusing 
them. It is fairly clear on how to implement it correctly, it just 
hasn't been done for a lack of resources.


Maybe we should revisit those items at one point in time.
Thanks
Edgar

On 11/7/2017 10:23 AM, George Bosilca wrote:

Samuel,

You are right, we use qsort to sort the keys, but the qsort only 
applies on participants with the same color. So while the complexity 
of the qsort might reach bottom only when most of the processes 
participate with the same color.


What I think is OMPI problem in this are is the selection of the next 
cid for the newly created communicator. We are doing the selection of 
the cid on the original communicator, and this basically counts for a 
significant increase in the duration, as will need to iterate a longer 
to converge to a common cid.


We haven't made any improvement in this area for the last few years, 
we simply transformed the code to use non-blocking communications 
instead of blocking, but this has little impact on the performance of 
the split itself.


  George.


On Tue, Nov 7, 2017 at 10:52 AM, Samuel Williams > wrote:


I'll ask my collaborators if they've submitted a ticket.
(they have the accounts; built the code; ran the code; observed
the issues)

I believe the issue on MPICH was a qsort issue and not a Allreduce
issue.  When this is coupled with the fact that it looked like
qsort is called in ompi_comm_split

(https://github.com/open-mpi/ompi/blob/a7a30424cba6482c97f8f2f7febe53aaa180c91e/ompi/communicator/comm.c

),
I wanted to raise the issue so that it may be investigated to
understand whether users can naively blunder into worst case
computational complexity issues.

We've been running hpgmg-fv (not -fe).  They were using the flux
variants (requires local.mk  build
operators.flux.c instead of operators.fv4.c) and they are a couple
commits behind.  Regardless, this issue has persisted on K for
several years.  By default, it will build log(N) subcommunicators
where N is the problem size.  Weak scaling experiments has shown
comm_split/dup times growing consistently with worst case
complexity.  That being said, AMR codes might rebuild the sub
communicators as they regrid/adapt.








> On Nov 7, 2017, at 8:33 AM, Gilles Gouaillardet
> wrote:
>
> Samuel,
>
> The default MPI library on the K computer is Fujitsu MPI, and
yes, it
> is based on Open MPI.
> /* fwiw, an alternative is RIKEN MPI, and it is MPICH based */
> From a support perspective, this should be reported to the HPCI
> helpdesk http://www.hpci-office.jp/pages/e_support

>
> As far as i understand, Fujitsu MPI currently available on K is not
> based on the latest Open MPI.
> I suspect most of the time is spent trying to find the new
> communicator ID (CID) when a communicator is created (vs
figuring out
> the new ranks)
> iirc, on older versions of Open MPI, that was implemented with
as many
> MPI_Allreduce(MPI_MAX) as needed to figure out the smallest common
> unused CID for the newly created communicator.
>
> So if you MPI_Comm_dup(MPI_COMM_WORLD) n times at the beginning of
> your program, only one MPI_Allreduce() should be involved per
> MPI_Comm_dup().
> But if you do the same thing in the middle of your run, and
after each
> rank has a different lower unused CID, the performances can be
(much)
> worst.
> If i understand correctly your description of the issue, that would
> explain the performance discrepancy between static vs dynamic
> communicator creation time.
>
> fwiw, this part has been (highly) improved in the latest
releases of Open MPI.
>
> If your benchmark is available for download, could you please
post a link ?
>
>
> Cheers,
>
> Gilles
>
> On Wed, Nov 8, 2017 at 12:04 AM, Samuel Williams
> wrote:
>> Some of my collaborators have had issues with one of my
benchmarks at high concurrency (82K MPI procs) 

Re: [OMPI devel] #warning "Including liblustreapi.h is deprecated. Include lustreapi.h directly."

[Edgar Gabriel]

I'll put it on my to do list to write the configure logic for that, shouldn't be too difficult. Thanks for the report.Edgar___
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel

Re: [OMPI devel] 2.0.0rc4 Crash in MPI_File_write_all_end

[Edgar Gabriel]

I think the decision was made to postpone the patch to 2.0.1, since the 
release of 2.0.0 is eminent.


Thanks
Edgar

On 7/12/2016 2:51 PM, Eric Chamberland wrote:

Hi Edgard,

I just saw that your patch got into ompi/master... any chances it goes
into ompi-release/v2.x before rc5?

thanks,

Eric


On 08/07/16 03:14 PM, Edgar Gabriel wrote:

I think I found the problem, I filed a pr towards master, and if that
passes I will file a pr for the 2.x branch.

Thanks!
Edgar


On 7/8/2016 1:14 PM, Eric Chamberland wrote:

On 08/07/16 01:44 PM, Edgar Gabriel wrote:

ok, but just to be able to construct a test case, basically what you are
doing is

MPI_File_write_all_begin (fh, NULL, 0, some datatype);

MPI_File_write_all_end (fh, NULL, ),

is this correct?

Yes, but with 2 processes:

rank 0 writes something, but not rank 1...

other info: rank 0 didn't wait for rank1 after MPI_File_write_all_end so
it continued to the next MPI_File_write_all_begin with a different
datatype but on the same file...

thanks!

Eric
___
devel mailing list
de...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post:
http://www.open-mpi.org/community/lists/devel/2016/07/19173.php

___
devel mailing list
de...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2016/07/19192.php


--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
--

Re: [OMPI devel] 2.0.0rc4 Crash in MPI_File_write_all_end

[Edgar Gabriel]

I think I found the problem, I filed a pr towards master, and if that 
passes I will file a pr for the 2.x branch.


Thanks!
Edgar


On 7/8/2016 1:14 PM, Eric Chamberland wrote:


On 08/07/16 01:44 PM, Edgar Gabriel wrote:

ok, but just to be able to construct a test case, basically what you are
doing is

MPI_File_write_all_begin (fh, NULL, 0, some datatype);

MPI_File_write_all_end (fh, NULL, ),

is this correct?

Yes, but with 2 processes:

rank 0 writes something, but not rank 1...

other info: rank 0 didn't wait for rank1 after MPI_File_write_all_end so
it continued to the next MPI_File_write_all_begin with a different
datatype but on the same file...

thanks!

Eric
___
devel mailing list
de...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2016/07/19173.php


--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
--

Re: [OMPI devel] 2.0.0rc4 Crash in MPI_File_write_all_end

[Edgar Gabriel]

ok, but just to be able to construct a test case, basically what you are 
doing is


MPI_File_write_all_begin (fh, NULL, 0, some datatype);

MPI_File_write_all_end (fh, NULL, ),

is this correct?

Thanks

Edgar


On 7/8/2016 12:19 PM, Eric Chamberland wrote:

Hi,

On 08/07/16 12:52 PM, Edgar Gabriel wrote:

The default MPI I/O library has changed in the 2.x release to OMPIO for

ok, I am now doing I/O on my own hard drive... but I can test over NFS
easily.  For Lustre, I will have to produce a reduced example out of our
test suite...


most file systems. I can look into that problem, any chance to get
access to the testsuite that you mentioned?

Yikes! Sounds interesting, but difficult to realize...  Our in-house
code is not public... :/

I however proposed (to myself) to add a nightly compilation of openmpi
(see http://www.open-mpi.org/community/lists/users/2016/06/29515.php) so
I can report problems before releases are made...

Anyway, I will work on the little script to automate the
MPI+PETSc+InHouseCode combination so I get give you a feedback as soon
as you will propose me to test a patch...

Hoping this will be enough convenient for you...

Thanks!

Eric


Thanks
Edgar


On 7/8/2016 11:32 AM, Eric Chamberland wrote:

Hi,

I am testing for the first time the 2.X release candidate.

I have a segmentation violation using  MPI_File_write_all_end(MPI_File
fh, const void *buf, MPI_Status *status)

The "special" thing, may be that in the faulty test cases, there are
processes that haven't written anything, so they a a zero length buffer,
so the second parameter (buf) passed is a null pointer.

Until now, it was a valid call, has it changed?

Thanks,

Eric

FWIW: We are using our test suite (~2000 nightly tests) successfully
with openmpi-1.{6,8,10}.* and MPICH since many years...
___
devel mailing list
de...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post:
http://www.open-mpi.org/community/lists/devel/2016/07/19169.php

___
devel mailing list
de...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2016/07/19171.php


--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
--

Re: [OMPI devel] 2.0.0rc4 Crash in MPI_File_write_all_end

[Edgar Gabriel]

The default MPI I/O library has changed in the 2.x release to OMPIO for 
most file systems. I can look into that problem, any chance to get 
access to the testsuite that you mentioned?


Thanks
Edgar


On 7/8/2016 11:32 AM, Eric Chamberland wrote:

Hi,

I am testing for the first time the 2.X release candidate.

I have a segmentation violation using  MPI_File_write_all_end(MPI_File
fh, const void *buf, MPI_Status *status)

The "special" thing, may be that in the faulty test cases, there are
processes that haven't written anything, so they a a zero length buffer,
so the second parameter (buf) passed is a null pointer.

Until now, it was a valid call, has it changed?

Thanks,

Eric

FWIW: We are using our test suite (~2000 nightly tests) successfully
with openmpi-1.{6,8,10}.* and MPICH since many years...
___
devel mailing list
de...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2016/07/19169.php


--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
--

Re: [OMPI devel] parameters for OMPIO

[Edgar Gabriel]

the parameters on the webpage are for ompio in 2.x.  For 1.10 its a bit 
more complicated, you would have to set the number of aggregators for 
each fcoll component separatly, e.g.


--mca fcoll two_phase_num_io_procs  x

I would however start without setting the number of aggregators, since 
we do have some algorithms in ompio trying to predict the optimal number 
of aggregators. If you set these parameters, you are basically the 
algorithm. There might be very good reasons to override the algorithm 
(we are well aware of patterns where it will produce bad results), but 
the tile I/O patterns was one of the patterns it was originally designed 
for.


The coll_timing_info option is there in 1.10, but requires an additional 
change for  the compilation (that was supposed to be a configure flag, 
but we just haven't gotten around to implement the configure flag).  I 
would consider this however an option which is more of interest for 
developers than for users. If you want to use it anyway, I can tell you 
what you need to do to activate it.


Thanks
Edgar


On 5/11/2016 9:29 AM, Michael Rezny wrote:

Hi,
I am looking at the online FAQ for ompio
which seems to show that the following parameters are defined:
io_ompio_num_aggregators
io_ompio_call_timing

But on OMPI version 1.10.1 or 1.8.3:
1: setting mpirun -mca io ompio -mca io_ompio_coll_timing_info
appears to not produce a summary.

2: io_ompio_num_aggregators is not listed as a parameter
as listed by
ompi_info -a | grep ompio

Am I doing something wrong, or are these options not supported in 
these versions?


kindest regards
Mike


--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
--

Re: [OMPI devel] OMPIO vs ROMIO

[Edgar Gabriel]

in the 1.7, 1.8 and 1.10 series ROMIO remains the default. In the 
upcomgin 2.x series, OMPIO will be the default, except for Lustre file 
systems, where we will stick with ROMIO as the primary resource.


Regarding performance comparison, we ran numerous tests late last year 
and early this year. It really depends on the application scenario and 
the platform that you are using. If you want to know which one should 
you use, I would definitely suggest to stick with ROMIO in the 1.10 
series, since many of the bug fixes of OMPIO that we did in the last two 
years could not be back-ported to the 1.10 series for technical reasons. 
If you plan to switch to the 2.x series, it might be easiest to just run 
a couple of tests and compare the performance for your application on 
your platform, and base your decision on that.


Edgar


On 5/10/2016 6:32 AM, Sreenidhi Bharathkar Ramesh wrote:

Hi,

1. During default build of OpenMPI, it looks like both ompio.la 
 and romio.la  are built.  Which I/O 
MCA library is used and based on what is the decision taken ?


2. Are there any statistics available to compare these two - OMPIO vs 
ROMIO ?


I am using OpenMPI v1.10.1.

Thanks,
- Sreenidhi.


--

Re: [OMPI devel] [2.0.0rc2] NetBSD build failure (ompio)

[Edgar Gabriel]

well, actually the line that causes the trouble is unnecessary. 
Detecting the file system has been abstracted out, and the struct statfs 
is completely unused. I will create a one-line patch for that.


Edgar


On 5/2/2016 10:54 PM, Paul Hargrove wrote:
On NetNSD-7 I am testing PR1128 to get past an issue Nathan fixed 
since my report earlier today (Monday).
It appears that OMPIO is not prepared for NetBSD's uses of statvfs() 
instead of statfs().

The error message appear at the bottom of this email.

FWIW the ROMIO code does have the necessary configure login and #ifdefs

Of course, passing --enable-mca-no-build=io-ompio to configure is 
sufficient to work around this issue.


-Paul


libtool: compile:  gcc -std=gnu99 -DHAVE_CONFIG_H -I. 
-I/home/phargrov/OMPI/openmpi-pr1128-netbsd7-amd64/openmpi-gitclone/ompi/mca/io/ompio 
-I../../../../opal/include -I../../../../ompi/include 
-I../../../../oshmem/include 
-I../../../../opal/mca/hwloc/hwloc1112/hwloc/include/private/autogen 
-I../../../../opal/mca/hwloc/hwloc1112/hwloc/include/hwloc/autogen 
-I../../../../ompi/mpiext/cuda/c 
-I/home/phargrov/OMPI/openmpi-pr1128-netbsd7-amd64/openmpi-gitclone 
-I../../../.. 
-I/home/phargrov/OMPI/openmpi-pr1128-netbsd7-amd64/openmpi-gitclone/opal/include 
-I/home/phargrov/OMPI/openmpi-pr1128-netbsd7-amd64/openmpi-gitclone/orte/include 
-I../../../../orte/include 
-I/home/phargrov/OMPI/openmpi-pr1128-netbsd7-amd64/openmpi-gitclone/ompi/include 
-I/home/phargrov/OMPI/openmpi-pr1128-netbsd7-amd64/openmpi-gitclone/oshmem/include 
-I/home/phargrov/OMPI/openmpi-pr1128-netbsd7-amd64/openmpi-gitclone/opal/mca/hwloc/hwloc1112/hwloc/include 
-I/home/phargrov/OMPI/openmpi-pr1128-netbsd7-amd64/BLD/opal/mca/hwloc/hwloc1112/hwloc/include 
-I/home/phargrov/OMPI/openmpi-pr1128-netbsd7-amd64/openmpi-gitclone/opal/mca/event/libevent2022/libevent 
-I/home/phargrov/OMPI/openmpi-pr1128-netbsd7-amd64/openmpi-gitclone/opal/mca/event/libevent2022/libevent/include 
-I/home/phargrov/OMPI/openmpi-pr1128-netbsd7-amd64/BLD/opal/mca/event/libevent2022/libevent/include 
-g -finline-functions -fno-strict-aliasing -pthread -MT 
io_ompio_component.lo -MD -MP -MF .deps/io_ompio_component.Tpo -c 
/home/phargrov/OMPI/openmpi-pr1128-netbsd7-amd64/openmpi-gitclone/ompi/mca/io/ompio/io_ompio_component.c 
 -fPIC -DPIC -o .libs/io_ompio_component.o
/home/phargrov/OMPI/openmpi-pr1128-netbsd7-amd64/openmpi-gitclone/ompi/mca/io/ompio/io_ompio_component.c: 
In function 'file_query':
/home/phargrov/OMPI/openmpi-pr1128-netbsd7-amd64/openmpi-gitclone/ompi/mca/io/ompio/io_ompio_component.c:275:19: 
error: storage size of 'fsbuf' isn't known

 struct statfs fsbuf;
   ^
*** Error code 1

Stop.

--
Paul H. Hargrove phhargr...@lbl.gov <mailto:phhargr...@lbl.gov>
Computer Languages & Systems Software (CLaSS) Group
Computer Science Department   Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory Fax: +1-510-486-6900


--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
--

Re: [OMPI devel] UH jenkins node seems out for the holidays

[Edgar Gabriel]

the scheduled work on the power infrastructure is finished, the UH 
jenkins node is available again.

Happy new year to everybody
Edgar

On 12/30/2015 4:29 PM, Edgar Gabriel wrote:


I apologize, I completely forgot that t this node is affected. There 
is work on the electrical infrastructure of the building that hosts 
the main router to the cs machines. The machines themselves are still 
up and running, but cannot be reached. It should be finished tomorrow.


I'll keep you posted.
Edgar

On Dec 30, 2015 14:48, Howard Pritchard <hpprit...@gmail.com> wrote:

Hi Folks,

As those of you working on OMPI PRs this week have noticed,
it appears that Univ. Houston's CS department may have shut
a number of systems down for the holidays.

So, for now, ignore the status of the LANL-distcheck and LANL-dlopen
jenkins tests.  Hopefully the UH server(s) will be back on line
next week.

Howard



--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
--

Re: [OMPI devel] UH jenkins node seems out for the holidays

[Edgar Gabriel]

I apologize, I completely forgot that t this node is affected. There is work on the electrical infrastructure of the building that hosts the main router to the cs machines. The machines themselves are still up and running, but cannot be reached. It should be finished tomorrow.
I'll keep you posted.
Edgar
On Dec 30, 2015 14:48, Howard Pritchard  wrote:Hi Folks,As those of you working on OMPI PRs this week have noticed,it appears that Univ. Houston's CS department may have shuta number of systems down for the holidays.So, for now, ignore the status of the LANL-distcheck and LANL-dlopenjenkins tests.  Hopefully the UH server(s) will be back on linenext week.Howard

Re: [OMPI devel] interfaces gone?

[Edgar Gabriel]

argh. Forget about it. Sorry for the noise. linked to the wrong version :-(
Edgar

On 11/12/2015 11:13 AM, Edgar Gabriel wrote:

I have an interesting observation on master, for whatever reason the new
non-blocking collective I/O interfaces don't seem to be generated
anymore correctly. Does anybody have an idea what could cause that?

/home/gabriel/ompi-tests/mpi2basic_tests/file/read_all.c:416: undefined
reference to `MPI_File_iread_all'
/home/gabriel/ompi-tests/mpi2basic_tests/file/read_all.c:463: undefined
reference to `MPI_File_iread_at_all'
/home/gabriel/ompi-tests/mpi2basic_tests/file/read_all.c:464: undefined
reference to `MPI_File_iread_at_all'
write_all.o: In function `write_all_test':
/home/gabriel/ompi-tests/mpi2basic_tests/file/write_all.c:612: undefined
reference to `MPI_File_iwrite_all'
/home/gabriel/ompi-tests/mpi2basic_tests/file/write_all.c:667: undefined
reference to `MPI_File_iwrite_at_all'
/home/gabriel/ompi-tests/mpi2basic_tests/file/write_all.c:668: undefined
reference to `MPI_File_iwrite_at_all'


Thanks
Edgar



--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
--

[OMPI devel] interfaces gone?

[Edgar Gabriel]

I have an interesting observation on master, for whatever reason the new 
non-blocking collective I/O interfaces don't seem to be generated 
anymore correctly. Does anybody have an idea what could cause that?


/home/gabriel/ompi-tests/mpi2basic_tests/file/read_all.c:416: undefined 
reference to `MPI_File_iread_all'
/home/gabriel/ompi-tests/mpi2basic_tests/file/read_all.c:463: undefined 
reference to `MPI_File_iread_at_all'
/home/gabriel/ompi-tests/mpi2basic_tests/file/read_all.c:464: undefined 
reference to `MPI_File_iread_at_all'

write_all.o: In function `write_all_test':
/home/gabriel/ompi-tests/mpi2basic_tests/file/write_all.c:612: undefined 
reference to `MPI_File_iwrite_all'
/home/gabriel/ompi-tests/mpi2basic_tests/file/write_all.c:667: undefined 
reference to `MPI_File_iwrite_at_all'
/home/gabriel/ompi-tests/mpi2basic_tests/file/write_all.c:668: undefined 
reference to `MPI_File_iwrite_at_all'



Thanks
Edgar

--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
--

Re: [OMPI devel] inter vs. intra communicator problem on master

[Edgar Gabriel]

yes, I did fresh pull this morning, for me it deadlocks reliably for 2 
and more processes.


Thanks
Edgar

On 9/16/2015 10:42 AM, Nathan Hjelm wrote:


The reproducer is working for me with master on OX 10.10. Some changes
to ompi_comm_set went in yesterday. Are you on the latest hash?

-Nathan

On Wed, Sep 16, 2015 at 08:49:59AM -0500, Edgar Gabriel wrote:

something is borked right now on master in the management of inter vs. intra
communicators. It looks like intra communicators are wrongly selecting the
inter coll module thinking that it is an inter communicator, and we have
hangs because of that. I attach a small replicator, where a bcast of a
duplicate of MPI_COMM_WORLD hangs, because the inter collective module is
being selected.

Thanks
Edgar



#include 
#include "mpi.h"

int main( int argc, char *argv[] )
{
   MPI_Comm comm1;
   int root=0;
   int rank2, size2, global_buf=1;
   int rank, size;

   MPI_Init ( ,  );

   MPI_Comm_rank ( MPI_COMM_WORLD,  );
   MPI_Comm_size ( MPI_COMM_WORLD,  );

/* Setting up a new communicator */
   MPI_Comm_dup ( MPI_COMM_WORLD,  );

   MPI_Comm_size ( comm1,  );
   MPI_Comm_rank ( comm1,  );


   MPI_Bcast ( _buf, 1, MPI_INT, root, MPI_COMM_WORLD );
   if ( rank == root ) {
   printf("Bcast on MPI_COMM_WORLD finished\n");
   }
   MPI_Bcast ( _buf, 1, MPI_INT, root, comm1 );
   if ( rank == root ) {
   printf("Bcast on duplicate of MPI_COMM_WORLD finished\n");
   }

   MPI_Comm_free (  );

   MPI_Finalize ();
   return ( 0 );
}



___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2015/09/18040.php




___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2015/09/18042.php

[OMPI devel] inter vs. intra communicator problem on master

[Edgar Gabriel]

something is borked right now on master in the management of inter vs. 
intra communicators. It looks like intra communicators are wrongly 
selecting the inter coll module thinking that it is an inter 
communicator, and we have hangs because of that. I attach a small 
replicator, where a bcast of a duplicate of MPI_COMM_WORLD hangs, 
because the inter collective module is being selected.


Thanks
Edgar
#include 
#include "mpi.h"

int main( int argc, char *argv[] )
{
  MPI_Comm comm1;
  int root=0;
  int rank2, size2, global_buf=1;
  int rank, size;

  MPI_Init ( ,  );

  MPI_Comm_rank ( MPI_COMM_WORLD,  );
  MPI_Comm_size ( MPI_COMM_WORLD,  );

/* Setting up a new communicator */
  MPI_Comm_dup ( MPI_COMM_WORLD,  );

  MPI_Comm_size ( comm1,  );
  MPI_Comm_rank ( comm1,  );


  MPI_Bcast ( _buf, 1, MPI_INT, root, MPI_COMM_WORLD );
  if ( rank == root ) {
  printf("Bcast on MPI_COMM_WORLD finished\n");
  }
  MPI_Bcast ( _buf, 1, MPI_INT, root, comm1 );
  if ( rank == root ) {
  printf("Bcast on duplicate of MPI_COMM_WORLD finished\n");
  }

  MPI_Comm_free (  );

  MPI_Finalize ();
  return ( 0 );
}

Re: [OMPI devel] Branch for v2.0.0

[Edgar Gabriel]

I am halfway through the infrastructure changes required for the 
non-blocking collective I/O operations. It will take me probably 2 more 
days of coding to finish what I wanted to get done before the branch, 
but unfortunately I can not guarantee that I will get to it this week. 
Since it is very well isolated area in the code base, I think however 
that a pull request from master to the 2.0 branch will probably not be 
dramatic.


Edgar

On 6/8/2015 7:50 PM, Jeff Squyres (jsquyres) wrote:

Developers --

Per our new release scheme, it's (slightly past) time to branch for v2.0.0.

Howard and I would like to talk about branching tomorrow (9 June 2015) on the 
weekly webex:

- what features are incomplete?
- what bug fixes are coming in immanently?
- what else is needed before v2.0.0?
- how much time will these things take?
- generally: what still needs to be done for v2.0.0?

And then -- assuming all goes well -- actually branch next Tuesday (16 June 
2015) after the weekly webex.



--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
--

Re: [OMPI devel] change in io_ompio.c

[Edgar Gabriel]

sorry if I sounded too harsh, there was a real memory leak that you 
fixed, I agree to that.  I committed the correct version.


Thanks
Edgar

On 5/28/2015 9:00 AM, Gilles Gouaillardet wrote:

Edgar,

i am sorry about that.

i fixed some memory leaks (some memory was leaking in some error cases).
i also moved (up) some malloc in order to group them and simplify the
handling
of error cases.

per your comment, one move was incorrect indeed :-(

Cheers,

Gilles

On 5/28/2015 12:14 PM, Edgar Gabriel wrote:

ok, I see you moved the malloc up, the malloc was originally just for
the receiving side of the communication, you moved it up to cover
both. That is however unfortunately not correct.

I will fix it in a couple of mins.
Thanks
Edgar

On 5/28/2015 8:25 AM, Edgar Gabriel wrote:

Gilles,

I saw you a fixed a couple of the coverty warnings in ompio, but
unfortunately it also broke some things.

--
Question to you: in io_ompio.c line 2345, you introduced a malloc for
f_procs_in_group that was not there before. That array is allocated a
couple of lines earlier in the subroutine merge_groups

Since the values are not set, we segfault right away a couple of lines
later in the subsequent isend, where f_procs_in_group[0] simply does not
have a value.

Can I ask what the problem was that you tried to fix?  Because purely
from the logic perspective, that malloc needs to go.
---
Thanks
Edgar






___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post:
http://www.open-mpi.org/community/lists/devel/2015/05/17459.php


--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
--

Re: [OMPI devel] change in io_ompio.c

[Edgar Gabriel]

looking at the old code, I understand what you tried to fix, I'll commit 
a proper version in a couple of min.


Thanks
Edgar

On 5/28/2015 8:44 AM, Edgar Gabriel wrote:

ok, I see you moved the malloc up, the malloc was originally just for
the receiving side of the communication, you moved it up to cover both.
That is however unfortunately not correct.

I will fix it in a couple of mins.
Thanks
Edgar

On 5/28/2015 8:25 AM, Edgar Gabriel wrote:

Gilles,

I saw you a fixed a couple of the coverty warnings in ompio, but
unfortunately it also broke some things.

--
Question to you: in io_ompio.c line 2345, you introduced a malloc for
f_procs_in_group that was not there before. That array is allocated a
couple of lines earlier in the subroutine merge_groups

Since the values are not set, we segfault right away a couple of lines
later in the subsequent isend, where f_procs_in_group[0] simply does not
have a value.

Can I ask what the problem was that you tried to fix?  Because purely
from the logic perspective, that malloc needs to go.
---
Thanks
Edgar






--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
--

Re: [OMPI devel] change in io_ompio.c

[Edgar Gabriel]

ok, I see you moved the malloc up, the malloc was originally just for 
the receiving side of the communication, you moved it up to cover both. 
That is however unfortunately not correct.


I will fix it in a couple of mins.
Thanks
Edgar

On 5/28/2015 8:25 AM, Edgar Gabriel wrote:

Gilles,

I saw you a fixed a couple of the coverty warnings in ompio, but
unfortunately it also broke some things.

--
Question to you: in io_ompio.c line 2345, you introduced a malloc for
f_procs_in_group that was not there before. That array is allocated a
couple of lines earlier in the subroutine merge_groups

Since the values are not set, we segfault right away a couple of lines
later in the subsequent isend, where f_procs_in_group[0] simply does not
have a value.

Can I ask what the problem was that you tried to fix?  Because purely
from the logic perspective, that malloc needs to go.
---
Thanks
Edgar




--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
--

[OMPI devel] change in io_ompio.c

[Edgar Gabriel]

Gilles,

I saw you a fixed a couple of the coverty warnings in ompio, but 
unfortunately it also broke some things.


--
Question to you: in io_ompio.c line 2345, you introduced a malloc for 
f_procs_in_group that was not there before. That array is allocated a 
couple of lines earlier in the subroutine merge_groups


Since the values are not set, we segfault right away a couple of lines 
later in the subsequent isend, where f_procs_in_group[0] simply does not 
have a value.


Can I ask what the problem was that you tried to fix?  Because purely 
from the logic perspective, that malloc needs to go.

---
Thanks
Edgar


--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
--

Re: [OMPI devel] Open MPI collectives algorithm selection

[Edgar Gabriel]

I could be wrong (George, please feel free to correct me), but I *think* 
this was the designed behavior. If you read Jelena's paper,


http://www.open-mpi.org/papers/euro-pvmmpi-2006-collective-alg-selection/euro-pvmmpi-2006-collective-alg-selection.pdf

you basically construct a new decision map with the input file, and the 
it replaces the previous decision map. I doubt that the original design 
intended to do what you ask to do.


Thanks
Edgar

On 5/19/2015 9:05 PM, Gilles Gouaillardet wrote:

Folks,

this is a follow-up of a discussion on the user ML started at
http://www.open-mpi.org/community/lists/users/2015/05/26882.php

1) it turns out the dynamic rule filename must be "sorted" :
- rules must be sorted by communicator size
- within a given communicator size, rules must be sorted by message size

if not, some rules are silently skipped, which is counter intuitive imho.


2) the algo picks the rule with the higher communicator size less or
equal than the current communicator size (same thing for message size).
The exception is if there are no such rule, the first rule is selected.
for example, if the config file has rules for comm size 4, 8 and 16
comm size 4 => pick rule for comm size 4
comm size 5 => pick rule for comm 4
comm size 8 => pick rule for comm 8
*but*
comm size 2 => pick rule for comm size 4 (!)
imho, this is also counter intuitive.
i would have expected no rule is picked and the default behaviour is used.

Same thing applies for message sizes.

Is this the intended design ?

1) can be solved by inserting some qsort calls after parsing the config
file.
2) can be solved by returning a NULL rule instead of the first rule ( or
by automatically inserting a rule for comm size 0 (and message size 0)
if no such rule is present in the config file).

any thoughts ?

Cheers,

Gilles
___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post:
http://www.open-mpi.org/community/lists/devel/2015/05/17425.php


--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
--

Re: [OMPI devel] 1.8.5 release

[Edgar Gabriel]

ok, thanks! I will look into it.

Edgar

On 5/5/2015 8:23 PM, Orion Poplawski wrote:

On 05/05/2015 01:12 PM, Edgar Gabriel wrote:

Orion,

could you provide a couple of more details? I might not be able to fix
the problem for ompio in the 1.8 series, but I would definitely like
make sure that it is not an issue in the master/1.9 series.

I compiled netcdf-4.3.3.1 and netcdf-fortran--4.4.2, using hdf-1.8.9,
parallel-tests enabled, and all tests pass (both on master and 1.8) if I
run

  make check.

Are these some other tests that you are running?


I think a key trigger of the tests failing is that we compile everything
with:

-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4

It's a fortify check that is failing in this case.


On 05/04/2015 09:34 PM, Orion Poplawski wrote:

On 05/04/2015 01:21 PM, Ralph Castain wrote:

Yo folks

We are on a final overnight soak of the 1.8.5 release as we added
some datatype fixes this morning. Pending overnight MTT results, and
any last minute testing you wish to do, we will release tomorrow
after the telecon.

Ralph


It seems that (at least with the Fedora builds) we've gone from a
default of romio with 1.8.3 to a default of ompio with 1.8.5. This is
breaking the netcdf-fortran tests as they crash when using ompio.  I
can work around the build issue by selecting romio, but this seems
like a big change in a stable branch.




___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post:
http://www.open-mpi.org/community/lists/devel/2015/05/17385.php

Re: [OMPI devel] 1.8.5 release

[Edgar Gabriel]

Orion,

could you provide a couple of more details? I might not be able to fix 
the problem for ompio in the 1.8 series, but I would definitely like 
make sure that it is not an issue in the master/1.9 series.


I compiled netcdf-4.3.3.1 and netcdf-fortran--4.4.2, using hdf-1.8.9, 
parallel-tests enabled, and all tests pass (both on master and 1.8) if I run


 make check.

Are these some other tests that you are running?

Thanks
Edgar

On 05/04/2015 09:34 PM, Orion Poplawski wrote:

On 05/04/2015 01:21 PM, Ralph Castain wrote:

Yo folks

We are on a final overnight soak of the 1.8.5 release as we added 
some datatype fixes this morning. Pending overnight MTT results, and 
any last minute testing you wish to do, we will release tomorrow 
after the telecon.


Ralph


It seems that (at least with the Fedora builds) we've gone from a 
default of romio with 1.8.3 to a default of ompio with 1.8.5. This is 
breaking the netcdf-fortran tests as they crash when using ompio.  I 
can work around the build issue by selecting romio, but this seems 
like a big change in a stable branch.

Re: [OMPI devel] mpi_test_suite question

[Edgar Gabriel]

this error message comes from ompio, the split collective are not 
properly implemented at this point in time, they are basically just a 
printf statement. Once I have the non-blocking collectives re-enabled, 
it should be trivial add these interfaces to the list as well.


That being said, since I spent quite some time debugging the I/O tests 
in mpi_test_suite, I would recommend *not* to use that for I/O 
validation. I found some fundamental mistakes in the test suite, (e.g. 
writing with a file view, reading back without the file view and 
assuming that the byte pattern is the same) that I can not easily fix.


Thanks
Edgar

On 3/6/2015 12:03 PM, Howard Pritchard wrote:

Hi Folks,

I"m trying to run the mpi_test_suite test from ompi-tests.  I'm noticing
that if I try to run the io tests, I get an error in the read_all_begin
test:

STATIC READ ALL BEGIN

STATIC READ ALL END

(io/tst_file_read_all_begin.c:82) ERROR: Error in position; Invalid
argument(22)(io/tst_file_read_all_begin.c:82) ERROR: Error in position;
Invalid argument(22)(io/tst_file_read_all_begin.c:82) ERROR: Error in
position; Invalid argument(22)(io/tst_file_read_all_begin.c:82) ERROR:
Error in position; Invalid argument(22)

---

Are there known issues with the io tests?  My first guess is yes
since I notice esslingen is excluding io from their MTT runs.

Thanks for any info,

Howard



___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2015/03/17116.php



--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

[OMPI devel] openib receive queue settings

[Edgar Gabriel]

I still see an issue with the openib receive queues settings. 
Interestingly, it seems to work if I pass the setting with the mpirun 
command, e.g.


mpirun  --mca btl_openib_receive_queues 
S,12288,128,64,32:S,65536,128,64,32 --npernode 1 -np 2 ./lat


but if I add it to the ${HOME}/.openmpi/mca-param.conf file, e.g.

---snip---
 cat  ~/.openmpi/mca-params.conf
...
btl_openib_receive_queues = S,12288,128,64,32:S,65536,128,64,32
...
--snip---

I receive the following error message:
gabriel@crill:~> mpirun  --npernode 1 -np 2 ./lat
--
The Open MPI receive queue configuration for the OpenFabrics devices
on two nodes are incompatible, meaning that MPI processes on two
specific nodes were unable to communicate with each other.  This
generally happens when you are using OpenFabrics devices from
different vendors on the same network.  You should be able to use the
mca_btl_openib_receive_queues MCA parameter to set a uniform receive
queue configuration for all the devices in the MPI job, and therefore
be able to run successfully.

  Local host:   crill-003
  Local adapter:mlx4_0 (vendor 0x2c9, part ID 26418)
  Local queues: S,12288,128,64,32:S,65536,128,64,32

  Remote host:  crill-004
  Remote adapter:   (vendor 0x2c9, part ID 26418)
  Remote queues: 
P,128,256,192,128:S,2048,1024,1008,64:S,12288,1024,1008,64:S,65536,1024,1008,64

--

Does anybody have an idea what I should be looking for to fix this? I 
can definitely confirm, that the home file system is mounted on all 
nodes correctly (i.e. all processes can access the same mca-params.conf 
file), and they have the identical IB hardware (in contrary to what the 
error message says).



Thanks
Edgar

--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] problem running jobs on ompi-master

[Edgar Gabriel]

I applied the patch manually and it seemed in fact to resolve the issue, 
thanks! I must have done the git clone just right before this patch was 
committed two days back, so I just missed it (redoing it right now as well).


Thanks
Edgar

On 12/26/2014 9:06 AM, Gilles Gouaillardet wrote:

Edgar,

First, make sure your master includes 
https://github.com/open-mpi/ompi/commit/05af80b3025dbb95bdd4280087450791291d7219


If this is not enough, try with --mca coll ^ml

Hope this helps

Gilles.

Edgar Gabriel <gabr...@cs.uh.edu>さんのメール:

I have some problems running jobs with ompi-master on one of our
clusters (after doing a major software update). Here are scenarios that
work and don't work.

1. Everything still seems to work with 1.8.x series without any issues
2. With master, I can run without any issues single node, multi-process jobs
3. With master, I can run without any issues multi node jobs, as long as
there is only a single MPI process per node

4. I can not run multi-node jobs with multiple processes per node, ompi
hangs for that scenario. This is independent of whether I enforce using
openib or tcp, and just for the sake of simplicity I attach the output
for tcp (there is another openib parameter issue that still linguers,
but I will report that later).

Here is the output that I receive if setting btl_base_verbose

---snip--
gabriel@crill:~> salloc -N 2 -n 4
gabriel@crill:~> mpirun --mca btl tcp,self --mca btl_base_verbose 30 -np
  4 ./hello_world
[crill-004:18161] mca: base: components_register: registering btl components
[crill-004:18161] mca: base: components_register: found loaded component
self
[crill-004:18161] mca: base: components_register: component self
register function successful
[crill-004:18161] mca: base: components_register: found loaded component tcp
[crill-004:18161] mca: base: components_register: component tcp register
function successful
[crill-004:18161] mca: base: components_open: opening btl components
[crill-004:18161] mca: base: components_open: found loaded component self
[crill-004:18161] mca: base: components_open: component self open
function successful
[crill-004:18161] mca: base: components_open: found loaded component tcp
[crill-004:18161] mca: base: components_open: component tcp open
function successful
[crill-004:18161] select: initializing btl component self
[crill-004:18161] select: init of component self returned success
[crill-004:18161] select: initializing btl component tcp
[crill-004:18161] btl: tcp: Searching for exclude address+prefix:
127.0.0.1 / 8
[crill-004:18161] btl: tcp: Found match: 127.0.0.1 (lo)
[crill-004:18161] select: init of component tcp returned success
[crill-003:18962] mca: base: components_register: registering btl components
[crill-003:18962] mca: base: components_register: found loaded component
self
[crill-003:18962] mca: base: components_register: component self
register function successful
[crill-003:18962] mca: base: components_register: found loaded component tcp
[crill-003:18962] mca: base: components_register: component tcp register
function successful
[crill-003:18962] mca: base: components_open: opening btl components
[crill-003:18962] mca: base: components_open: found loaded component self
[crill-003:18962] mca: base: components_open: component self open
function successful
[crill-003:18962] mca: base: components_open: found loaded component tcp
[crill-003:18962] mca: base: components_open: component tcp open
function successful
[crill-003:18963] mca: base: components_register: registering btl components
[crill-003:18963] mca: base: components_register: found loaded component
self
[crill-003:18963] mca: base: components_register: component self
register function successful
[crill-003:18963] mca: base: components_register: found loaded component tcp
[crill-003:18963] mca: base: components_register: component tcp register
function successful
[crill-003:18963] mca: base: components_open: opening btl components
[crill-003:18963] mca: base: components_open: found loaded component self
[crill-003:18963] mca: base: components_open: component self open
function successful
[crill-003:18963] mca: base: components_open: found loaded component tcp
[crill-003:18963] mca: base: components_open: component tcp open
function successful
[crill-003:18964] mca: base: components_register: registering btl components
[crill-003:18964] mca: base: components_register: found loaded component
self
[crill-003:18964] mca: base: components_register: component self
register function successful
[crill-003:18964] mca: base: components_register: found loaded component tcp
[crill-003:18964] mca: base: components_register: component tcp register
function successful
[crill-003:18964] mca: base: components_open: opening btl components
[crill-003:18964] mca: base: components_open: found loaded component self
[crill-003:18964] mca: base: components_open: component self open
function successful
[crill-003:18964] mca: base: com

[OMPI devel] problem running jobs on ompi-master

[Edgar Gabriel]

turned success
[crill-003:18963] select: initializing btl component self
[crill-003:18963] select: init of component self returned success
[crill-003:18963] select: initializing btl component tcp
[crill-003:18963] btl: tcp: Searching for exclude address+prefix: 
127.0.0.1 / 8

[crill-003:18963] btl: tcp: Found match: 127.0.0.1 (lo)
[crill-003:18963] select: init of component tcp returned success
[crill-003:18964] select: initializing btl component self
[crill-003:18964] select: init of component self returned success
[crill-003:18964] select: initializing btl component tcp
[crill-003:18964] btl: tcp: Searching for exclude address+prefix: 
127.0.0.1 / 8

[crill-003:18964] btl: tcp: Found match: 127.0.0.1 (lo)
[crill-003:18964] select: init of component tcp returned success
[crill-003:18964] mca: bml: Using self btl to [[3417,1],2] on node crill-003
[crill-003:18964] mca: bml: Using tcp btl to [[3417,1],0] on node crill-003
[crill-003:18964] mca: bml: Using tcp btl to [[3417,1],1] on node crill-003
[crill-003:18964] mca: bml: Using tcp btl to [[3417,1],3] on node crill-004
[crill-004:18161] mca: bml: Using self btl to [[3417,1],3] on node crill-004
[crill-004:18161] mca: bml: Using tcp btl to [[3417,1],0] on node crill-003
[crill-004:18161] mca: bml: Using tcp btl to [[3417,1],1] on node crill-003
[crill-004:18161] mca: bml: Using tcp btl to [[3417,1],2] on node crill-003
^C


and than it just hangs.

Does anybody have an idea/suggestion what to try or look for?

Thanks
Edgar

--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] Trunk warnings

[Edgar Gabriel]

I'll take care of the one ompio warning.
Edgar

On 12/12/2014 12:01 PM, Nathan Hjelm wrote:


The osc warnings will go away after the btl modifications are applied. I
made signifigant changes to the component.

-Nathan

On Fri, Dec 12, 2014 at 09:49:47AM -0800, Ralph Castain wrote:

While building optimized on Linux:
bcol_ptpcoll_allreduce.c: In function
'bcol_ptpcoll_allreduce_narraying_init':
bcol_ptpcoll_allreduce.c:236: warning: unused variable 'dtype'
bcol_ptpcoll_allreduce.c:235: warning: unused variable `count'
io_ompio_file_set_view.c: In function
'mca_io_ompio_finalize_initial_grouping':
io_ompio_file_set_view.c:363: warning: 'sendreq' may be used uninitialized
in this function
osc_rdma_comm.c: In function 'ompi_osc_rdma_rget_accumulate_internal':
osc_rdma_comm.c:1034: warning: 'ptr' may be used uninitialized in this
function
osc_rdma_comm.c:1031: warning: 'frag' may be used uninitialized in this
function
osc_rdma_data_move.c: In function 'ompi_osc_rdma_callback':
osc_rdma_data_move.c:1647: warning: unused variable 'incoming_length'
osc_rdma_data_move.c: In function 'ompi_osc_rdma_control_send':
osc_rdma_data_move.c:225: warning: 'ptr' may be used uninitialized in this
function
osc_rdma_data_move.c:224: warning: 'frag' may be used uninitialized in
this function
osc_rdma_comm.c: In function 'ompi_osc_rdma_rget':
osc_rdma_comm.c:813: warning: 'ptr' may be used uninitialized in this
function
osc_rdma_comm.c:810: warning: 'frag' may be used uninitialized in this
function
osc_rdma_data_move.c: In function 'ompi_osc_gacc_long_start':
osc_rdma_data_move.c:973: warning: 'acc_data' may be used uninitialized in
this function
osc_rdma_comm.c: In function 'ompi_osc_rdma_put_w_req':
osc_rdma_comm.c:296: warning: 'ptr' may be used uninitialized in this
function
osc_rdma_comm.c:289: warning: 'frag' may be used uninitialized in this
function
osc_rdma_data_move.c: In function 'ompi_osc_rdma_gacc_start':
osc_rdma_data_move.c:924: warning: 'acc_data' may be used uninitialized in
this function
osc_rdma_data_move.c: In function 'ompi_osc_rdma_acc_long_start':
osc_rdma_data_move.c:839: warning: 'acc_data' may be used uninitialized in
this function
osc_rdma_comm.c: In function 'ompi_osc_rdma_accumulate_w_req':
osc_rdma_comm.c:479: warning: 'ptr' may be used uninitialized in this
function
osc_rdma_comm.c:476: warning: 'frag' may be used uninitialized in this
function
osc_rdma_comm.c: In function 'ompi_osc_rdma_get':
osc_rdma_comm.c:813: warning: 'ptr' may be used uninitialized in this
function
osc_rdma_comm.c:810: warning: 'frag' may be used uninitialized in this
function
vt_plugin_cntr.c: In function 'vt_plugin_cntr_write_post_mortem':
vt_plugin_cntr.c:1139: warning: 'min_counter' may be used uninitialized in
this function
vt_plugin_cntr.c: In function 'vt_plugin_cntr_write_post_mortem':
vt_plugin_cntr.c:1139: warning: 'min_counter' may be used uninitialized in
this function
vt_plugin_cntr.c: In function 'vt_plugin_cntr_write_post_mortem':
vt_plugin_cntr.c:1139: warning: 'min_counter' may be used uninitialized in
this function
vt_plugin_cntr.c: In function 'vt_plugin_cntr_write_post_mortem':
vt_plugin_cntr.c:1139: warning: 'min_counter' may be used uninitialized in
this function



___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2014/12/16554.php




___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2014/12/16555.php



--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] RTLD_GLOBAL question

[Edgar Gabriel]

the mailing list refused to let me add the config.log file, since it is 
too large, I can forward the output to you directly as well (as I did to 
Jeff).


I honestly have not looked into the configure logic, I can just tell 
that OPAL_HAVE_LTDL_ADVISE is not set on my linux system for master, but 
is set on the 1.8 series (1.8 series checkout was from Nov. 20, so if 
something changed in between the result might be different).




On 12/2/2014 9:27 AM, Artem Polyakov wrote:


2014-12-02 20:59 GMT+06:00 Edgar Gabriel <gabr...@cs.uh.edu
<mailto:gabr...@cs.uh.edu>>:

didn't want to interfere with this thread, although I have a similar
issue, since I have the solution nearly fully cooked up. But anyway,
this last email gave the hint on why we have suddenly the problem in
ompio:

it looks like OPAL_HAVE_LTDL_ADVISE (at least on my systems) is not
set anymore, so the entire section is being skipped. I double
checked that with the 1.8 branch, it goes through the section, but
not with master.


Hi, Edgar.

Both master and ompi-release (isn't it 1.8?!) are equal in sence of my
fix. Something else!? I'd like to see config.log too but will look into
it only tomorrow.

Also I want to add that SLURM PMI2 communicates with local slurmstepd's
and doesn't need any authentification. All PMI1 processes otherwise
communicate to the srun process and thus need libslurm services for
communication and authentification.


Thanks
Edgar




On 12/2/2014 7:56 AM, Jeff Squyres (jsquyres) wrote:

Looks like I was totally lying in
http://www.open-mpi.org/__community/lists/devel/2014/12/__16381.php
<http://www.open-mpi.org/community/lists/devel/2014/12/16381.php> (where
I said we should not use RTLD_GLOBAL).  We *do* use RTLD_GLOBAL:


https://github.com/open-mpi/__ompi/blob/master/opal/mca/__base/mca_base_component___repository.c#L124

<https://github.com/open-mpi/ompi/blob/master/opal/mca/base/mca_base_component_repository.c#L124>

This ltdl advice object is passed to lt_dlopen() for all
components.  My mistake; sorry.

So the idea that using RTLD_GLOBAL will fix this SLURM bug is
incorrect.

I believe someone said earlier in the thread that adding the
right -llibs to the configure line will solve the issue, and
that sounds correct to me.  If there's a missing symbol because
the SLURM libraries are not automatically pulling in the right
dependent libraries, then *if* we put a workaround in OMPI to
fix this issue, then the right workaround is to add the relevant
-llibs when that component is linked.

*If* you add that workaround (which is a whole separate
discussion), I would suggest adding a configure.m4 test to see
if adding the additional -llibs are necessary.  Perhaps
AC_LINK_IFELSE looking for a symbol, and then if that fails,
AC_LINK_IFELSE again with the additional -llibs to see if that
works.

Or something like that.



On Dec 2, 2014, at 6:38 AM, Artem Polyakov <artpo...@gmail.com
<mailto:artpo...@gmail.com>> wrote:

Agree. First you should check is to what value
OPAL_HAVE_LTDL_ADVISE is set. If it is zero - very probably
this is the same bug as mine.

2014-12-02 17:33 GMT+06:00 Ralph Castain <r...@open-mpi.org
<mailto:r...@open-mpi.org>>:
It does look similar - question is: why didn’t this fix the
problem? Will have to investigate.

Thanks


On Dec 2, 2014, at 3:17 AM, Artem Polyakov
<artpo...@gmail.com <mailto:artpo...@gmail.com>> wrote:



2014-12-02 17:13 GMT+06:00 Ralph Castain
<r...@open-mpi.org <mailto:r...@open-mpi.org>>:
Hmmm…if that is true, then it didn’t fix this problem as
it is being reported in the master.

I had this problem on my laptop installation. You can
check my report it was detailed enough and see if you
hitting the same issue. My fix was also included into
1.8 branch. I am not sure that this is the same issue
but they looks similar.



On Dec 1, 2014, at 9:40 PM, Artem Polyakov
<artpo...@gmail.com <mailto:artpo...@gmail.com>> wrote:

I think this might be related to the configuration
problem I was fixing with Jeff few months ago. Refer
here:
https://github.com/open-mpi/__ompi/pull/240
<https://github.com/open-mpi/ompi/pull/240>

2014-12-02 10:15 GMT+06:00 Ralph Castain
<r...@open-mpi.org &l

Re: [OMPI devel] RTLD_GLOBAL question

[Edgar Gabriel]

I checked with the debugger, that it did skip the entire section

On 12/2/2014 9:04 AM, Jeff Squyres (jsquyres) wrote:

Oy -- I thought we fixed that.  :-(

Are you saying that configure output says that ltdladvise is not found?


On Dec 2, 2014, at 9:59 AM, Edgar Gabriel <gabr...@cs.uh.edu> wrote:


didn't want to interfere with this thread, although I have a similar issue, 
since I have the solution nearly fully cooked up. But anyway, this last email 
gave the hint on why we have suddenly the problem in ompio:

it looks like OPAL_HAVE_LTDL_ADVISE (at least on my systems) is not set 
anymore, so the entire section is being skipped. I double checked that with the 
1.8 branch, it goes through the section, but not with master.

Thanks
Edgar



On 12/2/2014 7:56 AM, Jeff Squyres (jsquyres) wrote:

Looks like I was totally lying in 
http://www.open-mpi.org/community/lists/devel/2014/12/16381.php (where I said 
we should not use RTLD_GLOBAL).  We *do* use RTLD_GLOBAL:

https://github.com/open-mpi/ompi/blob/master/opal/mca/base/mca_base_component_repository.c#L124

This ltdl advice object is passed to lt_dlopen() for all components.  My 
mistake; sorry.

So the idea that using RTLD_GLOBAL will fix this SLURM bug is incorrect.

I believe someone said earlier in the thread that adding the right -llibs to 
the configure line will solve the issue, and that sounds correct to me.  If 
there's a missing symbol because the SLURM libraries are not automatically 
pulling in the right dependent libraries, then *if* we put a workaround in OMPI 
to fix this issue, then the right workaround is to add the relevant -llibs when 
that component is linked.

*If* you add that workaround (which is a whole separate discussion), I would 
suggest adding a configure.m4 test to see if adding the additional -llibs are 
necessary.  Perhaps AC_LINK_IFELSE looking for a symbol, and then if that 
fails, AC_LINK_IFELSE again with the additional -llibs to see if that works.

Or something like that.



On Dec 2, 2014, at 6:38 AM, Artem Polyakov <artpo...@gmail.com> wrote:


Agree. First you should check is to what value OPAL_HAVE_LTDL_ADVISE is set. If 
it is zero - very probably this is the same bug as mine.

2014-12-02 17:33 GMT+06:00 Ralph Castain <r...@open-mpi.org>:
It does look similar - question is: why didn’t this fix the problem? Will have 
to investigate.

Thanks



On Dec 2, 2014, at 3:17 AM, Artem Polyakov <artpo...@gmail.com> wrote:



2014-12-02 17:13 GMT+06:00 Ralph Castain <r...@open-mpi.org>:
Hmmm…if that is true, then it didn’t fix this problem as it is being reported 
in the master.

I had this problem on my laptop installation. You can check my report it was 
detailed enough and see if you hitting the same issue. My fix was also included 
into 1.8 branch. I am not sure that this is the same issue but they looks 
similar.




On Dec 1, 2014, at 9:40 PM, Artem Polyakov <artpo...@gmail.com> wrote:

I think this might be related to the configuration problem I was fixing with 
Jeff few months ago. Refer here:
https://github.com/open-mpi/ompi/pull/240

2014-12-02 10:15 GMT+06:00 Ralph Castain <r...@open-mpi.org>:
If it isn’t too much trouble, it would be good to confirm that it remains 
broken. I strongly suspect it is based on Moe’s comments.

Obviously, other people are making this work. For Intel MPI, all you do is 
point it at libpmi and they can run. However, they do explicitly dlopen it in 
their code, and I don’t know what flags they might pass when they do so.

If necessary, I suppose we could follow that pattern. In other words, rather 
than specifically linking the “s1” component to libpmi, instead require that 
the user point us to a pmi library via an MCA param, then explicitly dlopen 
that library with RTLD_GLOBAL. This avoids the issues cited by Jeff, but 
resolves the pmi linkage problem.



On Dec 1, 2014, at 8:09 PM, Gilles Gouaillardet <gilles.gouaillar...@iferc.org> 
wrote:

$ srun --version
slurm 2.6.6-VENDOR_PROVIDED

$ srun --mpi=pmi2 -n 1 ~/hw
I am 0 / 1

$ srun -n 1 ~/hw
/csc/home1/gouaillardet/hw: symbol lookup error: 
/usr/lib64/slurm/auth_munge.so: undefined symbol: slurm_verbose
srun: error: slurm_receive_msg: Zero Bytes were transmitted or received
srun: error: slurm_receive_msg[10.0.3.15]: Zero Bytes were transmitted or 
received
srun: error: soleil: task 0: Exited with exit code 127

$ ldd /usr/lib64/slurm/auth_munge.so
 linux-vdso.so.1 =>  (0x7fff54478000)
 libmunge.so.2 => /usr/lib64/libmunge.so.2 (0x7f744760f000)
 libpthread.so.0 => /lib64/libpthread.so.0 (0x7f74473f1000)
 libc.so.6 => /lib64/libc.so.6 (0x7f744705d000)
 /lib64/ld-linux-x86-64.so.2 (0x003bf540)


now, if i reling auth_munge.so so it depends on libslurm :

$ srun -n 1 ~/hw
srun: symbol lookup error: /usr/lib64/slurm/auth_munge.so: undefined symbol: 
slurm_auth_get_arg_desc


i can give a try to the latest slurm if needed

Chee

Re: [OMPI devel] RTLD_GLOBAL question

[Edgar Gabriel]

.org/mailman/listinfo.cgi/devel 
<http://www.open-mpi.org/mailman/listinfo.cgi/devel> 
<http://www.open-mpi.org/mailman/listinfo.cgi/devel> 
<http://www.open-mpi.org/mailman/listinfo.cgi/devel>

Link to this post:
http://www.open-mpi.org/community/lists/devel/2014/12/16386.php 
<http://www.open-mpi.org/community/lists/devel/2014/12/16386.php> 
<http://www.open-mpi.org/community/lists/devel/2014/12/16386.php> 
<http://www.open-mpi.org/community/lists/devel/2014/12/16386.php>

___
devel mailing list

de...@open-mpi.org <mailto:de...@open-mpi.org>

Subscription:
http://www.open-mpi.org/mailman/listinfo.cgi/devel 
<http://www.open-mpi.org/mailman/listinfo.cgi/devel>

Link to this post:
http://www.open-mpi.org/community/lists/devel/2014/12/16387.php 
<http://www.open-mpi.org/community/lists/devel/2014/12/16387.php>

___
devel mailing list

de...@open-mpi.org

Subscription:
http://www.open-mpi.org/mailman/listinfo.cgi/devel

Link to this post:
http://www.open-mpi.org/community/lists/devel/2014/12/16388.php



___
devel mailing list

de...@open-mpi.org

Subscription:
http://www.open-mpi.org/mailman/listinfo.cgi/devel

Link to this post:
http://www.open-mpi.org/community/lists/devel/2014/12/16389.php


___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2014/12/16390.php



___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2014/12/16391.php



--
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov
___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2014/12/16393.php



___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2014/12/16395.php



--
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov
___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2014/12/16396.php



___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2014/12/16397.php



--
С Уважением, Поляков Артем Юрьевич
Best regards, Artem Y. Polyakov
___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2014/12/16398.php





--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] question to OMPI_DECLSPEC

[Edgar Gabriel]

On 11/26/2014 11:02 AM, George Bosilca wrote:


We had similar problems in the PML V, and we decided to try to minimize
the increase in size of the main library. Thus, instead of moving
everything in the base, we added a structure in the base that will
contain all the pointer to the functions we would need. This structure
is only initialized when our main module is loaded, and all sub-modules
will use this structure to get access to the pointers provided.


That is an interesting option, let me think about it. What it would give 
us is that we do not have artificially 'force' some code into the base 
of other frameworks, since in my opinion the ompio component is still 
the best place for these functions.


Thanks
Edgar




   George.

>
> 2. I will have to extend the io framework interfaces a bit ( I will try 
to minimize the number of new function as much as I can), but those function 
pointers will be NULL for ROMIO. Just want to make sure this is ok with everybody.

I’ll have to let others chime in here, but that would seem to fit
the OMPI architecture.

 >
 > Thanks
 > Edgar
 >
 > On 11/25/2014 11:43 AM, Ralph Castain wrote:
 >>
 >>> On Nov 25, 2014, at 9:36 AM, Edgar Gabriel <gabr...@cs.uh.edu
<mailto:gabr...@cs.uh.edu>> wrote:
 >>>
 >>> On 11/25/2014 11:31 AM, Ralph Castain wrote:
 >>>>
 >>>>> On Nov 25, 2014, at 8:24 AM, Edgar Gabriel <gabr...@cs.uh.edu
<mailto:gabr...@cs.uh.edu>
 >>>>> <mailto:gabr...@cs.uh.edu <mailto:gabr...@cs.uh.edu>>> wrote:
 >>>>>
 >>>>> On 11/25/2014 10:18 AM, Ralph Castain wrote:
 >>>>>> Hmmm…no, nothing has changed with regard to declspec that I know
 >>>>>> about. I’ll ask the obvious things to check:
 >>>>>>
 >>>>>> * does that component have the proper include to find this
function?
 >>>>>> Could be that it used to be found thru some chain, but the
chain is
 >>>>>> now broken and it needs to be directly included
 >>>>>
 >>>>> header is included, I double checked.
 >>>>>
 >>>>>> * is that function in the base code, or down in a component?
If the
 >>>>>> latter, then that’s a problem, but I’m assuming you didn’t
make that
 >>>>>> mistake.
 >>>>>
 >>>>>
 >>>>> I am not sure what you mean. The function is in a component,
but I am
 >>>>> not aware that it is illegal to call a function of a
component from
 >>>>> another component.
 >>>>
 >>>>
 >>>> Of course that is illegal - you can only access a function via the
 >>>> framework interface, not directly. You have no way of knowing
that the
 >>>> other component has been loaded. Doing it directly violates the
 >>>> abstraction rules.
 >>>
 >>> well, ok. I know that the other componen has been loaded
because that component triggered the initialization of these
sub-frameworks.
 >>
 >> I think we’ve seen that before, and run into problems with that
approach (i.e., components calling framework opens).
 >>
 >>>
 >>> I can move that functionality to the base, however, none of the
20+ functions are required for the other components of the io
framework (i.e. ROMIO). So I would basically add functionality
required for one component only into the base.
 >>
 >> Sounds like you’ve got an abstraction problem. If the fcoll
component requires certain functions from another framework, then
the framework should be exposing those APIs. If ROMIO doesn’t
provide them, then it needs to return an error if someone attempts
to call it.
 >>
 >> You are welcome to bring this up on next week’s call if you
like. IIRC, this has come up before when people have tried this hard
links between components. Maybe someone else will have a better
solution, but is just seems to me like you have to go thru the
framework to avoid the problem.
 >>
 >>>
 >>> Nevertheless, I think the original question is still valid. We
did not see this problem before, but it is now showing on all of our
platforms, and I am still wandering that is the case. I *know* that
the ompio component is loaded, and I still get the error message
about the missing symbol from the ompio component. I do not
understand why that happens.
 >>
 

Re: [OMPI devel] question to OMPI_DECLSPEC

[Edgar Gabriel]

ok, so I thought about it a bit, and while I am still baffled by the 
actual outcome and the missing symbol (for the main reason that the 
function of the fcoll component is being called from the ompio module, 
so the function of the ompio that was called from the fcoll component is 
guaranteed to be loaded, and does have the proper OMPI_DECLSPEC), I will 
do some restructuring of the code to handle that.


As an explanation on why there are so many functions in ompio that are 
being called from the sub-frameworks directly, ompio is more or less the 
glue between all the other frameworks, and contains a lot of the code 
that is jointly used by the fbtl, fcoll and the sharedfp components (fs 
to a lesser extent as well).


Before I start to move code around however, just want to confirm two things:

1. I can move some of functionality of ompio to the base of various 
frameworks (fcoll, fbtl and io). Just want to confirm that this will 
work, e.g. I can call without restrictions a function of the fcoll base 
from an fbtl or the io component.


2. I will have to extend the io framework interfaces a bit ( I will try 
to minimize the number of new function as much as I can), but those 
function pointers will be NULL for ROMIO. Just want to make sure this is 
ok with everybody.


Thanks
Edgar

On 11/25/2014 11:43 AM, Ralph Castain wrote:



On Nov 25, 2014, at 9:36 AM, Edgar Gabriel <gabr...@cs.uh.edu> wrote:

On 11/25/2014 11:31 AM, Ralph Castain wrote:



On Nov 25, 2014, at 8:24 AM, Edgar Gabriel <gabr...@cs.uh.edu
<mailto:gabr...@cs.uh.edu>> wrote:

On 11/25/2014 10:18 AM, Ralph Castain wrote:

Hmmm…no, nothing has changed with regard to declspec that I know
about. I’ll ask the obvious things to check:

* does that component have the proper include to find this function?
Could be that it used to be found thru some chain, but the chain is
now broken and it needs to be directly included


header is included, I double checked.


* is that function in the base code, or down in a component? If the
latter, then that’s a problem, but I’m assuming you didn’t make that
mistake.



I am not sure what you mean. The function is in a component, but I am
not aware that it is illegal to call a function of a component from
another component.



Of course that is illegal - you can only access a function via the
framework interface, not directly. You have no way of knowing that the
other component has been loaded. Doing it directly violates the
abstraction rules.


well, ok. I know that the other componen has been loaded because that component 
triggered the initialization of these sub-frameworks.


I think we’ve seen that before, and run into problems with that approach (i.e., 
components calling framework opens).



I can move that functionality to the base, however, none of the 20+ functions 
are required for the other components of the io framework (i.e. ROMIO). So I 
would basically add functionality required for one component only into the base.


Sounds like you’ve got an abstraction problem. If the fcoll component requires 
certain functions from another framework, then the framework should be exposing 
those APIs. If ROMIO doesn’t provide them, then it needs to return an error if 
someone attempts to call it.

You are welcome to bring this up on next week’s call if you like. IIRC, this 
has come up before when people have tried this hard links between components. 
Maybe someone else will have a better solution, but is just seems to me like 
you have to go thru the framework to avoid the problem.



Nevertheless, I think the original question is still valid. We did not see this 
problem before, but it is now showing on all of our platforms, and I am still 
wandering that is the case. I *know* that the ompio component is loaded, and I 
still get the error message about the missing symbol from the ompio component. 
I do not understand why that happens.


Probably because the fcoll component didn’t explicitly link against the ompio 
component. You were likely getting away with it out of pure luck.




Thanks
Edgar






Thanks
Edgar







On Nov 25, 2014, at 8:07 AM, Edgar Gabriel <gabr...@cs.uh.edu
<mailto:gabr...@cs.uh.edu>>
wrote:

Has something changed recently on the trunk/master regarding
OMPI_DECLSPEC? The reason I ask is because we get now errors about
unresolved symbols, e.g.

symbol lookup error:
/home/gabriel/OpenMPI/lib64/openmpi/mca_fcoll_dynamic.so: undefined
symbol: ompi_io_ompio_decode_datatype


and that problem was not there roughly two weeks back the last time
I tested. I did verify that the the function listed there has an
OMPI_DECLSPEC before its definition.

Thanks Edgar -- Edgar Gabriel Associate Professor Parallel Software
Technologies Lab http://pstl.cs.uh.edu Department of Computer
Science  University of Houston Philip G. Hoffman Hall, Room
524Houston, TX-77204, USA Tel: +1 (713) 743-3857
Fax: +1 (713) 743-3335

Re: [OMPI devel] question to OMPI_DECLSPEC

[Edgar Gabriel]

On 11/25/2014 11:31 AM, Ralph Castain wrote:



On Nov 25, 2014, at 8:24 AM, Edgar Gabriel <gabr...@cs.uh.edu
<mailto:gabr...@cs.uh.edu>> wrote:

On 11/25/2014 10:18 AM, Ralph Castain wrote:

Hmmm…no, nothing has changed with regard to declspec that I know
about. I’ll ask the obvious things to check:

* does that component have the proper include to find this function?
Could be that it used to be found thru some chain, but the chain is
now broken and it needs to be directly included


header is included, I double checked.


* is that function in the base code, or down in a component? If the
latter, then that’s a problem, but I’m assuming you didn’t make that
mistake.



I am not sure what you mean. The function is in a component, but I am
not aware that it is illegal to call a function of a component from
another component.



Of course that is illegal - you can only access a function via the
framework interface, not directly. You have no way of knowing that the
other component has been loaded. Doing it directly violates the
abstraction rules.


well, ok. I know that the other componen has been loaded because that 
component triggered the initialization of these sub-frameworks.


I can move that functionality to the base, however, none of the 20+ 
functions are required for the other components of the io framework 
(i.e. ROMIO). So I would basically add functionality required for one 
component only into the base.


Nevertheless, I think the original question is still valid. We did not 
see this problem before, but it is now showing on all of our platforms, 
and I am still wandering that is the case. I *know* that the ompio 
component is loaded, and I still get the error message about the missing 
symbol from the ompio component. I do not understand why that happens.



Thanks
Edgar






Thanks
Edgar







On Nov 25, 2014, at 8:07 AM, Edgar Gabriel <gabr...@cs.uh.edu
<mailto:gabr...@cs.uh.edu>>
wrote:

Has something changed recently on the trunk/master regarding
OMPI_DECLSPEC? The reason I ask is because we get now errors about
unresolved symbols, e.g.

symbol lookup error:
/home/gabriel/OpenMPI/lib64/openmpi/mca_fcoll_dynamic.so: undefined
symbol: ompi_io_ompio_decode_datatype


and that problem was not there roughly two weeks back the last time
I tested. I did verify that the the function listed there has an
OMPI_DECLSPEC before its definition.

Thanks Edgar -- Edgar Gabriel Associate Professor Parallel Software
Technologies Lab http://pstl.cs.uh.edu Department of Computer
Science  University of Houston Philip G. Hoffman Hall, Room
524Houston, TX-77204, USA Tel: +1 (713) 743-3857
Fax: +1 (713) 743-3335
___ devel mailing list
de...@open-mpi.org <mailto:de...@open-mpi.org> Subscription:
http://www.open-mpi.org/mailman/listinfo.cgi/devel Link to this
post:
http://www.open-mpi.org/community/lists/devel/2014/11/16332.php


___ devel mailing list
de...@open-mpi.org <mailto:de...@open-mpi.org>Subscription:
http://www.open-mpi.org/mailman/listinfo.cgi/develLink to this post:
http://www.open-mpi.org/community/lists/devel/2014/11/16333.php



--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab http://pstl.cs.uh.edu
<http://pstl.cs.uh.edu/>
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
___
devel mailing list
de...@open-mpi.org <mailto:de...@open-mpi.org>
Subscription:http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this
post:http://www.open-mpi.org/community/lists/devel/2014/11/16334.php




___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2014/11/16336.php



--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] question to OMPI_DECLSPEC

[Edgar Gabriel]

On 11/25/2014 10:18 AM, Ralph Castain wrote:

Hmmm…no, nothing has changed with regard to declspec that I know
about. I’ll ask the obvious things to check:

* does that component have the proper include to find this function?
Could be that it used to be found thru some chain, but the chain is
now broken and it needs to be directly included


header is included, I double checked.


* is that function in the base code, or down in a component? If the
latter, then that’s a problem, but I’m assuming you didn’t make that
mistake.



I am not sure what you mean. The function is in a component, but I am 
not aware that it is illegal to call a function of a component from 
another component.


Thanks
Edgar







On Nov 25, 2014, at 8:07 AM, Edgar Gabriel <gabr...@cs.uh.edu>
wrote:

Has something changed recently on the trunk/master regarding
OMPI_DECLSPEC? The reason I ask is because we get now errors about
unresolved symbols, e.g.

symbol lookup error:
/home/gabriel/OpenMPI/lib64/openmpi/mca_fcoll_dynamic.so: undefined
symbol: ompi_io_ompio_decode_datatype


and that problem was not there roughly two weeks back the last time
I tested. I did verify that the the function listed there has an
OMPI_DECLSPEC before its definition.

Thanks Edgar -- Edgar Gabriel Associate Professor Parallel Software
Technologies Lab  http://pstl.cs.uh.edu Department of Computer
Science  University of Houston Philip G. Hoffman Hall, Room
524Houston, TX-77204, USA Tel: +1 (713) 743-3857
Fax: +1 (713) 743-3335
___ devel mailing list
de...@open-mpi.org Subscription:
http://www.open-mpi.org/mailman/listinfo.cgi/devel Link to this
post:
http://www.open-mpi.org/community/lists/devel/2014/11/16332.php


___ devel mailing list
de...@open-mpi.org Subscription:
http://www.open-mpi.org/mailman/listinfo.cgi/devel Link to this post:
http://www.open-mpi.org/community/lists/devel/2014/11/16333.php



--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

[OMPI devel] question to OMPI_DECLSPEC

[Edgar Gabriel]

Has something changed recently on the trunk/master regarding 
OMPI_DECLSPEC? The reason I ask is because we get now errors about 
unresolved symbols, e.g.


symbol lookup error: 
/home/gabriel/OpenMPI/lib64/openmpi/mca_fcoll_dynamic.so: undefined 
symbol: ompi_io_ompio_decode_datatype



and that problem was not there roughly two weeks back the last time I 
tested. I did verify that the the function listed there has an 
OMPI_DECLSPEC before its definition.


Thanks
Edgar
--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] Open MPI Developers F2F Q1 2015 (poll closes on Friday, 7th of November)

[Edgar Gabriel]

to throw in my 0.02$, I am probably not able to attend the entire 
meeting. Dallas would be however in driving distance, I would try attend 
parts of the meeting as well.


Thanks
Edgar

On 11/5/2014 1:10 PM, Howard Pritchard wrote:

Hi Folks,

I think Dallas (either Love or DFW) is cheaper to fly in to than Atlanta.

Howard


2014-11-05 11:46 GMT-07:00 Jeff Squyres (jsquyres) <jsquy...@cisco.com
<mailto:jsquy...@cisco.com>>:

Isn't Dallas 1 flight away from Knoxville?  Dallas is a bit more
central (i.e., shorter flights for those coming from the west)



On Nov 5, 2014, at 1:35 PM, George Bosilca <bosi...@icl.utk.edu
<mailto:bosi...@icl.utk.edu>> wrote:

 > Even to US attendees Atlanta might seem more appealing, as it is
one hop away from most locations and it has reasonable weather
forecast for January/February (not as good as Dallas I concede).
 >
 >   George.
 >
 >
 > On Wed, Nov 5, 2014 at 1:18 PM, Jeff Squyres (jsquyres)
<jsquy...@cisco.com <mailto:jsquy...@cisco.com>> wrote:
 > SHORT VERSION
 > =
 >
 > Will anyone be attending from Europe?
 >
 > This may influence the location of the meeting.
 >
 > MORE DETAIL
 > ===
 >
 > We're tentatively thinking that Dallas, TX would be a good
location for the meeting (at the Cisco facility).  The rationale was
as follows:
 >
 > 1. Chicago is nice and central, but the weather in Jan/Feb can
lead to travel problems
 > 2. The San Francisco Bay area is also nice, but is not very central
 > 3. Dallas was seen as a compromise:
 >- it's central
 >- it's likely a 1-flight for most US attendees
 >- it's less likely to have weather-related travel problems in
Jan/Feb
 >
 > However, it was brought to my attention today that San Francisco
(or Atlanta or New York?) may be more attractive to European
attendees.  I.e., there's few direct flights from Europe to Dallas
(if any?).
 >
 >
 >
 > On Nov 5, 2014, at 11:20 AM, Howard Pritchard
<hpprit...@gmail.com <mailto:hpprit...@gmail.com>> wrote:
 >
 > > Hi Folks,
 > >
 > > We've gotten a number of responses to the doodle poll
 > > for a week to hold the next OMPI developers F2F.
 > >
 > > The responses are definitely favoring a meeting the
 > > week of January 26th.
 > >
 > > The poll will be kept open till COB (PST) Friday,
 > > the 7th of November.
 > >
 > > Thanks,
 > >
 > > Howard
 > >
 > > ___
 > > devel mailing list
 > > de...@open-mpi.org <mailto:de...@open-mpi.org>
 > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
 > > Link to this post:
http://www.open-mpi.org/community/lists/devel/2014/11/16203.php
 >
 >
 > --
 > Jeff Squyres
 > jsquy...@cisco.com <mailto:jsquy...@cisco.com>
 > For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/
 >
 > ___
 > devel mailing list
 > de...@open-mpi.org <mailto:de...@open-mpi.org>
 > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
 > Link to this post:
http://www.open-mpi.org/community/lists/devel/2014/11/16213.php
 >
 > ___
 > devel mailing list
 > de...@open-mpi.org <mailto:de...@open-mpi.org>
 > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
 > Link to this post:
http://www.open-mpi.org/community/lists/devel/2014/11/16214.php


--
Jeff Squyres
jsquy...@cisco.com <mailto:jsquy...@cisco.com>
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

___
devel mailing list
de...@open-mpi.org <mailto:de...@open-mpi.org>
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post:
http://www.open-mpi.org/community/lists/devel/2014/11/16216.php




___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2014/11/16218.php



--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] Deprecated call in sharedfp framework

[Edgar Gabriel]

Yes, will have a look at it next week.

Thanks
Edgar

On 10/24/2014 12:01 PM, Jeff Squyres (jsquyres) wrote:

Edgar -- can you have a look?


On Oct 24, 2014, at 12:04 PM, Ralph Castain <r...@open-mpi.org> wrote:


I’m not sure who owns that framework, but I’m seeing this warning:

sharedfp_sm_file_open.c: In function 'mca_sharedfp_sm_file_open':
sharedfp_sm_file_open.c:159:5: warning: 'sem_init' is deprecated (declared at 
/usr/include/sys/semaphore.h:55) [-Wdeprecated-declarations]
  if(sem_init(_offset_ptr->mutex, 1, 1) != -1){
  ^
sharedfp_sm_file_open.c: In function 'mca_sharedfp_sm_file_close':
sharedfp_sm_file_open.c:214:13: warning: 'sem_destroy' is deprecated (declared 
at /usr/include/sys/semaphore.h:53) [-Wdeprecated-declarations]
  sem_destroy(_data->sm_offset_ptr->mutex);
  ^


This is with gcc (MacPorts gcc49 4.9.1_0) 4.9.1
Ralph

___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2014/10/16088.php





--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] Need to know your Github ID

[Edgar Gabriel]

edgar -> edgargabriel

On 9/10/2014 5:46 AM, Jeff Squyres (jsquyres) wrote:

As the next step of the planned migration to Github, I need to know:

- Your Github ID (so that you can be added to the new OMPI git repo)
- Your SVN ID (so that I can map SVN->Github IDs, and therefore map Trac 
tickets to appropriate owners)

Here's the list of SVN IDs who have committed over the past year -- I'm 
guessing that most of these people will need Github IDs:

  adrian
  alekseys
  alex
  alinas
  amikheev
  bbenton
  bosilca (done)
  bouteill
  brbarret
  bwesarg
  devendar
  dgoodell (done)
  edgar
  eugene
  ggouaillardet
  hadi
  hjelmn
  hpcchris
  hppritcha
  igoru
  jjhursey (done)
  jladd
  jroman
  jsquyres (done)
  jurenz
  kliteyn
  manjugv
  miked (done)
  mjbhaskar
  mpiteam (done)
  naughtont
  osvegis
  pasha
  regrant
  rfaucett
  rhc (done)
  rolfv (done)
  samuel
  shiqing
  swise
  tkordenbrock
  vasily
  vvenkates
  vvenkatesan
  yaeld
  yosefe



--
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] segfault in openib component on trunk

[Edgar Gabriel]

to add another piece of information that I just found, the segfault only 
occurs if I have a particular mca parameter set in my mca-params.conf 
file, namely


btl_openib_receive_queues = S,12288,128,64,32:S,65536,128,64,3

Has the syntax for this parameter changed, or should/can I get rid of it?

Thanks
Edgar

On 08/28/2014 04:19 PM, Edgar Gabriel wrote:

we are having recently problems running trunk with openib component
enabled on one of our clusters. The problem occurs right in the
initialization part, here is the stack right before the segfault:

---snip---
(gdb) where
#0  mca_btl_openib_tune_endpoint (openib_btl=0x762a40,
endpoint=0x7d9660) at btl_openib.c:470
#1  0x7f1062f105c4 in mca_btl_openib_add_procs (btl=0x762a40,
nprocs=2, procs=0x759be0, peers=0x762440, reachable=0x7fff22dd16f0) at
btl_openib.c:1093
#2  0x7f106316102c in mca_bml_r2_add_procs (nprocs=2,
procs=0x759be0, reachable=0x7fff22dd16f0) at bml_r2.c:201
#3  0x7f10615c0dd5 in mca_pml_ob1_add_procs (procs=0x70dc00,
nprocs=2) at pml_ob1.c:334
#4  0x7f106823ed84 in ompi_mpi_init (argc=1, argv=0x7fff22dd1da8,
requested=0, provided=0x7fff22dd184c) at runtime/ompi_mpi_init.c:790
#5  0x7f1068273a2c in MPI_Init (argc=0x7fff22dd188c,
argv=0x7fff22dd1880) at init.c:84
#6  0x004008e7 in main (argc=1, argv=0x7fff22dd1da8) at
hello_world.c:13
---snip---


in line 538 of the file containing the mca_btl_openib_tune_endpoint
routine, the strcmp operation fails, because  recv_qps is a NULL pointer.


---snip---

if(0 != strcmp(mca_btl_openib_component.receive_queues, recv_qps)) {

---snip---

Does anybody have an idea on what might be going wrong and how to
resolve it? Just to confirm, everything works perfectly with the 1.8
series on that very same  cluster

Thanks
Edgar
___
devel mailing list
de...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post:
http://www.open-mpi.org/community/lists/devel/2014/08/15746.php

[OMPI devel] segfault in openib component on trunk

[Edgar Gabriel]

we are having recently problems running trunk with openib component 
enabled on one of our clusters. The problem occurs right in the 
initialization part, here is the stack right before the segfault:


---snip---
(gdb) where
#0  mca_btl_openib_tune_endpoint (openib_btl=0x762a40, 
endpoint=0x7d9660) at btl_openib.c:470
#1  0x7f1062f105c4 in mca_btl_openib_add_procs (btl=0x762a40, 
nprocs=2, procs=0x759be0, peers=0x762440, reachable=0x7fff22dd16f0) at 
btl_openib.c:1093
#2  0x7f106316102c in mca_bml_r2_add_procs (nprocs=2, 
procs=0x759be0, reachable=0x7fff22dd16f0) at bml_r2.c:201
#3  0x7f10615c0dd5 in mca_pml_ob1_add_procs (procs=0x70dc00, 
nprocs=2) at pml_ob1.c:334
#4  0x7f106823ed84 in ompi_mpi_init (argc=1, argv=0x7fff22dd1da8, 
requested=0, provided=0x7fff22dd184c) at runtime/ompi_mpi_init.c:790
#5  0x7f1068273a2c in MPI_Init (argc=0x7fff22dd188c, 
argv=0x7fff22dd1880) at init.c:84
#6  0x004008e7 in main (argc=1, argv=0x7fff22dd1da8) at 
hello_world.c:13

---snip---


in line 538 of the file containing the mca_btl_openib_tune_endpoint 
routine, the strcmp operation fails, because  recv_qps is a NULL pointer.



---snip---

if(0 != strcmp(mca_btl_openib_component.receive_queues, recv_qps)) {

---snip---

Does anybody have an idea on what might be going wrong and how to 
resolve it? Just to confirm, everything works perfectly with the 1.8 
series on that very same  cluster


Thanks
Edgar

Re: [OMPI devel] Agenda for next week

[Edgar Gabriel]

sorry, let me be more precise for Wednesday, I have time before 12pm on
Wednesday.

Thanks
Edgar

On 6/19/2014 2:52 PM, Edgar Gabriel wrote:
> the best time for me would be either Wednesday morning (basically any
> time), or Thursday morning before 11am central.
> 
> Thanks
> Edgar
> 
> On 6/19/2014 1:42 PM, Ralph Castain wrote:
>> I found it on the agenda under the 1.9 branch subject - let us know when you 
>> are available, Edgar
>>
>> On Jun 19, 2014, at 11:29 AM, Jeff Squyres (jsquyres) <jsquy...@cisco.com> 
>> wrote:
>>
>>> ...and pick a time that would work for you for a webex.  :-)
>>>
>>> On Jun 19, 2014, at 1:48 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>>
>>>> I don't see that on the agenda, Edgar - can you please add it to ensure it 
>>>> gets covered?
>>>>
>>>>
>>>> On Jun 19, 2014, at 10:36 AM, Edgar Gabriel <gabr...@cs.uh.edu> wrote:
>>>>
>>>>> If possible, I would like to attend remotely the discussion about OMPIO
>>>>> as well.
>>>>>
>>>>> Thanks
>>>>> Edgar
>>>>>
>>>>> On 6/19/2014 10:44 AM, Jeff Squyres (jsquyres) wrote:
>>>>>> We have a bunch of topics listed on the wiki, but no real set agenda:
>>>>>>
>>>>>>  https://svn.open-mpi.org/trac/ompi/wiki/Jun14Meeting
>>>>>>
>>>>>> We had remote-attendance requests for 2 topics, however, so I took the 
>>>>>> liberty of setting up some fixed-time webexes for them (see the wiki for 
>>>>>> the webex links):
>>>>>>
>>>>>> - Tuesday at 2pm US Central for the git discussion
>>>>>> - Thursday at 9am US Central for the FT discussion
>>>>>>
>>>>>> Are there any other topics that people wanted to remote in to?  Fair 
>>>>>> warning: remote attendance is "ok" via webex, but it's no substitute for 
>>>>>> being there.
>>>>>>
>>>>>
>>>>> -- 
>>>>> Edgar Gabriel
>>>>> Associate Professor
>>>>> Parallel Software Technologies Lab  http://pstl.cs.uh.edu
>>>>> Department of Computer Science  University of Houston
>>>>> Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
>>>>> Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
>>>>>
>>>>> ___
>>>>> devel mailing list
>>>>> de...@open-mpi.org
>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>>> Link to this post: 
>>>>> http://www.open-mpi.org/community/lists/devel/2014/06/15023.php
>>>>
>>>> ___
>>>> devel mailing list
>>>> de...@open-mpi.org
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>> Link to this post: 
>>>> http://www.open-mpi.org/community/lists/devel/2014/06/15024.php
>>>
>>>
>>> -- 
>>> Jeff Squyres
>>> jsquy...@cisco.com
>>> For corporate legal information go to: 
>>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>>
>>> ___
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post: 
>>> http://www.open-mpi.org/community/lists/devel/2014/06/15026.php
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2014/06/15027.php
>>
> 
> 
> 
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/06/15028.php
> 

-- 
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335



signature.asc
Description: OpenPGP digital signature

Re: [OMPI devel] Agenda for next week

[Edgar Gabriel]

the best time for me would be either Wednesday morning (basically any
time), or Thursday morning before 11am central.

Thanks
Edgar

On 6/19/2014 1:42 PM, Ralph Castain wrote:
> I found it on the agenda under the 1.9 branch subject - let us know when you 
> are available, Edgar
> 
> On Jun 19, 2014, at 11:29 AM, Jeff Squyres (jsquyres) <jsquy...@cisco.com> 
> wrote:
> 
>> ...and pick a time that would work for you for a webex.  :-)
>>
>> On Jun 19, 2014, at 1:48 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>
>>> I don't see that on the agenda, Edgar - can you please add it to ensure it 
>>> gets covered?
>>>
>>>
>>> On Jun 19, 2014, at 10:36 AM, Edgar Gabriel <gabr...@cs.uh.edu> wrote:
>>>
>>>> If possible, I would like to attend remotely the discussion about OMPIO
>>>> as well.
>>>>
>>>> Thanks
>>>> Edgar
>>>>
>>>> On 6/19/2014 10:44 AM, Jeff Squyres (jsquyres) wrote:
>>>>> We have a bunch of topics listed on the wiki, but no real set agenda:
>>>>>
>>>>>  https://svn.open-mpi.org/trac/ompi/wiki/Jun14Meeting
>>>>>
>>>>> We had remote-attendance requests for 2 topics, however, so I took the 
>>>>> liberty of setting up some fixed-time webexes for them (see the wiki for 
>>>>> the webex links):
>>>>>
>>>>> - Tuesday at 2pm US Central for the git discussion
>>>>> - Thursday at 9am US Central for the FT discussion
>>>>>
>>>>> Are there any other topics that people wanted to remote in to?  Fair 
>>>>> warning: remote attendance is "ok" via webex, but it's no substitute for 
>>>>> being there.
>>>>>
>>>>
>>>> -- 
>>>> Edgar Gabriel
>>>> Associate Professor
>>>> Parallel Software Technologies Lab  http://pstl.cs.uh.edu
>>>> Department of Computer Science  University of Houston
>>>> Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
>>>> Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
>>>>
>>>> ___
>>>> devel mailing list
>>>> de...@open-mpi.org
>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>>> Link to this post: 
>>>> http://www.open-mpi.org/community/lists/devel/2014/06/15023.php
>>>
>>> ___
>>> devel mailing list
>>> de...@open-mpi.org
>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>>> Link to this post: 
>>> http://www.open-mpi.org/community/lists/devel/2014/06/15024.php
>>
>>
>> -- 
>> Jeff Squyres
>> jsquy...@cisco.com
>> For corporate legal information go to: 
>> http://www.cisco.com/web/about/doing_business/legal/cri/
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post: 
>> http://www.open-mpi.org/community/lists/devel/2014/06/15026.php
> 
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/06/15027.php
> 

-- 
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335



signature.asc
Description: OpenPGP digital signature

Re: [OMPI devel] Agenda for next week

[Edgar Gabriel]

If possible, I would like to attend remotely the discussion about OMPIO
as well.

Thanks
Edgar

On 6/19/2014 10:44 AM, Jeff Squyres (jsquyres) wrote:
> We have a bunch of topics listed on the wiki, but no real set agenda:
> 
> https://svn.open-mpi.org/trac/ompi/wiki/Jun14Meeting
> 
> We had remote-attendance requests for 2 topics, however, so I took the 
> liberty of setting up some fixed-time webexes for them (see the wiki for the 
> webex links):
> 
> - Tuesday at 2pm US Central for the git discussion
> - Thursday at 9am US Central for the FT discussion
> 
> Are there any other topics that people wanted to remote in to?  Fair warning: 
> remote attendance is "ok" via webex, but it's no substitute for being there.
> 

-- 
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335



signature.asc
Description: OpenPGP digital signature

Re: [OMPI devel] RFC: add STCI component to OMPI/RTE framework

[Edgar Gabriel]

__
> >>> Thomas Naughton
>  naught...@ornl.gov <mailto:naught...@ornl.gov>
> >>> Research Associate   (865)
> 576-4184 <tel:%28865%29%20576-4184>
> >>>
> >>> ___
> >>> devel mailing list
> >>> de...@open-mpi.org <mailto:de...@open-mpi.org>
> >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> >>> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/05/14852.php
> >>
> >>
> >> --
> >> Jeff Squyres
> >> jsquy...@cisco.com <mailto:jsquy...@cisco.com>
> >> For corporate legal information go to:
> http://www.cisco.com/web/about/doing_business/legal/cri/
> >>
> >> ___
> >> devel mailing list
> >> de...@open-mpi.org <mailto:de...@open-mpi.org>
> >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> >> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/05/14904.php
> >>
> > ___
> > devel mailing list
> > de...@open-mpi.org <mailto:de...@open-mpi.org>
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> > Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/05/14905.php
> 
> ___
> devel mailing list
> de...@open-mpi.org <mailto:de...@open-mpi.org>
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/05/14906.php
> 
> 
> 
> 
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/05/14907.php
> 

-- 
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335



signature.asc
Description: OpenPGP digital signature

Re: [OMPI devel] RFC: add STCI component to OMPI/RTE framework

[Edgar Gabriel]

not really, I stated my case, there is not much more to add. Its up to
the group to decide, and I am fine with any decision.

Edgar

On 5/27/2014 2:57 PM, Ralph Castain wrote:
> Forgot to add: would it help to discuss this over the phone instead?
> 
> 
> On May 27, 2014, at 12:56 PM, Ralph Castain <r...@open-mpi.org
> <mailto:r...@open-mpi.org>> wrote:
> 
>>
>> On May 27, 2014, at 12:50 PM, Edgar Gabriel <gabr...@cs.uh.edu
>> <mailto:gabr...@cs.uh.edu>> wrote:
>>
>>>
>>>
>>> On 5/27/2014 2:46 PM, Ralph Castain wrote:
>>>>
>>>> On May 27, 2014, at 12:27 PM, Edgar Gabriel <gabr...@cs.uh.edu
>>>> <mailto:gabr...@cs.uh.edu>>
>>>> wrote:
>>>>
>>>>> I'll let ORNL talk about the STCI component itself (which might
>>>>> have additional reasons), but keeping the code in trunk vs. an
>>>>> outside github/mercurial repository has two advantages in my
>>>>> opinion: i) it simplifies the propagation of know-how between the
>>>>> groups,
>>>>
>>>> Afraid I don't understand that - this is just glue, right?
>>>
>>>
>>> yes, but its easier to look in one place vs. n places for every features.
>>>
>>>>> and ii) avoids having to keep a separate branch up to date. (We did
>>>>> the second part with OMPIO for a couple of years, and that was
>>>>> really painful).
>>>>
>>>> Ah, perhaps this is the "rub" - are you saying that you expect us to
>>>> propagate any changes in the RTE interface to your component? If so,
>>>> then that violates the original agreement about this framework. It
>>>> was solely to provide a point-of-interface for *external* groups to
>>>> connect their RTE's into OMPI. We agreed that we would notify people
>>>> of changes to that interface, and give them a chance to provide input
>>>> on those changes - but under no conditions were we wiling to accept
>>>> responsibility for maintaining those branch interfaces.
>>>>
>>>> Given that the interface is wholly contained in the ompi/rte
>>>> component, I guess I struggle to understand the code conflict issue.
>>>> There is no change in the OMPI code base that can possibly conflict
>>>> with your component. The only things that could impact you are
>>>> changes in the OMPI layer that require modification to your
>>>> component, which is something you'd have to do regardless. We will
>>>> not test nor update that component for you.
>>>
>>>
>>> no, not all. My point was that we invested enormous efforts at that time
>>> to just do the svn merge from the changes on trunk to our branch, that's
>>> all.
>>>
>>
>> If you are on a branch that contains an svn checkout of the trunk,
>> plus one component directory in one framework, then I'm afraid I
>> cannot understand how you get merge conflicts. I've been doing this
>> for years and haven't hit one yet. The only possible source of a
>> conflict is if I touch code that is common to the two repos - i.e.,
>> outside of the area that I'm adding. In this case, that should never
>> happen, yes?
>>
>> If it does, then you touched code outside your component, and you
>> either (a) are going to encounter this no matter what because you
>> haven't pushed it up yet, or (b) couldn't commit that up to the main
>> repo anyway if it impacted the RTE interface.
>>
>> Sorry, but I'm really struggling to understand how adding only this
>> one component, which you solely modify and control, can possibly help
>> with maintaining your branch.
>>
>>
>>> Thanks
>>> Edgar
>>>
>>>>
>>>>>
>>>>> In addition, IANAL, but I was actually wandering about the
>>>>> implications of using separate code repositories outside of ompi
>>>>> for sharing code, and whether that is truly still covered by the
>>>>> contributors agreement that we all signed.
>>>>
>>>> Of course not - OMPI's license only declares that anything you push
>>>> into the main OMPI code repo (and hence, our official releases) is
>>>> covered by that agreement. Anything you add or distribute externally
>>>> is on your own. You can *choose* to license that code in accordance
>>>> with the OMPI license, but you aren't *required* to do so.
>>>>
>>>>>
>>>>

Re: [OMPI devel] RFC: add STCI component to OMPI/RTE framework

[Edgar Gabriel]

On 5/27/2014 2:46 PM, Ralph Castain wrote:
> 
> On May 27, 2014, at 12:27 PM, Edgar Gabriel <gabr...@cs.uh.edu>
> wrote:
> 
>> I'll let ORNL talk about the STCI component itself (which might
>> have additional reasons), but keeping the code in trunk vs. an
>> outside github/mercurial repository has two advantages in my
>> opinion: i) it simplifies the propagation of know-how between the
>> groups,
> 
> Afraid I don't understand that - this is just glue, right?


yes, but its easier to look in one place vs. n places for every features.

>> and ii) avoids having to keep a separate branch up to date. (We did
>> the second part with OMPIO for a couple of years, and that was
>> really painful).
> 
> Ah, perhaps this is the "rub" - are you saying that you expect us to
> propagate any changes in the RTE interface to your component? If so,
> then that violates the original agreement about this framework. It
> was solely to provide a point-of-interface for *external* groups to
> connect their RTE's into OMPI. We agreed that we would notify people
> of changes to that interface, and give them a chance to provide input
> on those changes - but under no conditions were we wiling to accept
> responsibility for maintaining those branch interfaces.
> 
> Given that the interface is wholly contained in the ompi/rte
> component, I guess I struggle to understand the code conflict issue.
> There is no change in the OMPI code base that can possibly conflict
> with your component. The only things that could impact you are
> changes in the OMPI layer that require modification to your
> component, which is something you'd have to do regardless. We will
> not test nor update that component for you.


no, not all. My point was that we invested enormous efforts at that time
to just do the svn merge from the changes on trunk to our branch, that's
all.

Thanks
Edgar

> 
>> 
>> In addition, IANAL, but I was actually wandering about the
>> implications of using separate code repositories outside of ompi
>> for sharing code, and whether that is truly still covered by the
>> contributors agreement that we all signed.
> 
> Of course not - OMPI's license only declares that anything you push
> into the main OMPI code repo (and hence, our official releases) is
> covered by that agreement. Anything you add or distribute externally
> is on your own. You can *choose* to license that code in accordance
> with the OMPI license, but you aren't *required* to do so.
> 
>> 
>> Anyway, I don't have strong feelings either way as well, just would
>> see a couple of advantages (for us) if the code was in the trunk.
> 
> I'm still trying to understand those - sorry to be a pain, but my
> biggest fear at this point is that the perceived advantage is based
> on a misunderstanding, and I'd like to head that off before it causes
> problems.
> 
>> 
>> Thanks Edgar
>> 
>> On 5/27/2014 1:45 PM, Ralph Castain wrote:
>>> I think so long as we leave these components out of any release, 
>>> there is a limited potential for problems (probably most
>>> importantly, we sidestep all the issues about syncing
>>> releases!).
>>> 
>>> However, that said, I'm not sure what it gains anyone to include
>>> a component that *isn't* going in a release. Nobody outside your 
>>> organizations is going to build against it - so what did it 
>>> accomplish to push the code into the repo?
>>> 
>>> Mind you, I'm not saying I'm staunchly opposed - just trying to 
>>> understand how it benefits anyone.
>>> 
>>> 
>>> On May 27, 2014, at 11:28 AM, Edgar Gabriel <gabr...@cs.uh.edu> 
>>> wrote:
>>> 
>>>> To through in my $0.02, I would see a benefit in adding the 
>>>> component to the trunk. As I mentioned in the last teleconf, we
>>>> are currently working on adding support for the HPX runtime
>>>> environment to Open MPI, and for various reasons (that I can
>>>> explain if somebody is interested), we think at the moment that
>>>> using the RTE abstraction layer could be easier for achieving
>>>> what we want to do. That is not at all a judgment on ORTE, but
>>>> a combination of what HPX offers and what we want to achieve
>>>> within that project.
>>>> 
>>>> I do not foresee at this point that our component would be 
>>>> production quality or part of an upcoming OMPI release, having 
>>>> however another component in the rte framework  could be
>>>> useful from our point of view. (And yes, the person that asked
>

Re: [OMPI devel] RFC: add STCI component to OMPI/RTE framework

[Edgar Gabriel]

I'll let ORNL talk about the STCI component itself (which might have
additional reasons), but keeping the code in trunk vs. an outside
github/mercurial repository has two advantages in my opinion: i) it
simplifies the propagation of know-how between the groups, and ii)
avoids having to keep a separate branch up to date. (We did the second
part with OMPIO for a couple of years, and that was really painful).

In addition, IANAL, but I was actually wandering about the implications
of using separate code repositories outside of ompi for sharing code,
and whether that is truly still covered by the contributors agreement
that we all signed.

Anyway, I don't have strong feelings either way as well, just would see
a couple of advantages (for us) if the code was in the trunk.

Thanks
Edgar

On 5/27/2014 1:45 PM, Ralph Castain wrote:
> I think so long as we leave these components out of any release,
> there is a limited potential for problems (probably most importantly,
> we sidestep all the issues about syncing releases!).
> 
> However, that said, I'm not sure what it gains anyone to include a
> component that *isn't* going in a release. Nobody outside your
> organizations is going to build against it - so what did it
> accomplish to push the code into the repo?
> 
> Mind you, I'm not saying I'm staunchly opposed - just trying to
> understand how it benefits anyone.
> 
> 
> On May 27, 2014, at 11:28 AM, Edgar Gabriel <gabr...@cs.uh.edu>
> wrote:
> 
>> To through in my $0.02, I would see a benefit in adding the
>> component to the trunk. As I mentioned in the last teleconf, we are
>> currently working on adding support for the HPX runtime environment
>> to Open MPI, and for various reasons (that I can explain if
>> somebody is interested), we think at the moment that using the RTE
>> abstraction layer could be easier for achieving what we want to do.
>> That is not at all a judgment on ORTE, but a combination of what
>> HPX offers and what we want to achieve within that project.
>> 
>> I do not foresee at this point that our component would be
>> production quality or part of an upcoming OMPI release, having
>> however another component in the rte framework  could be useful
>> from our point of view. (And yes, the person that asked the pmi/rte
>> question on the mailing list was my student who tried to make the
>> pmi component work, and was confused about the fact that other
>> emails said that the pmi stuff is working, while he couldn't even
>> get it to compile :)
>> 
>> Edgar
>> 
>> On 5/27/2014 12:23 PM, Ralph Castain wrote:
>>> I have mixed thoughts on this request. We have a policy of only 
>>> including things in the code base that are of general utility -
>>> i.e., that should be generally distributed across the community.
>>> This component is only applicable to ORNL, and it would therefore
>>> seem more sensible to have it continue to be maintained there.
>>> 
>>> I'm unaware of anyone outside of ORNL that regularly tests for 
>>> ompi-rte abstraction violations, and I suspect that your
>>> internal tests are the right place for catching them as nobody
>>> else really seems to care. It isn't clear to me how adding this
>>> component directly to the general code base impacts that
>>> operation. The original PMI component in the ompi/rte framework
>>> wasn't intended to provide an alternative method for building
>>> OMPI - it was solely created to support the development of that
>>> framework and had no intended utility beyond that time (hence the
>>> RFC to remove it to avoid user confusion as we just saw on the
>>> user mailing list).
>>> 
>>> 
>>> On May 27, 2014, at 9:25 AM, Thomas Naughton
>>> <naught...@ornl.gov> wrote:
>>> 
>>>> 
>>>> WHAT:  add new component to ompi/rte framework
>>>> 
>>>> WHY:   because it will simplify our maintenance & provide an
>>>> alt. reference
>>>> 
>>>> WHEN:  no rush, soon-ish? (June 12?)
>>>> 
>>>> This is a component we currently maintain outside of the ompi
>>>> tree to support using OMPI with an alternate runtime system.
>>>> This will also provide an alternate component to ORTE, which
>>>> was motivation for PMI component in related RFC.   We
>>>> build/test nightly and it occasionally catches ompi-rte
>>>> abstraction violations, etc.
>>>> 
>>>> Thomas
>>>> 
>>>> _
>>>>
>>>

Re: [OMPI devel] RFC: add STCI component to OMPI/RTE framework

[Edgar Gabriel]

To through in my $0.02, I would see a benefit in adding the component to
the trunk. As I mentioned in the last teleconf, we are currently working
on adding support for the HPX runtime environment to Open MPI, and for
various reasons (that I can explain if somebody is interested), we think
at the moment that using the RTE abstraction layer could be easier for
achieving what we want to do. That is not at all a judgment on ORTE, but
a combination of what HPX offers and what we want to achieve within that
project.

I do not foresee at this point that our component would be production
quality or part of an upcoming OMPI release, having however another
component in the rte framework  could be useful from our point of view.
(And yes, the person that asked the pmi/rte question on the mailing list
was my student who tried to make the pmi component work, and was
confused about the fact that other emails said that the pmi stuff is
working, while he couldn't even get it to compile :)

Edgar

On 5/27/2014 12:23 PM, Ralph Castain wrote:
> I have mixed thoughts on this request. We have a policy of only
> including things in the code base that are of general utility - i.e.,
> that should be generally distributed across the community. This
> component is only applicable to ORNL, and it would therefore seem
> more sensible to have it continue to be maintained there.
> 
> I'm unaware of anyone outside of ORNL that regularly tests for
> ompi-rte abstraction violations, and I suspect that your internal
> tests are the right place for catching them as nobody else really
> seems to care. It isn't clear to me how adding this component
> directly to the general code base impacts that operation. The
> original PMI component in the ompi/rte framework wasn't intended to
> provide an alternative method for building OMPI - it was solely
> created to support the development of that framework and had no
> intended utility beyond that time (hence the RFC to remove it to
> avoid user confusion as we just saw on the user mailing list).
> 
> 
> On May 27, 2014, at 9:25 AM, Thomas Naughton <naught...@ornl.gov>
> wrote:
> 
>> 
>> WHAT:  add new component to ompi/rte framework
>> 
>> WHY:   because it will simplify our maintenance & provide an alt.
>> reference
>> 
>> WHEN:  no rush, soon-ish? (June 12?)
>> 
>> This is a component we currently maintain outside of the ompi tree
>> to support using OMPI with an alternate runtime system.  This will
>> also provide an alternate component to ORTE, which was motivation
>> for PMI component in related RFC.   We build/test nightly and it
>> occasionally catches ompi-rte abstraction violations, etc.
>> 
>> Thomas
>> 
>> _
>>
>> 
Thomas Naughton  naught...@ornl.gov
>> Research Associate   (865)
>> 576-4184
>> 
>> ___ devel mailing list 
>> de...@open-mpi.org Subscription:
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel Link to this
>> post:
>> http://www.open-mpi.org/community/lists/devel/2014/05/14852.php
> 
> _______ devel mailing list 
> de...@open-mpi.org Subscription:
> http://www.open-mpi.org/mailman/listinfo.cgi/devel Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/05/14854.php
> 

-- 
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335



signature.asc
Description: OpenPGP digital signature

Re: [OMPI devel] Wrong Endianness in Open MPI for external32 representation

[Edgar Gabriel]

the way you launch the app, you will be using ROMIO, and I am not 100%
sure about how the data representation stuff is integrated with OMPI. I
am pretty sure that we are not doing the right thing for OMPIO, but I
will look into later this week.

Thanks
Edgar

On 4/29/2014 7:03 AM, Christoph Niethammer wrote:
> Hello,
> 
> It seems for me that the endianness is wrong in Open MPI's I/O for the 
> external32 data representation. :O
> 
> Find attached my test programs which write the integer -30 and the double 
> 16.25 into a file.
> Here the output on my Linux system:
> 
> mpicc c-xdr.c   -o c-xdr
> mpicc mpi-io-external32.c   -o mpi-io-external32
> ./c-xdr
> Output file: c-xdr.dat
> hexdump c-xdr.dat
> 000  e2ff 3040 0040    
> 00c
> ./mpi-io-external32
> Output file: mpi-io-external32.dat
> hexdump mpi-io-external32.dat
> 000 ffe2    4000 4030  
> 00c
> 
> 
> Best regards
> Christoph Niethammer
> 
> --
> 
> Christoph Niethammer
> High Performance Computing Center Stuttgart (HLRS)
> Nobelstrasse 19
> 70569 Stuttgart
> 
> Tel: ++49(0)711-685-87203
> email: nietham...@hlrs.de
> http://www.hlrs.de/people/niethammer
> 
> 
> 
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2014/04/14648.php
> 

-- 
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335



signature.asc
Description: OpenPGP digital signature

Re: [OMPI devel] 1-question developer poll

[Edgar Gabriel]

mostly svn, sometimes mercurial, no git.

Edgar

On 4/16/2014 5:32 AM, Jeff Squyres (jsquyres) wrote:
> What source code repository technology(ies) do you use for Open MPI 
> development? (indicate all that apply)
> 
> - SVN
> - Mercurial
> - Git
> 
> I ask this question because there's serious discussions afoot to switch 
> OMPI's main SVN repo to Git, and I want to get a feel for the current 
> landscape out there.
> 

-- 
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335



signature.asc
Description: OpenPGP digital signature

Re: [OMPI devel] Annual OMPI membership review: SVN accounts

[Edgar Gabriel]

both accounts for UH should stay active.


Edgar
On 7/8/2013 5:32 PM, Jeff Squyres (jsquyres) wrote:
> According to https://svn.open-mpi.org/trac/ompi/wiki/Admistrative%20rules, it 
> is time for our annual review of Open MPI SVN accounts of these SVN repos: 
> hwloc, mtt, ompi-docs, ompi-tests, ompi-www, ompi.
> 
> *** Organizations must reply by COB Friday, 12 July, 2013 ***
> *** No reply means: delete all of my organization's SVN accounts
> 
> Each organization must reply and specify which of their accounts can stay and 
> which should go.  I cross-referenced the SVN logs from all of our SVN 
> repositories to see who has not committed anything in the past year.  
> 
> *** I strongly recommend deleting accounts who have not committed in the last 
> year.
> *** Other accounts can be deleted, too (e.g., those who have left a given 
> organization).
> 
> bakeyournoodle.com (???)
> ==
> tonyb:Tony Breeds <t...@bakeyournoodle.com> **NO COMMITS IN LAST YEAR**
> 
> Cisco
> =
> dgoodell: Dave Goodell <dgood...@cisco.com>
> jsquyres: Jeff Squyres <jsquy...@cisco.com>
> 
> Indiana
> ==
> lums: Andrew Lumsdaine <l...@cs.indiana.edu> **NO COMMITS IN LAST YEAR**
> adkulkar: Abhishek Kulkarni <adkul...@osl.iu.edu>
> afriedle: Andrew Friedley <afrie...@osl.iu.edu> **NO COMMITS IN LAST YEAR**
> timattox: Tim Mattox <timat...@open-mpi.org> **NO COMMITS IN LAST YEAR**
> 
> U. Houston
> =
> edgar:Edgar Gabriel <gabr...@cs.uh.edu>
> vvenkatesan:Vishwanath Venkatesan <venka...@cs.uh.edu>
> 
> Mellanox
> ==
> alekseys: Aleksey Senin <aleks...@dev.mellanox.co.il>
> kliteyn:  Yevgeny Kliteynik <klit...@dev.mellanox.co.il>
> miked:Mike Dubman <mi...@dev.mellanox.co.il>
> lennyve:  Lenny Verkhovsky <lenny.verkhov...@gmail.com> **NO COMMITS IN LAST 
> YEAR**
> yaeld:Yael Dayan <yaeld.mella...@gmail.com>
> vasily:   Vasily Philipov <vas...@mellanox.co.il>
> amikheev: Alex Mikheev <al...@mellanox.com>
> alex: Alexander Margolin <ale...@mellanox.com>
> alinas:   Alina Sklarevich <ali...@mellanox.com> **NO COMMITS IN LAST YEAR**
> igoru:Igor Usarov <ig...@mellanox.com>
> jladd:Joshua Ladd <josh...@mellanox.com>
> yosefe:   Yossi <yos...@mellanox.com>
> rlgraham: Rich Graham <rlgra...@ornl.gov> **NO COMMITS IN LAST YEAR**
> 
> Tennessee
> 
> bosilca:  George Bosilca <bosi...@eecs.utk.edu>
> bouteill: Aurelien Bouteiller <boute...@eecs.utk.edu>
> wbland:   Wesley Bland <wbl...@mcs.anl.gov> **NO COMMITS IN LAST YEAR**
> 
> hlrs.de
> ===
> shiqing:  Shiqing Fan <f...@hlrs.de>
> hpcchris: Christoph Niethammer <nietham...@hlrs.de>
> rusraink: Rainer Keller <rainer.kel...@hft-stuttgart.de> **NO COMMITS IN LAST 
> YEAR**
> 
> IBM
> ==
> jnysal:   Nysal Jan K A <jny...@in.ibm.com> **NO COMMITS IN LAST YEAR**
> cyeoh:Chris Yeoh <cy...@au1.ibm.com>
> bbenton:  Brad Benton <brad.ben...@us.ibm.com>
> 
> INRIA
> 
> bgoglin:  Brice Goglin <brice.gog...@inria.fr>
> arougier: Antoine Rougier <antoine.roug...@inria.fr>
> sthibaul: Samuel Thibault <samuel.thiba...@inria.fr>
> mercier:  Guillaume Mercier <merc...@labri.fr> **NO COMMITS IN LAST YEAR**
> nfurmento:Nathalie Furmento <nathalie.furme...@labri.fr> **NO COMMITS IN LAST 
> YEAR**
> herault:  Thomas Herault <thomas.hera...@lri.fr> **NO COMMITS IN LAST YEAR**
> 
> LANL
> 
> hjelmn:   Nathan Hjelm <hje...@lanl.gov>
> samuel:   Samuel K. Gutierrez <sam...@lanl.gov>
> 
> NVIDIA
> ==
> rolfv:Rolf Vandevaart <rvandeva...@nvidia.com>
> 
> U. Wisconsin La Crosse
> 
> jjhursey: Joshua Hursey <jjhur...@open-mpi.org>
> 
> Intel
> 
> rhc:  Ralph Castain <r...@open-mpi.org>
> 
> Chelsio / OGC
> =
> swise:Steve Wise <sw...@opengridcomputing.com>
> 
> Oracle
> ==
> emallove: Ethan Mallove <ethan.mall...@oracle.com> **NO COMMITS IN LAST YEAR**
> eugene:   Eugene Loh <eugene@oracle.com>
> tdd:  Terry Dontje <terry.don...@oracle.com>
> 
> ORNL
> 
> manjugv:  Manjunath, Gorentla Venkata <manj...@ornl.gov>
> naughtont:Thomas Naughton <naught...@ornl.gov>
> pasha:Pavel Shamis <sham...@ornl.gov>
> 
> Sandia
> ==
> brbarret: Brian Barrett <bwba...@sandia.gov>
> memoryhole:Kyle Wheeler <kbwh...@sandia.gov> **NO COMMITS IN LAST 

Re: [OMPI devel] sbgp problem

[Edgar Gabriel]

as far as I can tell right now, yes, its the final thing...

Thanks
Edgar

On 10/30/2012 2:05 PM, Ralph Castain wrote:
> Grrbloody verb @##$@$.
> 
> Okay, I'll make that edit. Is that the last thing required to fix this 
> problem?
> 
> On Oct 30, 2012, at 11:57 AM, Edgar Gabriel <gabr...@cs.uh.edu> wrote:
> 
>> so the sbgp problem that I mentioned on the call this morning
>> unfortunately is not resolved by just adding the common/verbs directory
>> into the 1.7 branch.
>>
>> We looked a bit into it, and the problem/difference between in the file
>> ompi/sbgp/ibnet/sbgp_ibnet_component.c which has the following include
>> statement:
>>
>> #include "ompi/mca/common/ofautils/common_ofautils.h"
>>
>> The same file on trunk includes however
>>
>> #include "ompi/mca/common/verbs/common_verbs.h"
>>
>> If I make this change in the 1.7, things compile properly, otherwise we
>> still have the error message
>>
>> bgp_ibnet_component.c: In function 'ibnet_load_devices':
>> sbgp_ibnet_component.c:527:5: error: implicit declaration of function
>> 'ompi_ibv_get_device_list'
>> sbgp_ibnet_component.c:527:13: warning: assignment makes pointer from
>> integer without a cast
>> sbgp_ibnet_component.c:553:5: error: implicit declaration of function
>> 'ompi_ibv_free_device_list'
>> make[2]: *** [sbgp_ibnet_component.lo] Error 1
>> make[2]: *** Waiting for unfinished jobs
>> make[1]: *** [all-recursive] Error 1
>> make: *** [all-recursive] Error 1
>>
>>
>> Thanks
>> Edgar
>> -- 
>> Edgar Gabriel
>> Associate Professor
>> Parallel Software Technologies Lab  http://pstl.cs.uh.edu
>> Department of Computer Science  University of Houston
>> Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
>> Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
>>
>> _______
>> devel mailing list
>> de...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
> 
> 
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
> 

-- 
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335



signature.asc
Description: OpenPGP digital signature

Re: [OMPI devel] [OMPI svn] svn:open-mpi r26934 - trunk/ompi/mca/io/ompio

[Edgar Gabriel]

oops, thanks Ralph, overlooked that one...

Edgar

On 7/31/2012 11:08 PM, svn-commit-mai...@open-mpi.org wrote:
> Author: rhc (Ralph Castain)
> Date: 2012-08-01 00:08:47 EDT (Wed, 01 Aug 2012)
> New Revision: 26934
> URL: https://svn.open-mpi.org/trac/ompi/changeset/26934
> 
> Log:
> Add missing include file
> 
> Text files modified: 
>trunk/ompi/mca/io/ompio/Makefile.am | 1 +  
>  
>1 files changed, 1 insertions(+), 0 deletions(-)
> 
> Modified: trunk/ompi/mca/io/ompio/Makefile.am
> ==
> --- trunk/ompi/mca/io/ompio/Makefile.am   Tue Jul 31 17:50:34 2012
> (r26933)
> +++ trunk/ompi/mca/io/ompio/Makefile.am   2012-08-01 00:08:47 EDT (Wed, 
> 01 Aug 2012)  (r26934)
> @@ -42,6 +42,7 @@
>  
>  sources = \
>  io_ompio.h \
> +io_ompio_nbc.h \
>  io_ompio.c \
>  io_ompio_component.c \
>  io_ompio_module.c \
> ___
> svn mailing list
> s...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/svn
> 

-- 
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335



signature.asc
Description: OpenPGP digital signature

Re: [OMPI devel] Warning in fcoll

[Edgar Gabriel]

I'll look into this...

Edgar

On 5/29/2012 3:24 PM, Ralph Castain wrote:
> Not entirely sure who this might belong to, but thought I should pass it 
> along - seen during an optimized build on Linux:
> 
> fcoll_static_file_read_all.c: In function ‘mca_fcoll_static_file_read_all’:
> fcoll_static_file_read_all.c:74: warning: ‘sorted_file_offsets’ may be used 
> uninitialized in this function
> 
> 
> 
> 
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

-- 
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335



signature.asc
Description: OpenPGP digital signature

Re: [OMPI devel] Trunk build problem

[Edgar Gabriel]

sorry, should be fixed with the last commit...

Thanks
Edgar

On 2/28/2012 8:37 AM, Edgar Gabriel wrote:
> I'll look into this...
> 
> Thanks
> Edgar
> 
> On 2/28/2012 8:36 AM, Ralph Castain wrote:
>> I tried to build the trunk this morning on a machine where the fcoll 
>> framework could build and hit this:
>>
>> mca/fcoll/dynamic/.libs/libmca_fcoll_dynamic.a(fcoll_dynamic_file_write_all.o):
>>  In function `local_heap_sort':
>> /users/rcastain/openmpi-1.7a1/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c::
>>  multiple definition of `local_heap_sort'
>> mca/fcoll/static/.libs/libmca_fcoll_static.a(fcoll_static_file_write_all.o):/users/rcastain/openmpi-1.7a1/ompi/mca/fcoll/static/fcoll_static_file_write_all.c:929:
>>  first defined here
>>
>>
>> Any suggestions?
>> Ralph
>>
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
> 
> 
> 
> _______
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

-- 
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335



signature.asc
Description: OpenPGP digital signature

Re: [OMPI devel] Trunk build problem

[Edgar Gabriel]

I'll look into this...

Thanks
Edgar

On 2/28/2012 8:36 AM, Ralph Castain wrote:
> I tried to build the trunk this morning on a machine where the fcoll 
> framework could build and hit this:
> 
> mca/fcoll/dynamic/.libs/libmca_fcoll_dynamic.a(fcoll_dynamic_file_write_all.o):
>  In function `local_heap_sort':
> /users/rcastain/openmpi-1.7a1/ompi/mca/fcoll/dynamic/fcoll_dynamic_file_write_all.c::
>  multiple definition of `local_heap_sort'
> mca/fcoll/static/.libs/libmca_fcoll_static.a(fcoll_static_file_write_all.o):/users/rcastain/openmpi-1.7a1/ompi/mca/fcoll/static/fcoll_static_file_write_all.c:929:
>  first defined here
> 
> 
> Any suggestions?
> Ralph
> 
> 
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

-- 
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335



signature.asc
Description: OpenPGP digital signature

Re: [OMPI devel] poor btl sm latency

[Edgar Gabriel]

just a stupid question: in another sm-related thread the value of the
$TMPDIR variable was the problem, could this be the problem here as well?

Edgar

On 2/16/2012 9:30 AM, Matthias Jurenz wrote:
> On Thursday 16 February 2012 16:21:10 Jeff Squyres wrote:
>> On Feb 16, 2012, at 9:39 AM, Matthias Jurenz wrote:
>>> However, the latencies are constant now but still too high:
>>>
>>> $ mpirun -np 2 --bind-to-core --cpus-per-proc 2 ./NPmpi_ompi1.5.5 -S -u
>>> 12 -n 10
>>
>> Can you run:
>>
>> mpirun -np 2 --bind-to-core --cpus-per-proc 2 hwloc-bind --get
>>
>> I want to verify that the processes are actually getting bound to the
>> correct places.
> 
> Sure:
> $ mpirun -np 2 --bind-to-core --cpus-per-proc 2 hwloc-bind --get
> 0x0003
> 0x000c
> 
> Matthias
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

-- 
Edgar Gabriel
Associate Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335



signature.asc
Description: OpenPGP digital signature

Re: [OMPI devel] RFC: adding OMPIO module and new parallel I/O frameworks to trunk

[Edgar Gabriel]

the code has been committed in r25079. Let us know if there are any
issues, compilation problems etc. I also work on an FAQ entry as has
been suggested on the teleconf last week.

Thanks
Edgar

On 8/12/2011 3:09 PM, Edgar Gabriel wrote:
> WHAT: add the ompio io module and new parallel io frameworks to trunk
> 
> TIMEOUT: 08/22/2011
> 
> DETAILS:
> 
> we would like to bring the ompio module and the new parallel io
> frameworks to the trunk to expose it to a wider set of platforms and
> more widespread testing. The current functionality - while not yet
> providing the full functionality of MPI I/O - is already useful, since
> all blocking individual and collective I/O operations (arbitrary file
> views, implicit and explicit offset) are supported. The missing
> functionality ( non-blocking individual I/O and shared file pointers)
> are currently being worked on and exist partially as stand-alone libraries.
> 
> The current goal is to enable compilation of the new modules and
> frameworks per default, but not use them unless explicitly requested by
> the user, i.e. ROMIO will remain the default selection.
> 
> Very little existing code is being touched, since most of the new code
> are new frameworks and modules. If somebody would like to have a look at
> the code and the changes, here isa public repository with the code.
> 
> https://bitbucket.org/mschaara/ompio
> 
> The list of modified files are:
> ?   ompi/mca/fbtl
> ?   ompi/mca/sharedfp
> ?   ompi/mca/fcoll
> ?   ompi/mca/fs
> ?   ompi/mca/fcache
> ?   ompi/mca/io/ompio
> M   ompi/mca/io/romio/src/io_romio_component.c
> M   ompi/mca/io/base/io_base_file_select.c
> M   ompi/mca/io/base/io_base_delete.c
> ?   ompi/config/ompi_check_pvfs2.m4
> ?   ompi/config/ompi_check_lustre.m4
> 
> Feedback is highly welcome.
> 
> Thanks
> Edgar
> 
> 
> 
> 
> ___________
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

-- 
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335



signature.asc
Description: OpenPGP digital signature

[OMPI devel] RFC: adding OMPIO module and new parallel I/O frameworks to trunk

[Edgar Gabriel]

WHAT: add the ompio io module and new parallel io frameworks to trunk

TIMEOUT: 08/22/2011

DETAILS:

we would like to bring the ompio module and the new parallel io
frameworks to the trunk to expose it to a wider set of platforms and
more widespread testing. The current functionality - while not yet
providing the full functionality of MPI I/O - is already useful, since
all blocking individual and collective I/O operations (arbitrary file
views, implicit and explicit offset) are supported. The missing
functionality ( non-blocking individual I/O and shared file pointers)
are currently being worked on and exist partially as stand-alone libraries.

The current goal is to enable compilation of the new modules and
frameworks per default, but not use them unless explicitly requested by
the user, i.e. ROMIO will remain the default selection.

Very little existing code is being touched, since most of the new code
are new frameworks and modules. If somebody would like to have a look at
the code and the changes, here isa public repository with the code.

https://bitbucket.org/mschaara/ompio

The list of modified files are:
?   ompi/mca/fbtl
?   ompi/mca/sharedfp
?   ompi/mca/fcoll
?   ompi/mca/fs
?   ompi/mca/fcache
?   ompi/mca/io/ompio
M   ompi/mca/io/romio/src/io_romio_component.c
M   ompi/mca/io/base/io_base_file_select.c
M   ompi/mca/io/base/io_base_delete.c
?   ompi/config/ompi_check_pvfs2.m4
?   ompi/config/ompi_check_lustre.m4

Feedback is highly welcome.

Thanks
Edgar

-- 
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335



signature.asc
Description: OpenPGP digital signature

[OMPI devel] problems compiling new ROMIO with PVFS2 support

[Edgar Gabriel]

we recently encountered a problem here at UH when compiling the new
ROMIO version in the trunk with PVFS2 support. The error that we are
getting is a list of

ad_pvfs2_io_dtype.c:581:13: error: switch quantity not an integer
ad_pvfs2_io_dtype.c:583:2: error: pointers are not permitted as case values

(with a lot more similar error messages).

The error is due to a code sequence in that file which does something like:

switch (mpi_dtype)
{
case MPI_CHAR:
do something;
break;
case MPI_BYTE:
do something;
break;
case MPI_SHORT:
 ...
 }

This works for MPICH, but not for Open MPI since the datatypes are
pointers. Anyway, I have a fix which converts this switch statement in
the according ROMIO file to an

   if ( MPI_CHAR == mpi_dtype )
   else if ( == mpi_dtype )
   else if ...
   else


sequence. I was just wandering whether to commit the code to trunk,
since it modifies a package that has been brought in from outside..

(There is btw. a second warning in the file which makes me a bit
nervous, but that is a warning and the file still compiles, while the
other one is an error and the compilation aborts...

ad_pvfs2_io_dtype.c:264:6: warning: passing argument 6 of
âPMPI_Type_get_contentsâ from incompatible pointer type
../../../../../../../ompi/include/mpi.h:1985:20: note: expected
âMPI_Aint *â but argument is of type âint *â

since an MPI_Aint really should be a long on this platform, not an int
)

Thanks
Edgar
-- 
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335



signature.asc
Description: OpenPGP digital signature

Re: [OMPI devel] MPI_File_get_size fails for files > 2 GB in Fortran

[Edgar Gabriel]

well, to contradict my own position from previously, this function was
part of the MPI-2 specification, and the only interfaces published for
this function were C, C++ and F90. Not sure what this means for f90
applications however.

Thanks
Edgar

On 12/20/2010 3:59 PM, N.M. Maclaren wrote:
> On Dec 20 2010, George Bosilca wrote:
> 
>> There is a hint for F77 users at the bottom of the page. It suggests
>> to use INTEGER*MPI_OFFSET_KIND as type for the SIZE. I guess if we
>> cast it correctly, and the users follow the MPI specification, this
>> should work.
> 
> Please tell me you are joking?
> 
> No, that will NOT work, in general.  Firstly, the xxx in INTEGER*xxx was
> introduced in Fortran IV as a digit string and not a constant, and all
> of the compilers I know of still do that.  Secondly, it is NOT always
> aligned with the KIND values, and there is no implication in any document
> I know of that it should be.
> 
> Regards,
> Nick Maclaren.
> 
> 
> 
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

-- 
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335



signature.asc
Description: OpenPGP digital signature

Re: [OMPI devel] MPI_File_get_size fails for files > 2 GB in Fortran

[Edgar Gabriel]

well, but the f77 interface defines that to be an integer, unless you
want to change the fortran API, you will have to map it to an MPI_Fint
in my opinion.

Edgar

On 12/20/2010 1:36 PM, George Bosilca wrote:
> Nice catch. The sizes are MPI_Offset in C, and therefore we should not cast 
> them as MPI_Fint. I'll take a look, but I doubt it will be before next year. 
> Meanwhile, patches are always welcomed.
> 
>   george.
> 
> On Dec 20, 2010, at 10:59 , William George wrote:
> 
>>
>> In Fortran, calls to MPI_File_get_size return a negative value
>> when the file is larger that 2GB.
>>
>> I am using Open MPI 1.4.3 on an x86_64 system.  This happens with OpenMPI
>> compiled with Intel compilers or GCC, so I don't think it has
>> anything to do with the particular compiler in use.
>>
>> I can fix this by removing the cast to MPI_Fint in
>> the function   mpi_file_get_size_ in ompi/mpi/f77/file_get_size_f.c.
>>
>> Changing:
>>  *size = (MPI_Fint) c_size;
>>
>> To:
>>
>>  *size = c_size;
>>
>>
>> But my guess is that this is not a proper fix.
>>
>> There are a few other suspicious casts to MPI_Fint in the f77
>> directory too that probably cause similar problems:
>>
>> $  $ grep \(MPI_Fint\) *.c
>> address_f.c:*address = (MPI_Fint) addr;
>> file_get_position_f.c:*offset = (MPI_Fint) c_offset;
>> file_get_position_shared_f.c:*offset = (MPI_Fint) c_offset;
>> file_get_size_f.c:*size = (MPI_Fint) c_size;
>> file_get_view_f.c:*disp = (MPI_Fint) c_disp;
>> type_extent_f.c:*extent = (MPI_Fint)c_extent;
>>
>>
>> I can also fix this problem by compiling OpenMPI with
>> the flag -i8, but promoting all Fortran INTEGERs to 8-bytes
>> does not seem correct either.
>>
>> So - is this a configuration problem, a compile problem.
>> a source code bug, or what?  Is there an MPI_FOffsetint
>> and/or MPI_FAddressint type that should be used
>> in these casts?
>>
>>
>> Regards,
>> --
>> Bill
>>
>>  William L. George
>>  National Institute of Standards and Technology
>>  ITL - Applied and Computational Mathematics Division, Stop 8911
>>  100 Bureau Drive
>>  Gaithersburg, MD  20899-8911
>>
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
> 
> 
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

-- 
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335



signature.asc
Description: OpenPGP digital signature

Re: [OMPI devel] OMPI #2477

[Edgar Gabriel]

I can look into this later this week...
Thanks
Edgar

On 7/13/2010 10:13 AM, Jeff Squyres wrote:
> Thomas / Edgar --
> 
> Can you guys look at #2477?  Ralph is not entirely sure what's going on here, 
> and it looks like it might be more than an ORTE issue (i.e., it might be 
> inside the communicator code).
> 
> https://svn.open-mpi.org/trac/ompi/ticket/2477
> 

-- 
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335



signature.asc
Description: OpenPGP digital signature

Re: [OMPI devel] RFC: revamp topo framework

[Edgar Gabriel]

One question that I had in the back of my had a while ago was whether 
the functionality of the topo framework needs to be adapted to support 
the new MPI 2.2 graph topology functions? Maybe this can be taken into 
consideration as well...


Thanks
Edgar



Jeff Squyres wrote:

WHAT: Revamp the topo base to make it like the rest of the OMPI frameworks.

WHY: topo was written way back at the beginning of time and is showing 
its age (i.e., other frameworks have advanced while topo has not).  
Someone is interested in possibly writing a new topo component, so it 
seems an opprotune time to revamp the framework (i.e., before they start).


WHERE: Mostly in ompi/mca/topo, but some in ompi/communicator/, too

WHEN: 1.5.x sometime

TIMEOUT: Next Tuesday teleconf; Nov 3

More details


Per http://www.open-mpi.org/community/lists/devel/2009/10/7041.php, 
there are some shortcomings to the topo framework.  It pretty much 
reflects the fact that it was written way back near the beginning of the 
ompi project and has not been updated since.


I'd like to revamp it to have OBJ-based modules, per-communicator 
component/module selections, etc.  This would be similar to (but simpler 
than) the coll framework.


I've started an hg for this work:

http://bitbucket.org/jsquyres/ompi-topo-fixes/

Comments?



--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] application hangs with multiple dup

[Edgar Gabriel]

it will be available in 1.3.4...
Thanks
Edgar

Chris Samuel wrote:

Hi Edgar,

- "Edgar Gabriel" <gabr...@cs.uh.edu> wrote:


just wanted to give a heads-up that I *think* I know what the problem
is. I should have a fix (with a description) either later today or 
tomorrow morning...


I see that changeset 21970 is on trunk to fix this issue,
is that backportable to the 1.3.x branch ?

Love to see if this fixes up our users issues with Gadget!

cheers,
Chris


--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] Deadlock with comm_create since cid allocator change

[Edgar Gabriel]

what version of OpenMPI did you use? Patch #21970 should have fixed this 
issue on the trunk...


Thanks
Edgar

Sylvain Jeaugey wrote:

Hi list,

We are currently experiencing deadlocks when using communicators other 
than MPI_COMM_WORLD. So we made a very simple reproducer (Comm_create 
then MPI_Barrier on the communicator - see end of e-mail).


We can reproduce the deadlock only with openib and with at least 8 cores 
(no success with sm) and after ~20 runs average. Using larger number of 
cores greatly increases the occurence of the deadlock. When the deadlock 
occurs, every even process is stuck in MPI_Finalize and every odd 
process is in MPI_Barrier.


So we tracked the bug in the changesets and found out that this patch 
seem to have introduced the bug :


user:brbarret
date:Tue Aug 25 15:13:31 2009 +
summary: Per discussion in ticket #2009, temporarily disable the 
block CID allocation

algorithms until they properly reuse CIDs.

Reverting to the non multi-thread cid allocator makes the deadlock 
disappear.


I tried to dig further and understand why this makes a difference, with 
no luck.


If anyone can figure out what's happening, that would be great ...

Thanks,
Sylvain

#include 
#include 

int main(int argc, char **argv) {
int rank, numTasks;
int range[3];
MPI_Comm testComm, dupComm;
MPI_Group orig_group, new_group;

MPI_Init(, );
MPI_Comm_rank(MPI_COMM_WORLD, );
MPI_Comm_size(MPI_COMM_WORLD, );
MPI_Comm_group(MPI_COMM_WORLD, _group);
range[0] = 0; /* first rank */
range[1] = numTasks - 1; /* last rank */
range[2] = 1; /* stride */
MPI_Group_range_incl(orig_group, 1, , _group);
MPI_Comm_create(MPI_COMM_WORLD, new_group, );
MPI_Barrier(testComm);
MPI_Finalize();
return 0;
}

___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] application hangs with multiple dup

[Edgar Gabriel]

just wanted to give a heads-up that I *think* I know what the problem 
is. I should have a fix (with a description) either later today or 
tomorrow morning...


Thanks
Edgar

Edgar Gabriel wrote:
so I can confirm that I can reproduce the hang, and we (George, Rainer 
and me) have looked into that and are continue digging.


I hate to say that, but it looked to us as if messages were 'lost' 
(sender clearly called send and but the data is not in any of the queues 
on the receiver side), which seems to be consistent with two other bug 
reports currently being discussed on the mailing list. I could reproduce 
the hang with both sm and tcp,  so its probably not a btl issue but 
somewhere higher.


Thanks
Edgar

Thomas Ropars wrote:

Edgar Gabriel wrote:
Two short questions: do you have any open MPI mca parameters set in a 
file or at runtime?

No
And second, is there any difference if you disable the hierarch coll 
module (which does communicate additionally as well?) e.g.


mpirun --mca coll ^hierarch -np 4 ./mytest

No, there is no difference.

I don't know if it can help but : I've first had the problem when 
launching bt.A.4 and sp.A.4 of the NAS Parallel Benchmarks (3.3 version).


Thomas


Thanks
Edgar

Thomas Ropars wrote:

Ashley Pittman wrote:

On Wed, 2009-09-09 at 17:44 +0200, Thomas Ropars wrote:

Thank you.  I think you missed the top three lines of the output but
that doesn't matter.

 

main() at ?:?
  PMPI_Comm_dup() at pcomm_dup.c:62
ompi_comm_dup() at communicator/comm.c:661
  -
  [0,2] (2 processes)
  -
  ompi_comm_nextcid() at communicator/comm_cid.c:264
ompi_comm_allreduce_intra() at communicator/comm_cid.c:619
  ompi_coll_tuned_allreduce_intra_dec_fixed() at 
coll_tuned_decision_fixed.c:61
ompi_coll_tuned_allreduce_intra_recursivedoubling() at 
coll_tuned_allreduce.c:223
  ompi_request_default_wait_all() at 
request/req_wait.c:262
opal_condition_wait() at 
../opal/threads/condition.h:99

  -
  [1,3] (2 processes)
  -
  ompi_comm_nextcid() at communicator/comm_cid.c:245
ompi_comm_allreduce_intra() at communicator/comm_cid.c:619
  ompi_coll_tuned_allreduce_intra_dec_fixed() at 
coll_tuned_decision_fixed.c:61
ompi_coll_tuned_allreduce_intra_recursivedoubling() at 
coll_tuned_allreduce.c:223
  ompi_request_default_wait_all() at 
request/req_wait.c:262
opal_condition_wait() at 
../opal/threads/condition.h:99



Lines 264 and 245 of comm_cid.c are both in a for loop which calls
allreduce() twice in a loop until a certain condition is met.  As such
it's hard to tell from this trace if it is processes [0,2] are "ahead"
or [1,3] are "behind".  Either way you look at it however the
all_reduce() should not deadlock like that so it's as likely to be 
a bug

in reduce as it is in ompi_comm_nextcid() from the trace.

I assume all four processes are actually in the same call to comm_dup,
re-compiling your program with -g and re-running padb would confirm 
this

as it would show the line numbers.
  

Yes they are all in the second call to comm_dup.

Thomas

Ashley,

  


___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel




___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel




--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] application hangs with multiple dup

[Edgar Gabriel]

there is a ticket on that topic already (#2009),  and I just added some 
comments to that...


Jeff Squyres wrote:

On Sep 10, 2009, at 7:12 PM, Edgar Gabriel wrote:


so I can confirm that I can reproduce the hang, and we (George, Rainer
and me) have looked into that and are continue digging.

I hate to say that, but it looked to us as if messages were 'lost'
(sender clearly called send and but the data is not in any of the queues
on the receiver side), which seems to be consistent with two other bug
reports currently being discussed on the mailing list. I could reproduce
the hang with both sm and tcp,  so its probably not a btl issue but
somewhere higher.



Is this is, indeed, happening, someone please file a bug in trac.

Thanks.



--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] application hangs with multiple dup

[Edgar Gabriel]

so I can confirm that I can reproduce the hang, and we (George, Rainer 
and me) have looked into that and are continue digging.


I hate to say that, but it looked to us as if messages were 'lost' 
(sender clearly called send and but the data is not in any of the queues 
on the receiver side), which seems to be consistent with two other bug 
reports currently being discussed on the mailing list. I could reproduce 
the hang with both sm and tcp,  so its probably not a btl issue but 
somewhere higher.


Thanks
Edgar

Thomas Ropars wrote:

Edgar Gabriel wrote:
Two short questions: do you have any open MPI mca parameters set in a 
file or at runtime?

No
And second, is there any difference if you disable the hierarch coll 
module (which does communicate additionally as well?) e.g.


mpirun --mca coll ^hierarch -np 4 ./mytest

No, there is no difference.

I don't know if it can help but : I've first had the problem when 
launching bt.A.4 and sp.A.4 of the NAS Parallel Benchmarks (3.3 version).


Thomas


Thanks
Edgar

Thomas Ropars wrote:

Ashley Pittman wrote:

On Wed, 2009-09-09 at 17:44 +0200, Thomas Ropars wrote:

Thank you.  I think you missed the top three lines of the output but
that doesn't matter.

 

main() at ?:?
  PMPI_Comm_dup() at pcomm_dup.c:62
ompi_comm_dup() at communicator/comm.c:661
  -
  [0,2] (2 processes)
  -
  ompi_comm_nextcid() at communicator/comm_cid.c:264
ompi_comm_allreduce_intra() at communicator/comm_cid.c:619
  ompi_coll_tuned_allreduce_intra_dec_fixed() at 
coll_tuned_decision_fixed.c:61
ompi_coll_tuned_allreduce_intra_recursivedoubling() at 
coll_tuned_allreduce.c:223
  ompi_request_default_wait_all() at 
request/req_wait.c:262
opal_condition_wait() at 
../opal/threads/condition.h:99

  -
  [1,3] (2 processes)
  -
  ompi_comm_nextcid() at communicator/comm_cid.c:245
ompi_comm_allreduce_intra() at communicator/comm_cid.c:619
  ompi_coll_tuned_allreduce_intra_dec_fixed() at 
coll_tuned_decision_fixed.c:61
ompi_coll_tuned_allreduce_intra_recursivedoubling() at 
coll_tuned_allreduce.c:223
  ompi_request_default_wait_all() at 
request/req_wait.c:262
opal_condition_wait() at 
../opal/threads/condition.h:99



Lines 264 and 245 of comm_cid.c are both in a for loop which calls
allreduce() twice in a loop until a certain condition is met.  As such
it's hard to tell from this trace if it is processes [0,2] are "ahead"
or [1,3] are "behind".  Either way you look at it however the
all_reduce() should not deadlock like that so it's as likely to be a 
bug

in reduce as it is in ompi_comm_nextcid() from the trace.

I assume all four processes are actually in the same call to comm_dup,
re-compiling your program with -g and re-running padb would confirm 
this

as it would show the line numbers.
  

Yes they are all in the second call to comm_dup.

Thomas

Ashley,

  


___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel




___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] application hangs with multiple dup

[Edgar Gabriel]

Two short questions: do you have any open MPI mca parameters set in a 
file or at runtime?  And second, is there any difference if you disable 
the hierarch coll module (which does communicate additionally as well?) e.g.


mpirun --mca coll ^hierarch -np 4 ./mytest

Thanks
Edgar

Thomas Ropars wrote:

Ashley Pittman wrote:

On Wed, 2009-09-09 at 17:44 +0200, Thomas Ropars wrote:

Thank you.  I think you missed the top three lines of the output but
that doesn't matter.

 

main() at ?:?
  PMPI_Comm_dup() at pcomm_dup.c:62
ompi_comm_dup() at communicator/comm.c:661
  -
  [0,2] (2 processes)
  -
  ompi_comm_nextcid() at communicator/comm_cid.c:264
ompi_comm_allreduce_intra() at communicator/comm_cid.c:619
  ompi_coll_tuned_allreduce_intra_dec_fixed() at 
coll_tuned_decision_fixed.c:61
ompi_coll_tuned_allreduce_intra_recursivedoubling() at 
coll_tuned_allreduce.c:223

  ompi_request_default_wait_all() at request/req_wait.c:262
opal_condition_wait() at ../opal/threads/condition.h:99
  -
  [1,3] (2 processes)
  -
  ompi_comm_nextcid() at communicator/comm_cid.c:245
ompi_comm_allreduce_intra() at communicator/comm_cid.c:619
  ompi_coll_tuned_allreduce_intra_dec_fixed() at 
coll_tuned_decision_fixed.c:61
ompi_coll_tuned_allreduce_intra_recursivedoubling() at 
coll_tuned_allreduce.c:223

  ompi_request_default_wait_all() at request/req_wait.c:262
opal_condition_wait() at ../opal/threads/condition.h:99



Lines 264 and 245 of comm_cid.c are both in a for loop which calls
allreduce() twice in a loop until a certain condition is met.  As such
it's hard to tell from this trace if it is processes [0,2] are "ahead"
or [1,3] are "behind".  Either way you look at it however the
all_reduce() should not deadlock like that so it's as likely to be a bug
in reduce as it is in ompi_comm_nextcid() from the trace.

I assume all four processes are actually in the same call to comm_dup,
re-compiling your program with -g and re-running padb would confirm this
as it would show the line numbers.
  

Yes they are all in the second call to comm_dup.

Thomas

Ashley,

  


___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

[OMPI devel] compile error on trunk

[Edgar Gabriel]

I have currently a problem compiling the trunk. configure runs through 
correctly, but when starting make I get the following error:


gabriel@salmon:~/ompi/trunk> make
make: *** No rule to make target `config/ompi_check_attributes.m4', 
needed by `Makefile.in'.  Stop.



svn up claims that I am at the newest revision. What am I doing wrong?
Thanks
Edgar
--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] MPI_Group_compare is broken

[Edgar Gabriel]

you are right. At first I thought that the loop above that should catch 
that condition but it doesn't. I will apply your patch and file a CMR 
for the 1.3 branch...


Thanks for the bug report and the fix right away...
Edgar

Geoffrey Irving wrote:

Hello,

MPI_Group_compare is broken in both 1.3.2 and svn.  Here is a patch
which fixes the problem:

diff --git a/ompi/mpi/c/group_compare.c b/ompi/mpi/c/group_compare.c
index 0d199c1..89c83f9 100644
--- a/ompi/mpi/c/group_compare.c
+++ b/ompi/mpi/c/group_compare.c
@@ -106,6 +106,7 @@ int MPI_Group_compare(MPI_Group group1, MPI_Group
group2, int *result) {
 } /* end proc2 loop */
 if( match== -1 ) {
 similar=false;
+identical=false;
 break;
 }
 } /* end proc1 loop */

and a C test program which illustrates it:

#include 
#include 

int main(int argc,char* argv[])
{
MPI_Init(,);

MPI_Group group;
MPI_Comm_group(MPI_COMM_WORLD, );

int r1[2] = {0, 1};
int r2[2] = {1, 2};
MPI_Group g1, g2;
MPI_Group_incl(group, 2, r1, );
MPI_Group_incl(group, 2, r2, );

int cmp;
MPI_Group_compare(g1, g2, );
printf("compare %d, ident %d\n", cmp, MPI_IDENT);
assert(cmp != MPI_IDENT);

MPI_Finalize();
return 0;
}

A quick glance through the history (thanks git log -pM --follow) seems
to indicate that MPI_Group_compare hasn't ever worked in OpenMPI, so
apparently I'm the only user of this function. :)

Geoffrey
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] Inherent limit on #communicators?

[Edgar Gabriel]

Christian Siebert wrote:

Hi Edgar,

cid's are in fact not recycled in the block algorithm. The problem is 
that comm_free is not collective, so you can not make any assumptions 
whether other procs have also released that communicator.
well, that's not quite correct. The MPI standard says the following 
about MPI_Comm_free (MPI 2.1, p 201, l 43): "This collective operation 
marks the communication object for deallocation.". So MPI_Comm_free is 
collective which makes the prescribed problem(s) easier so solve.


Christian,

you are right, but unfortunately it doesn't help in this scenario. 
Consider the following test case, which fulfills the MPI spec:


  MPI_Comm_dup ( MPI_COMM_WORLD,  );
  if ( rank == 0 ) {
MPI_Isend ( dest=1, comm1,  );
MPI_Wait ( req, stat);
MPI_Comm_free ( comm1);
MPI_Comm_dup ( MPI_COMM_WORLD,  2);
  }
  if ( rank == 1 ) {
MPI_Irecv ( src=0, comm1,  );
MPI_Comm_free (  );
MPI_Comm_dup ( MPI_COMM_WORLD, comm1 );
MPI_Wait ( , stat);
  }

In this case, rank=0 thinks that the CID of comm1 can be reused, but 
rank=1 can not reuse that CID yet, since the pending communication 
operation still has to finish. If you want determine the communicator id 
without communication (which is the goal of the block algorithm), you 
can not do that based on your local information. Collective != 
synchronous...


Thanks
Edgar



However, I admit that there exist an advice to implementors that 
anticipates a local implementation. Personally I find this advice rather 
strange and (if nobody can give a good reason for it) would encourage 
its removal...


Best regards,
   Christian


___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] Inherent limit on #communicators?

[Edgar Gabriel]

ok, r21142 should fix the problem for the app. I did test it with a 
number of scenarios (e.g. all intra-comm cases, inter-comm cases, 
intercomm_merge etc.), but I would suggest to let at least one night of 
MTT runs go over it before we file a CMR for 1.3 ...


Thanks
Edgar



On Fri, May 1, 2009 at 6:38 AM, Ralph Castain <r...@open-mpi.org> wrote:
  I'm not entirely sure if David is going to be in today, so I
  will answer for him (and let him correct me later!).

  This code is indeed representative of what the app is doing.
  Basically, the user repeatedly splits the communicator so he
  can run mini test cases before going on to the larger
  computation. So it is always the base communicator being
  repeatedly split and freed.

  I would suspect, therefore, that the quick fix would serve us
  just fine while the worst case is later resolved.

  Thanks
  Ralph


On Fri, May 1, 2009 at 6:08 AM, Edgar Gabriel <gabr...@cs.uh.edu>
wrote:
  David,

  is this code representative for what your app is doing?
  E.g. you have a base communicator (e.g. MPI_COMM_WORLD)
  which is being 'split', freed again, split, freed again
  etc. ? i.e. the important aspect is that the same
  'base' communicator is being used for deriving new
  communicators again and again?

  The reason I ask is two-fold: one, you would in that
  case be one of the ideal beneficiaries of the block cid
  algorithm :-) (even if it fails you right now);  two, a
  fix for this scenario which basically tries to reuse
  the last block used (and which would fix your case if
  the condition is true) is roughly five lines of code.
  This would give us the possibility to have a fix
  quickly in the trunk and v1.3 (keep in mind that the
  block-cid code is in the trunk since two years and this
  is the first problem that we have) and give us more
  time to develop a profound solution for the worst case
  - a chain of communicators being created, e.g.
  communicator 1 is basis to derive a new comm 2, comm 2
  is being used to derive comm 3 etc.

  Thanks
  Edgar

  David Gunter wrote:
Here is the test code reproducer:

 program test2
 implicit none
 include 'mpif.h'
 integer ierr, myid,
numprocs,i1,i2,n,local_comm,
$ icolor,ikey,rank,root

c
c...  MPI set-up
 ierr = 0
 call MPI_INIT(IERR)
 ierr = 1
 CALL MPI_COMM_SIZE(MPI_COMM_WORLD,
numprocs, ierr)
 print *, ierr

 ierr = -1

 CALL MPI_COMM_RANK(MPI_COMM_WORLD,
myid, ierr)

 ierr = -5
 i1 = ierr
 if (myid.eq.0) i1 = 1
 call mpi_allreduce(i1, i2,
1,MPI_integer,MPI_MIN,
$ MPI_COMM_WORLD,ierr)

 ikey = myid
 if (mod(myid,2).eq.0) then
icolor = 0
 else
icolor = MPI_UNDEFINED
 end if

 root = 0
 do n = 1, 10

call MPI_COMM_SPLIT(MPI_COMM_WORLD,
icolor,
$ikey, local_comm, ierr)

if (mod(myid,2).eq.0) then
   CALL MPI_COMM_RANK(local_comm,
rank, ierr)
   i2 = i1
   call mpi_reduce(i1, i2,
1,MPI_integer,MPI_MIN,
$   root, local_comm,ierr)

   if
(myid.eq.0.and.mod(n,10).eq.0)
$   print *, n, i1,
i2,icolor,ikey

   call mpi_comm_free(local_comm,
ierr)
end if

 end do
c  if (icolor.eq.0) call
mpi_comm_free(local_comm, ierr)



 call MPI_barrier(MPi_COMM_WORLD,ierr)

 call MPI_FINALIZE(IERR)

 print *, myid, ierr

 end



-david
--
David Gunter
HPC-3: Parallel Tools Team
Los Alamos National Laboratory



On Apr 30, 2009, at 12:43 PM, David Gunter
wrote:

  Just to throw out more info on
  this, the test code runs fine
  on previous versions of OMPI.
   It only hangs on the 1.3 line
  when the cid reaches 65536.

  -david
  --
  David Gunter
  HPC-3: Parallel Tools Team
  Los Alamos National Laboratory



  On Apr 30, 2009, at 12:28 PM,
      Edgar Gabriel wrote:

cid's

Re: [OMPI devel] Inherent limit on #communicators?

[Edgar Gabriel]

David,

is this code representative for what your app is doing? E.g. you have a 
base communicator (e.g. MPI_COMM_WORLD) which is being 'split', freed 
again, split, freed again etc. ? i.e. the important aspect is that the 
same 'base' communicator is being used for deriving new communicators 
again and again?


The reason I ask is two-fold: one, you would in that case be one of the 
ideal beneficiaries of the block cid algorithm :-) (even if it fails you 
right now);  two, a fix for this scenario which basically tries to reuse 
the last block used (and which would fix your case if the condition is 
true) is roughly five lines of code. This would give us the possibility 
to have a fix quickly in the trunk and v1.3 (keep in mind that the 
block-cid code is in the trunk since two years and this is the first 
problem that we have) and give us more time to develop a profound 
solution for the worst case - a chain of communicators being created, 
e.g. communicator 1 is basis to derive a new comm 2, comm 2 is being 
used to derive comm 3 etc.


Thanks
Edgar

David Gunter wrote:

Here is the test code reproducer:

  program test2
  implicit none
  include 'mpif.h'
  integer ierr, myid, numprocs,i1,i2,n,local_comm,
 $ icolor,ikey,rank,root

c
c...  MPI set-up
  ierr = 0
  call MPI_INIT(IERR)
  ierr = 1
  CALL MPI_COMM_SIZE(MPI_COMM_WORLD, numprocs, ierr)
  print *, ierr

  ierr = -1

  CALL MPI_COMM_RANK(MPI_COMM_WORLD, myid, ierr)

  ierr = -5
  i1 = ierr
  if (myid.eq.0) i1 = 1
  call mpi_allreduce(i1, i2, 1,MPI_integer,MPI_MIN,
 $ MPI_COMM_WORLD,ierr)

  ikey = myid
  if (mod(myid,2).eq.0) then
 icolor = 0
  else
 icolor = MPI_UNDEFINED
  end if

  root = 0
  do n = 1, 10

 call MPI_COMM_SPLIT(MPI_COMM_WORLD, icolor,
 $ikey, local_comm, ierr)

 if (mod(myid,2).eq.0) then
CALL MPI_COMM_RANK(local_comm, rank, ierr)
i2 = i1
call mpi_reduce(i1, i2, 1,MPI_integer,MPI_MIN,
 $   root, local_comm,ierr)

if (myid.eq.0.and.mod(n,10).eq.0)
 $   print *, n, i1, i2,icolor,ikey

call mpi_comm_free(local_comm, ierr)
 end if

  end do
c  if (icolor.eq.0) call mpi_comm_free(local_comm, ierr)



  call MPI_barrier(MPi_COMM_WORLD,ierr)

  call MPI_FINALIZE(IERR)

  print *, myid, ierr

  end



-david
--
David Gunter
HPC-3: Parallel Tools Team
Los Alamos National Laboratory



On Apr 30, 2009, at 12:43 PM, David Gunter wrote:

Just to throw out more info on this, the test code runs fine on 
previous versions of OMPI.  It only hangs on the 1.3 line when the cid 
reaches 65536.


-david
--
David Gunter
HPC-3: Parallel Tools Team
Los Alamos National Laboratory



On Apr 30, 2009, at 12:28 PM, Edgar Gabriel wrote:

cid's are in fact not recycled in the block algorithm. The problem is 
that comm_free is not collective, so you can not make any assumptions 
whether other procs have also released that communicator.



But nevertheless, a cid in the communicator structure is a uint32_t, 
so it should not hit the 16k limit there yet. this is not new, so if 
there is a discrepancy between what the comm structure assumes that a 
cid is and what the pml assumes, than this was in the code since the 
very first days of Open MPI...


Thanks
Edgar

Brian W. Barrett wrote:

On Thu, 30 Apr 2009, Ralph Castain wrote:

We seem to have hit a problem here - it looks like we are seeing a
built-in limit on the number of communicators one can create in a
program. The program basically does a loop, calling MPI_Comm_split 
each

time through the loop to create a sub-communicator, does a reduce
operation on the members of the sub-communicator, and then calls
MPI_Comm_free to release it (this is a minimized reproducer for the 
real

code). After 64k times through the loop, the program fails.

This looks remarkably like a 16-bit index that hits a max value and 
then

blocks.

I have looked at the communicator code, but I don't immediately see 
such
a field. Is anyone aware of some other place where we would have a 
limit

that would cause this problem?
There's a maximum of 32768 communicator ids when using OB1 (each PML 
can set the max contextid, although the communicator code is the 
part that actually assigns a cid).  Assuming that comm_free is 
actually properly called, there should be plenty of cids available 
for that pattern. However, I'm not sure I understand the block 
algorithm someone added to cid allocation - I'd have to guess that 
there's something funny with that routine and cids aren't being 
recycled properly.

Brian
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science

Re: [OMPI devel] Inherent limit on #communicators?

[Edgar Gabriel]

so I agree that we need to fix that, and we'll get a fix for that as 
soon as possible. It still strikes me as wrong however to we have 
fundamentally different types on two layers for the same 'item'.


I still think that going back to the original algorithm would be bad - 
especially for an application that creates such a large number of 
communicators potentially executed on a large number ( 1000s) of 
processors. I'll look into how to reuse an entire block of communicator 
cid respectively how to take the max_contextid into account.


Edgar

Brian W. Barrett wrote:

On Thu, 30 Apr 2009, Edgar Gabriel wrote:


Brian W. Barrett wrote:
When we added the CM PML, we added a pml_max_contextid field to the 
PML structure, which is the max size cid the PML can handle (because 
the matching interfaces don't allow 32 bits to be used for the cid.  
At the same time, the max cid for OB1 was shrunk significantly, so 
that the header on a short message would be packed tightly with no 
alignment padding.


At the time, we believed 32k simultaneous communicators was plenty, 
and that CIDs were reused (we checked, I'm pretty sure).  It sounds 
like someone removed the CID reuse code, which seems rather bad to me. 


yes, we added the block algorithm. Not reusing a CID actually doesn't 
bite me as that dramatic, and I am still not sure and convinced about 
that:-) We do not have an empty array or something like that, its just 
a number.


The reason for the block algorithm was that the performance of our 
communicator creation code sucked, and the cid allocation was one 
portion of that. We used to require *at least* 4 collective operations 
per communicator creation at that time. We are now potentially down to 
0, among others thanks to the block algorithm.


However, let me think about reusing entire blocks, its probably doable 
just requires a little more bookkeeping...


There have to be unused CIDs in Ralph's example - is there a way to 
fallback out of the block algorithm when it can't find a new CID and 
find one it can reuse?  Other than setting the multi-threaded case 
back on, that is?


remember that its not the communicator id allocation that is failing 
at this point, so the question is do we have to 'validate' a cid with 
the pml before we declare it to be ok?


well, that's only because the code's doing something it shouldn't.  Have 
a look at comm_cid.c:185 - there's the check we added to the 
multi-threaded case (which was the only case when we added it).  The cid 
generation should never generate a number larger than 
mca_pml.pml_max_contextid. I'm actually somewhat amazed this fails 
gracefully, as OB1 doesn't appear to check it got a valid cid in 
add_comm, which it should probably do.


Looking at the differences between v1.2 and v1.3, the max_contextid code 
was already in v1.2 and OB1 was setting it to 32k.  So the cid blocking 
code removed a rather critical feature and probably should be fixed or 
removed for v1.3.  On Portals, I only get 8k cids, so not having reuse 
is a rather large problem.


Brian
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] Inherent limit on #communicators?

[Edgar Gabriel]

Brian W. Barrett wrote:
When we added the CM PML, we added a pml_max_contextid field to the PML 
structure, which is the max size cid the PML can handle (because the 
matching interfaces don't allow 32 bits to be used for the cid.  At the 
same time, the max cid for OB1 was shrunk significantly, so that the 
header on a short message would be packed tightly with no alignment 
padding.


At the time, we believed 32k simultaneous communicators was plenty, and 
that CIDs were reused (we checked, I'm pretty sure).  It sounds like 
someone removed the CID reuse code, which seems rather bad to me.  


yes, we added the block algorithm. Not reusing a CID actually doesn't 
bite me as that dramatic, and I am still not sure and convinced about 
that:-) We do not have an empty array or something like that, its just a 
number.


The reason for the block algorithm was that the performance of our 
communicator creation code sucked, and the cid allocation was one 
portion of that. We used to require *at least* 4 collective operations 
per communicator creation at that time. We are now potentially down to 
0, among others thanks to the block algorithm.


However, let me think about reusing entire blocks, its probably doable 
just requires a little more bookkeeping...


There 
have to be unused CIDs in Ralph's example - is there a way to fallback 
out of the block algorithm when it can't find a new CID and find one it 
can reuse?  Other than setting the multi-threaded case back on, that is?


remember that its not the communicator id allocation that is failing at 
this point, so the question is do we have to 'validate' a cid with the 
pml before we declare it to be ok?


Thanks
Edgar





Brian

On Thu, 30 Apr 2009, Edgar Gabriel wrote:

cid's are in fact not recycled in the block algorithm. The problem is 
that comm_free is not collective, so you can not make any assumptions 
whether other procs have also released that communicator.



But nevertheless, a cid in the communicator structure is a uint32_t, 
so it should not hit the 16k limit there yet. this is not new, so if 
there is a discrepancy between what the comm structure assumes that a 
cid is and what the pml assumes, than this was in the code since the 
very first days of Open MPI...


Thanks
Edgar

Brian W. Barrett wrote:

On Thu, 30 Apr 2009, Ralph Castain wrote:


We seem to have hit a problem here - it looks like we are seeing a
built-in limit on the number of communicators one can create in a
program. The program basically does a loop, calling MPI_Comm_split each
time through the loop to create a sub-communicator, does a reduce
operation on the members of the sub-communicator, and then calls
MPI_Comm_free to release it (this is a minimized reproducer for the 
real

code). After 64k times through the loop, the program fails.

This looks remarkably like a 16-bit index that hits a max value and 
then

blocks.

I have looked at the communicator code, but I don't immediately see 
such
a field. Is anyone aware of some other place where we would have a 
limit

that would cause this problem?


There's a maximum of 32768 communicator ids when using OB1 (each PML 
can set the max contextid, although the communicator code is the part 
that actually assigns a cid).  Assuming that comm_free is actually 
properly called, there should be plenty of cids available for that 
pattern. However, I'm not sure I understand the block algorithm 
someone added to cid allocation - I'd have to guess that there's 
something funny with that routine and cids aren't being recycled 
properly.


Brian
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel




___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] Inherent limit on #communicators?

[Edgar Gabriel]

cid's are in fact not recycled in the block algorithm. The problem is 
that comm_free is not collective, so you can not make any assumptions 
whether other procs have also released that communicator.



But nevertheless, a cid in the communicator structure is a uint32_t, so 
it should not hit the 16k limit there yet. this is not new, so if there 
is a discrepancy between what the comm structure assumes that a cid is 
and what the pml assumes, than this was in the code since the very first 
days of Open MPI...


Thanks
Edgar

Brian W. Barrett wrote:

On Thu, 30 Apr 2009, Ralph Castain wrote:


We seem to have hit a problem here - it looks like we are seeing a
built-in limit on the number of communicators one can create in a
program. The program basically does a loop, calling MPI_Comm_split each
time through the loop to create a sub-communicator, does a reduce
operation on the members of the sub-communicator, and then calls
MPI_Comm_free to release it (this is a minimized reproducer for the real
code). After 64k times through the loop, the program fails.

This looks remarkably like a 16-bit index that hits a max value and then
blocks.

I have looked at the communicator code, but I don't immediately see such
a field. Is anyone aware of some other place where we would have a limit
that would cause this problem?


There's a maximum of 32768 communicator ids when using OB1 (each PML can 
set the max contextid, although the communicator code is the part that 
actually assigns a cid).  Assuming that comm_free is actually properly 
called, there should be plenty of cids available for that pattern. 
However, I'm not sure I understand the block algorithm someone added to 
cid allocation - I'd have to guess that there's something funny with 
that routine and cids aren't being recycled properly.


Brian
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] reduce_scatter bug with hierarch

[Edgar Gabriel]

so I am not entirely sure why the bug only happened on trunk, it could 
in theory also appear on v1.3 (is there a difference on how 
pointer_arrays are handled between the two versions?)


Anyway, it passes now on both with changeset 20275. We should probably 
move that over to 1.3 as well, whether for 1.3.0 or 1.3.1 I leave that 
up to others to decide...


Thanks
Edgar

Edgar Gabriel wrote:
I'm already debugging it. the good news is that it only seems to appear 
with trunk, with 1.3 (after copying the new tuned module over), all the 
tests pass.


Now if somebody can tell me a trick on how to tell mpirun not kill the 
debugger under my feet, then I could even see where the problem occurs:-)


Thanks
Edga

George Bosilca wrote:
All these errors are in the MPI_Finalize, it should not be that hard 
to find. I'll take a look later this afternoon.


  george.

On Jan 14, 2009, at 06:41 , Tim Mattox wrote:

Unfortunately, although this fixed some problems when enabling 
hierarch coll,
there is still a segfault in two of IU's tests that only shows up 
when we set

-mca coll_hierarch_priority 100

See this MTT summary to see how the failures improved on the trunk,
but that there are still two that segfault even at 1.4a1r20267:
http://www.open-mpi.org/mtt/index.php?do_redir=923

This link just has the remaining failures:
http://www.open-mpi.org/mtt/index.php?do_redir=922

So, I'll vote for applying the CMR for 1.3 since it clearly improved 
things,
but there is still more to be done to get coll_hierarch ready for 
regular

use.

On Wed, Jan 14, 2009 at 12:15 AM, George Bosilca 
<bosi...@eecs.utk.edu> wrote:

Here we go by the book :)

https://svn.open-mpi.org/trac/ompi/ticket/1749

george.

On Jan 13, 2009, at 23:40 , Jeff Squyres wrote:

Let's debate tomorrow when people are around, but first you have to 
file a

CMR... :-)

On Jan 13, 2009, at 10:28 PM, George Bosilca wrote:


Unfortunately, this pinpoint the fact that we didn't test enough the
collective module mixing thing. I went over the tuned collective 
functions
and changed all instances to use the correct module information. 
It is now

on the trunk, revision 20267. Simultaneously,I checked that all other
collective components do the right thing ... and I have to admit 
tuned was

the only faulty one.

This is clearly a bug in the tuned, and correcting it will allow 
people
to use the hierarch. In the current incarnation 1.3 will 
mostly/always
segfault when hierarch is active. I would prefer not to give a 
broken toy

out there. How about pushing r20267 in the 1.3?

george.


On Jan 13, 2009, at 20:13 , Jeff Squyres wrote:

Thanks for digging into this.  Can you file a bug?  Let's mark it 
for

v1.3.1.

I say 1.3.1 instead of 1.3.0 because this *only* affects 
hierarch, and
since hierarch isn't currently selected by default (you must 
specifically
elevate hierarch's priority to get it to run), there's no danger 
that users

will run into this problem in default runs.

But clearly the problem needs to be fixed, and therefore we need 
a bug

to track it.



On Jan 13, 2009, at 2:09 PM, Edgar Gabriel wrote:

I just debugged the Reduce_scatter bug mentioned previously. The 
bug is

unfortunately not in hierarch, but in tuned.

Here is the code snipplet causing the problems:

int reduce_scatter (, mca_coll_base_module_t *module)
{
...
err = comm->c_coll.coll_reduce (, module)
...
}


but should be
{
...
err = comm->c_coll.coll_reduce (..., 
comm->c_coll.coll_reduce_module);

...
}

The problem as it is right now is, that when using hierarch, only a
subset of the function are set, e.g. reduce,allreduce, bcast and 
barrier.

Thus, reduce_scatter is from tuned in most scenarios, and calls the
subsequent functions with the wrong module. Hierarch of course 
does not like

that :-)

Anyway, a quick glance through the tuned code reveals a significant
number of instances where this appears(reduce_scatter, 
allreduce, allgather,
allgatherv). Basic, hierarch and inter seem to do that mostly 
correctly.


Thanks
Edgar
--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel



--
Jeff Squyres
Cisco Systems

___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel



--
Jeff Squyres
Cisco Systems

___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/lis

Re: [OMPI devel] reduce_scatter bug with hierarch

[Edgar Gabriel]

I'm already debugging it. the good news is that it only seems to appear 
with trunk, with 1.3 (after copying the new tuned module over), all the 
tests pass.


Now if somebody can tell me a trick on how to tell mpirun not kill the 
debugger under my feet, then I could even see where the problem occurs:-)


Thanks
Edga

George Bosilca wrote:
All these errors are in the MPI_Finalize, it should not be that hard to 
find. I'll take a look later this afternoon.


  george.

On Jan 14, 2009, at 06:41 , Tim Mattox wrote:

Unfortunately, although this fixed some problems when enabling 
hierarch coll,
there is still a segfault in two of IU's tests that only shows up when 
we set

-mca coll_hierarch_priority 100

See this MTT summary to see how the failures improved on the trunk,
but that there are still two that segfault even at 1.4a1r20267:
http://www.open-mpi.org/mtt/index.php?do_redir=923

This link just has the remaining failures:
http://www.open-mpi.org/mtt/index.php?do_redir=922

So, I'll vote for applying the CMR for 1.3 since it clearly improved 
things,

but there is still more to be done to get coll_hierarch ready for regular
use.

On Wed, Jan 14, 2009 at 12:15 AM, George Bosilca 
<bosi...@eecs.utk.edu> wrote:

Here we go by the book :)

https://svn.open-mpi.org/trac/ompi/ticket/1749

george.

On Jan 13, 2009, at 23:40 , Jeff Squyres wrote:

Let's debate tomorrow when people are around, but first you have to 
file a

CMR... :-)

On Jan 13, 2009, at 10:28 PM, George Bosilca wrote:


Unfortunately, this pinpoint the fact that we didn't test enough the
collective module mixing thing. I went over the tuned collective 
functions
and changed all instances to use the correct module information. It 
is now

on the trunk, revision 20267. Simultaneously,I checked that all other
collective components do the right thing ... and I have to admit 
tuned was

the only faulty one.

This is clearly a bug in the tuned, and correcting it will allow 
people

to use the hierarch. In the current incarnation 1.3 will mostly/always
segfault when hierarch is active. I would prefer not to give a 
broken toy

out there. How about pushing r20267 in the 1.3?

george.


On Jan 13, 2009, at 20:13 , Jeff Squyres wrote:


Thanks for digging into this.  Can you file a bug?  Let's mark it for
v1.3.1.

I say 1.3.1 instead of 1.3.0 because this *only* affects hierarch, 
and
since hierarch isn't currently selected by default (you must 
specifically
elevate hierarch's priority to get it to run), there's no danger 
that users

will run into this problem in default runs.

But clearly the problem needs to be fixed, and therefore we need a 
bug

to track it.



On Jan 13, 2009, at 2:09 PM, Edgar Gabriel wrote:

I just debugged the Reduce_scatter bug mentioned previously. The 
bug is

unfortunately not in hierarch, but in tuned.

Here is the code snipplet causing the problems:

int reduce_scatter (, mca_coll_base_module_t *module)
{
...
err = comm->c_coll.coll_reduce (, module)
...
}


but should be
{
...
err = comm->c_coll.coll_reduce (..., 
comm->c_coll.coll_reduce_module);

...
}

The problem as it is right now is, that when using hierarch, only a
subset of the function are set, e.g. reduce,allreduce, bcast and 
barrier.

Thus, reduce_scatter is from tuned in most scenarios, and calls the
subsequent functions with the wrong module. Hierarch of course 
does not like

that :-)

Anyway, a quick glance through the tuned code reveals a significant
number of instances where this appears(reduce_scatter, allreduce, 
allgather,
allgatherv). Basic, hierarch and inter seem to do that mostly 
correctly.


Thanks
Edgar
--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel



--
Jeff Squyres
Cisco Systems

___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel



--
Jeff Squyres
Cisco Systems

___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel





--
Tim Mattox, Ph.D. - http://homepage.mac.com/tmattox/
tmat...@gmail.com || timat...@open-mpi.org
   I'm a bright... http://www.the-brights.net/
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/lis

[OMPI devel] reduce_scatter bug with hierarch

[Edgar Gabriel]

I just debugged the Reduce_scatter bug mentioned previously. The bug is 
unfortunately not in hierarch, but in tuned.


Here is the code snipplet causing the problems:

int reduce_scatter (, mca_coll_base_module_t *module)
{
...
   err = comm->c_coll.coll_reduce (, module)
...
}


but should be
{
...
  err = comm->c_coll.coll_reduce (..., comm->c_coll.coll_reduce_module);
...
}

The problem as it is right now is, that when using hierarch, only a 
subset of the function are set, e.g. reduce,allreduce, bcast and 
barrier. Thus, reduce_scatter is from tuned in most scenarios, and calls 
the subsequent functions with the wrong module. Hierarch of course does 
not like that :-)


Anyway, a quick glance through the tuned code reveals a significant 
number of instances where this appears(reduce_scatter, allreduce, 
allgather, allgatherv). Basic, hierarch and inter seem to do that mostly 
correctly.


Thanks
Edgar
--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] Dropped message for the non-existing communicator

[Edgar Gabriel]

Terry,

was this with the trunk or v1.3? If it was the trunk, was it before 
r19929 was applied? The reason I ask is because r19929 should remove all 
error messages related to 'non-existing communictors'. Hierarch btw. is 
not the cause for the error messages even before that, it just exposes 
it more frequently...


Thanks
Edgar

Terry Dontje wrote:
I am seeing the message "Dropped message for the non-existing 
communicator" when running hpcc with np=124 against r19845.  This seems 
to be pretty reproducible at np=124.  When the job prints out the 
message above some set of processes are in an MPI_Bcast and the 15 
processes reporting the message are stuck in MPI_Barrier.
I am not sure how related this is to #1408 since I am not invoking the 
hierarchical collectives.  I just wanted to see if anyone else has tried 
to run hpcc at such an np size with any success.


My next steps are to try to run this with the latest trunk and to narrow 
down the failing case.


--td
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] Affect of compression on modex and launch messages

[Edgar Gabriel]

actually, we used lzo a looong time ago with PACX-MPI, it was indeed 
faster then zlib. Our findings at that time were however similar to what 
George mentioned, namely a benefit from compression was only visible if 
the network latency was really high (e.g. multiple ms)...


Thanks
Edgar

Roland Dreier wrote:

 > Based on some discussion on this list, I integrated a zlib-based compression
 > ability into ORTE. Since the launch message sent to the orteds and the modex
 > between the application procs are the only places where messages of any size
 > are sent, I only implemented compression for those two exchanges.
 > 
 > I have found virtually no benefit to the compression. Essentially, the

 > overhead consumed in compression/decompressing the messages pretty much
 > balances out any transmission time differences. However, I could only test
 > this for 64 nodes, 8ppn, so perhaps there is some benefit at larger sizes.

A faster compression library might change the balance... eg LZO
(http://www.oberhumer.com/opensource/lzo) might be worth a look although
I'm not an expert on all the compression libraries that are out there.

 - R.
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] [RFC] Default hostfile MCA param

[Edgar Gabriel]

Tim Prins wrote:
We have used '^' elsewhere to indicate not, so maybe just have the 
syntax be if you put '^' at the beginning of a line, that node is not used.


So we could have:
n0
n1
^headnode
n3


this would sound fine for me.



I understand the idea of having a flag to indicate that all nodes below 
a certain point should be ignored, but I think this might get confusing, 
and I'm unsure how useful it would be. I just see the usefulness of this 
to block out a couple of nodes by default. Besides, if you do want to 
block out many nodes, any reasonable text editor allows you to insert 
'^' in front of any number of lines easily.


Alternatively, for the particular situation that Edgar mentions, it may 
be good enough just to set rmaps_base_no_schedule_local in the mca 
params default file.


hm, ok, here is another flag which I was not aware of. Anyway, I can 
think of other scenarios where this feature could be useful, e.g. when 
hunting down performance problems on a cluster and you would like to 
avoid to have to get a new allocation or do a major rewrite of the 
hostfile every time. Or including an I/O node into an allocation (in 
order to have it exclusively), but make sure that no MPI process gets 
scheduled onto the node.


Thanks
Edgar



One question though: If I am in a slurm allocation which contains n1, 
and there is a default hostfile that contains "^n1", will I run on 'n1'?


I'm not sure what the answer is, I know we talked about the precedence 
earlier...


Tim

Ralph H Castain wrote:

I personally have no objection, but I would ask then that the wiki be
modified to cover this case. All I require is that someone define the syntax
to be used to indicate "this is a node I do -not- want used", or
alternatively a flag that indicates "all nodes below are -not- to be used".

Implementation isn't too hard once I have that...


On 3/3/08 9:44 AM, "Edgar Gabriel" <gabr...@cs.uh.edu> wrote:


Ralph,

could this mechanism be used also to exclude a node, indicating to never
run a job there? Here is the problem that I face quite often: students
working on the homework forget to allocate a partition  on the cluster,
and just type mpirun. Because of that, all jobs end up running on the
front-end node.

If we would have now the ability to specify in a default hostfile, to
never run a job on a specified node (e.g. the front end node), users
would get an error message when trying to do that. I am aware that
that's a little ugly...

THanks
edgar

Ralph Castain wrote:

I forget all the formatting we are supposed to use, so I hope you'll all
just bear with me.

George brought up the fact that we used to have an MCA param to specify a
hostfile to use for a job. The hostfile behavior described on the wiki,
however, doesn't provide for that option. It associates a hostfile with a
specific app_context, and provides a detailed hierarchical layout of how
mpirun is to interpret that information.

What I propose to do is add an MCA param called "OMPI_MCA_default_hostfile"
to replace the deprecated capability. If found, the system's behavior will
be:

1. in a managed environment, the default hostfile will be used to filter the
discovered nodes to define the available node pool. Any hostfile and/or dash
host options provided to an app_context will be used to further filter the
node pool to define the specific nodes for use by that app_context. Thus,
nodes in the hostfile and dash host options given to an app_context -must-
also be in the default hostfile in order to be available for use by that
app_context - any nodes in the app_context options that are not in the
default hostfile will be ignored.

2. in an unmanaged environment, the default hostfile will be used to define
the available node pool. Any hostfile and/or dash host options provided to
an app_context will be used to filter the node pool to define the specific
nodes for use by that app_context, subject to the previous caveat. However,
add-hostfile and add-host options will add nodes to the node pool for use
-only- by the associated app_context.


I believe this proposed behavior is consistent with that described on the
wiki, and would be relatively easy to implement. If nobody objects, I will
do so by end-of-day 3/6.

Comments, suggestions, objections - all are welcome!
Ralph


___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77

Re: [OMPI devel] [OMPI svn] svn:open-mpi r17366

[Edgar Gabriel]

George,

I hate to say this, but I don't think that this is correct. Page 114 in 
MPI 1 says (in the section about MPI_Reduce)


"Thus, all processes provide input buffers and output buffers of the 
same length, with elements of the same time".


The content is only significant at the root, but all processes have to 
provide the buffer according to my understanding...


Thanks
Edgar

bosi...@osl.iu.edu wrote:

Author: bosilca
Date: 2008-02-03 20:44:41 EST (Sun, 03 Feb 2008)
New Revision: 17366
URL: https://svn.open-mpi.org/trac/ompi/changeset/17366

Log:
As the receive buffer is only significant at root, limit the
check only where it makes sense.

Text files modified: 
   trunk/ompi/mpi/c/reduce.c | 4 +---
   1 files changed, 1 insertions(+), 3 deletions(-)


Modified: trunk/ompi/mpi/c/reduce.c
==
--- trunk/ompi/mpi/c/reduce.c   (original)
+++ trunk/ompi/mpi/c/reduce.c   2008-02-03 20:44:41 EST (Sun, 03 Feb 2008)
@@ -59,9 +59,7 @@
 free(msg);
 return ret;
 } else if ((ompi_comm_rank(comm) != root && MPI_IN_PLACE == sendbuf) ||
-   (ompi_comm_rank(comm) == root && MPI_IN_PLACE == recvbuf)) {
-err = MPI_ERR_ARG;
-} else if( sendbuf == recvbuf ) {
+   (ompi_comm_rank(comm) == root && ((MPI_IN_PLACE == recvbuf) 
|| (sendbuf == recvbuf {
 err = MPI_ERR_ARG;
 } else {
 OMPI_CHECK_DATATYPE_FOR_SEND(err, datatype, count);
___
svn mailing list
s...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/svn


--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] Hierarchical Collectives Query

[Edgar Gabriel]

Rolf,

Whoowh! That's actually good news, since in our own tests hierarch is 
always slower. But this might be due to various reasons, including the 
fact, that we only have two cores per node. BTW: I actually would expect 
IMB test to have worse performance for hierarch compared to many other 
benchmarks, since the rotating root causes some additional work/overhead.


Rolf Vandevaart wrote:
I am curious if anyone is doing any work currently on the hierarchical 
collectives.  I ask this because I just did some runs on a cluster made 
up of 4 servers with 4 processors per server.  I used TCP over IB.  I 
was running with np=16 and using the IMB benchmark to test MPI_Bcast. 
What I am seeing is that the hierarchical collectives appear to boost 
performance.  The IMB test rotates the root so one could imagine that 
since the hierarchical minimizes internode communication, performance 
increases.  See the table at the end of this post with the comparison 
for MPI_Bcast between tuned and hierarchical.  This leads me to a few 
other questions.


1. From what I can tell from the debug messages, we still cannot stack 
the hierarchical on top of the tuned.  I know that Brian Barrett did 
some work after the collectives meeting to allow for this, but I could 
not figure out how to get it to work.


actually, this should be possible. We do however experience with the 
current trunk some problems, so I can not verify right now. So, just for 
the sake of clarity, did you run hierarch on top of tuned or on top of 
basic and/or sm?




2. Enabling the hierarchical collectives causes a massive slowdown 
during MPI_Init.  I know it was discussed a little at the collectives 
meeting and it appears that this is still something we need to solve. 
For a simple hello_world, np=4, 2 node cluster, I see around 5 seconds 
to run for tuned collectives, but I see around 18 seconds for

hierarchical.


yes. A faster, however simpler hierarchy detection is implemented, but 
not yet committed.




3. Apart from the MPI_Init issue, is hierarchical ready to go?


Clearly, the algorithms are very simple in hierarch, but they are still 
lacking large-scale testing, so this is something which would have to be 
incorporated.


We have also experimented with various other hierarchical algorithms for 
bcast, over the last few months. Our overall progress has however been 
significantly slower than I hoped. I know however, that various other 
groups also have interest in the hierarch component, and might also be 
ready to invest some time to bring it up to speed.




Thanks
Edgar



4. As the nodes get fatter, I assume the need for hierarchical
will increase, so this may become a larger issue for all of us?

RESULTS FROM TWO RUNS OF IMB-MPI1

#
# Benchmarking Bcast
# #processes = 16 TUNED HIERARCH
#
#bytes #repetitions  t_avg[usec]  t_avg[usec]
 0 1000 0.11 0.22
 1 1000   205.97   319.86
 2 1000   159.23   180.80
 4 1000   175.32   189.16
 8 1000   153.10   184.26
16 1000   170.98   192.33
32 1000   160.69   187.17
64 1000   159.75   182.62
   128 1000   175.47   185.19
   256 1000   160.77   194.68
   512 1000   265.45   313.89
  1024 1000   185.66   215.43
  2048 1000   815.97   257.37
  4096 1000  1208.48   442.93
  8192 1000  1521.23   530.54
 16384 1000  2357.45   813.44
 32768 1000  3341.29  1455.78
 65536  640  6485.70  3387.02
131072  320 13488.35  5261.65
262144  160 24783.09 10747.28
524288   80 50906.06 21817.64
   1048576   40 95466.82 41397.49
   2097152   20180759.72 81319.54
   4194304   10322327.71163274.55


=
rolf.vandeva...@sun.com
781-442-3043
=
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] GROUP_EMPTY fixes break intel tests :-(

[Edgar Gabriel]

well, the best I could find is the following in section 5.2.1

"MPI_GROUP_EMPTY, which is a valid handle to an empty group, should not 
be confused with MPI_GROUP_NULL, which in turn is an invalid handle. The 
former may be used as an argument to group operations; the latter, which 
is returned when a group is freed, in not a valid argument. ( End of 
advice to users.) "



Jeff Squyres wrote:
So the changes that we debated and had Edgar put in *do* break some  
intel tests.  Doh!  :-(


MPI_Group_compare_f
MPI_Group_intersection2_c
MPI_Group_intersection2_f

It looks like these tests are specifically calling MPI_GROUP_FREE on  
MPI_GROUP_EMPTY.


I note that there is code in the ompi/group/group_*.c code that  
specifically calls OBJ_RETAIN on ompi_group_empty when we return  
MPI_GROUP_EMPTY.  I wonder if this RETAIN was added (and the MPI param  
check removed) in reaction to the intel tests...?


Can someone cite again where we thought the spec said that we should  
not free GROUP_EMPTY?  Was is just on the argument that it's a  
predefined handle and therefore should never be freed?


I cannot find any specific text in 1.2 or the errata stating that it's  
bad to free GROUP_EMPTY.  I agree that this is somewhat counter to the  
rest of the MPI philosophy of not freeing predefined handles, though.




--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] Hybrid examples

[Edgar Gabriel]

Just to clarify, we had some conversation off-line with Jeff about that. 
You are eligible to access the ompi-tests directory, since you are a 
member of the group at UofDresden which signed the third party 
contribution, and thus is a member of Open MPI. For the sake of 
simplicity the svn accounts do not have automatically access to the 
svn-tests and svn-docs repository. For an existing svn account, it is as 
simple as sending an email to the svn repository maintainer ( Tim 
Mattox?) asking for that.


In order to issue a *new* svn account, a member has to submit a new 
Appendix A of the third party contribution -- which is a simple document 
listing all persons working on ompi and thus requiring an svn account.


Thanks
Edgar

Jeff Squyres wrote:
We've never made our ompi-tests SVN repository public mainly because  
it's mainly a collection of MPI benchmarks and tests that we've  
collected from around the net, but we've never looked into  
redistribution rights.  Hence, our SVN is not publicly readable.


As the SVN URL implies, the thread tests came from Sun, so I think  
it's up to them as to whether they want them to be public or not.




On Oct 4, 2007, at 7:22 AM, Tobias Hilbrich wrote:


Hello,

I am a developer of MARMOT (MPI Checker) and currently working on  
support for
MPI_THREAD_MULTIPLE programs. Therefore I am looking for test  
examples which
are realy rare. I heard from Matthias Müller that you have a suit  
of test

programs that use MPI_THREAD_MULTIPLE. They should be located at:
https://svn.open-mpi.org/svn/ompi-tests/trunk/sun/threads

Unfortunatly I and also the other people here in Dresden have no  
access to
these examples. So it would be nice to get a login or to get these  
examples

in some other way.

mfg Tobias Hilbrich

___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel





--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] VampirTrace integration / bundling 3rd party software in OMPI

[Edgar Gabriel]

Jeff Squyres wrote:


Is this in the production VT, or is this OMPI-specific functionality?

If it's OMPI-specific functionality, I would vote to not have it.

One of my big problems with this idea is that we lose the concept of  
shipping a single unit of Open MPI.  If someone sends us a bug report  
concerning VT, we no longer have a solid idea of what version they  
are running because they may have replaced the one inside their Open  
MPI software.


well, this issue could be however resolved, if ompi_info and friends 
would have a way to report the precise version number for VT, isn't it?


Without having any strong feelings one way or the other, I think that 
the functionality is great from the end-users perspective. Just my 0.02$...


Thanks
Edgar




Running an external VT install OMPI is a different thing; that's easy  
enough to tell that someone is not using the included VT vs. an  
external VT.  But if the user is able to arbitrarily (and perhaps  
accidentally) change the included VT, this becomes problematic for  
support and maintenance.


- about the two vampirtrace-specific spots in the .m4 files: they  
correspont
to two tasks: firstly, decide if you want vampirtrace at all or (if  
you might
want to update) and secondly, passing configure options to  
vampirtrace.
we need to do the first before the second, of course. maybe we can  
move

everything to "our" .m4 file, let me check ...


I would think that all OMPI-specific VT functionality should be in  
one .m4 file.  Per my other mail, I think it should be in contrib/vt/ 
configure.m4.  This makes a nice, clean separation of m4  
functionality and keeps it self-contained into the contrib/vt/ tree.


- btw: so far the vampirtrace distribution tarball is brought to  
openmpi

under ./tracing/vampirtrace with no modifications


Excellent.  That makes things considerably easier.


- the mpicc-vt (and friends) compiler wrappers: this is not part of
vampirtrace but a new thing that only makes sense together with  
openmpi.
therefore, they stay next to 'mpicc' and all others. in fact we're  
following
a earlier suggestion from you, Jeff: 'mpicc-vt' is just like  
'mpicc' but

calls the 'vtcc' compier wrapper instead of 'cc'.

this makes everything much simpler, because we can handle all  
special cases in
vtcc. the wrapper config for 'mpicc-vt' is almost a mere copy of  
mpicc's one.
therefore, I'd like to keep them where they are right now. is this  
o.k. with

everyone?


I like the idea of mpicc-vt (etc.) wrappers, but again, I think they  
should be consolidated in the contrib/vt tree.  There's no technical  
reason they need to be in the wrappers directory.


More specifically, I am uncomfortable with importing 3rd party  
packages that touch a whole bunch of places in the OMPI tree.  I am  
much more comfortable with 3rd party packages being self-contained.


I hope to have the libnbc integration done either this week or next  
as an example.  We're still far enough away from v1.3 release that  
this does not impact any release plans with VT.




--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r16088

[Edgar Gabriel]

Gleb Natapov wrote:

On Tue, Sep 11, 2007 at 10:00:07AM -0500, Edgar Gabriel wrote:

Gleb,

in the scenario which you describe in the comment to the patch, what 
should happen is, that the communicator with the cid which started 
already the allreduce will basically 'hang' until the other processes 
'allow' the lower cids to continue. It should basically be blocked in 
the allreduce.

Why? Two threads are allowed to run allreduce simultaneously for different
communicators. Are they?


They are, but they might never agree on the cid. This is simply how the 
algorithm was designed originally -- which does not mean, that it has to 
remain this way, just to explain its behavior and the intent. See the 
design doc for that in ompi-docs in the January 2004 repository.


Lets assume, that we have n procs with 2 threads, and both threads do a 
comm_create at the same time, input cid 1 and cid 2. N-1 processes let 
cid 1 start because that's the lower number. However, one process let 
cid 2 start because the other thread was late. What would happen in the 
algorithm is nobody responds to cid 2, so it would hang. As soon as the 
other thread with cid 1 enters the comm_create, it would be allowed to 
run, this operation would finish. The other threads would then allow cid 
2 to enter, the 'hanging' process would be released.





However, here is something, where we might have problems with the sun 
thread tests (and we discussed that with Terry already): the cid 
allocation algorithm as implemented in Open MPI assumes ( -- this was/is 
my/our understanding of the standard --) that a communicator creation is 
a collective operation. This means, you can not have a comm_create and 
another allreduce of the same communicator running in different threads, 
because these allreduces will mix up and produce non-sense results. We 
fixed the case, if all collective operations are comm_creates, but if 
some of the threads are in a comm_create and some are in allreduce on 
the same communicator, it won't work.

Correct, but this is not what happens with mt_coll test. mt_coll calls
commdup on the same communicator in different threads concurrently, but
we handle this case inside ompi_comm_nextcid().



Gleb Natapov wrote:

On Tue, Sep 11, 2007 at 10:14:30AM -0400, George Bosilca wrote:

Gleb,

This patch is not correct. The code preventing the registration of the same 
communicator twice is later in the code (same file in the function 
ompi_comm_register_cid line 326). Once the function ompi_comm_register_cid 

I saw this code and the comment. The problem is not with the same
communicator but with different communicators.

is called, we know that each communicator only handle one "communicator 
creation" function at the same time. Therefore, we want to give priority to 
the smallest com_id, which is what happens in the code you removed.

The code I removed was doing it wrongly. I.e the algorithm sometimes is executed
for different communicators simultaneously by different threads. Think
about the case where the function is running for cid 1 and then another
thread runs it for cid 0. cid 0 will proceed although the function is
executed on another CPU. And this is not something theoretical, that
is happening with sun's thread test suit mpi_coll test case.

Without the condition in the ompi_comm_register_cid (each communicator only 
get registered once) your comment make sense. However, with the condition 
your patch allow a dead end situation, while 2 processes try to create 
communicators in multiple threads, and they will never succeed, simply 
because they will not order the creation based on the com_id.

If the algorithm is really prone to deadlock in case it is concurrently
executed for several different communicators (I haven't check this),
then we may want to fix original code to really prevent two threads to
enter the function, but then I don't see the reason for all those
complications with ompi_comm_register_cid()/ompi_comm_unregister_cid()
The algorithm described here:
http://209.85.129.104/search?q=cache:5PV5MMRkBWkJ:ftp://info.mcs.anl.gov/pub/tech_reports/reports/P1382.pdf+MPI+communicator+dup+algorithm=en=clnk=2
in section 5.3 works without it and we can do something similar.


  george.



On Sep 11, 2007, at 9:23 AM, g...@osl.iu.edu wrote:


Author: gleb
Date: 2007-09-11 09:23:46 EDT (Tue, 11 Sep 2007)
New Revision: 16088
URL: https://svn.open-mpi.org/trac/ompi/changeset/16088

Log:
The code tries to prevent itself from running for more then one 
communicator
simultaneously, but is doing it incorrectly. If the function is running 
already
for one communicator and it is called from another thread for other 
communicator

with lower cid the check comm->c_contextid != ompi_comm_lowest_cid()
will fail and the function will be executed for two different 
communicators by
two threads simultaneously. There is nothing in the algorithm that prevent 
it
from been running simultaneously for different communi

Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r16088

[Edgar Gabriel]

_comm_lowest_cid() ) {
-/* if not lowest cid, we do not continue, but sleep and 
try again */

-OPAL_THREAD_UNLOCK(_cid_lock);
-continue;
-}
-OPAL_THREAD_UNLOCK(_cid_lock);
-
-
 for (i=start; i < mca_pml.pml_max_contextid ; i++) {
 
flag=ompi_pointer_array_test_and_set_item(_mpi_communicators,

   i, comm);
@@ -365,10 +357,18 @@

 static int ompi_comm_unregister_cid (uint32_t cid)
 {
-ompi_comm_reg_t *regcom=NULL;
-opal_list_item_t 
*item=opal_list_remove_first(_registered_comms);

+ompi_comm_reg_t *regcom;
+opal_list_item_t *item;

-regcom = (ompi_comm_reg_t *) item;
+for (item = opal_list_get_first(_registered_comms);
+ item != opal_list_get_end(_registered_comms);
+ item = opal_list_get_next(item)) {
+regcom = (ompi_comm_reg_t *)item;
+if(regcom->cid == cid) {
+opal_list_remove_item(_registered_comms, item);
+break;
+}
+}
 OBJ_RELEASE(regcom);
 return OMPI_SUCCESS;
 }
___
svn-full mailing list
svn-f...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/svn-full





___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


--
Gleb.
___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


--
Edgar Gabriel
Assistant Professor
Parallel Software Technologies Lab  http://pstl.cs.uh.edu
Department of Computer Science  University of Houston
Philip G. Hoffman Hall, Room 524Houston, TX-77204, USA
Tel: +1 (713) 743-3857  Fax: +1 (713) 743-3335

Re: [OMPI devel] some possible bugs

[Edgar Gabriel]

Lisandro,

do you have an example for the extended collective operations tests 
which fail? It would help track down the problem. I had a quick look at 
our implementation but I can not find an obvious problem, so an example 
would be extremely helpful.


Thanks
Edgar



 - Some extended collective communications failed (not by raising
   errors, but instead aborting tracing to stdout) when using
   intercommunicators. Sometimes, the problems appeared when
   size(local_group) != size(remote_group). However, MPI_Barrier and
   MPI_Bcast worked well. I still could not get the reasons for those
   failures. I've found a similar problem in MPICH2 when configured
   with error-cheking enabled (they had a bug in some error-cheking
   macros, I reported this issue and next they told me I was right).

Re: [OMPI devel] MPI_Comm_spawn_multiple() and MPI_ERRORCODES_IGNORE

[Edgar Gabriel]

Rolf,

this should actually be fixed already, I just checked on trunk, v1.0 and 
v1.1, in all three branches the according lines have been removed...


Thanks
Edgar

Rolf Vandevaart wrote:
We believe there is a minor bug in comm_spawn_multiple.c. 
If a user hands in an argument of MPI_ERRCODES_IGNORE for the

array_of_errcodes to MPI_Comm_spawn_multiple() and has
parameter checking on, then one will get an error like the
following:

[burl-ct-v440-4:03317] *** MPI_ERR_ARG: invalid argument of some other kind
[burl-ct-v440-4:03317] *** MPI_ERRORS_ARE_FATAL (goodbye)

I think lines 66-69 need to come out.

if ( NULL == array_of_errcodes ) {
return OMPI_ERRHANDLER_INVOKE(comm, MPI_ERR_ARG,
  FUNC_NAME);
}

Looks like this has already been fixed for MPI_Comm_spawn().

Rolf




--
Edgar Gabriel
Assistant Professor
Department of Computer Science  email:gabr...@cs.uh.edu
University of Houston   http://www.cs.uh.edu/~gabriel
Philip G. Hoffman Hall, Room 524Tel: +1 (713) 743-3857
Houston, TX-77204, USA  Fax: +1 (713) 743-3335

Re: [OMPI devel] Repost: MPI_Comm_spawn_multiple() and MPI_ERRORCODES_IGNORE

[Edgar Gabriel]

Rolf,

thanks for catching that, it is now fixed on the trunk and Jeff is 
moving it right now to v1.0 and v1.1


Best regards
Edgar


Rolf Vandevaart wrote:

Repost because I did not see it in the archives after a day.

Rolf Vandevaart wrote On 05/09/06 17:32,:

We believe there is a minor bug in comm_spawn_multiple.c. If a user 
hands in an argument of MPI_ERRCODES_IGNORE for the

array_of_errcodes to MPI_Comm_spawn_multiple() and has
parameter checking on, then one will get an error like the
following:

[burl-ct-v440-4:03317] *** MPI_ERR_ARG: invalid argument of some other 
kind

[burl-ct-v440-4:03317] *** MPI_ERRORS_ARE_FATAL (goodbye)

I think lines 66-69 need to come out.

if ( NULL == array_of_errcodes ) {
return OMPI_ERRHANDLER_INVOKE(comm, MPI_ERR_ARG,
  FUNC_NAME);
}

Looks like this has already been fixed for MPI_Comm_spawn().

Rolf

Re: [O-MPI devel] collectives discussion @LANL

[Edgar Gabriel]

don't forget Stuttgart in the list of remote sites :-). Rainer will be 
at the meeting, but I can only attend through Access Grid.


Thanks
Edgar

Jeff Squyres wrote:

Did we ever set a day/time for the collectives meeting at LANL next 
week?  (we may have and I've forgotten...?)


I ask because those of us involved in that meeting will need to reserve 
time on the Access Grid and coordinate with the remote sites for 
participation.


AFAIK, we have 2 remote sites, right?

- UTK, for Graham Fagg (and others?)
- ?? for Torsten (Torsten: you mentioned where before, but I don't 
recall the name of the site)


Are there others?



--
==
Dr.-Ing. Edgar Gabriel
Clusters and Distributed Units
High Performance Computing Center Stuttgart (HLRS)
University of Stuttgart
Tel: +49 711 685 8039http://www.hlrs.de/people/gabriel
Fax: +49 711 678 7626e-mail:gabr...@hlrs.de
==