[O-MPI users] ft state in ompi
hello list, i'd like to know about the state of fault-tolerance in open-mpi. it's said on the site that ompi takes features from ft-mpi and la-mpi (among others), but i couldn't find any reference on the site, nor during my superficial search in the sources. so, is ft implemented or en route? thanks in advance, mlistus __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
Re: [O-MPI users] ft state in ompi
mlistus, Thank you for your question. The integration of fault tolerant techniques into Open MPI is under active development at the moment. As the site notes, we are scheduled to incorporate many forms of fault tolerance into Open MPI which were previously supported by our core projects, and likely some new methods as we go along. So we will support: - FT-MPI: user directed, and communicator driven fault tolerance - LA-MPI: data reliability, and network fault tolerance - LAM/MPI: process fault tolerance by checkpoint and restart I cannot state a timeline for the completion of any of these styles, but I can say they are in active development. Look for updates on this users list, and on the website. Cheers, Josh On Feb 6, 2006, at 4:59 AM, m listus wrote: hello list, i'd like to know about the state of fault-tolerance in open-mpi. it's said on the site that ompi takes features from ft-mpi and la-mpi (among others), but i couldn't find any reference on the site, nor during my superficial search in the sources. so, is ft implemented or en route? thanks in advance, mlistus __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users Josh Hursey jjhur...@open-mpi.org http://www.open-mpi.org/
Re: [O-MPI users] Open-MPI all-to-all performance
Dear Galen, It actually turns out that there is a problem not only with MPI_Alltoall_Isend_Irecv, but also with another related operation insyncol_MPI_Alltoallv-nodes-long-SM.ski (this what seems to be holding down the FFTs, I checked the source code, and it uses alltoallv): #/*@insyncol_MPI_Alltoallv-nodes-long-SM.ski*/ 2 250.8 1.0 8 250.8 1.0 8 31779.6 27.0 81779.6 27.0 8 42975.1 45.8 82975.1 45.8 8 54413.1 76.0 84413.1 76.0 8 6 93370.6 42900.6 8 93370.6 42900.6 8 7 199634.4 43273.1 8 199634.4 43273.1 8 8 262469.6 5896.3 8 262469.6 5896.3 8 The file .skampi I am using is the standard one that came with version 4.1, with only one notable change: @STANDARDERRORDEFAULT 100.00 Thanks! Kostya --- "Galen M. Shipman" wrote: > Hi Konstantin, > > > MPI_Alltoall_Isend_Irecv > > This is a very unscalable algorithm in skampi as it simply posts N > MPI_Irecv's and MPI_Isend's and then does a Waitall. We shouldn't > have an issue though on 8 procs but in general I would expect the > performance of this algorithm to degrade quite quickly especially > compared to Open MPI's tuned collectives. I can dig into this a bit > more if you send me your .skampi file configured to run this > particular benchmark. > > Thanks, > > Galen > > > On Feb 4, 2006, at 9:37 AM, Konstantin Kudin wrote: > > > Dear Jeff and Galen, > > > > I have tried openmpi-1.1a1r8890. The good news is that it seems > like > > the freaky long latencies for certain packet sizes went away with > the > > options they showed up with before. Also, one version of all-to-all > > appears to behave nicer with a specified set of parameters. > However, I > > still get only 1cpu performance out of 8 with the actual > application, > > and all this time is spent doing parallel FFTs. What is interesting > is > > that even with the tuned parameters, the other version of > all-to-all > > still performs quite poorly (see below). > > > > #/*@insyncol_MPI_Alltoall-nodes-long-SM.ski*/ > > mpirun -np 8 -mca btl tcp -mca coll self,basic,tuned -mca \ > > mpi_paffinity_alone 1 skampi41 > >2 272.1 3.7 8 272.1 3.7 8 > >31800.5 72.9 81800.5 72.9 8 > >43074.0 61.0 83074.0 61.0 8 > >55705.5102.0 85705.5102.0 8 > >68054.2282.3 88054.2282.3 8 > >79462.9104.2 89462.9104.2 8 > >8 11245.8 66.9 8 11245.8 66.9 8 > > > > mpirun -np 8 -mca btl tcp -mca coll self,basic,tuned -mca \ > > mpi_paffinity_alone 1 -mca coll_basic_crossover 8 skampi41 > >2 267.7 1.5 8 267.7 1.5 8 > >31591.2 8.4 81591.2 8.4 8 > >42704.4 17.1 82704.4 17.1 8 > >54813.7307.9 34813.7307.9 3 > >65329.1 57.0 25329.1 57.0 2 > >7 198767.6 49076.2 5 198767.6 49076.2 5 > >8 254832.6 11235.3 5 254832.6 11235.3 5 > > > > > > Still poor performance: > > > > #/*@insyncol_MPI_Alltoall_Isend_Irecv-nodes-long-SM.ski*/ > >2 235.0 0.7 8 235.0 0.7 8 > >31565.6 15.3 81565.6 15.3 8 > >42694.8 24.3 82694.8 24.3 8 > >5 11389.9 6971.9 6 11389.9 6971.9 6 > >6 249612.0 21102.1 2 249612.0 21102.1 2 > >7 239051.9 3915.0 2 239051.9 3915.0 2 > >8 262356.5 12324.6 2 262356.5 12324.6 2 > > > > > > Kostya > > > > > > > > > > --- Jeff Squyres wrote: > > > >> Greetings Konstantin. > >> > >> Many thanks for this report. Another user submitted almost the > same > >> > >> issue earlier today (poor performance of Open MPI 1.0.x > collectives; > >> > >> see > http://www.open-mpi.org/community/lists/users/2006/02/0558.php). > >> > >> Let me provide an additional clarification on Galen's reply: > >> > >> The collectives in Open MPI 1.0.x are known to be sub-optimal -- > they > >> > >> return correct results, but they are not optimized at all. This > is > >> what Galen meant by "If I use the basic collectives then things do > >> fall apart with long messages, but this is expected". The > >> collectives in the Open MPI 1.1.x series (i.e., our current > >> development trunk) provide *much* better performance. > >> > >> Galen ran his tests using the "tuned" collective module in the > 1.1.x > >> > >> series -- these are the "better" collectives that I referred to > >> above. This "tuned" module does not exist in the 1.0.x series. > >> > >> You can download a 1.1.x nightly snapshot -- inc
[O-MPI users] mpirun sets umask to 0
mpirun (v1.0.1) sets the umask to 0, and hence creates world-writable
output files. Interestingly, adding the -d option to mpirun makes this
problem go away. To reproduce:
mpirun -np 1 --hostfile ./hostfile --mca pls_rsh_agent ssh ./a.out
where a.out is compiled from:
#include
#include
#include
int main ()
{
printf("%.4o\n", umask( 022 ) );
return 0;
}
Re: [O-MPI users] Configuring OPEN MPI 1.0.1 MAC OS X 10.4.4
Thank you Brian Barrett and Brian Granger. I downloaded and installed a new version of gfortran from http://gcc.gnu.org/wiki/GFortranBinaries and the install went without a problem. I had gfortran installed before from Fink. David * Correspondence * > From: Brian Barrett > Reply-To: Open MPI Users > Date: Sat, 4 Feb 2006 10:29:30 -0600 > To: Open MPI Users > Subject: Re: [O-MPI users] Configuring OPEN MPI 1.0.1 MAC OS X 10.4.4 > > Brian Granger is correct - the problem is with your Fortran > installation. Since you are trying to use gfortran instead of g77, > you do not need to use the GCC 3.3 toolchain - the GCC 4.0 chain is > the right one to use. > > However, gfortran is installed incorrectly on your system. Compiling > a simple code results in warnings that should not be there (from > config.log): > >configure:20505: gfortran -c conftest.f >&5 >gfortran: spec failure: unrecognized spec option 'y' > > And running a simple code results in even worse problems: > >configure:20670: gfortran conftestf.f conftest.o -o conftest >gfortran: spec failure: unrecognized spec option 'y' >gfortran: spec failure: unrecognized spec option 'M' >/usr/bin/ld: warning suggest use of -bind_at_load, as lazy binding > may result in > errors or different symbols being used >symbol _clogf used from dynamic library /usr/lib/libSystem.dylib > (complex.o) not > from earlier dynamic library /usr/lib/libmx.A.dylib(single module) >... and so on with a large number of symbols ... > > I'd try making sure you have the latest version of the gfortran build > you downloaded. I've found that the one on hpc.sourceforge.net works > pretty well for me, but I rarely do anything more intensive than > making sure the MPI bindings work properly. > > Slightly off-topic, but future releases of Open MPI will have a more > clear report when the compiler fails to produce useable executables. > We're aware that the current warnings are a bit misleading. > > Hope this helps, > > Brian > > > On Feb 3, 2006, at 4:55 PM, Brian Granger wrote: > >> Hi, >> >> It looks like it could be a problem with either using gcc 4.x or with >> the fortran you are using. I have compiled OpenMPI on 10.4.4. using >> the compilers: >> >> myhost$ g77 -v >> Reading specs from /usr/local/lib/gcc/powerpc-apple-darwin7.9.0/3.4.4/ >> specs >> Configured with: ../gcc/configure --enable-threads=posix --enable- >> languages=f77 >> Thread model: posix >> gcc version 3.4.4 >> >> myhost$ gcc -v >> Reading specs from /usr/libexec/gcc/darwin/ppc/3.3/specs >> Thread model: posix >> gcc version 3.3 20030304 (Apple Computer, Inc. build 1809) >> >> You might want to download g77 and switch gcc to version 3.3 using >> the command: >> >> sudo gcc_select 3.3 >> >> Brian >> >> On Feb 3, 2006, at 2:20 PM, Xiaoning (David) Yang wrote: >> >>> I'm trying to configure OPEN MPI 1.0.1 under MAC OS X 10.4.4 and >>> was not >>> successful. Attached are the output from './configure --prefix=/usr/ >>> local' >>> and the configure log flle in a tarball. Any help is appreciated! >>> >>> David >>> >>> * Correspondence * >>> >>> >>> >>> ___ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> ___ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > -- >Brian Barrett >Open MPI developer >http://www.open-mpi.org/ > > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
[O-MPI users] OpenMPI and Xgrid
Brian help!! :-) On Feb 5, 2006, at 9:00 AM, users-requ...@open-mpi.org wrote: If this is the case, my next question is, how do I supply the usual xgrid options, such as working directory, standard input file, etc? Or is that simply not possible? Do I simply have to have some other way (eg ssh) to get files to/ from agent machines, like I would for a batch system like PBS? It looks like I never implemented those options (shame on me). I've added that to my to-do list, although I can't give an accurate time- table for implementation at this point. One thing to note is that rather than using XGrid's standard input/output forwarding services, we use Open MPI's services. So if you do: mpirun -np 2 ./myapp < foo.txt Under Xgrid with Open MPI, I'm trying to run applications that require more than just reading standard input/output but also the creation and writing of other intermediate files. For an application that like HP Linpack that just reads and writes one file, things work fine. My guess is that this is where things are getting hung up. Below, my application was trying to write out a file called "testrun.nex.run1.p" and failed. The MrBayes application writes out two files for each mpi process. Initial log likelihoods for run 1: Chain 1 -- -429.987779 Chain 2 -- -386.761468 Could not open file "testrun.nex.run1.p" Memory allocation error on at least one processor Error in command "Mcmc" There was an error on at least one processor Error in command "Execute" Will exit with signal 1 (error) because quitonerror is set to yes Am I just misunderstanding how to set up Open MPI to work with Xgrid? Warner Yuen Research Computing Consultant Apple Computer email: wy...@apple.com Tel: 408.718.2859 Fax: 408.715.0133
Re: [O-MPI users] A few benchmarks
George Bosilca wrote: Glen, Thanks for the spending time benchmarking OpenMPI and for sending us the feedback. We know we have some issues on the 1.0.2 version, more precisely with the collective communications. We just look inside the CMAQ code, and there are a lot of reduce and Allreduce. As it look like the collective are intensively used it's normal that the 1.0.2a4 is slower than MPICH (I expect the same behaviour for both MPICH1 and MPICH2). The collective are now fixed in the nightly build, we are working toward moving them on the next stable release. Until then, if you can redo the benchmark with one of the nightly build that will be very usefull. I'm confident that the results will improve considerably. Hi. You're a brave guy even looking at CMAQ. =) Anyway, here are the times on a few runs I did with Open MPI 1.1a1r887. Basically what I'm seeing, my jobs run ok when they're local to one machine, but as soon as they're split up between multiple machines performance can vary: 4 cpu jobs: 2:16:27 2:01:35 1:24:20 1:03:55 1:22:43 1:31:53 8 cpu jobs: 1:02:53 1:08:52 1:46:25 1:17:39 0:43:44 1:02:31 And by the way. I was doing some maintenance work on my machines this weekend, so absolutely everyone was kicked off. I'm positive nothing else was interfering with these jobs. Also, someone had asked what my setup was, so here it is basically: HP Procurve 2848 gigabit ethernet switch. Tyan K8S boards, with dual opteron 246's, 2 gigs of ram, and built in broadcom gigabit ethernet adapters. Rocks 4.0, with the latest updates from Red Hat, running a 2.6.9-22.0.1.ELsmp kernel. Network attached storage via NFS. I don't think my setup is the problem though anyway, as these jobs have been running file for a while now with MPICH. Glen
