Wow! Great and useful explanation. Thanks Jeff . 2009/1/23 Jeff Squyres <jsquy...@cisco.com>: > FWIW, OMPI v1.3 is much better that registered memory usage than the 1.2 > series. We introduced some new things, to include being able to specify > exactly what receive queues you want. See: > > ...gaaah! It's not on our FAQ yet. :-( > > The main idea is that there is a new MCA parameter for the openib BTL: > btl_openib_receive_queues. It takes a colon-delimited string listing one or > more receive queues of specific sizes and characteristics. For now, all > processes in the job *must* use the same string. You can specify three > kinds of receive queues: > > - P: per-peer queues > - S: shared receive queues > - X: XRC queues (with OFED 1.4 and later with specific Mellanox hardware) > > Here's a copy-n-paste of our help file describing the format of each: > > Per-peer receive queues require between 1 and 5 parameters: > > 1. Buffer size in bytes (mandatory) > 2. Number of buffers (optional; defaults to 8) > 3. Low buffer count watermark (optional; defaults to (num_buffers / 2)) > 4. Credit window size (optional; defaults to (low_watermark / 2)) > 5. Number of buffers reserved for credit messages (optional; > defaults to (num_buffers*2-1)/credit_window) > > Example: P,128,256,128,16 > - 128 byte buffers > - 256 buffers to receive incoming MPI messages > - When the number of available buffers reaches 128, re-post 128 more > buffers to reach a total of 256 > - If the number of available credits reaches 16, send an explicit > credit message to the sender > - Defaulting to ((256 * 2) - 1) / 16 = 31; this many buffers are > reserved for explicit credit messages > > Shared receive queues can take between 1 and 4 parameters: > > 1. Buffer size in bytes (mandatory) > 2. Number of buffers (optional; defaults to 16) > 3. Low buffer count watermark (optional; defaults to (num_buffers / 2)) > 4. Maximum number of outstanding sends a sender can have (optional; > defaults to (low_watermark / 4) > > Example: S,1024,256,128,32 > - 1024 byte buffers > - 256 buffers to receive incoming MPI messages > - When the number of available buffers reaches 128, re-post 128 more > buffers to reach a total of 256 > - A sender will not send to a peer unless it has less than 32 > outstanding sends to that peer. > > IIRC, "X" takes the same parameters as "S"...? Note that if you you *any* > XRC queues, then *all* of your queues must be XRC. > > OMPI defaults to a btl_receive_queues value that may be specific to your > hardware. For example, connectx defaults to the following value: > > shell$ ompi_info --param btl openib --parsable | grep receive_queues > mca:btl:openib:param:btl_openib_receive_queues:value:P,128,256,192,128:S,2048,256,128,32:S,12288,256,128,32:S,65536,256,128,32 > mca:btl:openib:param:btl_openib_receive_queues:data_source:default value > mca:btl:openib:param:btl_openib_receive_queues:status:writable > mca:btl:openib:param:btl_openib_receive_queues:help:Colon-delimited, comma > delimited list of receive queues: P,4096,8,6,4:P,32768,8,6,4 > mca:btl:openib:param:btl_openib_receive_queues:deprecated:no > > Hope that helps! > > > > > On Jan 23, 2009, at 9:27 AM, Igor Kozin wrote: > >> Hi Gabriele, >> it might be that your message size is too large for available memory per >> node. >> I had a problem with IMB when I was not able to run to completion Alltoall >> on N=128, ppn=8 on our cluster with 16 GB per node. You'd think 16 GB is >> quite a lot but when you do the maths: >> 2* 4 MB * 128 procs * 8 procs/node = 8 GB/node plus you need to double >> because of buffering. I was told by Mellanox (our cards are ConnectX cards) >> that they introduced XRC in OFED 1.3 in addition to Share Receive Queue >> which should reduce memory foot print but I have not tested this yet. >> HTH, >> Igor >> 2009/1/23 Gabriele Fatigati <g.fatig...@cineca.it> >> Hi Igor, >> My message size is 4096kb and i have 4 procs per core. >> There isn't any difference using different algorithms.. >> >> 2009/1/23 Igor Kozin <i.n.ko...@googlemail.com>: >> > what is your message size and the number of cores per node? >> > is there any difference using different algorithms? >> > >> > 2009/1/23 Gabriele Fatigati <g.fatig...@cineca.it> >> >> >> >> Hi Jeff, >> >> i would like to understand why, if i run over 512 procs or more, my >> >> code stops over mpi collective, also with little send buffer. All >> >> processors are locked into call, doing nothing. But, if i add >> >> MPI_Barrier after MPI collective, it works! I run over Infiniband >> >> net. >> >> >> >> I know many people with this strange problem, i think there is a >> >> strange interaction between Infiniband and OpenMPI that causes it. >> >> >> >> >> >> >> >> 2009/1/23 Jeff Squyres <jsquy...@cisco.com>: >> >> > On Jan 23, 2009, at 6:32 AM, Gabriele Fatigati wrote: >> >> > >> >> >> I've noted that OpenMPI has an asynchronous behaviour in the >> >> >> collective >> >> >> calls. >> >> >> The processors, doesn't wait that other procs arrives in the call. >> >> > >> >> > That is correct. >> >> > >> >> >> This behaviour sometimes can cause some problems with a lot of >> >> >> processors in the jobs. >> >> > >> >> > Can you describe what exactly you mean? The MPI spec specifically >> >> > allows >> >> > this behavior; OMPI made specific design choices and optimizations to >> >> > support this behavior. FWIW, I'd be pretty surprised if any >> >> > optimized >> >> > MPI >> >> > implementation defaults to fully synchronous collective operations. >> >> > >> >> >> Is there an OpenMPI parameter to lock all process in the collective >> >> >> call until is finished? Otherwise i have to insert many MPI_Barrier >> >> >> in my code and it is very tedious and strange.. >> >> > >> >> > As you have notes, MPI_Barrier is the *only* collective operation >> >> > that >> >> > MPI >> >> > guarantees to have any synchronization properties (and it's a fairly >> >> > weak >> >> > guarantee at that; no process will exit the barrier until every >> >> > process >> >> > has >> >> > entered the barrier -- but there's no guarantee that all processes >> >> > leave >> >> > the >> >> > barrier at the same time). >> >> > >> >> > Why do you need your processes to exit collective operations at the >> >> > same >> >> > time? >> >> > >> >> > -- >> >> > Jeff Squyres >> >> > Cisco Systems >> >> > >> >> > _______________________________________________ >> >> > users mailing list >> >> > us...@open-mpi.org >> >> > http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> > >> >> > >> >> >> >> >> >> >> >> -- >> >> Ing. Gabriele Fatigati >> >> >> >> Parallel programmer >> >> >> >> CINECA Systems & Tecnologies Department >> >> >> >> Supercomputing Group >> >> >> >> Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy >> >> >> >> www.cineca.it Tel: +39 051 6171722 >> >> >> >> g.fatigati [AT] cineca.it >> >> _______________________________________________ >> >> users mailing list >> >> us...@open-mpi.org >> >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> > >> > >> > _______________________________________________ >> > users mailing list >> > us...@open-mpi.org >> > http://www.open-mpi.org/mailman/listinfo.cgi/users >> > >> >> >> >> -- >> Ing. Gabriele Fatigati >> >> Parallel programmer >> >> CINECA Systems & Tecnologies Department >> >> Supercomputing Group >> >> Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy >> >> www.cineca.it Tel: +39 051 6171722 >> >> g.fatigati [AT] cineca.it >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > -- > Jeff Squyres > Cisco Systems > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users > >
-- Ing. Gabriele Fatigati Parallel programmer CINECA Systems & Tecnologies Department Supercomputing Group Via Magnanelli 6/3, Casalecchio di Reno (BO) Italy www.cineca.it Tel: +39 051 6171722 g.fatigati [AT] cineca.it
