As far as I can tell, it appears the problem is somewhere in our communicator setup. The people knowledgeable on that area are going to look into it later this week.
I'm creating a ticket to track the problem and will copy you on it. On Jul 13, 2010, at 6:57 AM, Ralph Castain wrote: > > On Jul 13, 2010, at 3:36 AM, Grzegorz Maj wrote: > >> Bad news.. >> I've tried the latest patch with and without the prior one, but it >> hasn't changed anything. I've also tried using the old code but with >> the OMPI_DPM_BASE_MAXJOBIDS constant changed to 80, but it also didn't >> help. >> While looking through the sources of openmpi-1.4.2 I couldn't find any >> call of the function ompi_dpm_base_mark_dyncomm. > > It isn't directly called - it shows in ompi_comm_set as > ompi_dpm.mark_dyncomm. You were definitely overrunning that array, but I > guess something else is also being hit. Have to look further... > > >> >> >> 2010/7/12 Ralph Castain <r...@open-mpi.org>: >>> Just so you don't have to wait for 1.4.3 release, here is the patch >>> (doesn't include the prior patch). >>> >>> >>> >>> >>> On Jul 12, 2010, at 12:13 PM, Grzegorz Maj wrote: >>> >>>> 2010/7/12 Ralph Castain <r...@open-mpi.org>: >>>>> Dug around a bit and found the problem!! >>>>> >>>>> I have no idea who or why this was done, but somebody set a limit of 64 >>>>> separate jobids in the dynamic init called by ompi_comm_set, which builds >>>>> the intercommunicator. Unfortunately, they hard-wired the array size, but >>>>> never check that size before adding to it. >>>>> >>>>> So after 64 calls to connect_accept, you are overwriting other areas of >>>>> the code. As you found, hitting 66 causes it to segfault. >>>>> >>>>> I'll fix this on the developer's trunk (I'll also add that original patch >>>>> to it). Rather than my searching this thread in detail, can you remind me >>>>> what version you are using so I can patch it too? >>>> >>>> I'm using 1.4.2 >>>> Thanks a lot and I'm looking forward for the patch. >>>> >>>>> >>>>> Thanks for your patience with this! >>>>> Ralph >>>>> >>>>> >>>>> On Jul 12, 2010, at 7:20 AM, Grzegorz Maj wrote: >>>>> >>>>>> 1024 is not the problem: changing it to 2048 hasn't change anything. >>>>>> Following your advice I've run my process using gdb. Unfortunately I >>>>>> didn't get anything more than: >>>>>> >>>>>> Program received signal SIGSEGV, Segmentation fault. >>>>>> [Switching to Thread 0xf7e4c6c0 (LWP 20246)] >>>>>> 0xf7f39905 in ompi_comm_set () from /home/gmaj/openmpi/lib/libmpi.so.0 >>>>>> >>>>>> (gdb) bt >>>>>> #0 0xf7f39905 in ompi_comm_set () from >>>>>> /home/gmaj/openmpi/lib/libmpi.so.0 >>>>>> #1 0xf7e3ba95 in connect_accept () from >>>>>> /home/gmaj/openmpi/lib/openmpi/mca_dpm_orte.so >>>>>> #2 0xf7f62013 in PMPI_Comm_connect () from >>>>>> /home/gmaj/openmpi/lib/libmpi.so.0 >>>>>> #3 0x080489ed in main (argc=825832753, argv=0x34393638) at client.c:43 >>>>>> >>>>>> What's more: when I've added a breakpoint on ompi_comm_set in 66th >>>>>> process and stepped a couple of instructions, one of the other >>>>>> processes crashed (as usualy on ompi_comm_set) earlier than 66th did. >>>>>> >>>>>> Finally I decided to recompile openmpi using -g flag for gcc. In this >>>>>> case the 66 processes issue has gone! I was running my applications >>>>>> exactly the same way as previously (even without recompilation) and >>>>>> I've run successfully over 130 processes. >>>>>> When switching back to the openmpi compilation without -g it again >>>>>> segfaults. >>>>>> >>>>>> Any ideas? I'm really confused. >>>>>> >>>>>> >>>>>> >>>>>> 2010/7/7 Ralph Castain <r...@open-mpi.org>: >>>>>>> I would guess the #files limit of 1024. However, if it behaves the same >>>>>>> way when spread across multiple machines, I would suspect it is >>>>>>> somewhere in your program itself. Given that the segfault is in your >>>>>>> process, can you use gdb to look at the core file and see where and why >>>>>>> it fails? >>>>>>> >>>>>>> On Jul 7, 2010, at 10:17 AM, Grzegorz Maj wrote: >>>>>>> >>>>>>>> 2010/7/7 Ralph Castain <r...@open-mpi.org>: >>>>>>>>> >>>>>>>>> On Jul 6, 2010, at 8:48 AM, Grzegorz Maj wrote: >>>>>>>>> >>>>>>>>>> Hi Ralph, >>>>>>>>>> sorry for the late response, but I couldn't find free time to play >>>>>>>>>> with this. Finally I've applied the patch you prepared. I've launched >>>>>>>>>> my processes in the way you've described and I think it's working as >>>>>>>>>> you expected. None of my processes runs the orted daemon and they can >>>>>>>>>> perform MPI operations. Unfortunately I'm still hitting the 65 >>>>>>>>>> processes issue :( >>>>>>>>>> Maybe I'm doing something wrong. >>>>>>>>>> I attach my source code. If anybody could have a look on this, I >>>>>>>>>> would >>>>>>>>>> be grateful. >>>>>>>>>> >>>>>>>>>> When I run that code with clients_count <= 65 everything works fine: >>>>>>>>>> all the processes create a common grid, exchange some information and >>>>>>>>>> disconnect. >>>>>>>>>> When I set clients_count > 65 the 66th process crashes on >>>>>>>>>> MPI_Comm_connect (segmentation fault). >>>>>>>>> >>>>>>>>> I didn't have time to check the code, but my guess is that you are >>>>>>>>> still hitting some kind of file descriptor or other limit. Check to >>>>>>>>> see what your limits are - usually "ulimit" will tell you. >>>>>>>> >>>>>>>> My limitations are: >>>>>>>> time(seconds) unlimited >>>>>>>> file(blocks) unlimited >>>>>>>> data(kb) unlimited >>>>>>>> stack(kb) 10240 >>>>>>>> coredump(blocks) 0 >>>>>>>> memory(kb) unlimited >>>>>>>> locked memory(kb) 64 >>>>>>>> process 200704 >>>>>>>> nofiles 1024 >>>>>>>> vmemory(kb) unlimited >>>>>>>> locks unlimited >>>>>>>> >>>>>>>> Which one do you think could be responsible for that? >>>>>>>> >>>>>>>> I was trying to run all the 66 processes on one machine or spread them >>>>>>>> across several machines and it always crashes the same way on the 66th >>>>>>>> process. >>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Another thing I would like to know is if it's normal that any of my >>>>>>>>>> processes when calling MPI_Comm_connect or MPI_Comm_accept when the >>>>>>>>>> other side is not ready, is eating up a full CPU available. >>>>>>>>> >>>>>>>>> Yes - the waiting process is polling in a tight loop waiting for the >>>>>>>>> connection to be made. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Any help would be appreciated, >>>>>>>>>> Grzegorz Maj >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2010/4/24 Ralph Castain <r...@open-mpi.org>: >>>>>>>>>>> Actually, OMPI is distributed with a daemon that does pretty much >>>>>>>>>>> what you >>>>>>>>>>> want. Checkout "man ompi-server". I originally wrote that code to >>>>>>>>>>> support >>>>>>>>>>> cross-application MPI publish/subscribe operations, but we can >>>>>>>>>>> utilize it >>>>>>>>>>> here too. Have to blame me for not making it more publicly known. >>>>>>>>>>> The attached patch upgrades ompi-server and modifies the singleton >>>>>>>>>>> startup >>>>>>>>>>> to provide your desired support. This solution works in the >>>>>>>>>>> following >>>>>>>>>>> manner: >>>>>>>>>>> 1. launch "ompi-server -report-uri <filename>". This starts a >>>>>>>>>>> persistent >>>>>>>>>>> daemon called "ompi-server" that acts as a rendezvous point for >>>>>>>>>>> independently started applications. The problem with starting >>>>>>>>>>> different >>>>>>>>>>> applications and wanting them to MPI connect/accept lies in the >>>>>>>>>>> need to have >>>>>>>>>>> the applications find each other. If they can't discover contact >>>>>>>>>>> info for >>>>>>>>>>> the other app, then they can't wire up their interconnects. The >>>>>>>>>>> "ompi-server" tool provides that rendezvous point. I don't like that >>>>>>>>>>> comm_accept segfaulted - should have just error'd out. >>>>>>>>>>> 2. set OMPI_MCA_orte_server=file:<filename>" in the environment >>>>>>>>>>> where you >>>>>>>>>>> will start your processes. This will allow your singleton processes >>>>>>>>>>> to find >>>>>>>>>>> the ompi-server. I automatically also set the envar to connect the >>>>>>>>>>> MPI >>>>>>>>>>> publish/subscribe system for you. >>>>>>>>>>> 3. run your processes. As they think they are singletons, they will >>>>>>>>>>> detect >>>>>>>>>>> the presence of the above envar and automatically connect >>>>>>>>>>> themselves to the >>>>>>>>>>> "ompi-server" daemon. This provides each process with the ability >>>>>>>>>>> to perform >>>>>>>>>>> any MPI-2 operation. >>>>>>>>>>> I tested this on my machines and it worked, so hopefully it will >>>>>>>>>>> meet your >>>>>>>>>>> needs. You only need to run one "ompi-server" period, so long as >>>>>>>>>>> you locate >>>>>>>>>>> it where all of the processes can find the contact file and can >>>>>>>>>>> open a TCP >>>>>>>>>>> socket to the daemon. There is a way to knit multiple ompi-servers >>>>>>>>>>> into a >>>>>>>>>>> broader network (e.g., to connect processes that cannot directly >>>>>>>>>>> access a >>>>>>>>>>> server due to network segmentation), but it's a tad tricky - let me >>>>>>>>>>> know if >>>>>>>>>>> you require it and I'll try to help. >>>>>>>>>>> If you have trouble wiring them all into a single communicator, you >>>>>>>>>>> might >>>>>>>>>>> ask separately about that and see if one of our MPI experts can >>>>>>>>>>> provide >>>>>>>>>>> advice (I'm just the RTE grunt). >>>>>>>>>>> HTH - let me know how this works for you and I'll incorporate it >>>>>>>>>>> into future >>>>>>>>>>> OMPI releases. >>>>>>>>>>> Ralph >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Apr 24, 2010, at 1:49 AM, Krzysztof Zarzycki wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Ralph, >>>>>>>>>>> I'm Krzysztof and I'm working with Grzegorz Maj on this our small >>>>>>>>>>> project/experiment. >>>>>>>>>>> We definitely would like to give your patch a try. But could you >>>>>>>>>>> please >>>>>>>>>>> explain your solution a little more? >>>>>>>>>>> You still would like to start one mpirun per mpi grid, and then have >>>>>>>>>>> processes started by us to join the MPI comm? >>>>>>>>>>> It is a good solution of course. >>>>>>>>>>> But it would be especially preferable to have one daemon running >>>>>>>>>>> persistently on our "entry" machine that can handle several mpi >>>>>>>>>>> grid starts. >>>>>>>>>>> Can your patch help us this way too? >>>>>>>>>>> Thanks for your help! >>>>>>>>>>> Krzysztof >>>>>>>>>>> >>>>>>>>>>> On 24 April 2010 03:51, Ralph Castain <r...@open-mpi.org> wrote: >>>>>>>>>>>> >>>>>>>>>>>> In thinking about this, my proposed solution won't entirely fix the >>>>>>>>>>>> problem - you'll still wind up with all those daemons. I believe I >>>>>>>>>>>> can >>>>>>>>>>>> resolve that one as well, but it would require a patch. >>>>>>>>>>>> >>>>>>>>>>>> Would you like me to send you something you could try? Might take >>>>>>>>>>>> a couple >>>>>>>>>>>> of iterations to get it right... >>>>>>>>>>>> >>>>>>>>>>>> On Apr 23, 2010, at 12:12 PM, Ralph Castain wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hmmm....I -think- this will work, but I cannot guarantee it: >>>>>>>>>>>>> >>>>>>>>>>>>> 1. launch one process (can just be a spinner) using mpirun that >>>>>>>>>>>>> includes >>>>>>>>>>>>> the following option: >>>>>>>>>>>>> >>>>>>>>>>>>> mpirun -report-uri file >>>>>>>>>>>>> >>>>>>>>>>>>> where file is some filename that mpirun can create and insert its >>>>>>>>>>>>> contact info into it. This can be a relative or absolute path. >>>>>>>>>>>>> This process >>>>>>>>>>>>> must remain alive throughout your application - doesn't matter >>>>>>>>>>>>> what it does. >>>>>>>>>>>>> It's purpose is solely to keep mpirun alive. >>>>>>>>>>>>> >>>>>>>>>>>>> 2. set OMPI_MCA_dpm_orte_server=FILE:file in your environment, >>>>>>>>>>>>> where >>>>>>>>>>>>> "file" is the filename given above. This will tell your processes >>>>>>>>>>>>> how to >>>>>>>>>>>>> find mpirun, which is acting as a meeting place to handle the >>>>>>>>>>>>> connect/accept >>>>>>>>>>>>> operations >>>>>>>>>>>>> >>>>>>>>>>>>> Now run your processes, and have them connect/accept to each >>>>>>>>>>>>> other. >>>>>>>>>>>>> >>>>>>>>>>>>> The reason I cannot guarantee this will work is that these >>>>>>>>>>>>> processes >>>>>>>>>>>>> will all have the same rank && name since they all start as >>>>>>>>>>>>> singletons. >>>>>>>>>>>>> Hence, connect/accept is likely to fail. >>>>>>>>>>>>> >>>>>>>>>>>>> But it -might- work, so you might want to give it a try. >>>>>>>>>>>>> >>>>>>>>>>>>> On Apr 23, 2010, at 8:10 AM, Grzegorz Maj wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> To be more precise: by 'server process' I mean some process that >>>>>>>>>>>>>> I >>>>>>>>>>>>>> could run once on my system and it could help in creating those >>>>>>>>>>>>>> groups. >>>>>>>>>>>>>> My typical scenario is: >>>>>>>>>>>>>> 1. run N separate processes, each without mpirun >>>>>>>>>>>>>> 2. connect them into MPI group >>>>>>>>>>>>>> 3. do some job >>>>>>>>>>>>>> 4. exit all N processes >>>>>>>>>>>>>> 5. goto 1 >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2010/4/23 Grzegorz Maj <ma...@wp.pl>: >>>>>>>>>>>>>>> Thank you Ralph for your explanation. >>>>>>>>>>>>>>> And, apart from that descriptors' issue, is there any other way >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>> solve my problem, i.e. to run separately a number of processes, >>>>>>>>>>>>>>> without mpirun and then to collect them into an MPI intracomm >>>>>>>>>>>>>>> group? >>>>>>>>>>>>>>> If I for example would need to run some 'server process' (even >>>>>>>>>>>>>>> using >>>>>>>>>>>>>>> mpirun) for this task, that's OK. Any ideas? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Grzegorz Maj >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2010/4/18 Ralph Castain <r...@open-mpi.org>: >>>>>>>>>>>>>>>> Okay, but here is the problem. If you don't use mpirun, and >>>>>>>>>>>>>>>> are not >>>>>>>>>>>>>>>> operating in an environment we support for "direct" launch >>>>>>>>>>>>>>>> (i.e., starting >>>>>>>>>>>>>>>> processes outside of mpirun), then every one of those >>>>>>>>>>>>>>>> processes thinks it is >>>>>>>>>>>>>>>> a singleton - yes? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> What you may not realize is that each singleton immediately >>>>>>>>>>>>>>>> fork/exec's an orted daemon that is configured to behave just >>>>>>>>>>>>>>>> like mpirun. >>>>>>>>>>>>>>>> This is required in order to support MPI-2 operations such as >>>>>>>>>>>>>>>> MPI_Comm_spawn, MPI_Comm_connect/accept, etc. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> So if you launch 64 processes that think they are singletons, >>>>>>>>>>>>>>>> then >>>>>>>>>>>>>>>> you have 64 copies of orted running as well. This eats up a >>>>>>>>>>>>>>>> lot of file >>>>>>>>>>>>>>>> descriptors, which is probably why you are hitting this 65 >>>>>>>>>>>>>>>> process limit - >>>>>>>>>>>>>>>> your system is probably running out of file descriptors. You >>>>>>>>>>>>>>>> might check you >>>>>>>>>>>>>>>> system limits and see if you can get them revised upward. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Apr 17, 2010, at 4:24 PM, Grzegorz Maj wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Yes, I know. The problem is that I need to use some special >>>>>>>>>>>>>>>>> way for >>>>>>>>>>>>>>>>> running my processes provided by the environment in which I'm >>>>>>>>>>>>>>>>> working >>>>>>>>>>>>>>>>> and unfortunately I can't use mpirun. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2010/4/18 Ralph Castain <r...@open-mpi.org>: >>>>>>>>>>>>>>>>>> Guess I don't understand why you can't use mpirun - all it >>>>>>>>>>>>>>>>>> does is >>>>>>>>>>>>>>>>>> start things, provide a means to forward io, etc. It mainly >>>>>>>>>>>>>>>>>> sits there >>>>>>>>>>>>>>>>>> quietly without using any cpu unless required to support the >>>>>>>>>>>>>>>>>> job. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Sounds like it would solve your problem. Otherwise, I know >>>>>>>>>>>>>>>>>> of no >>>>>>>>>>>>>>>>>> way to get all these processes into comm_world. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Apr 17, 2010, at 2:27 PM, Grzegorz Maj wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>> I'd like to dynamically create a group of processes >>>>>>>>>>>>>>>>>>> communicating >>>>>>>>>>>>>>>>>>> via >>>>>>>>>>>>>>>>>>> MPI. Those processes need to be run without mpirun and >>>>>>>>>>>>>>>>>>> create >>>>>>>>>>>>>>>>>>> intracommunicator after the startup. Any ideas how to do >>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>> efficiently? >>>>>>>>>>>>>>>>>>> I came up with a solution in which the processes are >>>>>>>>>>>>>>>>>>> connecting >>>>>>>>>>>>>>>>>>> one by >>>>>>>>>>>>>>>>>>> one using MPI_Comm_connect, but unfortunately all the >>>>>>>>>>>>>>>>>>> processes >>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>> are already in the group need to call MPI_Comm_accept. This >>>>>>>>>>>>>>>>>>> means >>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>> when the n-th process wants to connect I need to collect >>>>>>>>>>>>>>>>>>> all the >>>>>>>>>>>>>>>>>>> n-1 >>>>>>>>>>>>>>>>>>> processes on the MPI_Comm_accept call. After I run about 40 >>>>>>>>>>>>>>>>>>> processes >>>>>>>>>>>>>>>>>>> every subsequent call takes more and more time, which I'd >>>>>>>>>>>>>>>>>>> like to >>>>>>>>>>>>>>>>>>> avoid. >>>>>>>>>>>>>>>>>>> Another problem in this solution is that when I try to >>>>>>>>>>>>>>>>>>> connect >>>>>>>>>>>>>>>>>>> 66-th >>>>>>>>>>>>>>>>>>> process the root of the existing group segfaults on >>>>>>>>>>>>>>>>>>> MPI_Comm_accept. >>>>>>>>>>>>>>>>>>> Maybe it's my bug, but it's weird as everything works fine >>>>>>>>>>>>>>>>>>> for at >>>>>>>>>>>>>>>>>>> most >>>>>>>>>>>>>>>>>>> 65 processes. Is there any limitation I don't know about? >>>>>>>>>>>>>>>>>>> My last question is about MPI_COMM_WORLD. When I run my >>>>>>>>>>>>>>>>>>> processes >>>>>>>>>>>>>>>>>>> without mpirun their MPI_COMM_WORLD is the same as >>>>>>>>>>>>>>>>>>> MPI_COMM_SELF. >>>>>>>>>>>>>>>>>>> Is >>>>>>>>>>>>>>>>>>> there any way to change MPI_COMM_WORLD and set it to the >>>>>>>>>>>>>>>>>>> intracommunicator that I've created? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>> Grzegorz Maj >>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> users mailing list >>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> users mailing list >>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> users mailing list >>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>> >>>>>>>>>> <client.c><server.c>_______________________________________________ >>>>>>>>>> users mailing list >>>>>>>>>> us...@open-mpi.org >>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> users mailing list >>>>>>>>> us...@open-mpi.org >>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> users mailing list >>>>>>>> us...@open-mpi.org >>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> users mailing list >>>>>>> us...@open-mpi.org >>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>> >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>> >> >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >
