Hm, so I am not sure how to approach this. First of all, the test case works for me. I used up to 80 clients, and for both optimized and non-optimized compilation. I ran the tests with trunk (not with 1.4 series, but the communicator code is identical in both cases). Clearly, the patch from Ralph is necessary to make it work.
Additionally, I went through the communicator creation code for dynamic communicators trying to find spots that could create problems. The only place that I found the number 64 appear is the fortran-to-c mapping arrays (e.g. for communicators), where the initial size of the table is 64. I looked twice over the pointer-array code to see whether we could have a problem their (since it is a key-piece of the cid allocation code for communicators), but I am fairly confident that it is correct. Note, that we have other (non-dynamic tests), were comm_set is called 100,000 times, and the code per se does not seem to have a problem due to being called too often. So I am not sure what else to look at. Edgar On 7/13/2010 8:42 PM, Ralph Castain wrote: > As far as I can tell, it appears the problem is somewhere in our communicator > setup. The people knowledgeable on that area are going to look into it later > this week. > > I'm creating a ticket to track the problem and will copy you on it. > > > On Jul 13, 2010, at 6:57 AM, Ralph Castain wrote: > >> >> On Jul 13, 2010, at 3:36 AM, Grzegorz Maj wrote: >> >>> Bad news.. >>> I've tried the latest patch with and without the prior one, but it >>> hasn't changed anything. I've also tried using the old code but with >>> the OMPI_DPM_BASE_MAXJOBIDS constant changed to 80, but it also didn't >>> help. >>> While looking through the sources of openmpi-1.4.2 I couldn't find any >>> call of the function ompi_dpm_base_mark_dyncomm. >> >> It isn't directly called - it shows in ompi_comm_set as >> ompi_dpm.mark_dyncomm. You were definitely overrunning that array, but I >> guess something else is also being hit. Have to look further... >> >> >>> >>> >>> 2010/7/12 Ralph Castain <r...@open-mpi.org>: >>>> Just so you don't have to wait for 1.4.3 release, here is the patch >>>> (doesn't include the prior patch). >>>> >>>> >>>> >>>> >>>> On Jul 12, 2010, at 12:13 PM, Grzegorz Maj wrote: >>>> >>>>> 2010/7/12 Ralph Castain <r...@open-mpi.org>: >>>>>> Dug around a bit and found the problem!! >>>>>> >>>>>> I have no idea who or why this was done, but somebody set a limit of 64 >>>>>> separate jobids in the dynamic init called by ompi_comm_set, which >>>>>> builds the intercommunicator. Unfortunately, they hard-wired the array >>>>>> size, but never check that size before adding to it. >>>>>> >>>>>> So after 64 calls to connect_accept, you are overwriting other areas of >>>>>> the code. As you found, hitting 66 causes it to segfault. >>>>>> >>>>>> I'll fix this on the developer's trunk (I'll also add that original >>>>>> patch to it). Rather than my searching this thread in detail, can you >>>>>> remind me what version you are using so I can patch it too? >>>>> >>>>> I'm using 1.4.2 >>>>> Thanks a lot and I'm looking forward for the patch. >>>>> >>>>>> >>>>>> Thanks for your patience with this! >>>>>> Ralph >>>>>> >>>>>> >>>>>> On Jul 12, 2010, at 7:20 AM, Grzegorz Maj wrote: >>>>>> >>>>>>> 1024 is not the problem: changing it to 2048 hasn't change anything. >>>>>>> Following your advice I've run my process using gdb. Unfortunately I >>>>>>> didn't get anything more than: >>>>>>> >>>>>>> Program received signal SIGSEGV, Segmentation fault. >>>>>>> [Switching to Thread 0xf7e4c6c0 (LWP 20246)] >>>>>>> 0xf7f39905 in ompi_comm_set () from /home/gmaj/openmpi/lib/libmpi.so.0 >>>>>>> >>>>>>> (gdb) bt >>>>>>> #0 0xf7f39905 in ompi_comm_set () from >>>>>>> /home/gmaj/openmpi/lib/libmpi.so.0 >>>>>>> #1 0xf7e3ba95 in connect_accept () from >>>>>>> /home/gmaj/openmpi/lib/openmpi/mca_dpm_orte.so >>>>>>> #2 0xf7f62013 in PMPI_Comm_connect () from >>>>>>> /home/gmaj/openmpi/lib/libmpi.so.0 >>>>>>> #3 0x080489ed in main (argc=825832753, argv=0x34393638) at client.c:43 >>>>>>> >>>>>>> What's more: when I've added a breakpoint on ompi_comm_set in 66th >>>>>>> process and stepped a couple of instructions, one of the other >>>>>>> processes crashed (as usualy on ompi_comm_set) earlier than 66th did. >>>>>>> >>>>>>> Finally I decided to recompile openmpi using -g flag for gcc. In this >>>>>>> case the 66 processes issue has gone! I was running my applications >>>>>>> exactly the same way as previously (even without recompilation) and >>>>>>> I've run successfully over 130 processes. >>>>>>> When switching back to the openmpi compilation without -g it again >>>>>>> segfaults. >>>>>>> >>>>>>> Any ideas? I'm really confused. >>>>>>> >>>>>>> >>>>>>> >>>>>>> 2010/7/7 Ralph Castain <r...@open-mpi.org>: >>>>>>>> I would guess the #files limit of 1024. However, if it behaves the >>>>>>>> same way when spread across multiple machines, I would suspect it is >>>>>>>> somewhere in your program itself. Given that the segfault is in your >>>>>>>> process, can you use gdb to look at the core file and see where and >>>>>>>> why it fails? >>>>>>>> >>>>>>>> On Jul 7, 2010, at 10:17 AM, Grzegorz Maj wrote: >>>>>>>> >>>>>>>>> 2010/7/7 Ralph Castain <r...@open-mpi.org>: >>>>>>>>>> >>>>>>>>>> On Jul 6, 2010, at 8:48 AM, Grzegorz Maj wrote: >>>>>>>>>> >>>>>>>>>>> Hi Ralph, >>>>>>>>>>> sorry for the late response, but I couldn't find free time to play >>>>>>>>>>> with this. Finally I've applied the patch you prepared. I've >>>>>>>>>>> launched >>>>>>>>>>> my processes in the way you've described and I think it's working as >>>>>>>>>>> you expected. None of my processes runs the orted daemon and they >>>>>>>>>>> can >>>>>>>>>>> perform MPI operations. Unfortunately I'm still hitting the 65 >>>>>>>>>>> processes issue :( >>>>>>>>>>> Maybe I'm doing something wrong. >>>>>>>>>>> I attach my source code. If anybody could have a look on this, I >>>>>>>>>>> would >>>>>>>>>>> be grateful. >>>>>>>>>>> >>>>>>>>>>> When I run that code with clients_count <= 65 everything works fine: >>>>>>>>>>> all the processes create a common grid, exchange some information >>>>>>>>>>> and >>>>>>>>>>> disconnect. >>>>>>>>>>> When I set clients_count > 65 the 66th process crashes on >>>>>>>>>>> MPI_Comm_connect (segmentation fault). >>>>>>>>>> >>>>>>>>>> I didn't have time to check the code, but my guess is that you are >>>>>>>>>> still hitting some kind of file descriptor or other limit. Check to >>>>>>>>>> see what your limits are - usually "ulimit" will tell you. >>>>>>>>> >>>>>>>>> My limitations are: >>>>>>>>> time(seconds) unlimited >>>>>>>>> file(blocks) unlimited >>>>>>>>> data(kb) unlimited >>>>>>>>> stack(kb) 10240 >>>>>>>>> coredump(blocks) 0 >>>>>>>>> memory(kb) unlimited >>>>>>>>> locked memory(kb) 64 >>>>>>>>> process 200704 >>>>>>>>> nofiles 1024 >>>>>>>>> vmemory(kb) unlimited >>>>>>>>> locks unlimited >>>>>>>>> >>>>>>>>> Which one do you think could be responsible for that? >>>>>>>>> >>>>>>>>> I was trying to run all the 66 processes on one machine or spread them >>>>>>>>> across several machines and it always crashes the same way on the 66th >>>>>>>>> process. >>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Another thing I would like to know is if it's normal that any of my >>>>>>>>>>> processes when calling MPI_Comm_connect or MPI_Comm_accept when the >>>>>>>>>>> other side is not ready, is eating up a full CPU available. >>>>>>>>>> >>>>>>>>>> Yes - the waiting process is polling in a tight loop waiting for the >>>>>>>>>> connection to be made. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Any help would be appreciated, >>>>>>>>>>> Grzegorz Maj >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2010/4/24 Ralph Castain <r...@open-mpi.org>: >>>>>>>>>>>> Actually, OMPI is distributed with a daemon that does pretty much >>>>>>>>>>>> what you >>>>>>>>>>>> want. Checkout "man ompi-server". I originally wrote that code to >>>>>>>>>>>> support >>>>>>>>>>>> cross-application MPI publish/subscribe operations, but we can >>>>>>>>>>>> utilize it >>>>>>>>>>>> here too. Have to blame me for not making it more publicly known. >>>>>>>>>>>> The attached patch upgrades ompi-server and modifies the singleton >>>>>>>>>>>> startup >>>>>>>>>>>> to provide your desired support. This solution works in the >>>>>>>>>>>> following >>>>>>>>>>>> manner: >>>>>>>>>>>> 1. launch "ompi-server -report-uri <filename>". This starts a >>>>>>>>>>>> persistent >>>>>>>>>>>> daemon called "ompi-server" that acts as a rendezvous point for >>>>>>>>>>>> independently started applications. The problem with starting >>>>>>>>>>>> different >>>>>>>>>>>> applications and wanting them to MPI connect/accept lies in the >>>>>>>>>>>> need to have >>>>>>>>>>>> the applications find each other. If they can't discover contact >>>>>>>>>>>> info for >>>>>>>>>>>> the other app, then they can't wire up their interconnects. The >>>>>>>>>>>> "ompi-server" tool provides that rendezvous point. I don't like >>>>>>>>>>>> that >>>>>>>>>>>> comm_accept segfaulted - should have just error'd out. >>>>>>>>>>>> 2. set OMPI_MCA_orte_server=file:<filename>" in the environment >>>>>>>>>>>> where you >>>>>>>>>>>> will start your processes. This will allow your singleton >>>>>>>>>>>> processes to find >>>>>>>>>>>> the ompi-server. I automatically also set the envar to connect the >>>>>>>>>>>> MPI >>>>>>>>>>>> publish/subscribe system for you. >>>>>>>>>>>> 3. run your processes. As they think they are singletons, they >>>>>>>>>>>> will detect >>>>>>>>>>>> the presence of the above envar and automatically connect >>>>>>>>>>>> themselves to the >>>>>>>>>>>> "ompi-server" daemon. This provides each process with the ability >>>>>>>>>>>> to perform >>>>>>>>>>>> any MPI-2 operation. >>>>>>>>>>>> I tested this on my machines and it worked, so hopefully it will >>>>>>>>>>>> meet your >>>>>>>>>>>> needs. You only need to run one "ompi-server" period, so long as >>>>>>>>>>>> you locate >>>>>>>>>>>> it where all of the processes can find the contact file and can >>>>>>>>>>>> open a TCP >>>>>>>>>>>> socket to the daemon. There is a way to knit multiple ompi-servers >>>>>>>>>>>> into a >>>>>>>>>>>> broader network (e.g., to connect processes that cannot directly >>>>>>>>>>>> access a >>>>>>>>>>>> server due to network segmentation), but it's a tad tricky - let >>>>>>>>>>>> me know if >>>>>>>>>>>> you require it and I'll try to help. >>>>>>>>>>>> If you have trouble wiring them all into a single communicator, >>>>>>>>>>>> you might >>>>>>>>>>>> ask separately about that and see if one of our MPI experts can >>>>>>>>>>>> provide >>>>>>>>>>>> advice (I'm just the RTE grunt). >>>>>>>>>>>> HTH - let me know how this works for you and I'll incorporate it >>>>>>>>>>>> into future >>>>>>>>>>>> OMPI releases. >>>>>>>>>>>> Ralph >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Apr 24, 2010, at 1:49 AM, Krzysztof Zarzycki wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi Ralph, >>>>>>>>>>>> I'm Krzysztof and I'm working with Grzegorz Maj on this our small >>>>>>>>>>>> project/experiment. >>>>>>>>>>>> We definitely would like to give your patch a try. But could you >>>>>>>>>>>> please >>>>>>>>>>>> explain your solution a little more? >>>>>>>>>>>> You still would like to start one mpirun per mpi grid, and then >>>>>>>>>>>> have >>>>>>>>>>>> processes started by us to join the MPI comm? >>>>>>>>>>>> It is a good solution of course. >>>>>>>>>>>> But it would be especially preferable to have one daemon running >>>>>>>>>>>> persistently on our "entry" machine that can handle several mpi >>>>>>>>>>>> grid starts. >>>>>>>>>>>> Can your patch help us this way too? >>>>>>>>>>>> Thanks for your help! >>>>>>>>>>>> Krzysztof >>>>>>>>>>>> >>>>>>>>>>>> On 24 April 2010 03:51, Ralph Castain <r...@open-mpi.org> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> In thinking about this, my proposed solution won't entirely fix >>>>>>>>>>>>> the >>>>>>>>>>>>> problem - you'll still wind up with all those daemons. I believe >>>>>>>>>>>>> I can >>>>>>>>>>>>> resolve that one as well, but it would require a patch. >>>>>>>>>>>>> >>>>>>>>>>>>> Would you like me to send you something you could try? Might take >>>>>>>>>>>>> a couple >>>>>>>>>>>>> of iterations to get it right... >>>>>>>>>>>>> >>>>>>>>>>>>> On Apr 23, 2010, at 12:12 PM, Ralph Castain wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hmmm....I -think- this will work, but I cannot guarantee it: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. launch one process (can just be a spinner) using mpirun that >>>>>>>>>>>>>> includes >>>>>>>>>>>>>> the following option: >>>>>>>>>>>>>> >>>>>>>>>>>>>> mpirun -report-uri file >>>>>>>>>>>>>> >>>>>>>>>>>>>> where file is some filename that mpirun can create and insert its >>>>>>>>>>>>>> contact info into it. This can be a relative or absolute path. >>>>>>>>>>>>>> This process >>>>>>>>>>>>>> must remain alive throughout your application - doesn't matter >>>>>>>>>>>>>> what it does. >>>>>>>>>>>>>> It's purpose is solely to keep mpirun alive. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2. set OMPI_MCA_dpm_orte_server=FILE:file in your environment, >>>>>>>>>>>>>> where >>>>>>>>>>>>>> "file" is the filename given above. This will tell your >>>>>>>>>>>>>> processes how to >>>>>>>>>>>>>> find mpirun, which is acting as a meeting place to handle the >>>>>>>>>>>>>> connect/accept >>>>>>>>>>>>>> operations >>>>>>>>>>>>>> >>>>>>>>>>>>>> Now run your processes, and have them connect/accept to each >>>>>>>>>>>>>> other. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The reason I cannot guarantee this will work is that these >>>>>>>>>>>>>> processes >>>>>>>>>>>>>> will all have the same rank && name since they all start as >>>>>>>>>>>>>> singletons. >>>>>>>>>>>>>> Hence, connect/accept is likely to fail. >>>>>>>>>>>>>> >>>>>>>>>>>>>> But it -might- work, so you might want to give it a try. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Apr 23, 2010, at 8:10 AM, Grzegorz Maj wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> To be more precise: by 'server process' I mean some process >>>>>>>>>>>>>>> that I >>>>>>>>>>>>>>> could run once on my system and it could help in creating those >>>>>>>>>>>>>>> groups. >>>>>>>>>>>>>>> My typical scenario is: >>>>>>>>>>>>>>> 1. run N separate processes, each without mpirun >>>>>>>>>>>>>>> 2. connect them into MPI group >>>>>>>>>>>>>>> 3. do some job >>>>>>>>>>>>>>> 4. exit all N processes >>>>>>>>>>>>>>> 5. goto 1 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2010/4/23 Grzegorz Maj <ma...@wp.pl>: >>>>>>>>>>>>>>>> Thank you Ralph for your explanation. >>>>>>>>>>>>>>>> And, apart from that descriptors' issue, is there any other >>>>>>>>>>>>>>>> way to >>>>>>>>>>>>>>>> solve my problem, i.e. to run separately a number of processes, >>>>>>>>>>>>>>>> without mpirun and then to collect them into an MPI intracomm >>>>>>>>>>>>>>>> group? >>>>>>>>>>>>>>>> If I for example would need to run some 'server process' (even >>>>>>>>>>>>>>>> using >>>>>>>>>>>>>>>> mpirun) for this task, that's OK. Any ideas? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Grzegorz Maj >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2010/4/18 Ralph Castain <r...@open-mpi.org>: >>>>>>>>>>>>>>>>> Okay, but here is the problem. If you don't use mpirun, and >>>>>>>>>>>>>>>>> are not >>>>>>>>>>>>>>>>> operating in an environment we support for "direct" launch >>>>>>>>>>>>>>>>> (i.e., starting >>>>>>>>>>>>>>>>> processes outside of mpirun), then every one of those >>>>>>>>>>>>>>>>> processes thinks it is >>>>>>>>>>>>>>>>> a singleton - yes? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> What you may not realize is that each singleton immediately >>>>>>>>>>>>>>>>> fork/exec's an orted daemon that is configured to behave just >>>>>>>>>>>>>>>>> like mpirun. >>>>>>>>>>>>>>>>> This is required in order to support MPI-2 operations such as >>>>>>>>>>>>>>>>> MPI_Comm_spawn, MPI_Comm_connect/accept, etc. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> So if you launch 64 processes that think they are singletons, >>>>>>>>>>>>>>>>> then >>>>>>>>>>>>>>>>> you have 64 copies of orted running as well. This eats up a >>>>>>>>>>>>>>>>> lot of file >>>>>>>>>>>>>>>>> descriptors, which is probably why you are hitting this 65 >>>>>>>>>>>>>>>>> process limit - >>>>>>>>>>>>>>>>> your system is probably running out of file descriptors. You >>>>>>>>>>>>>>>>> might check you >>>>>>>>>>>>>>>>> system limits and see if you can get them revised upward. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Apr 17, 2010, at 4:24 PM, Grzegorz Maj wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Yes, I know. The problem is that I need to use some special >>>>>>>>>>>>>>>>>> way for >>>>>>>>>>>>>>>>>> running my processes provided by the environment in which I'm >>>>>>>>>>>>>>>>>> working >>>>>>>>>>>>>>>>>> and unfortunately I can't use mpirun. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 2010/4/18 Ralph Castain <r...@open-mpi.org>: >>>>>>>>>>>>>>>>>>> Guess I don't understand why you can't use mpirun - all it >>>>>>>>>>>>>>>>>>> does is >>>>>>>>>>>>>>>>>>> start things, provide a means to forward io, etc. It mainly >>>>>>>>>>>>>>>>>>> sits there >>>>>>>>>>>>>>>>>>> quietly without using any cpu unless required to support >>>>>>>>>>>>>>>>>>> the job. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Sounds like it would solve your problem. Otherwise, I know >>>>>>>>>>>>>>>>>>> of no >>>>>>>>>>>>>>>>>>> way to get all these processes into comm_world. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Apr 17, 2010, at 2:27 PM, Grzegorz Maj wrote: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>> I'd like to dynamically create a group of processes >>>>>>>>>>>>>>>>>>>> communicating >>>>>>>>>>>>>>>>>>>> via >>>>>>>>>>>>>>>>>>>> MPI. Those processes need to be run without mpirun and >>>>>>>>>>>>>>>>>>>> create >>>>>>>>>>>>>>>>>>>> intracommunicator after the startup. Any ideas how to do >>>>>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>>>>> efficiently? >>>>>>>>>>>>>>>>>>>> I came up with a solution in which the processes are >>>>>>>>>>>>>>>>>>>> connecting >>>>>>>>>>>>>>>>>>>> one by >>>>>>>>>>>>>>>>>>>> one using MPI_Comm_connect, but unfortunately all the >>>>>>>>>>>>>>>>>>>> processes >>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>> are already in the group need to call MPI_Comm_accept. >>>>>>>>>>>>>>>>>>>> This means >>>>>>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>>> when the n-th process wants to connect I need to collect >>>>>>>>>>>>>>>>>>>> all the >>>>>>>>>>>>>>>>>>>> n-1 >>>>>>>>>>>>>>>>>>>> processes on the MPI_Comm_accept call. After I run about 40 >>>>>>>>>>>>>>>>>>>> processes >>>>>>>>>>>>>>>>>>>> every subsequent call takes more and more time, which I'd >>>>>>>>>>>>>>>>>>>> like to >>>>>>>>>>>>>>>>>>>> avoid. >>>>>>>>>>>>>>>>>>>> Another problem in this solution is that when I try to >>>>>>>>>>>>>>>>>>>> connect >>>>>>>>>>>>>>>>>>>> 66-th >>>>>>>>>>>>>>>>>>>> process the root of the existing group segfaults on >>>>>>>>>>>>>>>>>>>> MPI_Comm_accept. >>>>>>>>>>>>>>>>>>>> Maybe it's my bug, but it's weird as everything works fine >>>>>>>>>>>>>>>>>>>> for at >>>>>>>>>>>>>>>>>>>> most >>>>>>>>>>>>>>>>>>>> 65 processes. Is there any limitation I don't know about? >>>>>>>>>>>>>>>>>>>> My last question is about MPI_COMM_WORLD. When I run my >>>>>>>>>>>>>>>>>>>> processes >>>>>>>>>>>>>>>>>>>> without mpirun their MPI_COMM_WORLD is the same as >>>>>>>>>>>>>>>>>>>> MPI_COMM_SELF. >>>>>>>>>>>>>>>>>>>> Is >>>>>>>>>>>>>>>>>>>> there any way to change MPI_COMM_WORLD and set it to the >>>>>>>>>>>>>>>>>>>> intracommunicator that I've created? >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>> Grzegorz Maj >>>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> users mailing list >>>>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> users mailing list >>>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> users mailing list >>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> users mailing list >>>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>>>> >>>>>>>>>>> <client.c><server.c>_______________________________________________ >>>>>>>>>>> users mailing list >>>>>>>>>>> us...@open-mpi.org >>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> users mailing list >>>>>>>>>> us...@open-mpi.org >>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> users mailing list >>>>>>>>> us...@open-mpi.org >>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> users mailing list >>>>>>>> us...@open-mpi.org >>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> users mailing list >>>>>>> us...@open-mpi.org >>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> users mailing list >>>>>> us...@open-mpi.org >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> users mailing list >>>>> us...@open-mpi.org >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>>> >>>> _______________________________________________ >>>> users mailing list >>>> us...@open-mpi.org >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users >>>> >>> >>> _______________________________________________ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
signature.asc
Description: OpenPGP digital signature
