Okay, fixed in r23499. Thanks again...
On Jul 26, 2010, at 9:47 PM, Ralph Castain wrote:
> Doh - yes it should! I'll fix it right now.
>
> Thanks!
>
> On Jul 26, 2010, at 9:28 PM, Philippe wrote:
>
>> Ralph,
>>
>> i was able to test the generic module and it seems to be working.
>>
>> one question tho, the function orte_ess_generic_component_query in
>> "orte/mca/ess/generic/ess_generic_component.c" calls getenv with the
>> argument "OMPI_MCA_enc", which seems to cause the module to fail to
>> load. shouldnt it be "OMPI_MCA_ess" ?
>>
>> .....
>>
>> /* only pick us if directed to do so */
>> if (NULL != (pick = getenv("OMPI_MCA_env")) &&
>> 0 == strcmp(pick, "generic")) {
>> *priority = 1000;
>> *module = (mca_base_module_t *)&orte_ess_generic_module;
>>
>> ...
>>
>> p.
>>
>> On Thu, Jul 22, 2010 at 5:53 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>> Dev trunk looks okay right now - I think you'll be fine using it. My new
>>> component -might- work with 1.5, but probably not with 1.4. I haven't
>>> checked either of them.
>>>
>>> Anything at r23478 or above will have the new module. Let me know how it
>>> works for you. I haven't tested it myself, but am pretty sure it should
>>> work.
>>>
>>>
>>> On Jul 22, 2010, at 3:22 PM, Philippe wrote:
>>>
>>>> Ralph,
>>>>
>>>> Thank you so much!!
>>>>
>>>> I'll give it a try and let you know.
>>>>
>>>> I know it's a tough question, but how stable is the dev trunk? Can I
>>>> just grab the latest and run, or am I better off taking your changes
>>>> and copy them back in a stable release? (if so, which one? 1.4? 1.5?)
>>>>
>>>> p.
>>>>
>>>> On Thu, Jul 22, 2010 at 3:50 PM, Ralph Castain <r...@open-mpi.org> wrote:
>>>>> It was easier for me to just construct this module than to explain how to
>>>>> do so :-)
>>>>>
>>>>> I will commit it this evening (couple of hours from now) as that is our
>>>>> standard practice. You'll need to use the developer's trunk, though, to
>>>>> use it.
>>>>>
>>>>> Here are the envars you'll need to provide:
>>>>>
>>>>> Each process needs to get the same following values:
>>>>>
>>>>> * OMPI_MCA_ess=generic
>>>>> * OMPI_MCA_orte_num_procs=<number of MPI procs>
>>>>> * OMPI_MCA_orte_nodes=<a comma-separated list of nodenames where MPI
>>>>> procs reside>
>>>>> * OMPI_MCA_orte_ppn=<number of procs/node>
>>>>>
>>>>> Note that I have assumed this last value is a constant for simplicity. If
>>>>> that isn't the case, let me know - you could instead provide it as a
>>>>> comma-separated list of values with an entry for each node.
>>>>>
>>>>> In addition, you need to provide the following value that will be unique
>>>>> to each process:
>>>>>
>>>>> * OMPI_MCA_orte_rank=<MPI rank>
>>>>>
>>>>> Finally, you have to provide a range of static TCP ports for use by the
>>>>> processes. Pick any range that you know will be available across all the
>>>>> nodes. You then need to ensure that each process sees the following envar:
>>>>>
>>>>> * OMPI_MCA_oob_tcp_static_ports=6000-6010 <== obviously, replace this
>>>>> with your range
>>>>>
>>>>> You will need a port range that is at least equal to the ppn for the job
>>>>> (each proc on a node will take one of the provided ports).
>>>>>
>>>>> That should do it. I compute everything else I need from those values.
>>>>>
>>>>> Does that work for you?
>>>>> Ralph
>>>>>
>>>>>
>>>>> On Jul 22, 2010, at 6:48 AM, Philippe wrote:
>>>>>
>>>>>> On Wed, Jul 21, 2010 at 10:44 AM, Ralph Castain <r...@open-mpi.org>
>>>>>> wrote:
>>>>>>>
>>>>>>> On Jul 21, 2010, at 7:44 AM, Philippe wrote:
>>>>>>>
>>>>>>>> Ralph,
>>>>>>>>
>>>>>>>> Sorry for the late reply -- I was away on vacation.
>>>>>>>
>>>>>>> no problem at all!
>>>>>>>
>>>>>>>>
>>>>>>>> regarding your earlier question about how many processes where
>>>>>>>> involved when the memory was entirely allocated, it was only two, a
>>>>>>>> sender and a receiver. I'm still trying to pinpoint what can be
>>>>>>>> different between the standalone case and the "integrated" case. I
>>>>>>>> will try to find out what part of the code is allocating memory in a
>>>>>>>> loop.
>>>>>>>
>>>>>>> hmmm....that sounds like a bug in your program. let me know what you
>>>>>>> find
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Jul 20, 2010 at 12:51 AM, Ralph Castain <r...@open-mpi.org>
>>>>>>>> wrote:
>>>>>>>>> Well, I finally managed to make this work without the required
>>>>>>>>> ompi-server rendezvous point. The fix is only in the devel trunk
>>>>>>>>> right now - I'll have to ask the release managers for 1.5 and 1.4 if
>>>>>>>>> they want it ported to those series.
>>>>>>>>>
>>>>>>>>
>>>>>>>> great -- i'll give it a try
>>>>>>>>
>>>>>>>>> On the notion of integrating OMPI to your launch environment:
>>>>>>>>> remember that we don't necessarily require that you use mpiexec for
>>>>>>>>> that purpose. If your launch environment provides just a little info
>>>>>>>>> in the environment of the launched procs, we can usually devise a
>>>>>>>>> method that allows the procs to perform an MPI_Init as a single job
>>>>>>>>> without all this work you are doing.
>>>>>>>>>
>>>>>>>>
>>>>>>>> I'm working on creating operators using MPI for the IBM product
>>>>>>>> "InfoSphere Streams". It has its own launching mechanism to start the
>>>>>>>> processes. However I can pass some information to the processes that
>>>>>>>> belong to the same job (Streams job -- which should neatly map to MPI
>>>>>>>> job).
>>>>>>>>
>>>>>>>>> Only difference is that your procs will all block in MPI_Init until
>>>>>>>>> they -all- have executed that function. If that isn't a problem, this
>>>>>>>>> would be a much more scalable and reliable method than doing it thru
>>>>>>>>> massive calls to MPI_Port_connect.
>>>>>>>>>
>>>>>>>>
>>>>>>>> in the general case, that would be a problem, but for my prototype,
>>>>>>>> this is acceptable.
>>>>>>>>
>>>>>>>> In general, each process is composed of operators, some may be MPI
>>>>>>>> related and some may not. But in my case, I know ahead of time which
>>>>>>>> processes will be part of the MPI job, so I can easily deal with the
>>>>>>>> fact that they would block on MPI_init (actually -- MPI_thread_init
>>>>>>>> since its using a lot of threads).
>>>>>>>
>>>>>>> We have talked in the past about creating a non-blocking MPI_Init as an
>>>>>>> extension to the standard. It would lock you to Open MPI, though...
>>>>>>>
>>>>>>> Regardless, at some point you would have to know how many processes are
>>>>>>> going to be part of the job so you can know when MPI_Init is complete.
>>>>>>> I would think you would require that info for the singleton wireup
>>>>>>> anyway - yes? Otherwise, how would you know when to quit running
>>>>>>> connect-accept?
>>>>>>>
>>>>>>
>>>>>> the short answer is yes... although, the longer answer is a bit more
>>>>>> complicated. currently I do know the number of connect I need to do on
>>>>>> a per-port basis. a job can contains an arbitrary number of MPI
>>>>>> processes, each opening one or more ports. so i know the count port by
>>>>>> ports but I dont need to worry about how many MPI processes there is
>>>>>> globally. to make things a bit more complicated, each MPI operator can
>>>>>> be "fused" with other operators to make a process. each fused operator
>>>>>> may or may not require MPI. the bottom line is, to get the total
>>>>>> number of processes to calculate rank&size, I need to reverse engineer
>>>>>> the fusing that the compiler may do.
>>>>>>
>>>>>> but that's ok, I'm willing to do that for our prototype :-)
>>>>>>
>>>>>>>>
>>>>>>>> Is there a documentation or example I can use to see what information
>>>>>>>> I can pass to the processes to enable that? Is it just environment
>>>>>>>> variables?
>>>>>>>
>>>>>>> No real documentation - a lack I should probably fill. At the moment,
>>>>>>> we don't have a "generic" module for standalone launch, but I can
>>>>>>> create one as it is pretty trivial. I would then need you to pass each
>>>>>>> process envars telling it the total number of processes in the MPI job,
>>>>>>> its rank within that job, and a file where some rendezvous process (can
>>>>>>> be rank=0) has provided that port string. Armed with that info, I can
>>>>>>> wireup the job.
>>>>>>>
>>>>>>> Won't be as scalable as an mpirun-initiated startup, but will be much
>>>>>>> better than doing it from singletons.
>>>>>>
>>>>>> that would be great. I can definitely pass environment variables to
>>>>>> each process.
>>>>>>
>>>>>>>
>>>>>>> Or if you prefer, we could setup an "infosphere" module that we could
>>>>>>> customize for this system. Main thing here would be to provide us with
>>>>>>> some kind of regex (or access to a file containing the info) that
>>>>>>> describes the map of rank to node so we can construct the wireup
>>>>>>> communication pattern.
>>>>>>>
>>>>>>
>>>>>> i think for our prototype we are fine with the first method. I'd leave
>>>>>> the cleaner implementation as a task for the product team ;-)
>>>>>>
>>>>>> regarding the "generic" module, is that something you can put together
>>>>>> quickly? can I help in any way?
>>>>>>
>>>>>> Thanks!
>>>>>> p
>>>>>>
>>>>>>> Either way would work. The second is more scalable, but I don't know if
>>>>>>> you have (or can construct) the map info.
>>>>>>>
>>>>>>>>
>>>>>>>> Many thanks!
>>>>>>>> p.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Jul 18, 2010, at 4:09 PM, Philippe wrote:
>>>>>>>>>
>>>>>>>>>> Ralph,
>>>>>>>>>>
>>>>>>>>>> thanks for investigating.
>>>>>>>>>>
>>>>>>>>>> I've applied the two patches you mentioned earlier and ran with the
>>>>>>>>>> ompi server. Although i was able to runn our standalone test, when I
>>>>>>>>>> integrated the changes to our code, the processes entered a crazy
>>>>>>>>>> loop
>>>>>>>>>> and allocated all the memory available when calling MPI_Port_Connect.
>>>>>>>>>> I was not able to identify why it works standalone but not integrated
>>>>>>>>>> with our code. If I found why, I'll let your know.
>>>>>>>>>>
>>>>>>>>>> looking forward to your findings. We'll be happy to test any patches
>>>>>>>>>> if you have some!
>>>>>>>>>>
>>>>>>>>>> p.
>>>>>>>>>>
>>>>>>>>>> On Sat, Jul 17, 2010 at 9:47 PM, Ralph Castain <r...@open-mpi.org>
>>>>>>>>>> wrote:
>>>>>>>>>>> Okay, I can reproduce this problem. Frankly, I don't think this
>>>>>>>>>>> ever worked with OMPI, and I'm not sure how the choice of BTL makes
>>>>>>>>>>> a difference.
>>>>>>>>>>>
>>>>>>>>>>> The program is crashing in the communicator definition, which
>>>>>>>>>>> involves a communication over our internal out-of-band messaging
>>>>>>>>>>> system. That system has zero connection to any BTL, so it should
>>>>>>>>>>> crash either way.
>>>>>>>>>>>
>>>>>>>>>>> Regardless, I will play with this a little as time allows. Thanks
>>>>>>>>>>> for the reproducer!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Jun 25, 2010, at 7:23 AM, Philippe wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I'm trying to run a test program which consists of a server
>>>>>>>>>>>> creating a
>>>>>>>>>>>> port using MPI_Open_port and N clients using MPI_Comm_connect to
>>>>>>>>>>>> connect to the server.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm able to do so with 1 server and 2 clients, but with 1 server +
>>>>>>>>>>>> 3
>>>>>>>>>>>> clients, I get the following error message:
>>>>>>>>>>>>
>>>>>>>>>>>> [node003:32274] [[37084,0],0]:route_callback tried routing message
>>>>>>>>>>>> from [[37084,1],0] to [[40912,1],0]:102, can't find route
>>>>>>>>>>>>
>>>>>>>>>>>> This is only happening with the openib BTL. With tcp BTL it works
>>>>>>>>>>>> perfectly fine (ofud also works as a matter of fact...). This has
>>>>>>>>>>>> been
>>>>>>>>>>>> tested on two completely different clusters, with identical
>>>>>>>>>>>> results.
>>>>>>>>>>>> In either cases, the IB frabic works normally.
>>>>>>>>>>>>
>>>>>>>>>>>> Any help would be greatly appreciated! Several people in my team
>>>>>>>>>>>> looked at the problem. Google and the mailing list archive did not
>>>>>>>>>>>> provide any clue. I believe that from an MPI standpoint, my test
>>>>>>>>>>>> program is valid (and it works with TCP, which make me feel better
>>>>>>>>>>>> about the sequence of MPI calls)
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Philippe.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Background:
>>>>>>>>>>>>
>>>>>>>>>>>> I intend to use openMPI to transport data inside a much larger
>>>>>>>>>>>> application. Because of that, I cannot used mpiexec. Each process
>>>>>>>>>>>> is
>>>>>>>>>>>> started by our own "job management" and use a name server to find
>>>>>>>>>>>> about each others. Once all the clients are connected, I would like
>>>>>>>>>>>> the server to do MPI_Recv to get the data from all the client. I
>>>>>>>>>>>> dont
>>>>>>>>>>>> care about the order or which client are sending data, as long as I
>>>>>>>>>>>> can receive it with on call. Do do that, the clients and the server
>>>>>>>>>>>> are going through a series of
>>>>>>>>>>>> Comm_accept/Conn_connect/Intercomm_merge
>>>>>>>>>>>> so that at the end, all the clients and the server are inside the
>>>>>>>>>>>> same
>>>>>>>>>>>> intracomm.
>>>>>>>>>>>>
>>>>>>>>>>>> Steps:
>>>>>>>>>>>>
>>>>>>>>>>>> I have a sample program that show the issue. I tried to make it as
>>>>>>>>>>>> short as possible. It needs to be executed on a shared file system
>>>>>>>>>>>> like NFS because the server write the port info to a file that the
>>>>>>>>>>>> client will read. To reproduce the issue, the following steps
>>>>>>>>>>>> should
>>>>>>>>>>>> be performed:
>>>>>>>>>>>>
>>>>>>>>>>>> 0. compile the test with "mpicc -o ben12 ben12.c"
>>>>>>>>>>>> 1. ssh to the machine that will be the server
>>>>>>>>>>>> 2. run ./ben12 3 1
>>>>>>>>>>>> 3. ssh to the machine that will be the client #1
>>>>>>>>>>>> 4. run ./ben12 3 0
>>>>>>>>>>>> 5. repeat step 3-4 for client #2 and #3
>>>>>>>>>>>>
>>>>>>>>>>>> the server accept the connection from client #1 and merge it in a
>>>>>>>>>>>> new
>>>>>>>>>>>> intracomm. It then accept connection from client #2 and merge it.
>>>>>>>>>>>> when
>>>>>>>>>>>> the client #3 arrives, the server accept the connection, but that
>>>>>>>>>>>> cause client #1 and #2 to die with the error above (see the
>>>>>>>>>>>> complete
>>>>>>>>>>>> trace in the tarball).
>>>>>>>>>>>>
>>>>>>>>>>>> The exact steps are:
>>>>>>>>>>>>
>>>>>>>>>>>> - server open port
>>>>>>>>>>>> - server does accept
>>>>>>>>>>>> - client #1 does connect
>>>>>>>>>>>> - server and client #1 do merge
>>>>>>>>>>>> - server does accept
>>>>>>>>>>>> - client #2 does connect
>>>>>>>>>>>> - server, client #1 and client #2 do merge
>>>>>>>>>>>> - server does accept
>>>>>>>>>>>> - client #3 does connect
>>>>>>>>>>>> - server, client #1, client #2 and client #3 do merge
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> My infiniband network works normally with other test programs or
>>>>>>>>>>>> applications (MPI or others like Verbs).
>>>>>>>>>>>>
>>>>>>>>>>>> Info about my setup:
>>>>>>>>>>>>
>>>>>>>>>>>> openMPI version = 1.4.1 (I also tried 1.4.2, nightly snapshot of
>>>>>>>>>>>> 1.4.3, nightly snapshot of 1.5 --- all show the same error)
>>>>>>>>>>>> config.log in the tarball
>>>>>>>>>>>> "ompi_info --all" in the tarball
>>>>>>>>>>>> OFED version = 1.3 installed from RHEL 5.3
>>>>>>>>>>>> Distro = RedHat Entreprise Linux 5.3
>>>>>>>>>>>> Kernel = 2.6.18-128.4.1.el5 x86_64
>>>>>>>>>>>> subnet manager = built-in SM from the cisco/topspin switch
>>>>>>>>>>>> output of ibv_devinfo included in the tarball (there are no
>>>>>>>>>>>> "bad" nodes)
>>>>>>>>>>>> "ulimit -l" says "unlimited"
>>>>>>>>>>>>
>>>>>>>>>>>> The tarball contains:
>>>>>>>>>>>>
>>>>>>>>>>>> - ben12.c: my test program showing the behavior
>>>>>>>>>>>> - config.log / config.out / make.out / make-install.out /
>>>>>>>>>>>> ifconfig.txt / ibv-devinfo.txt / ompi_info.txt
>>>>>>>>>>>> - trace-tcp.txt: output of the server and each client when it
>>>>>>>>>>>> works
>>>>>>>>>>>> with TCP (I added "btl = tcp,self" in ~/.openmpi/mca-params.conf)
>>>>>>>>>>>> - trace-ib.txt: output of the server and each client when it fails
>>>>>>>>>>>> with IB (I added "btl = openib,self" in ~/.openmpi/mca-params.conf)
>>>>>>>>>>>>
>>>>>>>>>>>> I hope I provided enough info for somebody to reproduce the
>>>>>>>>>>>> problem...
>>>>>>>>>>>> <ompi-output.tar.bz2>_______________________________________________
>>>>>>>>>>>> users mailing list
>>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> users mailing list
>>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> users mailing list
>>>>>>>>>> us...@open-mpi.org
>>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> users mailing list
>>>>>>>>> us...@open-mpi.org
>>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> users mailing list
>>>>>>>> us...@open-mpi.org
>>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> users mailing list
>>>>>>> us...@open-mpi.org
>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> users mailing list
>>>>>> us...@open-mpi.org
>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> users mailing list
>>>>> us...@open-mpi.org
>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>>>
>>>>
>>>> _______________________________________________
>>>> users mailing list
>>>> us...@open-mpi.org
>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>>
>>> _______________________________________________
>>> users mailing list
>>> us...@open-mpi.org
>>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>>>
>>
>> _______________________________________________
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
>
