[OMPI users] CentOS 6.3 & OpenMPI 1.6.3
Howdy. I a have a stock CentOS 6.3 OS and a Mellanox MT26428 card. I installed the Mellanox OFED MLNX_OFED_LINUX-1.5.3-3.1.0-rhel6.3-x86_64 which installed just fine. Rebooted the system and when I try building OpenMPI 1.6.3, it aborts with: mtl_mxm_send.c: In function 'ompi_mtl_mxm_isend': mtl_mxm_send.c:183: error: 'MXM_REQ_FLAG_SEND_SYNC' undeclared (first use in this function) make[2]: *** [mtl_mxm_send.lo] Error 1 make[2]: *** Waiting for unfinished jobs make[2]: Leaving directory `/data/apps/sources/openmpi-1.6.3/ompi/mca/mtl/mxm' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/data/apps/sources/openmpi-1.6.3/ompi' make: *** [all-recursive] Error 1 My configure options are: --with-sge \ --with-threads \ --with-hwloc\ --with-openib \ --enable-mpi-thread-multiple\ --with-mxm=/opt/mellanox/mxm\ --with-mxm-libdir=/opt/mellanox/mxm/lib \ --with-fca=/opt/mellanox/fca\ --enable-heterogeneous \ --enable-openib-connectx-xrc\ Has anyone been able to compile OpenMPI 1.6.3 with the Mellanox OFED on CentOS 6.3? Joseph
Re: [OMPI users] Maximum number of MPI processes on a node + discovering faulty nodes
Dear Ralph, For the file descriptors the declared limit is over 65536 files but if OMPI needs several of them, then this can be interesting. Is there any source to read about it or I just should do trials? About the child processes again, can I do something? I have root access, so I can change the values. Best regards, George Markomanolis On 11/27/2012 05:58 PM, Ralph Castain wrote: Just glancing at the code, I don't see anything tied to 2**12 that pops out at me. I suspect the issue is that you are hitting a system limit on the number of child processes a process can spawn - this is different from the total number of processes allowed on the node - or the number of file descriptors a process can have open (we need several per process for I/O forwarding). On Nov 27, 2012, at 8:24 AM, George Markomanolis wrote: Dear Ralph, Thanks for the answer, I am using OMPI v1.4.1. Best regards, George Markomanolis On 11/26/2012 05:07 PM, Ralph Castain wrote: What version of OMPI are you using? On Nov 26, 2012, at 1:02 AM, George Markomanolis wrote: Dear all, Initially I would like an advice of how to identify the maximum number of MPI processes that can be executed on a node with oversubscribing. When I try to execute an application with 4096 MPI processes on a 24-cores node with 48GB of memory, I have an error "Unknown error: 1" while the memory is not even at the half. I can execute the same application with 2048 MPI processes in less than one minute. I have checked linux settings about maximum number of processes and it is much bigger than 4096. Another more generic question, is about discovering nodes with faulty memory. Is there any way to identify nodes with faulty memory? I found accidentally that a node with exact the same hardware couldn't execute an MPI application when it was using more than 12GB of ram while the second one could use all of the 48GB of memory. If I have 500+ nodes is difficult to check all of them and I am not familiar with any efficient solution. Initially I thought about memtester but it takes a lot of time. I know that this does not apply exactly on this mailing list but I thought that maybe an OpenMPI user knows something about. Best regards, George Markomanolis ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] Maximum number of MPI processes on a node + discovering faulty nodes
Just glancing at the code, I don't see anything tied to 2**12 that pops out at me. I suspect the issue is that you are hitting a system limit on the number of child processes a process can spawn - this is different from the total number of processes allowed on the node - or the number of file descriptors a process can have open (we need several per process for I/O forwarding). On Nov 27, 2012, at 8:24 AM, George Markomanolis wrote: > Dear Ralph, > > Thanks for the answer, I am using OMPI v1.4.1. > > Best regards, > George Markomanolis > > On 11/26/2012 05:07 PM, Ralph Castain wrote: >> What version of OMPI are you using? >> >> On Nov 26, 2012, at 1:02 AM, George Markomanolis >> wrote: >> >>> Dear all, >>> >>> Initially I would like an advice of how to identify the maximum number of >>> MPI processes that can be executed on a node with oversubscribing. When I >>> try to execute an application with 4096 MPI processes on a 24-cores node >>> with 48GB of memory, I have an error "Unknown error: 1" while the memory is >>> not even at the half. I can execute the same application with 2048 MPI >>> processes in less than one minute. I have checked linux settings about >>> maximum number of processes and it is much bigger than 4096. >>> >>> Another more generic question, is about discovering nodes with faulty >>> memory. Is there any way to identify nodes with faulty memory? I found >>> accidentally that a node with exact the same hardware couldn't execute an >>> MPI application when it was using more than 12GB of ram while the second >>> one could use all of the 48GB of memory. If I have 500+ nodes is difficult >>> to check all of them and I am not familiar with any efficient solution. >>> Initially I thought about memtester but it takes a lot of time. I know that >>> this does not apply exactly on this mailing list but I thought that maybe >>> an OpenMPI user knows something about. >>> >>> >>> Best regards, >>> George Markomanolis >>> ___ >>> users mailing list >>> us...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> ___ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users >> >
Re: [OMPI users] Maximum number of MPI processes on a node + discovering faulty nodes
Dear Jeff, Of course I was thinking to execute memtester on each node on the same time and gather the outputs. However executing memtester on a node with 48GB memory it takes a lot of time (more than 1-2 hours, I don't remember exactly, maybe even more because I cancelled its execution) and I have to consume resources just for testing. So I was curious if you know a tool/procedure that works much faster. Of course filling the memory with an application works also but I don't know how right it is. Best regards, George Markomanolis On 11/26/2012 06:09 PM, Jeff Squyres wrote: On Nov 26, 2012, at 4:02 AM, George Markomanolis wrote: Another more generic question, is about discovering nodes with faulty memory. Is there any way to identify nodes with faulty memory? I found accidentally that a node with exact the same hardware couldn't execute an MPI application when it was using more than 12GB of ram while the second one could use all of the 48GB of memory. If I have 500+ nodes is difficult to check all of them and I am not familiar with any efficient solution. Initially I thought about memtester but it takes a lot of time. I know that this does not apply exactly on this mailing list but I thought that maybe an OpenMPI user knows something about. You really do want something like a memory tester. MPI applications *might* beat on your memory to identify errors, but that's really just a side effect of HPC access patterns. You really want a dedicated memory tester. If such a memory tester takes a long time, you might want to use mpirun to launch it on multiple nodes simultaneously to save some time...?
Re: [OMPI users] Maximum number of MPI processes on a node + discovering faulty nodes
Dear Ralph, Thanks for the answer, I am using OMPI v1.4.1. Best regards, George Markomanolis On 11/26/2012 05:07 PM, Ralph Castain wrote: What version of OMPI are you using? On Nov 26, 2012, at 1:02 AM, George Markomanolis wrote: Dear all, Initially I would like an advice of how to identify the maximum number of MPI processes that can be executed on a node with oversubscribing. When I try to execute an application with 4096 MPI processes on a 24-cores node with 48GB of memory, I have an error "Unknown error: 1" while the memory is not even at the half. I can execute the same application with 2048 MPI processes in less than one minute. I have checked linux settings about maximum number of processes and it is much bigger than 4096. Another more generic question, is about discovering nodes with faulty memory. Is there any way to identify nodes with faulty memory? I found accidentally that a node with exact the same hardware couldn't execute an MPI application when it was using more than 12GB of ram while the second one could use all of the 48GB of memory. If I have 500+ nodes is difficult to check all of them and I am not familiar with any efficient solution. Initially I thought about memtester but it takes a lot of time. I know that this does not apply exactly on this mailing list but I thought that maybe an OpenMPI user knows something about. Best regards, George Markomanolis ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
[OMPI users] How to reduce the use of the network by Open MPI job?
Hi all, I have a question related to Open MPI btl_tcp module. How to slow down OMPI in terms of network utilization ? My OpenMPI based application (lets call it APP1) is sharing hardware components with other software (lets call it APP2) which is very sensitive on network saturation. The APP2 suffers from lost “KEEP_ALIVE_RESPONSE” like tcp packages. Things are more complicated because of below described architecture constraints or other reasons: - We are using Open MPI version 1.4.3 and we rather do not want to migrate to newer version right now - computational nodes have a some company internal Linux distribution (2.6.x core version) - I've done some external patches on OMPI 1.4.3 source codes in order to address problem with lack of support for the multiple NIC aliases (in conjunction with bonds) by OMPI. 1. Ifconfig says us that on every node we have 3 NIC aliases (3 different IP and names with pattern such as: xxx0, xxx0:0, xxx0:1) in conjunction with bond configuration (all have the same HWaddr) - My patch wasn't so sophisticated, the main idea was to just store a local NIC index instead of kernel NIC index. Then I modified the mca_btl_tcp_proc_accept() method to iterate over all published by peer endpoint NIC IP's (I made some changes to the * mca_btl_tcp_component_exchange* method ...) . - I'm not allowed to add some new Linux core modules which could be used to limit network bandwidth on OS level :( Any tips are very appreciated Best Regards, Sylvester
