[OMPI users] Anyone has OpenMPI Logo (Vector Graphics only)
Hello All, I know that this is something out of topic here: but does anyone have OpenMPI logo in vector graphics (preferably SVG), that I can use in my documents and presentations? Thanks a lot for any help. Devendra
Re: [OMPI users] Anyone has OpenMPI Logo (Vector Graphics only)
Hi, Am 11.03.2013 um 12:04 schrieb devendra rai: > Hello All, > > I know that this is something out of topic here: but does anyone have OpenMPI > logo in vector graphics (preferably SVG), that I can use in my documents and > presentations? there is a search box on Open MPI's homepage and the second hit will bring you there. -- Reuti > Thanks a lot for any help. > > Devendra > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] Segmentation fault error with IB support when number of processes is greater 129
Did the check the FAQ entries listed on all the warning messages that you're getting? You should probably fix those first. Sent from my phone. No type good. On Mar 10, 2013, at 4:30 AM, "Svyatoslav Korneev" wrote: > Dear Community, > > I have 4 computing nodes and front-end. Computing nodes connected via > IB and Ethernet and fron-end has Ethernet only. Computing node has 4 > CPU on board, each CPU has 16 cores, total number of cores per node is > 64. The IB network controller is Mellanox MT26428, IB switch is Qlogic > 12000. I installed Rock Cluster Linux 6.1 on my cluster, and this > system has OpenMPI from the box. ompi_info gives the version of OpenMP > is 1.6.2 > > Good news that IB and OpendMPI with IB support working from the box, > but I face a really strange bug. If I try to run HelloWord on more > then 129 processes with IB support it gives me Segmentation fault > error. I have this error even if I try to start it on one node. Below > 129 processes everything is working fine, on 1 node or on 4 nodes > except warning messages (listing bellow). Without IB support > everything working fine on arbitrary number of processes. > > Anybody have an idea regarding my issue? > > Thank you. > > > Warning messages for HelloWord, running on two processes (each process > per node): > > mpirun --mca btl_openib_verbose 1 --mca btl ^tcp -hostfile > machinefile -n 2 a.out > [compute-0-0.local][[43740,1],0][btl_openib_ini.c:166:ompi_btl_openib_ini_query] > Querying INI files for vendor 0x02c9, part ID 26428 > [compute-0-0.local][[43740,1],0][btl_openib_ini.c:185:ompi_btl_openib_ini_query] > Found corresponding INI values: Mellanox Hermon > [compute-0-0.local][[43740,1],0][btl_openib_ini.c:166:ompi_btl_openib_ini_query] > Querying INI files for vendor 0x, part ID 0 > [compute-0-0.local][[43740,1],0][btl_openib_ini.c:185:ompi_btl_openib_ini_query] > Found corresponding INI values: default > -- > WARNING: There are more than one active ports on host > 'compute-0-0.local', but the > default subnet GID prefix was detected on more than one of these > ports. If these ports are connected to different physical IB > networks, this configuration will fail in Open MPI. This version of > Open MPI requires that every physically separate IB subnet that is > used between connected MPI processes must have different subnet ID > values. > > Please see this FAQ entry for more details: > > http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid > > NOTE: You can turn off this warning by setting the MCA parameter > btl_openib_warn_default_gid_prefix to 0. > -- > [compute-0-1.local][[43740,1],1][btl_openib_ini.c:166:ompi_btl_openib_ini_query] > Querying INI files for vendor 0x02c9, part ID 26428 > [compute-0-1.local][[43740,1],1][btl_openib_ini.c:185:ompi_btl_openib_ini_query] > Found corresponding INI values: Mellanox Hermon > [compute-0-1.local][[43740,1],1][btl_openib_ini.c:166:ompi_btl_openib_ini_query] > Querying INI files for vendor 0x, part ID 0 > [compute-0-1.local][[43740,1],1][btl_openib_ini.c:185:ompi_btl_openib_ini_query] > Found corresponding INI values: default > -- > WARNING: Failed to open "OpenIB-cma-1" [DAT_INVALID_ADDRESS:]. > This may be a real error or it may be an invalid entry in the uDAPL > Registry which is contained in the dat.conf file. Contact your local > System Administrator to confirm the availability of the interfaces in > the dat.conf file. > -- > compute-0-0.local:20229: open_hca: getaddr_netdev ERROR: Success. Is > ib1 configured? > compute-0-0.local:20229: open_hca: device mthca0 not found > compute-0-0.local:20229: open_hca: device mthca0 not found > compute-0-1.local:58701: open_hca: getaddr_netdev ERROR: Success. Is > ib1 configured? > compute-0-1.local:58701: open_hca: device mthca0 not found > compute-0-1.local:58701: open_hca: device mthca0 not found > DAT: library load failure: libdaplscm.so.2: cannot open shared object > file: No such file or directory > DAT: library load failure: libdaplscm.so.2: cannot open shared object > file: No such file or directory > DAT: library load failure: libdaplscm.so.2: cannot open shared object > file: No such file or directory > DAT: library load failure: libdaplscm.so.2: cannot open shared object > file: No such file or directory > [compute-0-0.local][[43740,1],0][btl_openib_ini.c:166:ompi_btl_openib_ini_query] > Querying INI files for vendor 0x02c9, part ID 26428 > [compute-0-0.local][[43740,1],0][btl_openib_ini.c:185:ompi_btl_openib_ini_query] > Found corresponding INI values: Mellanox Hermon > [compute-0-1.local][[43740,1],1][btl_openib_ini.c:166:ompi_btl_openib_ini_query] > Querying INI files for vendor 0x02c9, part I
Re: [OMPI users] Segmentation fault error with IB support when number of processes is greater 129
Hello, Thank you for your answer. But below 129 code runs well even with these warnings. I have following warnings: 1. WARNING: There are more than one active ports on host 'compute-0-0.local' 2. WARNING: Failed to open "OpenIB-cma-1" [DAT_INVALID_ADDRESS:]. 3. open_hca: getaddr_netdev ERROR: Success. Is ib1 configured? 4. open_hca: device mthca0 not found 5. library load failure: libdaplscm.so.2: cannot open shared object file: No such file or directory 6. WARNING: It appears that your OpenFabrics subsystem is configured to only allow registering part of your physical memory. It's looks messy, but these warnings are not critical, or I'm not right ? Warnings 1-3 denote the unconfigured second port, should I configure network interface for it ? I tried to solve warning 6 by manual, but it does not work. What about warning 5, what is this library libdaplscm.so.2? How do you think, may be it would be better to install MLNX_OFED on my nodes as I have NIC from this brand ? Regards, Svyatoslav On Mon, Mar 11, 2013 at 4:04 PM, Jeff Squyres (jsquyres) wrote: > Did the check the FAQ entries listed on all the warning messages that you're > getting? You should probably fix those first. > > Sent from my phone. No type good. > > On Mar 10, 2013, at 4:30 AM, "Svyatoslav Korneev" > wrote: > >> Dear Community, >> >> I have 4 computing nodes and front-end. Computing nodes connected via >> IB and Ethernet and fron-end has Ethernet only. Computing node has 4 >> CPU on board, each CPU has 16 cores, total number of cores per node is >> 64. The IB network controller is Mellanox MT26428, IB switch is Qlogic >> 12000. I installed Rock Cluster Linux 6.1 on my cluster, and this >> system has OpenMPI from the box. ompi_info gives the version of OpenMP >> is 1.6.2 >> >> Good news that IB and OpendMPI with IB support working from the box, >> but I face a really strange bug. If I try to run HelloWord on more >> then 129 processes with IB support it gives me Segmentation fault >> error. I have this error even if I try to start it on one node. Below >> 129 processes everything is working fine, on 1 node or on 4 nodes >> except warning messages (listing bellow). Without IB support >> everything working fine on arbitrary number of processes. >> >> Anybody have an idea regarding my issue? >> >> Thank you. >> >> >> Warning messages for HelloWord, running on two processes (each process >> per node): >> >> mpirun --mca btl_openib_verbose 1 --mca btl ^tcp -hostfile >> machinefile -n 2 a.out >> [compute-0-0.local][[43740,1],0][btl_openib_ini.c:166:ompi_btl_openib_ini_query] >> Querying INI files for vendor 0x02c9, part ID 26428 >> [compute-0-0.local][[43740,1],0][btl_openib_ini.c:185:ompi_btl_openib_ini_query] >> Found corresponding INI values: Mellanox Hermon >> [compute-0-0.local][[43740,1],0][btl_openib_ini.c:166:ompi_btl_openib_ini_query] >> Querying INI files for vendor 0x, part ID 0 >> [compute-0-0.local][[43740,1],0][btl_openib_ini.c:185:ompi_btl_openib_ini_query] >> Found corresponding INI values: default >> -- >> WARNING: There are more than one active ports on host >> 'compute-0-0.local', but the >> default subnet GID prefix was detected on more than one of these >> ports. If these ports are connected to different physical IB >> networks, this configuration will fail in Open MPI. This version of >> Open MPI requires that every physically separate IB subnet that is >> used between connected MPI processes must have different subnet ID >> values. >> >> Please see this FAQ entry for more details: >> >> http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid >> >> NOTE: You can turn off this warning by setting the MCA parameter >> btl_openib_warn_default_gid_prefix to 0. >> -- >> [compute-0-1.local][[43740,1],1][btl_openib_ini.c:166:ompi_btl_openib_ini_query] >> Querying INI files for vendor 0x02c9, part ID 26428 >> [compute-0-1.local][[43740,1],1][btl_openib_ini.c:185:ompi_btl_openib_ini_query] >> Found corresponding INI values: Mellanox Hermon >> [compute-0-1.local][[43740,1],1][btl_openib_ini.c:166:ompi_btl_openib_ini_query] >> Querying INI files for vendor 0x, part ID 0 >> [compute-0-1.local][[43740,1],1][btl_openib_ini.c:185:ompi_btl_openib_ini_query] >> Found corresponding INI values: default >> -- >> WARNING: Failed to open "OpenIB-cma-1" [DAT_INVALID_ADDRESS:]. >> This may be a real error or it may be an invalid entry in the uDAPL >> Registry which is contained in the dat.conf file. Contact your local >> System Administrator to confirm the availability of the interfaces in >> the dat.conf file. >> -- >> compute-0-0.local:20229: open_hca: getaddr_netdev ERROR: Success. Is >> ib1 configured? >> co
Re: [OMPI users] Segmentation fault error with IB support when number of processes is greater 129
If you have multiple active ports, you might as well utilize them. After all, you paid for them! You should disable udapl; it's really not useful unless you're going to run intel MPI. You do need to fix the registration memory issue - see the FAQ for that. So yes, none of these may be contributing to the segv. But they should sill be cleared up before chasing the segv, if for no other reason than to decease the number of variables and side effects. that you're chasing. Also, please upgrade to the latest certain of open MPI. Sent from my phone. No type good. On Mar 11, 2013, at 10:29 AM, "Svyatoslav Korneev" wrote: > Hello, > > Thank you for your answer. > > But below 129 code runs well even with these warnings. > > I have following warnings: > > 1. WARNING: There are more than one active ports on host > 'compute-0-0.local' > > 2. WARNING: Failed to open "OpenIB-cma-1" [DAT_INVALID_ADDRESS:]. > > 3. open_hca: getaddr_netdev ERROR: Success. Is > ib1 configured? > > 4. open_hca: device mthca0 not found > > 5. library load failure: libdaplscm.so.2: cannot open shared object > file: No such file or directory > > 6. WARNING: It appears that your OpenFabrics subsystem is configured to only > allow registering part of your physical memory. > > > It's looks messy, but these warnings are not critical, or I'm not right ? > > Warnings 1-3 denote the unconfigured second port, should I configure > network interface for it ? I tried to solve warning 6 by manual, but > it does not work. What about warning 5, what is this library > libdaplscm.so.2? > > How do you think, may be it would be better to install MLNX_OFED on my > nodes as I have NIC from this brand ? > > Regards, > Svyatoslav > > On Mon, Mar 11, 2013 at 4:04 PM, Jeff Squyres (jsquyres) > wrote: >> Did the check the FAQ entries listed on all the warning messages that you're >> getting? You should probably fix those first. >> >> Sent from my phone. No type good. >> >> On Mar 10, 2013, at 4:30 AM, "Svyatoslav Korneev" >> wrote: >> >>> Dear Community, >>> >>> I have 4 computing nodes and front-end. Computing nodes connected via >>> IB and Ethernet and fron-end has Ethernet only. Computing node has 4 >>> CPU on board, each CPU has 16 cores, total number of cores per node is >>> 64. The IB network controller is Mellanox MT26428, IB switch is Qlogic >>> 12000. I installed Rock Cluster Linux 6.1 on my cluster, and this >>> system has OpenMPI from the box. ompi_info gives the version of OpenMP >>> is 1.6.2 >>> >>> Good news that IB and OpendMPI with IB support working from the box, >>> but I face a really strange bug. If I try to run HelloWord on more >>> then 129 processes with IB support it gives me Segmentation fault >>> error. I have this error even if I try to start it on one node. Below >>> 129 processes everything is working fine, on 1 node or on 4 nodes >>> except warning messages (listing bellow). Without IB support >>> everything working fine on arbitrary number of processes. >>> >>> Anybody have an idea regarding my issue? >>> >>> Thank you. >>> >>> >>> Warning messages for HelloWord, running on two processes (each process >>> per node): >>> >>> mpirun --mca btl_openib_verbose 1 --mca btl ^tcp -hostfile >>> machinefile -n 2 a.out >>> [compute-0-0.local][[43740,1],0][btl_openib_ini.c:166:ompi_btl_openib_ini_query] >>> Querying INI files for vendor 0x02c9, part ID 26428 >>> [compute-0-0.local][[43740,1],0][btl_openib_ini.c:185:ompi_btl_openib_ini_query] >>> Found corresponding INI values: Mellanox Hermon >>> [compute-0-0.local][[43740,1],0][btl_openib_ini.c:166:ompi_btl_openib_ini_query] >>> Querying INI files for vendor 0x, part ID 0 >>> [compute-0-0.local][[43740,1],0][btl_openib_ini.c:185:ompi_btl_openib_ini_query] >>> Found corresponding INI values: default >>> -- >>> WARNING: There are more than one active ports on host >>> 'compute-0-0.local', but the >>> default subnet GID prefix was detected on more than one of these >>> ports. If these ports are connected to different physical IB >>> networks, this configuration will fail in Open MPI. This version of >>> Open MPI requires that every physically separate IB subnet that is >>> used between connected MPI processes must have different subnet ID >>> values. >>> >>> Please see this FAQ entry for more details: >>> >>> http://www.open-mpi.org/faq/?category=openfabrics#ofa-default-subnet-gid >>> >>> NOTE: You can turn off this warning by setting the MCA parameter >>> btl_openib_warn_default_gid_prefix to 0. >>> -- >>> [compute-0-1.local][[43740,1],1][btl_openib_ini.c:166:ompi_btl_openib_ini_query] >>> Querying INI files for vendor 0x02c9, part ID 26428 >>> [compute-0-1.local][[43740,1],1][btl_openib_ini.c:185:ompi_btl_openib_ini_query] >>> Found corresponding INI values: Mellanox Hermon >>> [compute-0-1.local
[OMPI users] status of cuda across multiple IO hubs?
I'm running into issues when trying to use GPUs in a multiprocessor
system when using the latest release candidate (1.7rc8).
Specifically, it looks like the OpenMPI code is still assuming that
all GPUs are on the same IOH, as in this message from a few months
ago:
http://www.open-mpi.org/community/lists/users/2012/07/19879.php
I couldn't determine what happened to the ticket mentioned in that thread.
For the moment, I'm just constraining myself to using the GPUs
attached to one processor, but obviously that's less then ideal :).
Curiously, the eager send path doesn't seem to have the same issue -
if I adjust btl_smcuda_eager_limit up, sends work up to that
threshold. Unfortunately, if I increase it beyond 10 megabytes I
start seeing bus errors.
I can manually breakup my own sends to be below the eager limit, but
that seems non-optimal.
Any other recommendations?
Thanks,
R
The testing code and output is pasted below.
---
#include
#include
#include
# define CUDA_SAFE_CALL( call) {\
cudaError err = call;\
if( cudaSuccess != err) {\
fprintf(stderr, "Cuda error in file '%s' in line %i : %s.\n",\
__FILE__, __LINE__, cudaGetErrorString( err) ); \
exit(EXIT_FAILURE); \
} }
int recv(int src) {
cudaSetDevice(0);
for (int bSize = 1; bSize < 100e6; bSize *= 2) {
fprintf(stderr, "Recv: %d\n", bSize);
void* buffer;
CUDA_SAFE_CALL(cudaMalloc(&buffer, bSize));
auto world = MPI::COMM_WORLD;
world.Recv(buffer, bSize, MPI::BYTE, src, 0);
CUDA_SAFE_CALL(cudaFree(buffer))
}
}
int send(int dst) {
cudaSetDevice(2);
for (int bSize = 1; bSize < 100e6; bSize *= 2) {
fprintf(stderr, "Send: %d\n", bSize);
void* buffer;
CUDA_SAFE_CALL(cudaMalloc(&buffer, bSize));
auto world = MPI::COMM_WORLD;
world.Send(buffer, bSize, MPI::BYTE, dst, 0);
CUDA_SAFE_CALL(cudaFree(buffer))
}
}
void checkPeerAccess() {
fprintf(stderr, "Access capability: gpu -> gpu\n");
for (int a = 0; a < 3; ++a) {
for (int b = a; b < 3; ++b) {
if (a == b) { continue; }
int res;
cudaDeviceCanAccessPeer(&res, a, b);
fprintf(stderr, "%d <-> %d: %d\n", a, b, res);
}
}
}
int main() {
MPI::Init_thread(MPI::THREAD_MULTIPLE);
if (MPI::COMM_WORLD.Get_rank() == 0) {
checkPeerAccess();
recv(1);
} else {
send(0);
}
MPI::Finalize();
}
output from running:
mpirun -mca btl_smcuda_eager_limit 64 -n 2 ./a.out
Access capability: gpu -> gpu
0 <-> 1: 1
0 <-> 2: 0
1 <-> 2: 0
Send: 1
Recv: 1
Send: 2
Recv: 2
Send: 4
Recv: 4
Send: 8
Recv: 8
Send: 16
Recv: 16
--
The call to cuIpcOpenMemHandle failed. This is an unrecoverable error
and will cause the program to abort.
cuIpcOpenMemHandle return value: 217
address: 0x230020
Check the cuda.h file for what the return value means. Perhaps a reboot
of the node will clear the problem.
--
Re: [OMPI users] status of cuda across multiple IO hubs?
Yes, unfortunately, that issue is still unfixed. I just created the ticket and
included a possible workaround.
https://svn.open-mpi.org/trac/ompi/ticket/3531
>-Original Message-
>From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org]
>On Behalf Of Russell Power
>Sent: Monday, March 11, 2013 11:28 AM
>To: us...@open-mpi.org
>Subject: [OMPI users] status of cuda across multiple IO hubs?
>
>I'm running into issues when trying to use GPUs in a multiprocessor system
>when using the latest release candidate (1.7rc8).
>Specifically, it looks like the OpenMPI code is still assuming that all GPUs
>are
>on the same IOH, as in this message from a few months
>ago:
>
>http://www.open-mpi.org/community/lists/users/2012/07/19879.php
>
>I couldn't determine what happened to the ticket mentioned in that thread.
>
>For the moment, I'm just constraining myself to using the GPUs attached to
>one processor, but obviously that's less then ideal :).
>
>Curiously, the eager send path doesn't seem to have the same issue - if I
>adjust btl_smcuda_eager_limit up, sends work up to that threshold.
>Unfortunately, if I increase it beyond 10 megabytes I start seeing bus errors.
>
>I can manually breakup my own sends to be below the eager limit, but that
>seems non-optimal.
>
>Any other recommendations?
>
>Thanks,
>
>R
>
>The testing code and output is pasted below.
>
>---
>
>#include
>#include
>#include
>
>
># define CUDA_SAFE_CALL( call) {\
>cudaError err = call;\
>if( cudaSuccess != err) {\
>fprintf(stderr, "Cuda error in file '%s' in line %i : %s.\n",\
>__FILE__, __LINE__, cudaGetErrorString( err) ); \
>exit(EXIT_FAILURE); \
>} }
>
>int recv(int src) {
> cudaSetDevice(0);
> for (int bSize = 1; bSize < 100e6; bSize *= 2) {
>fprintf(stderr, "Recv: %d\n", bSize);
>void* buffer;
>CUDA_SAFE_CALL(cudaMalloc(&buffer, bSize));
>auto world = MPI::COMM_WORLD;
>world.Recv(buffer, bSize, MPI::BYTE, src, 0);
>CUDA_SAFE_CALL(cudaFree(buffer))
> }
>}
>
>int send(int dst) {
> cudaSetDevice(2);
> for (int bSize = 1; bSize < 100e6; bSize *= 2) {
>fprintf(stderr, "Send: %d\n", bSize);
>void* buffer;
>CUDA_SAFE_CALL(cudaMalloc(&buffer, bSize));
>auto world = MPI::COMM_WORLD;
>world.Send(buffer, bSize, MPI::BYTE, dst, 0);
>CUDA_SAFE_CALL(cudaFree(buffer))
> }
>}
>
>void checkPeerAccess() {
> fprintf(stderr, "Access capability: gpu -> gpu\n");
> for (int a = 0; a < 3; ++a) {
>for (int b = a; b < 3; ++b) {
> if (a == b) { continue; }
> int res;
> cudaDeviceCanAccessPeer(&res, a, b);
> fprintf(stderr, "%d <-> %d: %d\n", a, b, res);
>}
> }
>}
>
>int main() {
> MPI::Init_thread(MPI::THREAD_MULTIPLE);
> if (MPI::COMM_WORLD.Get_rank() == 0) {
>checkPeerAccess();
>recv(1);
> } else {
>send(0);
> }
> MPI::Finalize();
>}
>
>output from running:
>mpirun -mca btl_smcuda_eager_limit 64 -n 2 ./a.out Access capability: gpu ->
>gpu
>0 <-> 1: 1
>0 <-> 2: 0
>1 <-> 2: 0
>Send: 1
>Recv: 1
>Send: 2
>Recv: 2
>Send: 4
>Recv: 4
>Send: 8
>Recv: 8
>Send: 16
>Recv: 16
>--
>The call to cuIpcOpenMemHandle failed. This is an unrecoverable error and
>will cause the program to abort.
> cuIpcOpenMemHandle return value: 217
> address: 0x230020
>Check the cuda.h file for what the return value means. Perhaps a reboot of
>the node will clear the problem.
>--
>___
>users mailing list
>us...@open-mpi.org
>http://www.open-mpi.org/mailman/listinfo.cgi/users
---
This email message is for the sole use of the intended recipient(s) and may
contain
confidential information. Any unauthorized review, use, disclosure or
distribution
is prohibited. If you are not the intended recipient, please contact the
sender by
reply email and destroy all copies of the original message.
---
[OMPI users] Send Recv Bandwidth
Hi, I was comparing a simple Send-Recv program to another program with two memcpy's to/from shared memory. Number of processes = 2 and different array sizes (from 10^6 - 10^8 doubles) on IA64. With the --mca btl sm,self options I get almost twice the bandwidth compared to the two memcpy's. I looked at openmpi source and I cannot figure out if it's using anything other than simple glibc memcpy. I must be missing something. Can somebody please help? Thanks, Nilesh. P.S. I was not sure if I should post this message in the users or the devl list, so I posted to both. Apologies for the multiple postings.
