NIC = network interface controller? There is QDR Infiniband 4x/10G Ethernet/Gigabit Ethernet. I want to use QDR Infiniband. Here is a new output: $ mpirun -mca mca_base_env_list 'LD_PRELOAD' --debug-daemons --mca plm_base_verbose 5 -mca oob_base_verbose 10 -mca rml_base_verbose 10 -np 2 hello_c |tee hello.out Warning: Conflicting CPU frequencies detected, using: 2927.000000. [compiler-2:30735] mca:base:select:( plm) Querying component [isolated] [compiler-2:30735] mca:base:select:( plm) Query of component [isolated] set priority to 0 [compiler-2:30735] mca:base:select:( plm) Querying component [rsh] [compiler-2:30735] mca:base:select:( plm) Query of component [rsh] set priority to 10 [compiler-2:30735] mca:base:select:( plm) Querying component [slurm] [compiler-2:30735] mca:base:select:( plm) Query of component [slurm] set priority to 75 [compiler-2:30735] mca:base:select:( plm) Selected component [slurm] [compiler-2:30735] mca: base: components_register: registering oob components [compiler-2:30735] mca: base: components_register: found loaded component tcp [compiler-2:30735] mca: base: components_register: component tcp register function successful [compiler-2:30735] mca: base: components_open: opening oob components [compiler-2:30735] mca: base: components_open: found loaded component tcp [compiler-2:30735] mca: base: components_open: component tcp open function successful [compiler-2:30735] mca:oob:select: checking available component tcp [compiler-2:30735] mca:oob:select: Querying component [tcp] [compiler-2:30735] oob:tcp: component_available called [compiler-2:30735] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4 [compiler-2:30735] WORKING INTERFACE 2 KERNEL INDEX 3 FAMILY: V4 [compiler-2:30735] [[65177,0],0] oob:tcp:init adding 10.0.251.53 to our list of V4 connections [compiler-2:30735] WORKING INTERFACE 3 KERNEL INDEX 4 FAMILY: V4 [compiler-2:30735] [[65177,0],0] oob:tcp:init adding 10.0.0.4 to our list of V4 connections [compiler-2:30735] WORKING INTERFACE 4 KERNEL INDEX 5 FAMILY: V4 [compiler-2:30735] [[65177,0],0] oob:tcp:init adding 10.2.251.14 to our list of V4 connections [compiler-2:30735] WORKING INTERFACE 5 KERNEL INDEX 6 FAMILY: V4 [compiler-2:30735] [[65177,0],0] oob:tcp:init adding 10.128.0.4 to our list of V4 connections [compiler-2:30735] WORKING INTERFACE 6 KERNEL INDEX 7 FAMILY: V4 [compiler-2:30735] [[65177,0],0] oob:tcp:init adding 93.180.7.38 to our list of V4 connections [compiler-2:30735] [[65177,0],0] TCP STARTUP [compiler-2:30735] [[65177,0],0] attempting to bind to IPv4 port 0 [compiler-2:30735] [[65177,0],0] assigned IPv4 port 49759 [compiler-2:30735] mca:oob:select: Adding component to end [compiler-2:30735] mca:oob:select: Found 1 active transports [compiler-2:30735] mca: base: components_register: registering rml components [compiler-2:30735] mca: base: components_register: found loaded component oob [compiler-2:30735] mca: base: components_register: component oob has no register or open function [compiler-2:30735] mca: base: components_open: opening rml components [compiler-2:30735] mca: base: components_open: found loaded component oob [compiler-2:30735] mca: base: components_open: component oob open function successful [compiler-2:30735] orte_rml_base_select: initializing rml component oob [compiler-2:30735] [[65177,0],0] posting recv [compiler-2:30735] [[65177,0],0] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD] [compiler-2:30735] [[65177,0],0] posting recv [compiler-2:30735] [[65177,0],0] posting persistent recv on tag 15 for peer [[WILDCARD],WILDCARD] [compiler-2:30735] [[65177,0],0] posting recv [compiler-2:30735] [[65177,0],0] posting persistent recv on tag 32 for peer [[WILDCARD],WILDCARD] [compiler-2:30735] [[65177,0],0] posting recv [compiler-2:30735] [[65177,0],0] posting persistent recv on tag 33 for peer [[WILDCARD],WILDCARD] [compiler-2:30735] [[65177,0],0] posting recv [compiler-2:30735] [[65177,0],0] posting persistent recv on tag 5 for peer [[WILDCARD],WILDCARD] [compiler-2:30735] [[65177,0],0] posting recv [compiler-2:30735] [[65177,0],0] posting persistent recv on tag 10 for peer [[WILDCARD],WILDCARD] [compiler-2:30735] [[65177,0],0] posting recv [compiler-2:30735] [[65177,0],0] posting persistent recv on tag 12 for peer [[WILDCARD],WILDCARD] [compiler-2:30735] [[65177,0],0] posting recv [compiler-2:30735] [[65177,0],0] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD] [compiler-2:30735] [[65177,0],0] posting recv [compiler-2:30735] [[65177,0],0] posting persistent recv on tag 34 for peer [[WILDCARD],WILDCARD] [compiler-2:30735] [[65177,0],0] posting recv [compiler-2:30735] [[65177,0],0] posting persistent recv on tag 2 for peer [[WILDCARD],WILDCARD] [compiler-2:30735] [[65177,0],0] posting recv [compiler-2:30735] [[65177,0],0] posting persistent recv on tag 21 for peer [[WILDCARD],WILDCARD] [compiler-2:30735] [[65177,0],0] posting recv [compiler-2:30735] [[65177,0],0] posting persistent recv on tag 22 for peer [[WILDCARD],WILDCARD] [compiler-2:30735] [[65177,0],0] posting recv [compiler-2:30735] [[65177,0],0] posting persistent recv on tag 45 for peer [[WILDCARD],WILDCARD] [compiler-2:30735] [[65177,0],0] posting recv [compiler-2:30735] [[65177,0],0] posting persistent recv on tag 46 for peer [[WILDCARD],WILDCARD] [compiler-2:30735] [[65177,0],0] posting recv [compiler-2:30735] [[65177,0],0] posting persistent recv on tag 1 for peer [[WILDCARD],WILDCARD] [compiler-2:30735] [[65177,0],0] posting recv [compiler-2:30735] [[65177,0],0] posting persistent recv on tag 27 for peer [[WILDCARD],WILDCARD] Daemon was launched on node1-128-17 - beginning to initialize Daemon was launched on node1-128-18 - beginning to initialize [node1-128-17:14779] mca: base: components_register: registering oob components [node1-128-17:14779] mca: base: components_register: found loaded component tcp [node1-128-17:14779] mca: base: components_register: component tcp register function successful [node1-128-17:14779] mca: base: components_open: opening oob components [node1-128-17:14779] mca: base: components_open: found loaded component tcp [node1-128-17:14779] mca: base: components_open: component tcp open function successful [node1-128-17:14779] mca:oob:select: checking available component tcp [node1-128-17:14779] mca:oob:select: Querying component [tcp] [node1-128-17:14779] oob:tcp: component_available called [node1-128-17:14779] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4 [node1-128-17:14779] WORKING INTERFACE 2 KERNEL INDEX 3 FAMILY: V4 [node1-128-17:14779] [[65177,0],1] oob:tcp:init adding 10.0.128.17 to our list of V4 connections [node1-128-17:14779] WORKING INTERFACE 3 KERNEL INDEX 4 FAMILY: V4 [node1-128-17:14779] [[65177,0],1] oob:tcp:init adding 10.128.128.17 to our list of V4 connections [node1-128-17:14779] [[65177,0],1] TCP STARTUP [node1-128-17:14779] [[65177,0],1] attempting to bind to IPv4 port 0 [node1-128-17:14779] [[65177,0],1] assigned IPv4 port 46441 [node1-128-17:14779] mca:oob:select: Adding component to end [node1-128-17:14779] mca:oob:select: Found 1 active transports [node1-128-17:14779] mca: base: components_register: registering rml components [node1-128-17:14779] mca: base: components_register: found loaded component oob [node1-128-17:14779] mca: base: components_register: component oob has no register or open function [node1-128-17:14779] mca: base: components_open: opening rml components [node1-128-17:14779] mca: base: components_open: found loaded component oob [node1-128-17:14779] mca: base: components_open: component oob open function successful [node1-128-17:14779] orte_rml_base_select: initializing rml component oob [node1-128-18:17849] mca: base: components_register: registering oob components [node1-128-18:17849] mca: base: components_register: found loaded component tcp [node1-128-18:17849] mca: base: components_register: component tcp register function successful [node1-128-18:17849] mca: base: components_open: opening oob components [node1-128-18:17849] mca: base: components_open: found loaded component tcp [node1-128-18:17849] mca: base: components_open: component tcp open function successful [node1-128-18:17849] mca:oob:select: checking available component tcp [node1-128-18:17849] mca:oob:select: Querying component [tcp] [node1-128-18:17849] oob:tcp: component_available called [node1-128-18:17849] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4 [node1-128-18:17849] WORKING INTERFACE 2 KERNEL INDEX 3 FAMILY: V4 [node1-128-18:17849] [[65177,0],2] oob:tcp:init adding 10.0.128.18 to our list of V4 connections [node1-128-18:17849] WORKING INTERFACE 3 KERNEL INDEX 4 FAMILY: V4 [node1-128-18:17849] [[65177,0],2] oob:tcp:init adding 10.128.128.18 to our list of V4 connections [node1-128-18:17849] [[65177,0],2] TCP STARTUP [node1-128-18:17849] [[65177,0],2] attempting to bind to IPv4 port 0 [node1-128-18:17849] [[65177,0],2] assigned IPv4 port 60695 [node1-128-18:17849] mca:oob:select: Adding component to end [node1-128-18:17849] mca:oob:select: Found 1 active transports [node1-128-18:17849] mca: base: components_register: registering rml components [node1-128-18:17849] mca: base: components_register: found loaded component oob [node1-128-18:17849] mca: base: components_register: component oob has no register or open function [node1-128-18:17849] mca: base: components_open: opening rml components [node1-128-18:17849] mca: base: components_open: found loaded component oob [node1-128-18:17849] mca: base: components_open: component oob open function successful [node1-128-18:17849] orte_rml_base_select: initializing rml component oob Daemon [[65177,0],1] checking in as pid 14779 on host node1-128-17 [node1-128-17:14779] [[65177,0],1] orted: up and running - waiting for commands! [node1-128-17:14779] [[65177,0],1] posting recv [node1-128-17:14779] [[65177,0],1] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD] [node1-128-17:14779] [[65177,0],1] posting recv [node1-128-17:14779] [[65177,0],1] posting persistent recv on tag 15 for peer [[WILDCARD],WILDCARD] [node1-128-17:14779] [[65177,0],1] posting recv [node1-128-17:14779] [[65177,0],1] posting persistent recv on tag 32 for peer [[WILDCARD],WILDCARD] [node1-128-17:14779] [[65177,0],1] posting recv [node1-128-17:14779] [[65177,0],1] posting persistent recv on tag 11 for peer [[WILDCARD],WILDCARD] [node1-128-17:14779] [[65177,0],1] posting recv [node1-128-17:14779] [[65177,0],1] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD] [node1-128-17:14779] [[65177,0],1]: set_addr to uri 4271439872.0;tcp://10.0.251.53,10.0.0.4,10.2.251.14,10.128.0.4,93.180.7.38:49759 [node1-128-17:14779] [[65177,0],1]:set_addr checking if peer [[65177,0],0] is reachable via component tcp [node1-128-17:14779] [[65177,0],1] oob:tcp: working peer [[65177,0],0] address tcp://10.0.251.53,10.0.0.4,10.2.251.14,10.128.0.4,93.180.7.38:49759 [node1-128-17:14779] [[65177,0],1] PASSING ADDR 10.0.251.53 TO MODULE [node1-128-17:14779] [[65177,0],1]:tcp set addr for peer [[65177,0],0] [node1-128-17:14779] [[65177,0],1] PASSING ADDR 10.0.0.4 TO MODULE [node1-128-17:14779] [[65177,0],1]:tcp set addr for peer [[65177,0],0] [node1-128-17:14779] [[65177,0],1] PASSING ADDR 10.2.251.14 TO MODULE [node1-128-17:14779] [[65177,0],1]:tcp set addr for peer [[65177,0],0] [node1-128-17:14779] [[65177,0],1] PASSING ADDR 10.128.0.4 TO MODULE [node1-128-17:14779] [[65177,0],1]:tcp set addr for peer [[65177,0],0] [node1-128-17:14779] [[65177,0],1] PASSING ADDR 93.180.7.38 TO MODULE [node1-128-17:14779] [[65177,0],1]:tcp set addr for peer [[65177,0],0] [node1-128-17:14779] [[65177,0],1]: peer [[65177,0],0] is reachable via component tcp [node1-128-17:14779] [[65177,0],1] posting recv [node1-128-17:14779] [[65177,0],1] posting persistent recv on tag 3 for peer [[WILDCARD],WILDCARD] [node1-128-17:14779] [[65177,0],1] posting recv [node1-128-17:14779] [[65177,0],1] posting persistent recv on tag 21 for peer [[WILDCARD],WILDCARD] [node1-128-17:14779] [[65177,0],1] posting recv [node1-128-17:14779] [[65177,0],1] posting persistent recv on tag 45 for peer [[WILDCARD],WILDCARD] [node1-128-17:14779] [[65177,0],1] posting recv [node1-128-17:14779] [[65177,0],1] posting persistent recv on tag 46 for peer [[WILDCARD],WILDCARD] [node1-128-17:14779] [[65177,0],1] posting recv [node1-128-17:14779] [[65177,0],1] posting persistent recv on tag 1 for peer [[WILDCARD],WILDCARD] [node1-128-17:14779] [[65177,0],1] OOB_SEND: rml_oob_send.c:199 [node1-128-17:14779] [[65177,0],1]:tcp:processing set_peer cmd [node1-128-17:14779] [[65177,0],1]:tcp:processing set_peer cmd [node1-128-17:14779] [[65177,0],1]:tcp:processing set_peer cmd [node1-128-17:14779] [[65177,0],1]:tcp:processing set_peer cmd [node1-128-17:14779] [[65177,0],1]:tcp:processing set_peer cmd [node1-128-17:14779] [[65177,0],1] oob:base:send to target [[65177,0],0] [node1-128-17:14779] [[65177,0],1] oob:tcp:send_nb to peer [[65177,0],0]:10 [node1-128-17:14779] [[65177,0],1] tcp:send_nb to peer [[65177,0],0] [node1-128-17:14779] [[65177,0],1]:[oob_tcp.c:484] post send to [[65177,0],0] [node1-128-17:14779] [[65177,0],1]:[oob_tcp.c:421] processing send to peer [[65177,0],0]:10 [node1-128-17:14779] [[65177,0],1]:[oob_tcp.c:455] queue pending to [[65177,0],0] [node1-128-17:14779] [[65177,0],1] tcp:send_nb: initiating connection to [[65177,0],0] [node1-128-17:14779] [[65177,0],1]:[oob_tcp.c:469] connect to [[65177,0],0] [node1-128-17:14779] [[65177,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[65177,0],0] [node1-128-17:14779] [[65177,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[65177,0],0] on socket 10 [node1-128-17:14779] [[65177,0],1] orte_tcp_peer_try_connect: attempting to connect to proc [[65177,0],0] on 10.0.251.53:49759 - 0 retries [node1-128-17:14779] [[65177,0],1] waiting for connect completion to [[65177,0],0] - activating send event Daemon [[65177,0],2] checking in as pid 17849 on host node1-128-18 [node1-128-18:17849] [[65177,0],2] orted: up and running - waiting for commands! [node1-128-18:17849] [[65177,0],2] posting recv [node1-128-18:17849] [[65177,0],2] posting persistent recv on tag 30 for peer [[WILDCARD],WILDCARD] [node1-128-18:17849] [[65177,0],2] posting recv [node1-128-18:17849] [[65177,0],2] posting persistent recv on tag 15 for peer [[WILDCARD],WILDCARD] [node1-128-18:17849] [[65177,0],2] posting recv [node1-128-18:17849] [[65177,0],2] posting persistent recv on tag 32 for peer [[WILDCARD],WILDCARD] [node1-128-18:17849] [[65177,0],2] posting recv [node1-128-18:17849] [[65177,0],2] posting persistent recv on tag 11 for peer [[WILDCARD],WILDCARD] [node1-128-18:17849] [[65177,0],2] posting recv [node1-128-18:17849] [[65177,0],2] posting persistent recv on tag 9 for peer [[WILDCARD],WILDCARD] [node1-128-18:17849] [[65177,0],2]: set_addr to uri 4271439872.0;tcp://10.0.251.53,10.0.0.4,10.2.251.14,10.128.0.4,93.180.7.38:49759 [node1-128-18:17849] [[65177,0],2]:set_addr checking if peer [[65177,0],0] is reachable via component tcp [node1-128-18:17849] [[65177,0],2] oob:tcp: working peer [[65177,0],0] address tcp://10.0.251.53,10.0.0.4,10.2.251.14,10.128.0.4,93.180.7.38:49759 [node1-128-18:17849] [[65177,0],2] PASSING ADDR 10.0.251.53 TO MODULE [node1-128-18:17849] [[65177,0],2]:tcp set addr for peer [[65177,0],0] [node1-128-18:17849] [[65177,0],2] PASSING ADDR 10.0.0.4 TO MODULE [node1-128-18:17849] [[65177,0],2]:tcp set addr for peer [[65177,0],0] [node1-128-18:17849] [[65177,0],2] PASSING ADDR 10.2.251.14 TO MODULE [node1-128-18:17849] [[65177,0],2]:tcp set addr for peer [[65177,0],0] [node1-128-18:17849] [[65177,0],2] PASSING ADDR 10.128.0.4 TO MODULE [node1-128-18:17849] [[65177,0],2]:tcp set addr for peer [[65177,0],0] [node1-128-18:17849] [[65177,0],2] PASSING ADDR 93.180.7.38 TO MODULE [node1-128-18:17849] [[65177,0],2]:tcp set addr for peer [[65177,0],0] [node1-128-18:17849] [[65177,0],2]: peer [[65177,0],0] is reachable via component tcp [node1-128-18:17849] [[65177,0],2] posting recv [node1-128-18:17849] [[65177,0],2] posting persistent recv on tag 3 for peer [[WILDCARD],WILDCARD] [node1-128-18:17849] [[65177,0],2] posting recv [node1-128-18:17849] [[65177,0],2] posting persistent recv on tag 21 for peer [[WILDCARD],WILDCARD] [node1-128-18:17849] [[65177,0],2] posting recv [node1-128-18:17849] [[65177,0],2] posting persistent recv on tag 45 for peer [[WILDCARD],WILDCARD] [node1-128-18:17849] [[65177,0],2] posting recv [node1-128-18:17849] [[65177,0],2] posting persistent recv on tag 46 for peer [[WILDCARD],WILDCARD] [node1-128-18:17849] [[65177,0],2] posting recv [node1-128-18:17849] [[65177,0],2] posting persistent recv on tag 1 for peer [[WILDCARD],WILDCARD] [node1-128-18:17849] [[65177,0],2] OOB_SEND: rml_oob_send.c:199 [node1-128-18:17849] [[65177,0],2]:tcp:processing set_peer cmd [node1-128-18:17849] [[65177,0],2]:tcp:processing set_peer cmd [node1-128-18:17849] [[65177,0],2]:tcp:processing set_peer cmd [node1-128-18:17849] [[65177,0],2]:tcp:processing set_peer cmd [node1-128-18:17849] [[65177,0],2]:tcp:processing set_peer cmd [node1-128-18:17849] [[65177,0],2] oob:base:send to target [[65177,0],0] [node1-128-18:17849] [[65177,0],2] oob:tcp:send_nb to peer [[65177,0],0]:10 [node1-128-18:17849] [[65177,0],2] tcp:send_nb to peer [[65177,0],0] [node1-128-18:17849] [[65177,0],2]:[oob_tcp.c:484] post send to [[65177,0],0] [node1-128-18:17849] [[65177,0],2]:[oob_tcp.c:421] processing send to peer [[65177,0],0]:10 [node1-128-18:17849] [[65177,0],2]:[oob_tcp.c:455] queue pending to [[65177,0],0] [node1-128-18:17849] [[65177,0],2] tcp:send_nb: initiating connection to [[65177,0],0] [node1-128-18:17849] [[65177,0],2]:[oob_tcp.c:469] connect to [[65177,0],0] [node1-128-18:17849] [[65177,0],2] orte_tcp_peer_try_connect: attempting to connect to proc [[65177,0],0] [node1-128-18:17849] [[65177,0],2] orte_tcp_peer_try_connect: attempting to connect to proc [[65177,0],0] on socket 10 [node1-128-18:17849] [[65177,0],2] orte_tcp_peer_try_connect: attempting to connect to proc [[65177,0],0] on 10.0.251.53:49759 - 0 retries [node1-128-18:17849] [[65177,0],2] waiting for connect completion to [[65177,0],0] - activating send event [node1-128-18:17837] [[61806,0],2] tcp:send_handler called to send to peer [[61806,0],0] [node1-128-18:17837] [[61806,0],2] tcp:send_handler CONNECTING [node1-128-18:17837] [[61806,0],2]:tcp:complete_connect called for peer [[61806,0],0] on socket 10 [node1-128-18:17837] [[61806,0],2]-[[61806,0],0] tcp_peer_complete_connect: connection failed: Connection timed out (110) [node1-128-18:17837] [[61806,0],2] tcp_peer_close for [[61806,0],0] sd 10 state CONNECTING [node1-128-18:17837] [[61806,0],2] tcp:lost connection called for peer [[61806,0],0] [node1-128-18:17837] mca: base: close: component oob closed [node1-128-18:17837] mca: base: close: unloading component oob [node1-128-18:17837] [[61806,0],2] TCP SHUTDOWN [node1-128-18:17837] [[61806,0],2] RELEASING PEER OBJ [[61806,0],0] [node1-128-18:17837] [[61806,0],2] CLOSING SOCKET 10 [node1-128-18:17837] mca: base: close: component tcp closed [node1-128-18:17837] mca: base: close: unloading component tcp srun: error: node1-128-18: task 1: Exited with exit code 1 srun: Terminating job step 647191.1 [node1-128-17:14767] [[61806,0],1] tcp:send_handler called to send to peer [[61806,0],0] [node1-128-17:14767] [[61806,0],1] tcp:send_handler CONNECTING [node1-128-17:14767] [[61806,0],1]:tcp:complete_connect called for peer [[61806,0],0] on socket 10 [node1-128-17:14767] [[61806,0],1]-[[61806,0],0] tcp_peer_complete_connect: connection failed: Connection timed out (110) [node1-128-17:14767] [[61806,0],1] tcp_peer_close for [[61806,0],0] sd 10 state CONNECTING [node1-128-17:14767] [[61806,0],1] tcp:lost connection called for peer [[61806,0],0] [node1-128-17:14767] mca: base: close: component oob closed [node1-128-17:14767] mca: base: close: unloading component oob [node1-128-17:14767] [[61806,0],1] TCP SHUTDOWN [node1-128-17:14767] [[61806,0],1] RELEASING PEER OBJ [[61806,0],0] [node1-128-17:14767] [[61806,0],1] CLOSING SOCKET 10 [node1-128-17:14767] mca: base: close: component tcp closed [node1-128-17:14767] mca: base: close: unloading component tcp srun: error: node1-128-17: task 0: Exited with exit code 1 [node1-128-17:14779] [[65177,0],1] tcp:send_handler called to send to peer [[65177,0],0] [node1-128-17:14779] [[65177,0],1] tcp:send_handler CONNECTING [node1-128-17:14779] [[65177,0],1]:tcp:complete_connect called for peer [[65177,0],0] on socket 10 [node1-128-17:14779] [[65177,0],1]-[[65177,0],0] tcp_peer_complete_connect: connection failed: Connection timed out (110) [node1-128-17:14779] [[65177,0],1] tcp_peer_close for [[65177,0],0] sd 10 state CONNECTING [node1-128-17:14779] [[65177,0],1] tcp:lost connection called for peer [[65177,0],0] [node1-128-17:14779] mca: base: close: component oob closed [node1-128-17:14779] mca: base: close: unloading component oob [node1-128-17:14779] [[65177,0],1] TCP SHUTDOWN [node1-128-17:14779] [[65177,0],1] RELEASING PEER OBJ [[65177,0],0] [node1-128-17:14779] [[65177,0],1] CLOSING SOCKET 10 [node1-128-17:14779] mca: base: close: component tcp closed [node1-128-17:14779] mca: base: close: unloading component tcp [node1-128-18:17849] [[65177,0],2] tcp:send_handler called to send to peer [[65177,0],0] [node1-128-18:17849] [[65177,0],2] tcp:send_handler CONNECTING [node1-128-18:17849] [[65177,0],2]:tcp:complete_connect called for peer [[65177,0],0] on socket 10 [node1-128-18:17849] [[65177,0],2]-[[65177,0],0] tcp_peer_complete_connect: connection failed: Connection timed out (110) [node1-128-18:17849] [[65177,0],2] tcp_peer_close for [[65177,0],0] sd 10 state CONNECTING [node1-128-18:17849] [[65177,0],2] tcp:lost connection called for peer [[65177,0],0] [node1-128-18:17849] mca: base: close: component oob closed [node1-128-18:17849] mca: base: close: unloading component oob [node1-128-18:17849] [[65177,0],2] TCP SHUTDOWN [node1-128-18:17849] [[65177,0],2] RELEASING PEER OBJ [[65177,0],0] [node1-128-18:17849] [[65177,0],2] CLOSING SOCKET 10 [node1-128-18:17849] mca: base: close: component tcp closed [node1-128-18:17849] mca: base: close: unloading component tcp srun: error: node1-128-17: task 0: Exited with exit code 1 srun: Terminating job step 647191.2 srun: error: node1-128-18: task 1: Exited with exit code 1 -------------------------------------------------------------------------- An ORTE daemon has unexpectedly failed after launch and before communicating back to mpirun. This could be caused by a number of factors, including an inability to create a connection back to mpirun due to a lack of common network interfaces and/or no route found between them. Please check network connectivity (including firewalls and network routing requirements). -------------------------------------------------------------------------- [compiler-2:30735] [[65177,0],0] orted_cmd: received halt_vm cmd [compiler-2:30735] mca: base: close: component oob closed [compiler-2:30735] mca: base: close: unloading component oob [compiler-2:30735] [[65177,0],0] TCP SHUTDOWN [compiler-2:30735] mca: base: close: component tcp closed [compiler-2:30735] mca: base: close: unloading component tcp
Sun, 20 Jul 2014 13:11:19 -0700 от Ralph Castain <r...@open-mpi.org>: >Yeah, we aren't connecting back - is there a firewall running? You need to >leave the "--debug-daemons --mca plm_base_verbose 5" on there as well to see >the entire problem. > >What you can see here is that mpirun is listening on several interfaces: >>[access1:24264] [[55095,0],0] oob:tcp:init adding 10.0.251.51 to our list of >>V4 connections >>[access1:24264] [[55095,0],0] oob:tcp:init adding 10.2.251.11 to our list of >>V4 connections >>[access1:24264] [[55095,0],0] oob:tcp:init adding 10.0.0.111 to our list of >>V4 connections >>[access1:24264] [[55095,0],0] oob:tcp:init adding 10.128.0.1 to our list of >>V4 connections >>[access1:24264] [[55095,0],0] oob:tcp:init adding 93.180.7.36 to our list of >>V4 connections > >It looks like you have multiple interfaces connected to the same subnet - this >is generally a bad idea. I also saw that the last one in the list shows up >twice in the kernel array - not sure why, but is there something special about >that NIC? > >What do the NICs look like on the remote hosts? > >On Jul 20, 2014, at 10:59 AM, Timur Ismagilov < tismagi...@mail.ru > wrote: >> >> >> >>-------- Пересылаемое сообщение -------- >>От кого: Timur Ismagilov < tismagi...@mail.ru > >>Кому: Ralph Castain < r...@open-mpi.org > >>Дата: Sun, 20 Jul 2014 21:58:41 +0400 >>Тема: Re[2]: [OMPI users] Fwd: Re[4]: Salloc and mpirun problem >> >>Here it is: >>$ salloc -N2 --exclusive -p test -J ompi >>salloc: Granted job allocation 647049 >> >>$ mpirun -mca mca_base_env_list 'LD_PRELOAD' -mca oob_base_verbose 10 -mca >>rml_base_verbose 10 -np 2 hello_c >>[access1:24264] mca: base: components_register: registering oob components >>[access1:24264] mca: base: components_register: found loaded component tcp >>[access1:24264] mca: base: components_register: component tcp register >>function successful >>[access1:24264] mca: base: components_open: opening oob components >>[access1:24264] mca: base: components_open: found loaded component tcp >>[access1:24264] mca: base: components_open: component tcp open function >>successful >>[access1:24264] mca:oob:select: checking available component tcp >>[access1:24264] mca:oob:select: Querying component [tcp] >>[access1:24264] oob:tcp: component_available called >>[access1:24264] WORKING INTERFACE 1 KERNEL INDEX 1 FAMILY: V4 >>[access1:24264] WORKING INTERFACE 2 KERNEL INDEX 3 FAMILY: V4 >>[access1:24264] [[55095,0],0] oob:tcp:init adding 10.0.251.51 to our list of >>V4 connections >>[access1:24264] WORKING INTERFACE 3 KERNEL INDEX 4 FAMILY: V4 >>[access1:24264] [[55095,0],0] oob:tcp:init adding 10.0.0.111 to our list of >>V4 connections >>[access1:24264] WORKING INTERFACE 4 KERNEL INDEX 5 FAMILY: V4 >>[access1:24264] [[55095,0],0] oob:tcp:init adding 10.2.251.11 to our list of >>V4 connections >>[access1:24264] WORKING INTERFACE 5 KERNEL INDEX 6 FAMILY: V4 >>[access1:24264] [[55095,0],0] oob:tcp:init adding 10.128.0.1 to our list of >>V4 connections >>[access1:24264] WORKING INTERFACE 6 KERNEL INDEX 7 FAMILY: V4 >>[access1:24264] [[55095,0],0] oob:tcp:init adding 93.180.7.36 to our list of >>V4 connections >>[access1:24264] WORKING INTERFACE 7 KERNEL INDEX 7 FAMILY: V4 >>[access1:24264] [[55095,0],0] oob:tcp:init adding 93.180.7.36 to our list of >>V4 connections >>[access1:24264] [[55095,0],0] TCP STARTUP >>[access1:24264] [[55095,0],0] attempting to bind to IPv4 port 0 >>[access1:24264] [[55095,0],0] assigned IPv4 port 47756 >>[access1:24264] mca:oob:select: Adding component to end >>[access1:24264] mca:oob:select: Found 1 active transports >>[access1:24264] mca: base: components_register: registering rml components >>[access1:24264] mca: base: components_register: found loaded component oob >>[access1:24264] mca: base: components_register: component oob has no register >>or open function >>[access1:24264] mca: base: components_open: opening rml components >>[access1:24264] mca: base: components_open: found loaded component oob >>[access1:24264] mca: base: components_open: component oob open function >>successful >>[access1:24264] orte_rml_base_select: initializing rml component oob >>[access1:24264] [[55095,0],0] posting recv >>[access1:24264] [[55095,0],0] posting persistent recv on tag 30 for peer >>[[WILDCARD],WILDCARD] >>[access1:24264] [[55095,0],0] posting recv >>[access1:24264] [[55095,0],0] posting persistent recv on tag 15 for peer >>[[WILDCARD],WILDCARD] >>[access1:24264] [[55095,0],0] posting recv >>[access1:24264] [[55095,0],0] posting persistent recv on tag 32 for peer >>[[WILDCARD],WILDCARD] >>[access1:24264] [[55095,0],0] posting recv >>[access1:24264] [[55095,0],0] posting persistent recv on tag 33 for peer >>[[WILDCARD],WILDCARD] >>[access1:24264] [[55095,0],0] posting recv >>[access1:24264] [[55095,0],0] posting persistent recv on tag 5 for peer >>[[WILDCARD],WILDCARD] >>[access1:24264] [[55095,0],0] posting recv >>[access1:24264] [[55095,0],0] posting persistent recv on tag 10 for peer >>[[WILDCARD],WILDCARD] >>[access1:24264] [[55095,0],0] posting recv >>[access1:24264] [[55095,0],0] posting persistent recv on tag 12 for peer >>[[WILDCARD],WILDCARD] >>[access1:24264] [[55095,0],0] posting recv >>[access1:24264] [[55095,0],0] posting persistent recv on tag 9 for peer >>[[WILDCARD],WILDCARD] >>[access1:24264] [[55095,0],0] posting recv >>[access1:24264] [[55095,0],0] posting persistent recv on tag 34 for peer >>[[WILDCARD],WILDCARD] >>[access1:24264] [[55095,0],0] posting recv >>[access1:24264] [[55095,0],0] posting persistent recv on tag 2 for peer >>[[WILDCARD],WILDCARD] >>[access1:24264] [[55095,0],0] posting recv >>[access1:24264] [[55095,0],0] posting persistent recv on tag 21 for peer >>[[WILDCARD],WILDCARD] >>[access1:24264] [[55095,0],0] posting recv >>[access1:24264] [[55095,0],0] posting persistent recv on tag 22 for peer >>[[WILDCARD],WILDCARD] >>[access1:24264] [[55095,0],0] posting recv >>[access1:24264] [[55095,0],0] posting persistent recv on tag 45 for peer >>[[WILDCARD],WILDCARD] >>[access1:24264] [[55095,0],0] posting recv >>[access1:24264] [[55095,0],0] posting persistent recv on tag 46 for peer >>[[WILDCARD],WILDCARD] >>[access1:24264] [[55095,0],0] posting recv >>[access1:24264] [[55095,0],0] posting persistent recv on tag 1 for peer >>[[WILDCARD],WILDCARD] >>[access1:24264] [[55095,0],0] posting recv >>[access1:24264] [[55095,0],0] posting persistent recv on tag 27 for peer >>[[WILDCARD],WILDCARD] >>-------------------------------------------------------------------------- >>An ORTE daemon has unexpectedly failed after launch and before >>communicating back to mpirun. This could be caused by a number >>of factors, including an inability to create a connection back >>to mpirun due to a lack of common network interfaces and/or no >>route found between them. Please check network connectivity >>(including firewalls and network routing requirements). >>-------------------------------------------------------------------------- >>[access1:24264] mca: base: close: component oob closed >>[access1:24264] mca: base: close: unloading component oob >>[access1:24264] [[55095,0],0] TCP SHUTDOWN >>[access1:24264] mca: base: close: component tcp closed >>[access1:24264] mca: base: close: unloading component tcp >>When i use srun i got: >>$ salloc -N2 --exclusive -p test -J ompi >>.... >>$srun -N 2 ./hello_c >>Hello, world, I am 0 of 1, (Open MPI v1.9a1, package: Open MPI >>semenov@compiler-2 Distribution, ident: 1.9a1r32252, repo rev: r32252, Jul >>16, 2014 (nightly snapshot tarball), 146) >>Hello, world, I am 0 of 1, (Open MPI v1.9a1, package: Open MPI >>semenov@compiler-2 Distribution, ident: 1.9a1r32252, repo rev: r32252, Jul >>16, 2014 (nightly snapshot tarball), 146) >> >>Sun, 20 Jul 2014 09:28:13 -0700 от Ralph Castain < r...@open-mpi.org >: >>>Try adding -mca oob_base_verbose 10 -mca rml_base_verbose 10 to your cmd >>>line. It looks to me like we are unable to connect back to the node where >>>you are running mpirun for some reason. >>> >>> >>>On Jul 20, 2014, at 9:16 AM, Timur Ismagilov < tismagi...@mail.ru > wrote: >>>>I have the same problem in openmpi 1.8.1( Apr 23, 2014 ). >>>>Does the srun command have a --map-by<foo> mpirun parameter, or can i >>>>chage it from bash enviroment? >>>> >>>> >>>>-------- Пересылаемое сообщение -------- >>>>От кого: Timur Ismagilov < tismagi...@mail.ru > >>>>Кому: Mike Dubman < mi...@dev.mellanox.co.il > >>>>Копия: Open MPI Users < us...@open-mpi.org > >>>>Дата: Thu, 17 Jul 2014 16:42:24 +0400 >>>>Тема: Re[4]: [OMPI users] Salloc and mpirun problem >>>> >>>>With Open MPI 1.9a1r32252 (Jul 16, 2014 (nightly snapshot tarball)) i got >>>>this output (same?): >>>>$ salloc -N2 --exclusive -p test -J ompi >>>>salloc: Granted job allocation 645686 >>>> >>>>$LD_PRELOAD=/mnt/data/users/dm2/vol3/semenov/_scratch/mxm/mxm-3.0/lib/libmxm.so >>>> mpirun -mca mca_base_env_list 'LD_PRELOAD' --mca plm_base_verbose 10 >>>>--debug-daemons -np 1 hello_c >>>>[access1:04312] mca: base: components_register: registering plm components >>>>[access1:04312] mca: base: components_register: found loaded component >>>>isolated >>>>[access1:04312] mca: base: components_register: component isolated has no >>>>register or open function >>>>[access1:04312] mca: base: components_register: found loaded component rsh >>>>[access1:04312] mca: base: components_register: component rsh register >>>>function successful >>>>[access1:04312] mca: base: components_register: found loaded component slurm >>>>[access1:04312] mca: base: components_register: component slurm register >>>>function successful >>>>[access1:04312] mca: base: components_open: opening plm components >>>>[access1:04312] mca: base: components_open: found loaded component isolated >>>>[access1:04312] mca: base: components_open: component isolated open >>>>function successful >>>>[access1:04312] mca: base: components_open: found loaded component rsh >>>>[access1:04312] mca: base: components_open: component rsh open function >>>>successful >>>>[access1:04312] mca: base: components_open: found loaded component slurm >>>>[access1:04312] mca: base: components_open: component slurm open function >>>>successful >>>>[access1:04312] mca:base:select: Auto-selecting plm components >>>>[access1:04312] mca:base:select:( plm) Querying component [isolated] >>>>[access1:04312] mca:base:select:( plm) Query of component [isolated] set >>>>priority to 0 >>>>[access1:04312] mca:base:select:( plm) Querying component [rsh] >>>>[access1:04312] mca:base:select:( plm) Query of component [rsh] set >>>>priority to 10 >>>>[access1:04312] mca:base:select:( plm) Querying component [slurm] >>>>[access1:04312] mca:base:select:( plm) Query of component [slurm] set >>>>priority to 75 >>>>[access1:04312] mca:base:select:( plm) Selected component [slurm] >>>>[access1:04312] mca: base: close: component isolated closed >>>>[access1:04312] mca: base: close: unloading component isolated >>>>[access1:04312] mca: base: close: component rsh closed >>>>[access1:04312] mca: base: close: unloading component rsh >>>>Daemon was launched on node1-128-09 - beginning to initialize >>>>Daemon was launched on node1-128-15 - beginning to initialize >>>>Daemon [[39207,0],1] checking in as pid 26240 on host node1-128-09 >>>>[node1-128-09:26240] [[39207,0],1] orted: up and running - waiting for >>>>commands! >>>>Daemon [[39207,0],2] checking in as pid 30129 on host node1-128-15 >>>>[node1-128-15:30129] [[39207,0],2] orted: up and running - waiting for >>>>commands! >>>>srun: error: node1-128-09: task 0: Exited with exit code 1 >>>>srun: Terminating job step 645686.3 >>>>srun: error: node1-128-15: task 1: Exited with exit code 1 >>>>-------------------------------------------------------------------------- >>>>An ORTE daemon has unexpectedly failed after launch and before >>>>communicating back to mpirun. This could be caused by a number >>>>of factors, including an inability to create a connection back >>>>to mpirun due to a lack of common network interfaces and/or no >>>>route found between them. Please check network connectivity >>>>(including firewalls and network routing requirements). >>>>-------------------------------------------------------------------------- >>>>[access1:04312] [[39207,0],0] orted_cmd: received halt_vm cmd >>>>[access1:04312] mca: base: close: component slurm closed >>>>[access1:04312] mca: base: close: unloading component slurm >>>> >>>> >> >> >>---------------------------------------------------------------------- >> >> >>_______________________________________________ >>users mailing list >>us...@open-mpi.org >>Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >>Link to this post: >>http://www.open-mpi.org/community/lists/users/2014/07/24828.php
